S3 support in Kubeflow Pipelines #3405

Jeffwan · 2020-03-31T20:38:02Z

Create this parent issue to includes all known S3 problems in KFP and I also like to talk about more feature requests in this ticket.

Use cases

Replace Minio with S3 for Argo artifact store and KFP Pipeline store.

[] Manifest changes in Kubeflow/manifest and standalone kubeflow/pipeline/manifest to easily change from minio to S3
[] IRSA - bump minio-sdk-go version to support IRSA
[] IRSA - bump argo workflow version to make sure runner can persist artifact to S3

Relate issues and PRs

UI

[x] KFP UI should be able to read source file in S3, for example

mlpipeline-ui-metadata -> minio://mlpipeline/artifacts/pipeline-A/pipeline-A-run-id/mlpipeline-ui-metadata.tgz

{
  "outputs": [
    {
      "source": "s3://your_bucket/README.md",
      "type": "markdown"
    }
  ]
}

[] KFP UI should be able to read artifact files in S3

mlpipeline-ui-metadata -> s3://mlpipeline/artifacts/pipeline-A/pipeline-A-run-id/mlpipeline-ui-metadata.tgz

{
  "outputs": [
    {
      "source": "s3://your_bucket/README.md",
      "type": "markdown"
    }
  ]
}

[] IRSA - Bump minio-js version to support IRSA

SDK

[x] Use can declare custom S3 artifact location inside the pipeline.
[x] User can apply AWS credentials to pipeline pods to get access to S3.
[x] User can specify service account for pipeline. (IRSA)

More examples

/kind improvement
/area frontend
/area backend

/cc @eterna2 @discordianfish

The text was updated successfully, but these errors were encountered:

k8s-ci-robot · 2020-03-31T20:38:05Z

@Jeffwan: The label(s) kind/improvement cannot be applied, because the repository doesn't have them

In response to this:

Create this parent issue to includes all known S3 problems in KFP and I also like to talk about more feature requests in this ticket.

Use cases

Replace Minio with S3 for Argo artifact store and KFP Pipeline store.

[] Manifest changes in Kubeflow/manifest and standalone kubeflow/pipeline/manifest to easily change from minio to S3
[] IRSA - bump minio-sdk-go version to support IRSA
[] IRSA - bump argo workflow version to make sure runner can persist artifact to S3

Relate issues and PRs

[api-server] Object store folder path is configurable and can work with AWS (secure and region flag, and IAM credentials) #2080

[pipeline-ui] Retrieve pod logs from argo archive #2081

Could not parse metadata file from S3 #3388

Show small artifacts in UX inputs/outputs #2172

Bump minio client version to support assume-web-identity-role for AWS #3398

loadNodeOutputPaths can not infer the right object storage #3387

Pipeline UI unable to access artifacts stored in S3 #3366

UI

[x] KFP UI should be able to read source file in S3, for example

mlpipeline-ui-metadata -> minio://mlpipeline/artifacts/pipeline-A/pipeline-A-run-id/mlpipeline-ui-metadata.tgz
{
 "outputs": [
   {
     "source": "s3://your_bucket/README.md",
     "type": "markdown"
   }
 ]
}
[] KFP UI should be able to read artifact files in S3

mlpipeline-ui-metadata -> s3://mlpipeline/artifacts/pipeline-A/pipeline-A-run-id/mlpipeline-ui-metadata.tgz
{
 "outputs": [
   {
     "source": "s3://your_bucket/README.md",
     "type": "markdown"
   }
 ]
}
[] IRSA - Bump minio-js version to support IRSA

Frontend - Add support for artifacts stored in S3 #1278

https://github.com/kubeflow/pipelines/pull/1285/files

SDK

[x] Use can declare custom S3 artifact location inside the pipeline.
[x] User can apply AWS credentials to pipeline pods to get access to S3.
[x] User can specify service account for pipeline. (IRSA)

[Feature] Supports parameterized S3Artifactory for Pipeline and ContainerOp in kfp package #1064

Add helper to easily add aws secrets #1133

More examples

Update taxi pipeline example with latest beam with S3 support #3185

S3 errors in Pipeline examples for reading training data and artifact storage #596

https://github.com/kubeflow/pipelines/pull/1338/files

/kind improvement
/area frontend
/area backend

/cc @eterna2 @discordianfish

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Ark-kun · 2020-04-01T21:18:05Z

SDK: Use can declare custom S3 artifact location inside the pipeline.

This is deprecated. The supported method is to configure the artifact location in the workflow-configmap

There is also an option to use Minio gateway to proxy requests to the actual storage system (Google Cloud Storage or Amazon S3). WDYT?

Jeffwan · 2020-04-04T02:01:49Z

@Ark-kun artifact location in workflow-configmap works for us now.
Do you have any docs or code reference on minio gateway?

Ark-kun · 2020-04-06T21:48:49Z

Do you have any docs or code reference on minio gateway?

Here is the Minio doc: https://docs.minio.io/docs/minio-gateway-for-s3.html

eterna2 · 2020-04-10T02:17:50Z

@Jeffwan

Update metadata writer service to support S3 artifacts
Metadata writer service currently hardcode metadata artifacts url to minio. Need to investigate how to infer whether the artifact store endpoint is s3 or minio for metadata service.

pipelines/backend/metadata_writer/src/metadata_writer.py

Line 85 in 6b358d0

def argo_artifact_to_uri(artifact: dict) -> str:

@discordianfish Gateway is kind of easy in current manifest structure. KFP just need S3 permission and minio will becomes a proxy to S3. There's no need to make code changes everywhere to check protocol. If you see above PRs, It's kind of painful now. We also need to give credential or attach roles in every KFP component talks to S3.

I agree this won't meet everyone's need, we definitely need low level object storage configuration.

discordianfish · 2020-04-28T14:56:35Z

@Jeffwan What I don't understand is why we need to distinguish between minio and s3 at all, instead of just tracking the bucket name and the endpoint. The minio client would be configured with either auth credentials or fall back to the instance role. That should be the same config for all components, no?

But okay, if people want to go down the gateway route: How do all components that need s3/minio access authenticate against the gateway? Not using auth here would be a no go for us.

Jeffwan · 2020-04-29T15:35:51Z

@discordianfish

What I don't understand is why we need to distinguish between minio and s3 at all, instead of just tracking the bucket name and the endpoint.

I see. It's not consistency across all components. KFP has lots of references has fixed protocol like minio:// which won't work for S3.

But okay, if people want to go down the gateway route: How do all components that need s3/minio access authenticate against the gateway? Not using auth here would be a no go for us.

The only difference between minio and minio gateway with S3 is, user need to give credential with s3 policy to minio pod. instance profile will work as well. I have not checked with IRSA or Kube2IAM for fine grain control yet.

eterna2 · 2020-04-29T16:56:12Z

Just want to add a note.

One main reason why we moved to s3 instead of minio gateway (this was our original deployment) is because tensorboard doesn't work with minio.

It only support gcs and s3. It is using boto3 instead of minio the last time I checked (maybe 6 months ago?).

Other than that I think minio gateway is fine as long as the throughput is fine. But the concern is that u can't have fine grain rbac control with minio gateway.

And it is confusing to our data scientists when they have to mix between s3 and minio (i.e. s3 for our data, minio for data generated within kfp).

discordianfish · 2020-04-30T11:12:55Z

I see. It's not consistency across all components. KFP has lots of references has fixed protocol like minio:// which won't work for S3.

Yes, but why? The protocol part in that url is just for the client to know where to connect so. So for minio and s3, I suggest we always use the endpoint, bucket name and path to refer to an object. It shouldn't matter for the client whether this is s3 or minio. Does that make sense?

The only difference between minio and minio gateway with S3 is, user need to give credential with s3 policy to minio pod. instance profile will work as well. I have not checked with IRSA or Kube2IAM for fine grain control yet.

I understand that, but we need to authenticate access from kubeflow components to the minio gateway. If I understand you correctly, with minion gw everything that is able to reach minion will have access to all s3 content without auth. That's not acceptable for us.

Jeffwan · 2020-05-01T21:30:41Z

I see. It's not consistency across all components. KFP has lots of references has fixed protocol like minio:// which won't work for S3.

Yes, but why? The protocol part in that url is just for the client to know where to connect so. So for minio and s3, I suggest we always use the endpoint, bucket name and path to refer to an object. It shouldn't matter for the client whether this is s3 or minio. Does that make sense?

https://github.com/kubeflow/pipelines/pull/3531/files#diff-4e5a9c52bbed342532f527bae3aae002R15-R20
This is one example, lots of codes rely on protocol to have custom logic. I get your point and that would be ideal case, we don't alway have endpoint passed with artifact url everywhere. If there's a refactor to have a object will endpoint, bucket, path. Then all the customization can be changed to reply on endpoint. It will work but have more efforts.

The only difference between minio and minio gateway with S3 is, user need to give credential with s3 policy to minio pod. instance profile will work as well. I have not checked with IRSA or Kube2IAM for fine grain control yet.

I understand that, but we need to authenticate access from kubeflow components to the minio gateway. If I understand you correctly, with minion gw everything that is able to reach minion will have access to all s3 content without auth. That's not acceptable for us.

Right, that's what @eterna2 mention as well. Fine grain control doesn't work with minio gateway.

HassanOuda · 2021-03-11T18:28:24Z

Also, why is this line hardcoding S3 endpoint as the amazon endpoint? We have a different S3 instance not in Amazon and minio endpoint doesn't work.

pipelines/backend/metadata_writer/src/metadata_writer.py

Line 89 in 1577bdb

return re.search('^.*s3.*amazonaws.com.*$', endpoint)

and here:

pipelines/frontend/server/aws-helper.ts

Line 55 in 1577bdb

return !!endpoint.match(/s3.{0,}\.amazonaws\.com\.?.{0,}/i);

lastly:

pipelines/frontend/src/lib/AwsHelper.ts

Line 23 in 1577bdb

return !!endpoint.match(/s3.{0,}\.amazonaws\.com\.?.{0,}/i);

aaron-arellano · 2021-05-07T17:18:26Z

Is assume-web-identity-role to use IAM Role for Service account still an open issue? The thread is not clear on whether this works or not. I am looking to use IAM Role for service account to connect ML Pipeline and Argo artifacts to S3 using this annotation eks.amazonaws.com/role-arn: <your-arn-role> @Jeffwan

karlschriek · 2021-05-14T18:03:23Z

Doesn't work yet. Minio version used by pipelines is currently still sitting at github.com/minio/minio-go v6.0.14+incompatible. I think this needs to be merged: #3549

karlschriek · 2021-05-14T18:06:01Z

Also this issue also needs to be resolved: minio/minio-js#841. As far as I can tell from piecing together the various PRs and issues on this topic we need both minio-go and minio-js to implement this and then we need the versions to be bumped on KFP's side

aaron-arellano · 2021-05-14T20:07:05Z

thanks for the response. hopefully there is resolution to this soon. I know a lot of people would love to use IRSA for S3 to avid using credentials or third party K8s IAM providers...

Bobgy · 2021-05-17T05:18:50Z

Hi everyone, for KFP v2 compatible mode, we've switched to use https://gocloud.dev/ to access S3. Can someone help me verify whether it supports your use-cases especially IRSA and how?
You can find more about the design in bit.ly/kfp-v2-compatible

karlschriek · 2021-05-17T06:08:05Z

Hey @Bobgy, I could help with this. Could you give me some more info on what you would like to verify exactly? I guess this will replace the Minio Go and JS clients that are currently in use?

The link bit.ly/kfp-v2-compatible gives me a 404 error.

Bobgy · 2021-05-17T06:19:26Z

Thanks @karlschriek! Yes, we'd like to replace MinIO Go with Go CDK (also KFP tasks in pipelines will use Go CDK to upload/download from storage). So it'll be helpful to verify whether Go CDK (as an alternative client for S3) can support the IRSA you are talking about.

Re the 404 error, sorry I made a mistake, fixed the link.

karlschriek · 2021-05-17T06:28:51Z

Ok, it looks promising at least: google/go-cloud#2773.

IRSA works by attaching annotations to the ServiceAccounts that get attached to Pods. So all we would need to do is attach the right annotations to the ServiceAccounts for the pipelines api etc, as well as whatever sa is used for the pipeline jobs.

I'll provide you with some small examples and also try out the Go SDK.

karlschriek · 2021-05-17T06:30:49Z

Is there already an issue that tracks this? I guess the current issue is probably not the right place to continue this discussion.

Bobgy · 2021-05-17T06:49:41Z

@karlschriek WDUT about #5656?

RakeshRaj97 · 2021-06-15T02:26:26Z

I'm struggling to access my object storage bucket created using rook-ceph (s3) and would like to use this as my default artifact and metadata store instead of MinIO. Can anyone please share me what all files to be modified to use MinIO as s3 interface?

I've tried updating workflow-controller-configmap to use my object store

apiVersion: v1
data:
  artifactRepository: |
    archiveLogs: true
    s3:
      endpoint: "rook-ceph-rgw-my-store.rook-ceph.svc.cluster.local:80"
      bucket: "ceph-bkt-xxxx-xxxxx"
      keyFormat: "artifacts/{{workflow.name}}/{{pod.name}}"
      # insecure will disable TLS. Primarily used for minio installs not configured with TLS
      insecure: true
      accessKeySecret:
        name: mlpipeline-minio-artifact
        key: accesskey
      secretKeySecret:
        name: mlpipeline-minio-artifact
        key: secretkey
  containerRuntimeExecutor: docker

Also replace accesskey and secretkey in mlpipeline-minio-artifact secret

Any clue on this will be really helpful.

vinayan3 · 2021-08-20T01:39:56Z

@Jeffwan Did you have a PR or an example for using minio as a gateway to s3? I've been standing up Kubeflow on AWS and I'm a bit concerned about the number of places the patches that need to go in for S3. Instead of patching all these deployments I'd prefer to just patch the minio components and then be done with it.

Jeffwan · 2021-09-04T00:45:02Z

@Jeffwan Did you have a PR or an example for using minio as a gateway to s3? I've been standing up Kubeflow on AWS and I'm a bit concerned about the number of places the patches that need to go in for S3. Instead of patching all these deployments I'd prefer to just patch the minio components and then be done with it.

@vinayan3 Have you checked in this https://docs.min.io/docs/minio-gateway-for-s3.html? You can make changes in minio manifests to achieve that. minio gateway works with S3 out of box.

MaximumQuasar · 2022-02-16T04:06:10Z

Hi,

Currently is the assume-role-identity sts (temporary access) supported for the workflow to upload into s3 and is there any reference docs for the same?

By temporary access i mean with access key, secret key & security Token.

xwt-ml · 2022-05-10T07:09:09Z

Is there a time plan, when and in which release will this following feature be supported? @Jeffwan
UI
KFP UI should be able to read artifact files in S3
mlpipeline-ui-metadata -> s3://mlpipeline/artifacts/pipeline-A/pipeline-A-run-id/mlpipeline-ui-metadata.tgz

surajkota · 2022-12-01T20:35:07Z

Hi everyone, please see the proposal/design for using IRSA/IAM roles to access S3 on #8502 and let us know your thoughts.

For anyone looking to use S3 with Kubeflow pipelines(using IAM user credentials), please refer to the latest AWS distribution of Kubeflow docs - https://awslabs.github.io/kubeflow-manifests/

thesuperzapper · 2023-07-10T23:24:16Z

Hey all, I just wanted to share that deployKF uses S3 directly (no minio gateway), which also means that it comes with support for IRSA!

It's going to be documented better soon, but these are the values you need to configure to use S3, happy to help anyone who has trouble getting it working.

If you want, you can also read a bit more about deployKF in this comment: kubeflow/manifests#2451 (comment)

rimolive · 2024-03-24T14:05:45Z

Closing this issue. Looks like this was already implemented.

/close

google-oss-prow · 2024-03-24T14:05:49Z

@rimolive: Closing this issue.

In response to this:

Closing this issue. Looks like this was already implemented.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the area/frontend label Mar 31, 2020

k8s-ci-robot added the area/backend label Mar 31, 2020

Ark-kun assigned eterna2, Ark-kun, IronPan, Bobgy and rmgogogo Apr 1, 2020

Bobgy added platform/aws help wanted The community is welcome to contribute. status/triaged Whether the issue has been explicitly triaged labels Apr 2, 2020

This was referenced Apr 17, 2020

Infer artifact store endpoint in metadata writer #3530

Merged

Generate clickable artifact url for s3 URI #3531

Merged

Bobgy mentioned this issue May 17, 2021

[feature] consistently support object stores #5656

Closed

goswamig mentioned this issue Mar 3, 2022

Remove IAM user usage in Kubeflow and replace that with IAM role awslabs/kubeflow-manifests#44

Open

ryansteakley mentioned this issue Nov 28, 2022

[feature] S3 Integration for Artifact Storage (IRSA) #8502

Closed

IronPan assigned chensun Jan 27, 2023

surajkota mentioned this issue Mar 7, 2023

Proposal to release 2.0-alpha7 with bug fixes and awaited features for Kubeflow 1.7 release #8938

Closed

surajkota mentioned this issue Apr 24, 2023

Deprecating the use of static IAM User credentials for S3 support in Kubeflow Pipelines awslabs/kubeflow-manifests#704

Open

google-oss-prow bot closed this as completed Mar 24, 2024

S3 support in Kubeflow Pipelines #3405

S3 support in Kubeflow Pipelines #3405

Comments

Jeffwan commented Mar 31, 2020

Use cases

Replace Minio with S3 for Argo artifact store and KFP Pipeline store.

UI

SDK

More examples

k8s-ci-robot commented Mar 31, 2020

Use cases

Replace Minio with S3 for Argo artifact store and KFP Pipeline store.

UI

SDK

More examples

Ark-kun commented Apr 1, 2020

Jeffwan commented Apr 4, 2020

Ark-kun commented Apr 6, 2020

eterna2 commented Apr 10, 2020

See also

Ark-kun commented Apr 14, 2020

Jeffwan commented Apr 16, 2020

discordianfish commented Apr 20, 2020

rmgogogo commented Apr 23, 2020

Ark-kun commented Apr 24, 2020 • edited Loading

Jeffwan commented Apr 24, 2020

discordianfish commented Apr 27, 2020

Jeffwan commented Apr 28, 2020

discordianfish commented Apr 28, 2020

Jeffwan commented Apr 29, 2020

eterna2 commented Apr 29, 2020

discordianfish commented Apr 30, 2020

Jeffwan commented May 1, 2020

HassanOuda commented Mar 11, 2021 • edited Loading

aaron-arellano commented May 7, 2021 • edited Loading

karlschriek commented May 14, 2021

karlschriek commented May 14, 2021

aaron-arellano commented May 14, 2021 • edited Loading

Bobgy commented May 17, 2021 • edited Loading

karlschriek commented May 17, 2021

Bobgy commented May 17, 2021

karlschriek commented May 17, 2021

karlschriek commented May 17, 2021

Bobgy commented May 17, 2021

RakeshRaj97 commented Jun 15, 2021 • edited Loading

vinayan3 commented Aug 20, 2021

Jeffwan commented Sep 4, 2021 • edited Loading

MaximumQuasar commented Feb 16, 2022

xwt-ml commented May 10, 2022

surajkota commented Dec 1, 2022

thesuperzapper commented Jul 10, 2023

rimolive commented Mar 24, 2024

google-oss-prow bot commented Mar 24, 2024

Ark-kun commented Apr 24, 2020 •

edited

Loading

HassanOuda commented Mar 11, 2021 •

edited

Loading

aaron-arellano commented May 7, 2021 •

edited

Loading

aaron-arellano commented May 14, 2021 •

edited

Loading

Bobgy commented May 17, 2021 •

edited

Loading

RakeshRaj97 commented Jun 15, 2021 •

edited

Loading

Jeffwan commented Sep 4, 2021 •

edited

Loading