[feature] consistently support object stores

[Bobgy]

Feature Area

/area backend

What feature would you like to see?

Consistently support different object stores (MinIO, S3, GCS, Azure blob, ...) in KFP, no matter it's for UI preview / visualization, metrics or in pipeline tasks.

Propose to unify using Go CDK: https://gocloud.dev/.

Changes:

1. In https://bit.ly/kfp-v2-compatible mode, KFP pipeline tasks will use Go CDK to upload/download artifacts in pipeline containers.
2. Implement support for different artifact stores in KFP API server, also make it deployable standalone in user namespaces.
3. Remove artifact API implementation in KFP UI.

What is the use case or pain point?

1. Reduce tech debt

KFP UI uses MinIO js client, GCS client etc to access some object stores.
KFP API server uses MinIO Go client to only access MinIO.
Argo workflow supports several types of object stores natively too (but not sure what they use).

The difference in implementation makes it hard to achieve the same level of support for different object stores across KFP features. This also brings duplicate efforts in maintaining object access.

e.g. KFP v1 metrics is only supported with MinIO, because KFP API server only supports MinIO.

2. Consistently support object store features:

e.g. IRSA for S3: #3405 (comment)
or workload identity for GCS (needed an upgrade for argo)

Is there a workaround currently?

No

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[karlschriek]

Sorry for the slow response on this, but I can confirm that go-cloud supports IAM Roles for Service Accounts in order to access objects on S3.

Support was added with this PR: google/go-cloud#2773

I've also tested it independently. The IAM role needs to be attached to a ServiceAccount via an annotation as follows:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: go-cloud-test
  namespace: mynamespace
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/go-cloud-test

Any Pod that is started with this ServiceAccount will then inherit the role. (go-cloud uses the aws-cli in order to resolve the attached role into a temporary session). As long as the role arn:aws:iam::123456789012:role/go-cloud-test has the necessary policies and trust relationships attached to it, the following psuedo-deployment should work out of the box:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-cloud-test
  namespace: mynamespace
  labels:
    app: go-cloud-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-cloud-test
  template:
    metadata:
      labels:
        app: go-cloud-test
    spec:
      serviceAccount: go-cloud-test
      containers:
      - name: go-cloud-test
        image: my/testimage
        workingDir: /go/src/go-cloud/samples/gocdk-blob #see https://github.com/google/go-cloud/tree/v0.23.0/samples/gocdk-blob
        command: ["/bin/sh","-c"]
        args: ["go run main.go download s3://my-test-bucket-4134451 hello.txt > foo.txt]

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[davidspek]

/lifecycle freeze

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.