Use Google Auth and Cloud Python APIs instead of `gcloud` CLI #2083

swastik959 · 2023-10-22T16:30:35Z

fixes #2019

What does this implement/fix?

Put a x in the boxes that apply

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds a feature)
Breaking change (fix or feature that would cause existing features not to work as expected)
Documentation Update
Code style update (formatting, renaming)
Refactoring (no functional changes, no API changes)
Build related changes
Other (please describe):

Testing

Did you test the pull request locally?
Did you add new tests?

Any other comments?

Signed-off-by: swastik959 <[email protected]>

fangchenli

You need to add the required GCP libraries to pyproject.toml first.

iameskild

Thanks @swastik959 for the contribution! We really appreciate it. This is definitely a good start, as @fangchenli mentioned, we will need to add the required google python library as a dependency in the pyproject.toml.

swastik959 · 2023-10-25T04:56:30Z

@iameskild I wanted to ask should I modify the check credential method to point to credential json file for explicit parameter input

iameskild · 2023-10-29T05:40:46Z

@swastik959 that's a good question. I think we can start by supporting OAuth2 authentication for service accounts, this might be helpful: https://googleapis.github.io/google-api-python-client/docs/oauth.html#credentials

Signed-off-by: swastik959 <[email protected]>

swastik959 · 2023-11-03T10:59:30Z

@iameskild hi I have made few commits can you please review them . also can you please tell me how can I test them locally

iameskild

Amazing work @swastik959! I will have a closer look later today but in the meantime, do you mind making sure the tests/CI are passing?

marcelovilla · 2024-08-02T17:42:43Z

I'm picking up this PR to get it across the finish line. It has not seen any activity in almost a year but @swastik959 if you're still interested in working on it, feel free to leave a message here and I can point you in the right direction.

…wastik

…ment variable in provider functions

…supposed to be a file

marcelovilla · 2024-08-09T13:46:22Z

This PR is ready for review.

Companion docs PR: nebari-dev/nebari-docs#500

How to test this PR

Run:

export GOOGLE_CREDENTIALS="path/to/JSON/file/with/credentials"
export PROJECT_ID="Project ID"

Use the nebari init --guided-init command to create a Google Cloud nebari-config.yaml file and make sure to provide a GitHub repo name and auto-provision it
Run:

nebari validate -c nebari-config.yaml

Run:

nebari render -c nebari-config.yaml

Run:

nebari deploy -c nebari-config.yaml

Change anything in the config file, push to the auto-provisioned GitHub repo and let the deploy GHA workflow run
Run the gcp_cleanup function located in src/_nebari/provider/cloud/google_cloud.py using your config.

I tested all of this, making sure things were working correctly. Furthermore, I triggered our GCP deployment workflow using a branch with the same contents of this PR.

Notes

You'll notice that many functions were removed. Those were functions that were defined but not used anywhere in our codebase.
You'll notice that one testing file was removed. It was a file containing only tests for one of the functions that were removed.
We don't have any tests for any of our provider functions, which not only includes Google Cloud but also AWS, Azure and DO. If we want to address this, I suggest we do it in a separate PR. I can open an issue about it.

marcelovilla · 2024-08-09T13:48:06Z

pyproject.toml

+    "google-cloud-container==2.49.0",
+    "google-cloud-iam==2.15.1",
+    "google-cloud-storage==2.18.0",
+    "grpc-google-iam-v1==0.13.1",


I needed to add this library explicitly as our conda tests where failing because of it.

marcelovilla · 2024-08-09T13:49:21Z

src/_nebari/stages/infrastructure/__init__.py

@@ -339,7 +339,6 @@ class GoogleCloudPlatformProvider(schema.Base):
    @model_validator(mode="before")
    @classmethod
    def _check_input(cls, data: Any) -> Any:
-        google_cloud.check_credentials()


This is not needed anymore as the regions() function below will call it under the hood.

viniciusdc · 2024-08-15T15:57:25Z

Just as a reminder, once this is merged, we will need to remember to update the conda-forge package recipe to reflect the dependencies changes. Its passing our conda test, so we should be okay when doing the release.

marcelovilla · 2024-09-04T22:25:45Z

@viniciusdc just a friendly reminder to take a look at this PR when you get a chance. Thanks!

src/_nebari/provider/cicd/github.py

Adam-D-Lewis · 2024-09-11T20:33:47Z

src/_nebari/provider/cloud/google_cloud.py

-
-
-@functools.lru_cache()
-def instances(project: str) -> Dict[str, str]:


This function would be useful to do the same thing as AWS and Digital Ocean do and check instance types are valid before deploying. I'd be okay if you just open an issue to re-add that functionality at some point though. Search the repo for instances( and see how its used in infrastructure's __init__.py file.

@Adam-D-Lewis great catch. I just created an issue to track that and make sure we add that functionality for Google Cloud.

Adam-D-Lewis · 2024-09-11T20:54:33Z

@marcelovilla Looks like the local integration test failed too. Is there an issue already open for that flaky test. I couldn't find one by searching the error message.

viniciusdc · 2024-09-12T13:05:25Z

Uhm, this is interesting. #2641 should've addressed this particular issue.

marcelovilla · 2024-09-13T01:23:53Z

@Adam-D-Lewis tests seem indeed to be flaky again. I've reopened #2641 to track that as it should have anything to do with this PR. I re-ran the tests and they're all passing now.

Adam-D-Lewis

LGTM!

marcelovilla · 2024-09-17T13:51:10Z

Thanks for the review @Adam-D-Lewis ! Can you take a look at this companion docs PR when you get a chance? It's a fairly small PR.

swastik959 and others added 2 commits October 22, 2023 21:57

modified the projects and zones function

0f79dfb

Signed-off-by: swastik959 <[email protected]>

[pre-commit.ci] Apply automatic pre-commit fixes

3051574

fangchenli suggested changes Oct 23, 2023

View reviewed changes

iameskild suggested changes Oct 24, 2023

View reviewed changes

iameskild added provider: GCP type: maintenance 🛠 Day-to-day maintenance tasks area:developer-experience 👩🏻‍💻 labels Oct 29, 2023

swastik959 and others added 2 commits November 3, 2023 11:43

modified whole code

5d593b1

Signed-off-by: swastik959 <[email protected]>

[pre-commit.ci] Apply automatic pre-commit fixes

53ca190

iameskild reviewed Nov 7, 2023

View reviewed changes

iameskild mentioned this pull request Nov 13, 2023

added ability to control the prompt colour #2108

Closed

10 tasks

dcmcand requested a review from viniciusdc March 18, 2024 15:15

marcelovilla and others added 5 commits July 25, 2024 17:52

Merge branch 'develop' into swastik

2105e6e

Fix google dependency versions

7d319b5

Remove unused functions and fix calls to the Google Python APIs

1088d94

Merge branch 'develop' into swastik

a0694d1

[pre-commit.ci] Apply automatic pre-commit fixes

97677ec

marcelovilla added 9 commits August 5, 2024 08:45

Add grpc-google-iam-v1

6f3d813

Merge branch 'swastik' of https://github.com/swastik959/nebari into s…

d6498f9

…wastik

Load credentials explictly from file and get project ID from environe…

4daa377

…ment variable in provider functions

Check if credentials are a file or not before reading them

ab7f216

Remove gcloud step

4ac3911

Add google-auth as an explicit dependency

8a47dcd

Use string ending instead of Path.isfile to check whether env var is …

7be8fff

…supposed to be a file

Fix dependency version specifier

bdddbe2

Fix cleanup functions.

6338899

Add explicit google auth scopes

1c2fec3

marcelovilla changed the title ~~modified the projects and zones function~~ Use Google Auth and Cloud Python APIs instead of gcloud CLI Aug 9, 2024

marcelovilla requested review from dcmcand and Adam-D-Lewis August 9, 2024 13:46

marcelovilla reviewed Aug 9, 2024

View reviewed changes

marcelovilla modified the milestone: Nebari Maintenance Team - September 2024 Aug 29, 2024

Merge branch 'develop' into swastik

d54739b

Merge branch 'develop' into swastik

88785a1

marcelovilla added this to the Next Release milestone Sep 5, 2024

Merge branch 'develop' into swastik

f507f2d

Adam-D-Lewis requested changes Sep 11, 2024

View reviewed changes

marcelovilla requested a review from Adam-D-Lewis September 13, 2024 01:23

Adam-D-Lewis approved these changes Sep 17, 2024

View reviewed changes

marcelovilla merged commit dc91617 into nebari-dev:develop Sep 17, 2024
12 checks passed

marcelovilla mentioned this pull request Sep 18, 2024

[ENH] - Validate instance types for Google Cloud #2718

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Google Auth and Cloud Python APIs instead of `gcloud` CLI #2083

Use Google Auth and Cloud Python APIs instead of `gcloud` CLI #2083

swastik959 commented Oct 22, 2023

fangchenli left a comment

iameskild left a comment

swastik959 commented Oct 25, 2023

iameskild commented Oct 29, 2023

swastik959 commented Nov 3, 2023

iameskild left a comment

marcelovilla commented Aug 2, 2024

marcelovilla commented Aug 9, 2024

marcelovilla Aug 9, 2024

marcelovilla Aug 9, 2024

viniciusdc commented Aug 15, 2024

marcelovilla commented Sep 4, 2024

Adam-D-Lewis Sep 11, 2024

marcelovilla Sep 13, 2024

Adam-D-Lewis commented Sep 11, 2024 •

edited

Loading

viniciusdc commented Sep 12, 2024

marcelovilla commented Sep 13, 2024

Adam-D-Lewis left a comment

marcelovilla commented Sep 17, 2024



		@functools.lru_cache()
		def instances(project: str) -> Dict[str, str]:

Use Google Auth and Cloud Python APIs instead of gcloud CLI #2083

Use Google Auth and Cloud Python APIs instead of gcloud CLI #2083

Conversation

swastik959 commented Oct 22, 2023

What does this implement/fix?

Testing

Any other comments?

fangchenli left a comment

Choose a reason for hiding this comment

iameskild left a comment

Choose a reason for hiding this comment

swastik959 commented Oct 25, 2023

iameskild commented Oct 29, 2023

swastik959 commented Nov 3, 2023

iameskild left a comment

Choose a reason for hiding this comment

marcelovilla commented Aug 2, 2024

marcelovilla commented Aug 9, 2024

How to test this PR

Notes

marcelovilla Aug 9, 2024

Choose a reason for hiding this comment

marcelovilla Aug 9, 2024

Choose a reason for hiding this comment

viniciusdc commented Aug 15, 2024

marcelovilla commented Sep 4, 2024

Adam-D-Lewis Sep 11, 2024

Choose a reason for hiding this comment

marcelovilla Sep 13, 2024

Choose a reason for hiding this comment

Adam-D-Lewis commented Sep 11, 2024 • edited Loading

viniciusdc commented Sep 12, 2024

marcelovilla commented Sep 13, 2024

Adam-D-Lewis left a comment

Choose a reason for hiding this comment

marcelovilla commented Sep 17, 2024

Use Google Auth and Cloud Python APIs instead of `gcloud` CLI #2083

Use Google Auth and Cloud Python APIs instead of `gcloud` CLI #2083

Adam-D-Lewis commented Sep 11, 2024 •

edited

Loading