Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dagster-aws] [docs] add docs for PipesEMRClient #25011

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[dagster-aws] [docs] add docs for PipesEMRClient #25011

wants to merge 2 commits into from

Conversation

danielgafni
Copy link
Contributor

@danielgafni danielgafni commented Oct 2, 2024 •
edited
Loading

Summary & Motivation

Adding API docs & tutorial for AWS EMR Pipes

How I Tested These Changes

@danielgafni Graphite App
Copy link
Contributor Author

danielgafni commented Oct 2, 2024 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @danielgafni and the rest of your teammates on Graphite Graphite

This was referenced Oct 2, 2024
Merged
Open
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 2, 2024 •
edited
Loading

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-n0gnapxuy-elementl.vercel.app
https://08-30--dagster-aws-docs-add-docs-for-pipesemrclient.dagster.dagster-docs.io

Direct link to changed pages:

@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 3331660 to 2e6ae16 Compare October 2, 2024 22:38
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 2e6ae16 to d9eb9cf Compare October 2, 2024 22:54
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from d9eb9cf to dc793c9 Compare October 3, 2024 13:01
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from dc793c9 to 5231c8f Compare October 3, 2024 13:24
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 5231c8f to 79d63ab Compare October 3, 2024 13:30
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 79d63ab to e26cc22 Compare October 3, 2024 13:34
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from e26cc22 to fe2f8b6 Compare October 4, 2024 13:10
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from fe2f8b6 to 4e9beda Compare October 4, 2024 15:52
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 4e9beda to e9a2b0e Compare October 4, 2024 16:12
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from e9a2b0e to 3297448 Compare October 4, 2024 16:18
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 3297448 to 633284d Compare October 4, 2024 17:17
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 633284d to 2c1d9a9 Compare October 7, 2024 15:51
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 2c1d9a9 to 56ba07e Compare October 8, 2024 10:19
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch 2 times, most recently from 40c2508 to 96a5f2c Compare October 14, 2024 20:28
Base automatically changed from 08-27-add_pipesemrclient to master October 14, 2024 20:31
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 96a5f2c to 77c836b Compare October 18, 2024 12:26
@danielgafni danielgafni marked this pull request as ready for review October 18, 2024 12:28
@graphite-app graphite-app bot added the area: docs Related to documentation in general label Oct 18, 2024
danielgafni
danielgafni commented Oct 18, 2024
View reviewed changes
python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/emr_serverless.py
@@ -93,7 +93,7 @@ def run(
Args:
context (OpExecutionContext): The context of the currently executing Dagster op or asset.
params (dict): Parameters for the ``start_job_run`` boto3 AWS EMR Serverless client call.
See `Boto3 API Documentation <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr-serverless/client/start_job_run.html>`_
See `Boto3 EMR Serverless API Documentation <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr-serverless/client/start_job_run.html>`_
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sphinx was complaining about this entry being identical to another one so I decided to inject service name here

danielgafni
danielgafni commented Oct 18, 2024
View reviewed changes
docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst
@@ -122,6 +122,10 @@ Clients

.. autoclass:: dagster_aws.pipes.PipesECSClient

.. autoclass:: dagster_aws.pipes.PipesEMRClient

.. autoclass:: dagster_aws.pipes.PipesEMRServerlessClient
Copy link
Contributor Author

@danielgafni danielgafni Oct 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops... was missing

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 18, 2024 •
edited
Loading

Deploy preview for dagster-docs-beta ready!

Preview available at https://dagster-docs-beta-j9nl5efgy-elementl.vercel.app

Direct link to changed pages:

@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 77c836b to 614c8da Compare October 18, 2024 12:31
cmpadden
cmpadden requested changes Oct 18, 2024
View reviewed changes
Copy link
Contributor

@cmpadden cmpadden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks great--just a couple of small comments.

docs/content/concepts/dagster-pipes/aws-emr.mdx Outdated Show resolved Hide resolved
docs/content/concepts/dagster-pipes/aws-emr.mdx Outdated Show resolved Hide resolved
docs/content/concepts/dagster-pipes/aws-emr.mdx Outdated Show resolved Hide resolved
@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from 614c8da to bd54eec Compare October 18, 2024 23:10
@danielgafni Graphite App
Copy link
Contributor Author

Merge activity

  • Oct 18, 7:11 PM EDT: Graphite disabled "merge when ready" on this PR due to: a merge conflict with the target branch; resolve the conflict and try again..

@danielgafni danielgafni force-pushed the 08-30-_dagster-aws_docs_add_docs_for_pipesemrclient branch from bd54eec to 5c1b2f9 Compare October 18, 2024 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PedramNavid PedramNavid Awaiting requested review from PedramNavid

@smackesey smackesey Awaiting requested review from smackesey

@C00ldudeNoonan C00ldudeNoonan Awaiting requested review from C00ldudeNoonan

@cmpadden cmpadden Awaiting requested review from cmpadden

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
area: docs Related to documentation in general
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@danielgafni @cmpadden