Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update the store flow code docs #15604

Merged
merged 3 commits into from
Oct 7, 2024
Merged

Update the store flow code docs #15604

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d78a987
move secret example to store-flow-code and fix GH example
discdiver Oct 7, 2024
dca86e8
fix typo
discdiver Oct 7, 2024
366b2b8
minor edits
discdiver Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 17 additions & 7 deletions docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ If using the Python `deploy` method with a private repository that references a

github_repo = GitRepository(
url="https://github.com/org/my-private-repo.git",
credentials=GitHubCredentials.load("my-github-credentials-block")
credentials=GitHubCredentials.load("my-github-credentials-block"),
)

flow.from_source(
Expand All @@ -95,7 +95,9 @@ If using the Python `deploy` method with a private repository that references a

github_repo = GitRepository(
url="https://github.com/org/my-private-repo.git",
credentials=Secret.load("my-secret-block-with-my-gh-credentials")
credentials={
"access_token": Secret.load("my-secret-block-with-my-gh-credentials")
},
)

flow.from_source(
Expand Down Expand Up @@ -187,7 +189,9 @@ If using the Python `deploy` method with a private repository that references a
if __name__ == "__main__":
github_repo=GitRepository(
url="https://bitbucket.com/org/my-private-repo.git",
credentials=Secret.load("my-secret-block-with-my-bb-credentials")
credentials={
"access_token": Secret.load("my-secret-block-with-my-bb-credentials")
},
)

flow.from_source(
Expand Down Expand Up @@ -259,6 +263,7 @@ If using the Python `deploy` method with a private repository that references a
gitlab_repo = GitRepository(
url="https://gitlab.com/org/my-private-repo.git",
credentials=GitLabCredentials.load("my-gitlab-credentials-block")
)

flow.from_source(
source=gitlab_repo,
Expand All @@ -279,7 +284,9 @@ If using the Python `deploy` method with a private repository that references a
if __name__ == "__main__":
gitlab_repo = GitRepository(
url="https://gitlab.com/org/my-private-repo.git",
credentials=Secret.load("my-secret-block-with-my-gl-credentials")
credentials={
"access_token": Secret.load("my-secret-block-with-my-gl-credentials")
},
)

flow.from_source(
Expand Down Expand Up @@ -319,6 +326,9 @@ If using the Python `deploy` method with a private repository that references a
</Tab>
</Tabs>

Note that you can specify a `branch` if creating a `GitRepository` object.
The default is `"main"`.

<Warning>
**Push your code**

Expand Down Expand Up @@ -470,7 +480,7 @@ We also include Python code that shows how to use an existing storage block and
)
```

```python s3_blocks.py
```python s3_block.py

from prefect import flow
from prefect_aws.s3 import S3Bucket
Expand Down Expand Up @@ -561,7 +571,7 @@ We also include Python code that shows how to use an existing storage block and
)
```

```python azure_blocks.py
```python azure_block.py
from prefect import flow
from prefect_azure import AzureBlobCredentials, AzureBlobStorage

Expand Down Expand Up @@ -648,7 +658,7 @@ We also include Python code that shows how to use an existing storage block and
)
```

```python gcs_blocks.py
```python gcs_block.py
from prefect import flow
from prefect_gcp import GcpCredentials, GCSBucket

Expand Down
63 changes: 25 additions & 38 deletions docs/3.0/deploy/run-flows-in-local-processes.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ title: Run flows in local processes
description: Create a deployment for a flow by calling the `serve` method.
---

The simplest way to create a [deployment](/3.0/deploy/infrastructure-examples/docker/) for your
flow is by calling its `serve` method.
The simplest way to create a [deployment](/3.0/deploy/infrastructure-examples/docker/) for your flow is by calling its `serve` method.

## Serve a flow

Expand All @@ -29,9 +28,10 @@ if __name__ == "__main__":
# generated on the server

hello_world.serve(name="my-first-deployment",
tags=["onboarding"],
parameters={"goodbye": True},
interval=60)
tags=["onboarding"],
parameters={"goodbye": True},
interval=60
)
```

This interface provides the configuration for a deployment (with no
Expand Down Expand Up @@ -156,8 +156,7 @@ Read more about our [hybrid model](https://www.prefect.io/security/overview/#hyb

You can retrieve flows from remote storage with the `flow.from_source` method.

`flow.from_source` accepts a git repository URL and an entrypoint pointing to the
flow to load from the repository:
`flow.from_source` can accept a git repository URL and an entrypoint pointing to the flow to load from the repository:

```python load_from_url.py
from prefect import flow
Expand All @@ -179,38 +178,14 @@ if __name__ == "__main__":
16:40:34.706 | INFO | Flow run 'muscular-perch' - Finished in state Completed()
```

A flow entrypoint is the path to the file where the flow is located, and the name of the
flow function separated by a colon.

For additional configuration, such as specifying a private repository,
provide a `GitRepository` object instead of URL:
A flow entrypoint is the path to the file where the flow is located, and the name of the flow function separated by a colon.

```python load_from_storage.py
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret


my_flow = flow.from_source(
source=GitRepository(
url="https://github.com/org/private-repo.git",
branch="dev",
credentials={
"access_token": Secret.load("github-access-token").get()
}
),
entrypoint="flows.py:my_flow"
)


if __name__ == "__main__":
my_flow()
```
For more ways to store and access flow code, see the [Retrieve code from storage page](/3.0/deploy/infrastructure-concepts/store-flow-code).

<Tip>
**You can serve loaded flows**

You can serve a flow loaded from remote storage with the same [`serve`](#serving-a-flow) method as a local flow:
You can serve a flow loaded from remote storage with the same [`serve`](#serve-a-flow) method as a local flow:

```python serve_loaded_flow.py
from prefect import flow
Expand All @@ -224,7 +199,19 @@ if __name__ == "__main__":
```
</Tip>

When you serve a flow loaded from remote storage, the serving process
periodically polls your remote storage for updates to the flow's code.
This pattern allows you to update your flow code without restarting the serving
process.
When you serve a flow loaded from remote storage, the serving process periodically polls your remote storage for updates to the flow's code.
This pattern allows you to update your flow code without restarting the serving process.
Note that if you change the parameters to your flow, you'll need to restart the serving process.

## If you need dynamic infrastructure

For more configuration, you can create a deployment that uses a work pool.
Reasons to create a work-pool based deployment include:

- Wanting to run your flow on dynamically provisioned infrastructure
- Needing more control over the execution environment on a per-flow run basis
- Creating an infrastructure template to use across deployments

Work pools are popular with data platform teams because they allow you to manage infrastructure configuration across an organization.

Learn more about work-pool based deployments on the [Configure dynamic infrastructure with work pools page](/3.0/deploy/infrastructure-concepts/work-pools)
Loading