adding documentation and guide on push work pools (#10194)

PrefectHQ · Jul 13, 2023 · 0fe0660 · 0fe0660
1 parent 890f875
commit 0fe0660
Show file tree

Hide file tree

Showing 7 changed files with 76 additions and 3 deletions.
diff --git a/docs/concepts/work-pools.md b/docs/concepts/work-pools.md
@@ -18,7 +18,7 @@ search:
 
 ![flow-deployment-end-to-end](/img/concepts/flow-deployment-end-to-end.png)
 
-Work pools and the services that poll them, workers and agents, bridge the Prefect _orchestration environment_ with your _execution environment_. When a [deployment](/concepts/deployments/) creates a flow run, it is submitted to a specific work pool for scheduling. A worker or agent running in the execution environment polls its respective work pool for new runs to execute.
+Work pools, workers and agents, bridge the Prefect _orchestration environment_ with your _execution environment_. When a [deployment](/concepts/deployments/) creates a flow run, it is submitted to a specific work pool for scheduling. A worker or agent running in the execution environment can poll its respective work pool for new runs to execute, or the work pool can submit flow runs to serverless infrastructure directly, depending on your configuration.
 
 Each work pool has a default queue that all runs will be sent to. Work queues are automatically created whenever they are referenced by either a deployment or an agent. For most applications, this automatic behavior will be sufficient to run flows as expected. For advanced needs, additional queues can be created to enable a greater degree of control over work delivery. See [work pool configuration](#work-pool-configuration) for more information.
 
@@ -109,7 +109,7 @@ By default, the agent begins submission of flow runs a short time (10 seconds) b
 
 ## Work pool overview
 
-Work pools organize work that [agents](#agent-overview) or [workers](#worker-overview) pick up for execution. Deployments and agents coordinate through a shared work pool name. 
+Work pools organize work for execution. Work pools have types corresponding to the infrastructure will execute the flow code, as well as the delivery method of work to that environment. Pull work pools require [agents](#agent-overview) or [workers](#worker-overview) to poll the work pool for flow runs to execute. Push work pools can submit runs directly to serverless infrastructure providers like Cloud Run, Azure Container Instances, and AWS ECS without the need for an agent or worker.
 
 !!! tip "Work pools are like pub/sub topics"
     It's helpful to think of work pools as a way to coordinate (potentially many) deployments with (potentially many) agents through a known channel: the pool itself. This is similar to how "topics" are used to connect producers and consumers in a pub/sub or message-based system. By switching a deployment's work pool, users can quickly change the agent that will execute their runs, making it easy to promote runs through environments or even debug locally.
@@ -170,7 +170,7 @@ Inspect the work pool:
 ```
 </div>
 
-On success, the command returns the details of the newly created work pool, which can then be used to start agents that poll this pool for work or perform additional configuration of the pool.
+On success, the command returns the details of the newly created work pool.
 
 #### Base Job Template
 

diff --git a/docs/guides/cloud-run-push-work-pools.md b/docs/guides/cloud-run-push-work-pools.md
@@ -0,0 +1,71 @@
+---
+description: Learn how to use Prefect push work pools to schedule work on serverless infrastructure without having to run a worker.
+tags:
+    - work pools
+    - deployments
+    - Cloud Run
+search:
+  boost: 2
+---
+
+# Push Work to Google Cloud Run <span class="badge cloud"></span>
+Push [work pools](/concepts/work-pools/#work-pool-overview) are a special type of work pool that allows Prefect Cloud to submit flow runs for execution to serverless computing infrastructure without a user running a worker. Push work pools currently support execution in GCP Cloud Run Jobs, Azure Container Instances, and AWS ECS Tasks.
+
+In this guide we'll:
+
+- Create a push work pool that sends work to Google Cloud Run
+- Deploy a flow to that work pool
+- Execute our flow without having to run a worker or agent process to poll for flow runs
+
+## Google Cloud Run Setup
+
+To push work to Cloud Run, a GCP service account and an API Key are required.
+
+Create a service account by navigating to the service accounts page and clicking *Create*. Name and describe your service account, and click *continue* to configure permissions.
+
+The service account must have two roles at a minimum, *Cloud Run Developer*, and *Service Account User*.
+
+![Configuring service account permissions in GCP](/img/guides/gcr-service-account-setup.png)
+
+Once the Service account is created, navigate to its *Keys* page to add an API key. Create a JSON type key, download it, and store it somewhere safe for use in the next section.
+
+## Work Pool Configuration
+
+Our push work pool will store information about what type of infrastructure we're running on, what default values to provide to compute jobs, and other important execution environment parameters. Because our push work pool needs to integrate securely with your Cloud Run, we need to start by storing our credentials in Prefect Cloud, which we'll do by making a block.
+
+### Creating a GCP Credentials Block
+
+Navigate to the blocks page, click create new block, and select GCP Credentials for the type.
+
+For use in a push work pool, this block must have the contents of the JSON key stored in the Service Account Info field, as such:
+
+![Configuring GCP Credentials block for use in cloud run push work pools](/img/guides/gcp-creds-block-setup.png)
+
+Provide any other optional information and create your block.
+
+### Create your Cloud Run - Push Work Pool
+
+Now navigate to work pools and click create to start configuring your Cloud Run - Push work pool.
+
+Each step has several optional fields that are detailed in the [work pools](/concepts/work-pools/) documentation. For our purposes. Select the block you created under the GCP Credentials field. This will allow Prefect Cloud to securely interact with your GCP project.
+
+Create your pool and you are ready to deploy flows to your Cloud Run - Push work pool.
+
+## Deployment
+
+Deployment details are described in the deployments [concept section](/concepts/deployments/). Your deployment needs to be configured to send flow runs to our push work pool. For example, if you create a deployment through the interactive command line experience, choose the work pool you just created. If you are deploying an existing `prefect.yaml` file,  the deployment would contain:
+
+```yaml
+  work_pool:
+    name: my-google-cloud-run-push-pool
+```
+
+Deploying your flow to the `my-google-cloud-run-push-pool` work pool will ensure that runs that are ready for execution will be submitted immediately, without the need for a worker to poll for them.
+
+## Putting it all Together
+
+With your deployment created, navigate to its detail page and create a new flow run. You'll see the flow start running without ever having to poll the work pool, because Prefect Cloud securely connected to your GCP project, created a job, ran the job, and began reporting on its execution.
+
+![A flow running on a cloud run push work pool](/img/guides/push-flow-running.png)
+
+
diff --git a/docs/guides/index.md b/docs/guides/index.md
@@ -17,6 +17,7 @@ search:
 - Check out many common workflows with [Prefect Recipes](../recipes/recipes/).
 - Learn how to migrate from Prefect 1 to Prefect 2 with the [Migration Guide](/guides/migration-guide/).
 - Execute code in response to state changes with [state change hooks](/guides/state-change-hooks/).
+- Execute serverless flows without running a worker using [Push Work Pools](/guides/cloud-run-push-work-pools/).
 
 !!! tip "Need help?"
     Get your questions answered with a Prefect product advocate by [Booking A Rubber Duck](https://calendly.com/prefect-experts/prefect-product-advocates)!
diff --git a/docs/img/guides/gcp-creds-block-setup.png b/docs/img/guides/gcp-creds-block-setup.png
diff --git a/docs/img/guides/gcr-service-account-setup.png b/docs/img/guides/gcr-service-account-setup.png
diff --git a/docs/img/guides/push-flow-running.png b/docs/img/guides/push-flow-running.png
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -195,6 +195,7 @@ nav:
           - Recipes: recipes/recipes.md
           - Migration Guide: guides/migration-guide.md
           - State Change Hooks: guides/state-change-hooks.md
+          - Cloud Run Push Work Pools: guides/cloud-run-push-work-pools.md
     - Integrations:
           - integrations/index.md
           - Using Integrations: integrations/usage.md