Skip to content

Latest commit

 

History

History
391 lines (301 loc) · 14.5 KB

README.md

File metadata and controls

391 lines (301 loc) · 14.5 KB
Raw

Autoscaler tool for Cloud Spanner

Autoscaler

Set up the Autoscaler in a distributed deployment using Terraform
Home · Scaler function · Poller function · Forwarder function · Terraform configuration
Single Project · Centralized · Distributed

Table of Contents

Overview

This directory contains Terraform configuration files to quickly set up the infrastructure for your Autoscaler with a distributed deployment.

In this deployment option all the components of the Autoscaler reside in a single project, with the exception of Cloud Scheduler (step 1) and the Forwarder topic and function

This deployment is the best of both worlds between the per-project and the centralized deployments: Teams who own the Spanner instances, called Application teams, are able to manage the Autoscaler configuration parameters for their instances with their own Cloud Scheduler jobs. On the other hand, the rest of the Autoscaler infrastructure is managed by a central team.

Architecture

architecture-distributed

For an explanation of the components of the Autoscaler and the interaction flow, please read the main Architecture section.

Cloud Scheduler can only publish messages to topics in the same project. Therefore in step 2, we transparently introduce an intermediate component to make this architecture possible. For more information, see the Forwarder function.

The distributed deployment has the following pros and cons:

Pros

  • Configuration and infrastructure: application teams are in control of their config and schedules
  • Maintenance: Scaler infrastructure is centralized, reducing up-keep overhead
  • Policies and audit: Best practices across teams might be easier to specify and enact. Audits might be easier to execute.

Cons

  • Configuration: application teams need to provide service accounts to write to the polling topic.
  • Risk: the centralized team itself may become a single point of failure even if the infrastructure is designed with high availability in mind.

Before you begin

  1. Open the Cloud Console

  2. Activate Cloud Shell
    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

  3. In Cloud Shell, clone this repository

    git clone https://github.com/cloudspannerecosystem/autoscaler.git

  4. Export variables for the working directories

    export AUTOSCALER_DIR="$(pwd)/autoscaler/terraform/distributed/autoscaler-project"
export APP_DIR="$(pwd)/autoscaler/terraform/distributed/app-project"

Preparing the Autoscaler Project

In this section you prepare the deployment of the project where the centralized Autoscaler infrastructure, with the exception of Cloud Scheduler, lives.

  1. Go to the project selector page in the Cloud Console. Select or create a Cloud project.

  2. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  3. In Cloud Shell, set environment variables with the ID of your autoscaler project:

    export AUTO_SCALER_PROJECT_ID=<INSERT_YOUR_PROJECT_ID>
gcloud config set project "${AUTO_SCALER_PROJECT_ID}"

  4. Choose the region and zone and App Engine Location where the Autoscaler infrastructure will be located. sh export AUTO_SCALER_REGION=us-central1 export AUTO_SCALER_ZONE=us-central1-c export AUTO_SCALER_APP_ENGINE_LOCATION=us-central

  5. Enable the required Cloud APIs :

    gcloud services enable iam.googleapis.com \
    cloudresourcemanager.googleapis.com \
    spanner.googleapis.com \
    appengine.googleapis.com \
    firestore.googleapis.com \
    pubsub.googleapis.com \
    cloudfunctions.googleapis.com  \
    cloudbuild.googleapis.com

  6. Create a service account that will be used by Terraform to create all the resources in your infrastructure.

    gcloud iam service-accounts create terraformer --display-name "Terraform service account"

  7. Give the project owner role to the service account

    gcloud projects add-iam-policy-binding "${AUTO_SCALER_PROJECT_ID}" \
  --member "serviceAccount:terraformer@${AUTO_SCALER_PROJECT_ID}.iam.gserviceaccount.com" \
  --role roles/owner

  8. Create a service account key file

    gcloud iam service-accounts keys create \
  --iam-account "terraformer@${AUTO_SCALER_PROJECT_ID}.iam.gserviceaccount.com" "${AUTOSCALER_DIR}/key.json"

  9. If your project does not have a Firestore instance yet, create one

    gcloud app create --region="${AUTO_SCALER_APP_ENGINE_LOCATION}"
gcloud alpha firestore databases create --region="${AUTO_SCALER_APP_ENGINE_LOCATION}"

Deploy the Autoscaler

  1. Set the project ID, region, zone and App Engine location in the corresponding Terraform environment variables

    export TF_VAR_project_id="${AUTO_SCALER_PROJECT_ID}"
export TF_VAR_region="${AUTO_SCALER_REGION}"
export TF_VAR_zone="${AUTO_SCALER_ZONE}"
export TF_VAR_location="${AUTO_SCALER_APP_ENGINE_LOCATION}"

  2. Change directory into the Terraform scaler-project directory and initialize it.

    cd "${AUTOSCALER_DIR}"
terraform init

  3. Create the Autoscaler infrastructure. Answer yes when prompted, after reviewing the resources that Terraform intends to create.

    terraform apply -parallelism=2

If you are running this command in Cloud Shell and encounter errors of the form "Error: cannot assign requested address", this is a known issue in the Terraform Google provider, please retry with -parallelism=1:

Preparing the Application Project

In this section you prepare the deployment of the Cloud Scheduler, Forwarder topic and function in the project where the Spanner instances live.

  1. Go to the project selector page in the Cloud Console. Select or create a Cloud project.

  2. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  3. In Cloud Shell, set the environment variables with the ID of your application project:

    export APP_PROJECT_ID=<INSERT_YOUR_APP_PROJECT_ID>
gcloud config set project "${APP_PROJECT_ID}"

  4. Choose the region and zone and App Engine Location where the Application project will be located. sh export APP_REGION=us-central1 export APP_ZONE=us-central1-c export APP_APP_ENGINE_LOCATION=us-central

  5. Use the following command to enable the Cloud APIs:

    gcloud services enable iam.googleapis.com \
    cloudresourcemanager.googleapis.com \
    appengine.googleapis.com \
    spanner.googleapis.com \
    pubsub.googleapis.com \
    cloudfunctions.googleapis.com \
    cloudscheduler.googleapis.com \
    cloudbuild.googleapis.com

  6. Create a service account that will be used by Terraform to create all the resources in your infrastructure.

    gcloud iam service-accounts create terraformer --display-name "Terraform service account"

  7. Give the project owner role to the service account

    gcloud projects add-iam-policy-binding "${APP_PROJECT_ID}" \
  --member "serviceAccount:terraformer@${APP_PROJECT_ID}.iam.gserviceaccount.com" \
  --role roles/owner

  8. Create a service account key file

    gcloud iam service-accounts keys create \
  --iam-account "terraformer@${APP_PROJECT_ID}.iam.gserviceaccount.com" "${APP_DIR}/key.json"

  9. Create an App to enable Cloud Scheduler, but do not create a Firestore database:

    gcloud app create --region="${APP_APP_ENGINE_LOCATION}"

Deploy the Application infrastructure

  1. Set the project ID, region, zone and App Engine location in the corresponding Terraform environment variables

    export TF_VAR_project_id="${APP_PROJECT_ID}"
export TF_VAR_region="${APP_REGION}"
export TF_VAR_zone="${APP_ZONE}"
export TF_VAR_location="${APP_APP_ENGINE_LOCATION}"

  2. Set the project ID where the Autoscaler state will be stored. The Autoscaler state includes the timestamps when the scaling events were triggered for each instance.

    export TF_VAR_state_project_id="${AUTO_SCALER_PROJECT_ID}"

  3. If you want to create a new Spanner instance for testing the Autoscaler, set the following variable. The spanner instance that Terraform creates is named autoscale-test.

    export TF_VAR_terraform_spanner=true

    On the other hand, if you do not want to create a new Spanner instance because you already have an instance for the Autoscaler to monitor, set the name name of your instance in the following variable

    export TF_VAR_spanner_name=<INSERT_YOUR_SPANNER_INSTANCE_NAME>

    For more information on how to make your Spanner instance to be managed by Terraform, see Import your Spanner instances

  4. Change directory into the Terraform app-project directory and initialize it.

    cd "${APP_DIR}"
terraform init

  5. Create the infrastructure in the application project. Answer yes when prompted, after reviewing the resources that Terraform intends to create.

    terraform import module.scheduler.google_app_engine_application.app "${APP_PROJECT_ID}"
terraform apply -parallelism=2

    If you are running this command in Cloud Shell and encounter errors of the form "Error: cannot assign requested address", this is a known issue in the Terraform Google provider, please retry with -parallelism=1

Authorize the Forwarder function to publish to the Poller topic

  1. Switch back to the Autoscaler project and ensure that Terraform variables are correctly set. ```sh cd "${AUTOSCALER_DIR}"

    export TF_VAR_project_id="${AUTO_SCALER_PROJECT_ID}"
export TF_VAR_region="${AUTO_SCALER_REGION}"
export TF_VAR_zone="${AUTO_SCALER_ZONE}"
export TF_VAR_location="${AUTO_SCALER_APP_ENGINE_LOCATION}"

  2. Set the Terraform variables for your Forwarder service accounts, updating and adding your service accounts as needed. Answer yes when prompted, after reviewing the resources that Terraform intends to create.

    export TF_VAR_forwarder_sa_emails='["serviceAccount:forwarder-sa@'"${APP_PROJECT_ID}"'.iam.gserviceaccount.com"]'
terraform apply -parallelism=2

If you are running this command in Cloud Shell and encounter errors of the form "Error: cannot assign requested address", this is a known issue in the Terraform Google provider, please retry with -parallelism=1

Verifying your deployment

Your Autoscaler infrastructure is ready, follow the instructions in the main page to configure your Autoscaler. Please take in account that In a distributed deployment: Logs from the Poller and Scaler functions will appear in the Logs Viewer for the Autoscaler project. Logs about syntax errors in the JSON configuration of the Cloud Scheduler payload will appear in the Logs viewer of each Application project, so that the team responsible for a specific Cloud Spanner instance can troubleshoot its configuration issues independently.