Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deployer: add aws cluster.yaml file generation for parity with gcp #4752

Merged
merged 1 commit into from
Sep 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions config/clusters/templates/aws/cluster.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: {{ cluster_name }}
provider: aws # {{ sign_in_url }}
aws:
key: enc-deployer-credentials.secret.json
clusterType: eks
clusterName: {{ cluster_name }}
region: {{ cluster_region }}
billing:
paid_by_us: {{ paid_by_us }}
support:
helm_chart_values_files:
- support.values.yaml
- enc-support.secret.values.yaml
hubs: []
# Uncomment the lines below once the support infrastructure was deployed and
# you are ready to add the first cluster

# - name: {{ hub_name }}
# # Tip: consider changing this to something more human friendly
# display_name: "{{ cluster_name }} - {{ hub_name }}"
# domain: {{ hub_name }}.{{ cluster_name }}.2i2c.cloud
# helm_chart: {{ hub_type }}
# helm_chart_values_files:
# - common.values.yaml
# - {{ hub_name }}.values.yaml
# - enc-{{ hub_name }}.secret.values.yaml
22 changes: 22 additions & 0 deletions deployer/commands/generate/dedicated_cluster/aws.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

from .common import (
check_before_continuing_with_generate_command,
generate_cluster_config_file,
generate_config_directory,
generate_support_files,
)
Expand Down Expand Up @@ -105,6 +106,10 @@ def aws(
cluster_region: str = typer.Option(
..., prompt="The region where to deploy the cluster"
),
account_id: str = typer.Option(
...,
prompt="The AWS account id or alias. Declare 2i2c for 2i2c's SSO based accounts and paid_by_us=true",
),
force: bool = typer.Option(
False,
"--force",
Expand All @@ -115,6 +120,14 @@ def aws(
Automatically generate the files required to setup a new cluster on AWS if they don't exist.
Use --force to force existing configuration files to be overwritten by this command.
"""

if account_id == "2i2c":
sign_in_url = "https://2i2c.awsapps.com/start#/"
paid_by_us = True
else:
sign_in_url = f"https://{account_id}.signin.aws.amazon.com/console"
paid_by_us = False

# These are the variables needed by the templates used to generate the cluster config file
# and support files
vars = {
Expand All @@ -124,6 +137,8 @@ def aws(
"hub_type": "basehub",
"cluster_name": cluster_name,
"cluster_region": cluster_region,
"sign_in_url": sign_in_url,
"paid_by_us": str(paid_by_us).lower(),
}

if not check_before_continuing_with_generate_command(
Expand All @@ -133,8 +148,15 @@ def aws(

# If we are here, then either no existing infrastructure files for this cluster have been found
# or the `--force` flag was provided and we can override existing files.

# Automatically generate eksctl and terraform files
generate_infra_files(vars)

# Automatically generate the config directory
cluster_config_directory = generate_config_directory(vars)

# Create the cluster config directory and initial `cluster.yaml` file
generate_cluster_config_file(cluster_config_directory, "aws", vars)

# Generate the support files
generate_support_files(cluster_config_directory, vars)
29 changes: 5 additions & 24 deletions docs/hub-deployment-guide/new-cluster/new-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ We automatically generate the files required to setup a new cluster:
- A ssh public key used by `eksctl` to grant access to the private key.
- A `.tfvars` terraform variables file that will setup most of the non EKS infrastructure.
- The cluster config directory in `./config/cluster/<new-cluster>`
- The `cluster.yaml` config file
- The support values file `support.values.yaml`
- The the support credentials encrypted file `enc-support.values.yaml`
````
Expand Down Expand Up @@ -135,12 +136,13 @@ You can generate these with:
:sync: aws-key

```bash
export CLUSTER_NAME=<cluster-name>;
export CLUSTER_NAME=<cluster-name>
export CLUSTER_REGION=<cluster-region-like ca-central-1>
export ACCOUNT_ID=<declare 2i2c for clusters under 2i2c SSO, otherwise an account id or alias>
```

```bash
deployer generate dedicated-cluster aws --cluster-name=$CLUSTER_NAME --cluster-region=$CLUSTER_REGION
deployer generate dedicated-cluster aws --cluster-name=$CLUSTER_NAME --cluster-region=$CLUSTER_REGION --account-id=$ACCOUNT_ID
```

After running this command, you will be asked to provide the type of hub that will be deployed in the cluster, i.e. `basehub` or `daskhub`.
Expand Down Expand Up @@ -547,29 +549,8 @@ Create a `cluster.yaml` file under the `config/cluster/$CLUSTER_NAME>` folder an

````{tab-item} AWS
:sync: aws-key
```yaml
name: <your-cluster-name>
provider: aws # <copy paste link to sign in url here>
aws:
key: enc-deployer-credentials.secret.json
clusterType: eks
clusterName: $CLUSTER_NAME
region: $CLUSTER_REGION
billing:
# For an AWS account explicitly configured to have the cloud bill
# paid directly by the community and not through 2i2c, declare
# paid_by_us to false
paid_by_us: true
support:
helm_chart_values_files:
- support.values.yaml
- enc-support.secret.values.yaml
hubs: []
```

```{note}
The `aws.key` file is defined _relative_ to the location of the `cluster.yaml` file.
```
A `cluster.yaml` file should already have been generated as part of [](new-cluster:generate-cluster-files).
````

````{tab-item} Google Cloud
Expand Down