Skip to content

Commit

Permalink
Merge pull request #3 from Nike-Inc/improve-error-message-for-incorre…
Browse files Browse the repository at this point in the history
…ct-alias-lookup

Improve error message when using AWS alias not inline with current the current AWS account being used.
  • Loading branch information
marcinjzalewski authored Dec 21, 2021
2 parents 7c5eeb5 + 1d70f0c commit 062e613
Show file tree
Hide file tree
Showing 33 changed files with 304 additions and 135 deletions.
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,6 @@ Run unit tests with coverage report in HTML format.
```bash
poetry run pytest --cov-report=html --cov=pterradactyl --cov-fail-under=80 tests/
```
Note: keep line code coverage on at least 80% level.

# <a name="examples"></a> Examples of creating new projects/prodcuts:

Expand Down Expand Up @@ -130,7 +129,7 @@ containing downloaded given Terraform version with all required plugins, and wor
│ ├── terraform
│ └── terraform-provider-kubectl_v1.13.1
└── workspace
└── vt-projectc0-na-useast1
└── bt-projectc0-na-useast1
├── facts.json
└── main.tf.json
```
Expand Down
71 changes: 44 additions & 27 deletions examples/complex/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
## pterradyctal setup for complex project having remote s3 backend, encrypted files and streamlined tags support.
Pterradyctal setup for complex project having remote s3 backend, encrypted files and streamlined tags support.
---

Table of content
* [Creating infrastructure for a new AWS account](#newaccount)
* [Infra folder](#infrafolder)
* [Creating stacks for new projects](#newstack)
* [Remote S3 backend support](#remotebackend)
* [Encrypted files support (sops)](#sops)
* [Streamlined tags support](#tags)

## Creating infrastructure for a new AWS account
# <a name="newaccount"></a> Creating infrastructure for a new AWS account
Before we start creating stacks for our projects, we need to define a new AWS account and create basic infrastructure for it. See [infra](../complex/vars/project/infra/) example.
Let's call new AWS account a `common` one

Expand All @@ -10,16 +19,16 @@ Let's call new AWS account a `common` one
account_family_code = {
common = "c" # --> new account
restricted = "r"
team1 = "e"
team2 = "v"
teama = "a"
teamb = "b"
}
```
- edit [deployment/main.tf](../complex/terraform/modules/deployment/main.tf):
```terraform
account_families = {
"_default_" = "common" # <-- add default for common account.
"projecta" = "team1"
"projectb" = "team2"
"projecta" = "teama"
"projectb" = "teamb"
}
}
```
Expand All @@ -29,8 +38,8 @@ Let's call new AWS account a `common` one
complex:
family:
c: common # <-- new account entry
e: team1
v: team2
a: teama
b: teamb
account_type:
t: test
p: production
Expand Down Expand Up @@ -71,27 +80,32 @@ Let's call new AWS account a `common` one
- Again run the pt apply command:
- `pt apply ct-infra0-uswest2`

## Creating stacks for new projects
# <a name="infrafolder"></a> Infra folder
We are supposed to create a basic deployment to keep our stacks backend remotely on S3 bucket.
In general, we deploy only one infra and share it among stacks.
Take a look into [infra](../complex/vars/project/infra) folder to see what is the minimum for deploying infra stack.

# <a name="newstack"></a> Creating stacks for new projects
A below complex example of creating new projects based on AWS provider.

We have an organization named complex\
They have 2 teams
- team1
- team1 has only one project going on, named projecta and it's deployed both in test and prod account
- team2
- team2 have 2 projects
- teama
- teama has only one project going on, named projecta and it's deployed both in test and prod account
- teamb
- teamb have 2 projects
- projectb - deployed in test and prod
- projectc - This is a long project to put out the projectc, it's still in test.

This is how the stack looks like for teams

| Team | Project | Account Type | Stack Name | Stack contents |
|---|---|---|---|---|
| team1 |projecta | test | et-projecta | s3 (bucket-1), dynamodb(stream_enabled: false) |
| teama |projecta | test | at-projecta | s3 (bucket-1), dynamodb(stream_enabled: false) |
| | | prod | ep-projecta | s3 (bucket-1), dynamodb(stream_enabled: true) |
| team2 |projectb | test | vt-projectb | s3 (bucket-1), |
| teamb |projectb | test | bt-projectb | s3 (bucket-1), |
| | | prod | vp-projectb | s3 (bucket-1), SQS ( fifo: false) |
| |projectc | test | vt-projectc | s3 (bucket-1), s3 (bucket-2), dynamodb(stream_enabled: false), SQS ( fifo: true) |
| |projectc | test | bt-projectc | s3 (bucket-1), s3 (bucket-2), dynamodb(stream_enabled: false), SQS ( fifo: true) |

As you could see [here](../complex/terraform/modules). We are using much more modules than in the Simple project [here](../simple/terraform/modules)
Additional modules used here:
Expand All @@ -106,11 +120,11 @@ Each account can have more than one version of the same stack, meaning you can h
These different stacks could be effectively just copies of one another or could have overrides like in prod instance the size of the node or db might be different than the test account.\
The above is a very typical setup where every account needs some common things and Prod and Test are slightly different from each other.\
Now, its time you open up the examples directory and check out the structure for the simple project.\
Lets checkout the state of the stack for team1 in the test accounts using the following command
Lets checkout the state of the stack for teama in the test accounts using the following command

`pt plan et-projecta0-uswest2.yaml`
`pt plan at-projecta0-uswest2.yaml`

- e - team1
- e - teama
- projecta - project
- 0 - version of the stack
- uswest2 - region of the deployment
Expand Down Expand Up @@ -152,11 +166,11 @@ hierarchy:
datadir: vars

# Specify acronyms here
# This is derived from stack name e.g. vt acronym in vt-projecta-na-uswest2 stack name.
# This is derived from stack name e.g. vt acronym in bt-projecta-na-uswest2 stack name.
simple:
family:
e: team1
v: team2
a: teama
b: teamb
account_type:
t: test
p: production
Expand Down Expand Up @@ -230,12 +244,12 @@ pterradactyl: <-- version of the pterradctyl library

Similarly, you can do all the terraform commands (apply, state, etc) on any of these stacks.

e.g. do apply on team2 projectc project
`pt apply vt-projectc0-useast1.yaml`
e.g. do apply on teamb projectc project
`pt apply bt-projectc0-useast1.yaml`

This stack has a remote backend, type: S3.

### Remote S3 backend support
# <a name="remotebackend"></a> Remote S3 backend support

To store your backend remotely, in one of your common.yaml top hierarchy file, e.g. [here](../complex/vars/common.yaml), one have to provide below information:
```yaml
Expand All @@ -257,7 +271,10 @@ Variables to provide:
Under S3 bucket '_%{account_prefix}-infra0-global-terraform_' you will see backend state file e.g. _ct-infra0-uswest2.tfstate.json_
### Encrypted files support (sops)
# <a name="sops"></a> Encrypted files support (sops)
Installation guide for installing sops under Mac OS can be found [here](https://formulae.brew.sh/formula/sops)
Installation guide for installing sops from pypi can be found [here](https://pypi.org/project/sops/)
If you want to provide encrypted credentials as YAML configuration one should create separate file with .enc extension.
E.g. [here](../complex/vars/account/projectb-prod.yaml.enc)
Expand Down Expand Up @@ -319,7 +336,7 @@ Decrypting is as simple as running below sops command:
sops -d -i --input-type=yaml --output-type=yaml vars/account/projectb-prod.yaml.enc
```

### Streamlined tags support.
# <a name="tags"></a> Streamlined tags support.
To propagate tags, metadata across all resources look at [resource_metadata](../complex/terraform/modules/resource_metadata) generic module.

Any tags you want to propagate across all resources, simply create metadata.tf file like for [s3](../complex/terraform/modules/s3/metadata.tf) module.
Expand Down
8 changes: 4 additions & 4 deletions examples/complex/pterra.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ hiera:
datadir: vars

complex:
# This mapping allows us to get family and account_type form stack name: e.g. vt-projecta0-uswest2
# vt refers to team1 and test environment.
# This mapping allows us to get family and account_type form stack name: e.g. bt-projecta0-uswest2
# vt refers to teama and test environment.
family:
c: common
e: team1
v: team2
a: teama
b: teamb
account_type:
t: test
p: production
Expand Down
4 changes: 2 additions & 2 deletions examples/complex/terraform/modules/deployment/main.tf
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
locals {
account_families = {
"_default_" = "common"
"projecta" = "team1"
"projectb" = "team2"
"projecta" = "teama"
"projectb" = "teamb"
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ locals {
account_family_code = {
common = "c"
restricted = "r"
team1 = "e"
team2 = "v"
teama = "a"
teamb = "b"
}

# defined by naming convention
Expand Down
2 changes: 1 addition & 1 deletion examples/complex/vars/project/projecta/common.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ manifest:

module:
dynamodb:
name: team1-test-table
name: teama-test-table
hash_key: node_id
stream_enabled: false
attributes:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module:
name: ep-projecta-bucket

dynamodb:
name: team1-prod-table
name: teama-prod-table
stream_enabled: true
stream_view_type: "NEW_AND_OLD_IMAGES"
metadata: ${module.deployment.metadata}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module:
basic_bucket:
name: at-projecta-bucket

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module:
basic_bucket:
name: bt-projectb-bucket

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ manifest:
- s3 as not_basic_bucket
module:
basic_bucket:
name: vt-projectc-bucket
name: bt-projectc-bucket
not_basic_bucket:
name: vt-projectc-bucket-not-basic
name: bt-projectc-bucket-not-basic
sqs:
name: vt-projectc-sqs
name: bt-projectc-sqs
fifo_queue: true
projectc_table:
name: team2-projectc-table
name: teamb-projectc-table
stream_enabled: true
stream_view_type: "NEW_AND_OLD_IMAGES"
hash_key: node_id
Expand Down
36 changes: 18 additions & 18 deletions examples/simple/README.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,36 @@
# Simple pterradyctal setup

Simple Pterradyctal setup
---

We have an organization named simple
They have 2 teams
- team1
- team1 has only one project going on, named projecta and its deployed both in test and prod account
- team2
- team2 have 2 projects
- teama
- teama has only one project going on, named projecta and its deployed both in test and prod account
- teamb
- teamb have 2 projects
- projectb - deployed in test and prod
- projectc - This is a long project to put out the projectc, its still in test.

This is how the stack looks like for teams

| Team | Project | Account Type | Stack Name | Stack contents |
|---|---|---|---|---|
| team1 | projecta | test | et-projecta | s3 (bucket-1), dynamodb(stream_enabled: false) |
| teama | projecta | test | at-projecta | s3 (bucket-1), dynamodb(stream_enabled: false) |
| | | prod | ep-projecta | s3 (bucket-1), dynamodb(stream_enabled: true) |
| team2 | projectb | test | vt-projectb | s3 (bucket-1), |
| teamb | projectb | test | bt-projectb | s3 (bucket-1), |
| | | prod | vp-projectb | s3 (bucket-1), SQS ( fifo: false) |
| |projectc | test | vt-projectc | s3 (bucket-1), s3 (bucket-2), dynamodb(stream_enabled: false), SQS ( fifo: true) |
| |projectc | test | bt-projectc | s3 (bucket-1), s3 (bucket-2), dynamodb(stream_enabled: false), SQS ( fifo: true) |



Each account can have more than one version of the same stack, meaning you can have intergation stack running in the test account. \
These different stacks could be effectively just copies of one another or could have overrides like in prod instance the size of the node or db might be different than the test account.\
The above is a very typical setup where every account needs some common things and Prod and Test are slightly different from each other.\
Now, its time you open up the examples directory and check out the structure for the simple project.\
lets checkout the state of the stack for team1 in the test accounts using the following command
lets checkout the state of the stack for teama in the test accounts using the following command

`pt plan et-projecta0-na-uswest2.yaml`
`pt plan at-projecta0-na-uswest2.yaml`

- e - team1
- e - teama
- t - test environment
- projecta - project name
- 0 - version of the stack
Expand Down Expand Up @@ -72,11 +72,11 @@ hierarchy:
datadir: vars

# Specify acronyms here
# This is derived from stack name e.g. vt acronym in vt-projecta-na-uswest2 stack name.
# This is derived from stack name e.g. vt acronym in bt-projecta-na-uswest2 stack name.
simple:
family:
e: team1
v: team2
a: teama
b: teamb
account_type:
t: test
p: production
Expand Down Expand Up @@ -112,7 +112,7 @@ facter:
- shell:
git_sha: git rev-parse HEAD
aws_account_alias:
command: echo "team1-test" # e.g current AWS alias: `aws iam list-account-aliases`
command: echo "teama-test" # e.g current AWS alias: `aws iam list-account-aliases`
#jsonpath: $.AccountAliases[0]

# Facts set from the value of an environment variable
Expand Down Expand Up @@ -150,7 +150,7 @@ pterradactyl: <-- version of the pterradctyl library

Similarly, you can do all the terraform commands (apply, state, etc) on any of these stacks.

e.g. do apply on team2 projectc project
`pt apply vt-projectc0-na-useast1`
e.g. do apply on teamb projectc project
`pt apply bt-projectc0-na-useast1`

This stack does not have a remote bucket or encrypted credentials or even a streamlined tags support.
8 changes: 4 additions & 4 deletions examples/simple/pterra.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,12 @@ hiera:
datadir: vars

simple:
# This mapping allows us to get family and account_type form stack name: e.g. vt-projecta0-uswest2
# vt refers to team1 and test environment.
# This mapping allows us to get family and account_type form stack name: e.g. bt-projecta0-uswest2
# vt refers to teama and test environment.
family:
c: common
e: team1
v: team2
a: teama
b: teamb
account_type:
t: test
p: production
Expand Down
2 changes: 1 addition & 1 deletion examples/simple/vars/project/projecta/common.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ manifest:

module:
dynamodb:
name: team1-test-table
name: teama-test-table
metadata: ${module.deployment.metadata}
hash_key: node_id
stream_enabled: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@ module:
name: ep-projecta-bucket

dynamodb:
name: team1-prod-table
name: teama-prod-table
stream_enabled: true
stream_view_type: "NEW_AND_OLD_IMAGES"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module:
basic_bucket:
name: at-projecta-bucket

This file was deleted.

Loading

0 comments on commit 062e613

Please sign in to comment.