Scripts

CI docker images

Images that can be used in CI systems that run just docker containers.

Install all required dependencies in your Dockerfile.ci and publish an image:

    cd ci
    bash build.sh --push

Then run any command:

    ./ci.sh java -version

SBT

Dockerized sbt ci image witch cached dependencies from your project.

To build:

0. Copy ci folder into your project.
1. Move sbt Dockerfile.ci into ci folder.
2. Publish a ci image.
3. Run any sbt command inside: ./ci.sh sbt about.

DB utils

• connect to remote tunnel
• decrypt local .pgpass
• dump between two remotes

Airflow contrib

• AWS SES email sending module. Put this file into Airflow PYTHONPATH and set AIRFLOW__EMAIL__EMAIL_BACKEND=aws_ses.send_raw_email env variable.
• Operators: -- BigQuery insert operator using native python SDK and streaming API

Run tests

    PYTHONPATH=airflow python3 -m unittest discover test

To run a specific test:

    PYTHONPATH=airflow python3 -m unittest test.test_operators

AWS shortlist

Encrypt kms key:

    aws kms encrypt --key-id=<key_id> --plaintext "<secret>"

Decrypt kms key:

    aws kms decrypt --ciphertext-blob fileb://<(echo "<key_string>" | base64 --decode) --output text --query Plaintext | base64 --decode