Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-haul this example with more realistic example #2

Open
wants to merge 143 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
143 commits
Select commit Hold shift + click to select a range
ddbcf72
parent 8a400fe4b839f7d8407729f8791707b55db652c5
Sep 3, 2019
335806c
add more comments to cloudbuild.yaml
Mar 25, 2020
5202415
fix repo reference to github
Mar 25, 2020
8bca8ac
Merge pull request #11 from jaketf/patch/repoReference
Mar 25, 2020
88afa95
Merge branch 'master' into feature/comments
Mar 25, 2020
6ce3d86
add linters
Mar 26, 2020
7a92ab0
Merge pull request #9 from jaketf/feature/comments
Mar 26, 2020
34c334c
add dir
Mar 26, 2020
6ea5438
test without explicit pull
Mar 26, 2020
937efa5
remove REPO_ROOT
Mar 26, 2020
16f18d0
relative paths
Mar 26, 2020
425d74c
fix java linter check
Mar 26, 2020
5b4638b
fix helper scripts and yapf python
Mar 26, 2020
293bdb2
Merge pull request #13 from jaketf/feature/linters
Mar 26, 2020
9f5f33b
terraform and bash linting
Mar 28, 2020
f5c87cf
fix Dockerfile
Mar 28, 2020
7e18e88
shellcheck format.sh
Mar 28, 2020
c871fcd
Merge pull request #14 from jaketf/feature/linters
Mar 28, 2020
f68d3ef
add run terraform
Mar 28, 2020
1c00bc0
update terraform
Mar 28, 2020
a2c4abf
bucket location
Mar 28, 2020
c6353b0
update composer add timeouts
Mar 29, 2020
75c9716
remove terraform apply
Mar 29, 2020
0f16cd1
more docker more tests
Mar 29, 2020
cac62a4
remove errant line in cloud build
Mar 29, 2020
07670ac
fix cloud build typo
Mar 29, 2020
2c47c2f
remove timeout
Mar 29, 2020
9aedb04
fix entry point, new composer env
Mar 30, 2020
bb98c70
fail fast on missing flags and fix flag typo
Mar 30, 2020
cf7925c
set project for gcloud commands, don't bake dags into deployer container
Mar 30, 2020
b2ca6b2
update docs
Mar 30, 2020
1ca8d03
s/is/id
Mar 30, 2020
53d08ca
Merge pull request #15 from jaketf/feature/terraform
Mar 30, 2020
e22553a
always unpause airflow monitoring
Mar 30, 2020
c00cf70
Merge pull request #17 from jaketf/patch/unpauseMonitoring
Mar 31, 2020
50979cd
make runt_tests.sh more configurable
Mar 31, 2020
4af04f0
fix typo in envsubst
Mar 31, 2020
5d49c2d
fix gcs references
Mar 31, 2020
becbea3
fix gcs dags path
Apr 1, 2020
6d98828
Merge pull request #18 from jaketf/feature/moreFlags
Apr 2, 2020
a03557f
pause if not replace
Apr 3, 2020
5ec7ec2
improve dag validation tests
Apr 3, 2020
75843d2
use official mvn
Apr 3, 2020
69fd599
Merge pull request #23 from jaketf/feature/pauseIfNotReplace
Apr 3, 2020
0fe6f88
Merge branch 'master' into feature/improveDagValidationTests
Apr 3, 2020
930aec1
Merge pull request #24 from jaketf/feature/improveDagValidationTests
Apr 3, 2020
5e4ebbc
parallelize some build steps
Apr 3, 2020
f02d2f3
fix style typo
Apr 3, 2020
2e42119
fix wordcount typo
Apr 3, 2020
e750719
fix tests typo
Apr 3, 2020
2a2fbfc
Merge pull request #25 from jaketf/optimization/parallelizeBuildSteps
Apr 3, 2020
c7dc04e
standardize cloudbuild.yaml for readability
Apr 6, 2020
1da91a5
fix pull-deoploydags ref
Apr 6, 2020
ec8accb
Merge pull request #26 from jaketf/pretty/yaml
Apr 6, 2020
24cc776
handle nested dags folder
Apr 11, 2020
5769465
go fmt
Apr 11, 2020
3435bd9
fix deploydags always starts stoped dags bug
Apr 11, 2020
7595410
add missing return statement
Apr 13, 2020
d8c6935
wip
Apr 15, 2020
6e877e0
retab
Apr 21, 2020
656ea1c
go fmt
Apr 21, 2020
643953f
4 space tabs
Apr 21, 2020
7a393ac
fix linter
Apr 22, 2020
1ef7178
handling of nested dags with .airflowignore tests passing
Apr 22, 2020
c0f9247
remove overly verbose logs statements in dag finding
Apr 22, 2020
3e174fa
Merge pull request #27 from jaketf/feature/nestedDagsFolder
Apr 22, 2020
fa498b3
add variables for ci and prod
May 27, 2020
bf016c8
all branch prs
May 27, 2020
95f2297
restructure for nested cloudbuild
Jun 10, 2020
f36fa02
Revert "restructure for nested cloudbuild"
Jun 10, 2020
27b5dc6
Revert "Revert "restructure for nested cloudbuild""
Jun 10, 2020
940b0e8
fix cloudbuild missing comma
Jun 10, 2020
0f83bfc
remove unused refs
Jun 10, 2020
f25ab89
terragrunt multi-project
Jun 10, 2020
5b3018a
remove merge dep in precommit
Jun 10, 2020
337f62d
init git repo
Jun 10, 2020
ce6e3b8
revise pre-commit strategy
Jun 10, 2020
16f172a
checkout pr
Jun 11, 2020
8830fd3
add empty commit
Jun 10, 2020
1f03188
add pr number env var
Jun 11, 2020
4a050bf
add pr num to precommit
Jun 11, 2020
70e9742
patch post commit refs
Jun 11, 2020
6ec0bc6
fix post commit refs and remove terragrunt cache
Jun 11, 2020
cbafe2b
missing comma
Jun 11, 2020
13d5174
fix init repo
Jun 11, 2020
5ce5698
try agian
Jun 11, 2020
816c184
walk dagsRoot
Jun 11, 2020
e8f4fe0
patch composer ops
Jun 11, 2020
c7d6b2b
revent dags deployer changes
Jun 11, 2020
88648bd
debug print
Jun 11, 2020
d59a1f7
add func
Jun 11, 2020
1c29f57
more debug
Jun 11, 2020
03d2576
func
Jun 11, 2020
474f5ee
where does deploy dags run?
Jun 11, 2020
ca1c1fa
fixed dags deployer
Jun 11, 2020
8311e64
run precommits
Jun 11, 2020
dc8583b
fix redundant dir in dataflow cloudbuild.yaml
Jun 11, 2020
bbab516
dataflow builds
Jun 11, 2020
e41e0ad
fix makefile
Jun 12, 2020
8bcbc5f
fix cloud build finder
Jun 12, 2020
6547f0c
make extra args optional
Jun 12, 2020
0e7929e
add separate ids
Jun 12, 2020
cd1e29e
use unittest cuz pytest not in airflow image
Jun 12, 2020
92c86a2
package the whole repo for precommits
Jun 12, 2020
400be9a
yapf
Jun 12, 2020
253eb71
remove jinja templating
Jun 12, 2020
c065b3f
fix test_sql.sh
Jun 12, 2020
ba9469f
fix root cloudbuild typo
Jun 12, 2020
6cc22f7
fix root cloudbuild
Jun 12, 2020
960c049
fix reference to helper script
Jun 12, 2020
4869d1b
fix passing of additional build args
Jun 12, 2020
fb9d760
make additional args optional
Jun 12, 2020
4a59d05
use default machine type in precommit
Jun 12, 2020
ad251f9
fix wordcount cloudbuild
Jun 12, 2020
d91dee7
fix wordcount cloudbuild
Jun 12, 2020
0837329
improve wordcount cloudbuild variable overriding
Jun 12, 2020
f8806b4
fix word variable setting
Jun 12, 2020
74ec191
restructure for nested cloudbuild (#31)
Jun 12, 2020
4b6bb0a
[skip ci] update readmes
Jun 13, 2020
6b2ccb6
Merge pull request #39 from jaketf/docs
Jun 13, 2020
61d8897
bad demo commit
Jun 26, 2020
b0d92f0
refactor running dags
Jul 8, 2020
3a965c5
log
Jul 8, 2020
82f6653
more log
Jul 8, 2020
ebd8091
sleep
Jul 8, 2020
c4c3b2c
deploy patched examples
Jul 8, 2020
ada99b0
test changes
Jul 8, 2020
0a8febc
fixup
Jul 8, 2020
62df806
fix precommit project sub
Jul 8, 2020
b8d7008
deploydags image
Jul 8, 2020
061e33d
remove duplicate dag
Jul 8, 2020
e6857f7
modify helper
Jul 8, 2020
62dd816
use google/cloud-sdk
Jul 9, 2020
39d164a
remove extra yaml
Jul 9, 2020
d1a09a1
remove double run of airflow tests
Jul 9, 2020
ce5dd27
remove double run bq tests
Jul 9, 2020
fae4072
don't reinstall airflow
Jul 9, 2020
52d891f
parallel pre-commit steps
Jul 9, 2020
a71ee24
Merge pull request #45 from jaketf/demo
Jul 9, 2020
7862c02
remove example dags
Jul 9, 2020
f498b97
Merge pull request #46 from jaketf/removeDags
Jul 9, 2020
6f82e22
remove all running_dags.txt
Jul 9, 2020
87b9384
Merge pull request #47 from jaketf/revertDagRemoval
Jul 9, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[flake8]
max-line-length=80
exclude=.git,__pycache__,.venv
146 changes: 146 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
target/
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# static files generated from Django application using `collectstatic`
media
static

.idea/
28 changes: 0 additions & 28 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,28 +0,0 @@
# How to Contribute

We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.

## Contributor License Agreement

Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution;
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to <https://cla.developers.google.com/> to see
your current agreements on file or to sign a new one.

You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.

## Code reviews

All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.

## Community Guidelines

This project follows [Google's Open Source Community
Guidelines](https://opensource.google.com/conduct/).
26 changes: 25 additions & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
Expand Down Expand Up @@ -175,3 +174,28 @@
of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

APPENDIX: How to apply the Apache License to your work.

To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright 2019 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
40 changes: 40 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Make will use bash instead of sh
SHELL := /usr/bin/env bash

# The .PHONY directive tells make that this isn't a file target
.PHONY: fmt
fmt: ## Format files, including README
@$$SHELL ./helpers/format.sh

help: ## Prints help for targets with comments
@grep -E '^[a-zA-Z._-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "make \033[36m%- 30s\033[0m %s\n", $$1, $$2}'

.PHONY: test
test: ## Test if all files are properly formatted
@$$SHELL ./helpers/check_format.sh && python3 -m flake8 --max-line-length=100 && ./helpers/run_tests.sh

.PHONY: precommit
precommit: ## Test if all files are properly formatted
@$$SHELL ./helpers/check_format.sh && python3 -m flake8 --max-line-length=100 && ./helpers/run_relevant_cloudbuilds.sh precommit_cloudbuild.yaml

.PHONY: push_ci_image
push_ci_image:
@cd ci && gcloud builds submit --project=datapipelines-ci-282719 --tag gcr.io/datapipelines-ci-282719/make .

.PHONY: push_deploydags_image
push_deploydags_image:
@cd composer/cloudbuild/go/dagsdeployer && gcloud builds submit --project=datapipelines-ci-282719 --tag gcr.io/datapipelines-ci-282719/deploydags .

Loading