Skip to content

Commit

Permalink
feat: attempt v2 of gwas catalog batch job
Browse files Browse the repository at this point in the history
  • Loading branch information
Szymon Szyszkowski committed Aug 5, 2024
1 parent 3cbdfde commit 701ffa2
Show file tree
Hide file tree
Showing 5 changed files with 496 additions and 45 deletions.
34 changes: 34 additions & 0 deletions config/configv2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
DAG: GWAS_Catalog
mode: CONTINUE
gcp:
GCP_PROJECT: "open-targets-genetics-dev"
GCP_REGION: "europe-west1"
GCP_ZONE: "europe-west1-d"
GCP_DATAPROC_IMAGE: "2.1"
GCP_AUTOSCALING_POLICY: "otg-etl"

steps:

manifest_preparation:
raw_sumstat_path_pattern: gs://ot_orchestration/raw_summary_statistics/**/*.h.tsv.gz
staging_manifest_path_pattern: gs://ot_orchestration/staging/**/manifest.json
harmonised_prefix: harmonised
qc_prefix: qc

gwas-catalog-etl:
googlebatch:
resource_specs:
cpu_milli: 2000
memory_mib: 2000
boot_disk_mib: 10000
task_specs:
max_retry_count: 0
max_run_duration: "10m"
policy_specs:
machine_type: n1-standard-2
image: europe-west1-docker.pkg.dev/open-targets-genetics-dev/ot-orchestration/genetics_etl:dev
commands: ["exit 1"]
spark:
spark_uri: yarn
write_mode: errorifexists
2 changes: 1 addition & 1 deletion src/ot_orchestration/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from ot_orchestration.cli.process_in_batch import gwas_catalog_pipeline

logging.basicConfig(level=logging.INFO)
asci_art = """
asci_art = r"""
____ ______ ____ __ __ __ _
/ __ \/_ __/ / __ \__________/ /_ ___ _____/ /__________ _/ /_(_)___ ____
/ / / / / / / / / / ___/ ___/ __ \/ _ \/ ___/ __/ ___/ __ `/ __/ / __ \/ __ \\
Expand Down
Loading

0 comments on commit 701ffa2

Please sign in to comment.