feat: WDL Script Overhaul and Squash #1006
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
September 9, 2022 16:01
Through regular meetings with a potential user, the default behavior of a basic build is being redesigned for the base case: If a user only provides a sequence and metadata file pair (no build.yaml file), then: * An open reference file will be included in the builds.yaml to provide phylogenetic context * All the sequences will be included by default * The user provided sequencees will be colored by a "custom data" field as described by: https://docs.nextstrain.org/projects/ncov/en/latest/tutorial/custom-data.html#break-down-the-command
Squash several wdl Terra ingest changes into one commit. Changes listed below: * Splits the ncov_ingest task into gisaid_ingest and genbank_ingest * Can optionally take a compressed or decompressed nextclade cache file to reduce runtime * Parameterizes cpu and memory usage depending on new or cached run * Captures log files in the results.zip folder * Adds an optional tsv filter to replicate the regional datasets * Logs the date in a LAST_RUN output string * Compress with zstd, however can decompress xz or zst cache
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
Major overhaul of the WDL scripts motivated by #1005 with summarized changes below:
Through regular meetings with a potential user, the default behavior of a basic build was redesigned for the base case where a user only provides a sequence and metadata file pair (no build.yaml file):
https://docs.nextstrain.org/projects/ncov/en/latest/tutorial/custom-data.html#break-down-the-command
Squash several Terra WDL ingest changes (historically on
wdl/genbank_ingestandwdl/gisaid_ingest) into one commit. Changes listed below:Related issue(s)
Related to #1005
Testing
Testing here might be premature, at least until I can subsequently update the documentation (#999) to point at the new Dockstore entries.
However for the reviewer with time/incentive, testing is possible via a separate repo: https://github.com/j23414/wdl_pathogen_build where all three dockstore entries are available.
Assuming the reviewer has access to our development Terra workspace:
Can kick off three tests from the "WORKFLOWS" tab, by starting with the "test_genbank_ingest", "test_gisaid_ingest", and "ncov" cards.
I'll leave this up til end of day Tuesday just in case there are comments, otherwise will merge so I can start editing documentation.