[1pt] PR: Copy files to scratch directory for processing, and remove upon completion #1045
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to avoid file system collisions on AWS (keep the reads from the same file on disk to a minimum) four files (
HUC6_dem_domain.gpkg
,nws_lid.gpkg
,reformat_ras_rating_curve_points_rel_101.gpkg
, &usgs_gages.gpkg
) are now copied from the inputs directory into a scratch directory (temporary working directory), and removed after processing steps are completed.Resolves #1043 - (Merged
dependabot/pip/jupyter-server-2.11.2
branch and tested)Changes
config
/deny_unit.lst
: Add files to remove list - repetitive copies needed for processing step (run_unit_wb.sh
)src
bash_variables.env
: Add a new variable for the ras rating curve filename. It will be easier to track the filename in the.env
, and pull intorun_unit_wb.sh
, rather than hardcode it.run_unit_wb.sh
: Copy files and update references from$inputsDir
to$tempHucDataDir
.Testing
Merged in PR #1043, built new docker container.
Ran fim_pipeline using new container and this branch.
No difference in hashes, using
tools/hash_compare.py
with outputs from05030104
using dev branch, and this branch.Ran BED Domain on AWS, using the Step Function Workflow successfully.
Issuer Checklist (For developer use)
You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.
[_pt] PR: <description>
dev
branch (the default branch), you have a descriptive Feature Branch name using the format:dev-<description-of-change>
(e.g.dev-revise-levee-masking
)dev
branchpre-commit
hooks were run locally/foss_fim/
, run:pytest unit_tests/
)4.x.x.x
Merge Checklist (For Technical Lead use only)