Archiving all-files scheduler #388
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
3coins
reviewed
Jun 28, 2023
for more information, see https://pre-commit.ci
JasonWeill
force-pushed
the
all-files-scheduler
branch
from
202fe09
to
5ac1f74
Compare
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
dlqqq
requested changes
Aug 9, 2023
There was a problem hiding this comment.
@JasonWeill Thank you for working on this! Left some feedback.
dlqqq
requested changes
Aug 9, 2023
There was a problem hiding this comment.
There is also a failing unit test blocking CI:
FAILED jupyter_scheduler/tests/test_job_files_manager.py::test_downloader_download[output_formats1-output_filenames1-staging_paths1-/home/runner/work/jupyter-scheduler/jupyter-scheduler/jupyter_scheduler/tests/test_files_output-False] - AssertionError: assert False
+ where False = <function exists at 0x7feb182f7420>('/home/runner/work/jupyter-scheduler/jupyter-scheduler/jupyter_scheduler/tests/test_files_output/helloworld-out.ipynb')
+ where <function exists at 0x7feb182f7420> = <module 'posixpath' (frozen)>.exists
+ where <module 'posixpath' (frozen)> = os.path
No need for an urgent fix here, since CI is failing on main anyways until we merge #417. However, this definitely should be fixed after rebasing and before merging.
Co-authored-by: david qiu <[email protected]>
for more information, see https://pre-commit.ci
|
Simplifies archiving and unarchiving logic per @dlqqq. Seeking external feedback about having these new classes replace the old |
JasonWeill
changed the title
JasonWeill
added a commit
to JasonWeill/jupyter-scheduler
that referenced
this pull request
Aug 11, 2023
* Fix typo in comment * WIP: Adds new scheduler * writes individual files * WIP: Write zip file * WIP: Trying to get zip file to be written only on scheduled job runs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * WIP: Removes zip type, incremental work for archiving work dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Create tar.gz in staging subdir * Capture side effect files in staging dir * Extracts files * Add filter * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update jupyter_scheduler/job_files_manager.py Co-authored-by: david qiu <[email protected]> * Simplifies cleanup logic * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates docs, deletes old Archiving*, renames AllFilesArchiving --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: david qiu <[email protected]>
This was referenced Aug 11, 2023
dlqqq
added a commit
that referenced
this pull request
Aug 15, 2023
* Archiving all-files scheduler (#388) * Fix typo in comment * WIP: Adds new scheduler * writes individual files * WIP: Write zip file * WIP: Trying to get zip file to be written only on scheduled job runs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * WIP: Removes zip type, incremental work for archiving work dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Create tar.gz in staging subdir * Capture side effect files in staging dir * Extracts files * Add filter * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update jupyter_scheduler/job_files_manager.py Co-authored-by: david qiu <[email protected]> * Simplifies cleanup logic * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates docs, deletes old Archiving*, renames AllFilesArchiving --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: david qiu <[email protected]> * Avoids option compatible only with Python 3.11 * Fix JFM tests (#424) * fix JFM tests * pre-commit * add minor comment --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: david qiu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP fix for #349.
Creates an
AllFilesArchivingExecutionManagerandAllFilesArchivingSchedulerclass, with the intention of archiving all output files and side effects on job runs.You can use these classes by running:
After the user chooses to download output files from a job, the side effect files are unpacked to wherever the first output file is located.
In the screen shot below, after downloading the output of a job that ran
writefile.ipynb, which produces the side effect filepun.txt, the side effect file is in thejobssubdirectory.