Skip to content
This repository has been archived by the owner on Dec 11, 2023. It is now read-only.

Adds Jupyter Notebook replay driver #105

Closed
wants to merge 42 commits into from
Closed

Conversation

ChrisKeefe
Copy link
Collaborator

@ChrisKeefe ChrisKeefe commented Mar 12, 2023

Like it says on the tin, this enables provenance replay to produce Jupyter Notebook (.ipynb) files.

Notes

  • Exposes a new usage_driver='jn' choice for provenance replay at the CLI and Python API
  • A Jupyter Notebook is generated (in addition to the CLI and Python3 scripts) within reproducibility supplement zip archives
  • Jupyter-notebook specific helptext is included in the notebook's "header" cell
  • Within the generated notebook, each QIIME 2 Action and any new immediate dependencies (e.g. Metadata.load() and get_column() calls) are enclosed in the same notebook cell.
  • use.comment() is no longer used internally to generate inline comments, as this will break JN cell semantics

Draft status

This code has been extensively playtested (and the CLI and Python drivers have been unit tested and diffed against regressions), but no unit tests have been written for the new driver at this time.

Review comments and playtesting feedback are welcome.

@gregcaporaso gregcaporaso self-assigned this May 3, 2023
@ChrisKeefe ChrisKeefe marked this pull request as ready for review September 25, 2023 04:43
@ChrisKeefe
Copy link
Collaborator Author

I incorporated the notebook driver into some basic tests this evening, and added a set of tests covering line breaks in comments which is a kinda gory place in the code to deal with. This changeset still introduces a small loss in coverage, as the notebook driver isn't tested against e.g. parsing multiple results, or certain metadata handling paths.

I need to wind down work on this PR for now - feel free to add those tests, playtest and merge, or make any changes required to get this across the finish line. Thanks, CK

@ChrisKeefe
Copy link
Collaborator Author

Looks like there's a build issue related to a missing seed-environment-conda.yml - lmk if there's anything needed on my end to resolve.

Traceback (most recent call last):
 == Using inputs == 
{
  "conda-activate": "source '/home/runner/work/provenance-lib/provenance-lib/alp-base-env/etc/activate.sh'",
  "seed-environment": "./distro/2023.9/core/staged/seed-environment-conda.yml",
  "conda-build-config": "conda_build_config.yml",
  "channels": "[\"./built-channel\"]",
  "skip-first-channel": "false"
}
 == Evaluated keyword arguments == 
{
  "conda_activate": "source '/home/runner/work/provenance-lib/provenance-lib/alp-base-env/etc/activate.sh'",
  "seed_environment": "./distro/2023.9/core/staged/seed-environment-conda.yml",
  "conda_build_config": "conda_build_config.yml",
  "channels": [
    "./built-channel"
  ],
  "skip_first_channel": "false"
}
 == Executing == 
  File "/home/runner/work/_actions/qiime2/action-library-packaging/beta/make-conda-config/bin/make-cbc.py", line 34, in <module>
    ActionAdapter(main)
  File "/home/runner/work/_actions/qiime2/action-library-packaging/beta/src/alp/common.py", line 39, in ActionAdapter
    results = function(**kwargs, **extras)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/_actions/qiime2/action-library-packaging/beta/make-conda-config/bin/make-cbc.py", line 11, in main
    with open(seed_environment) as fh:
         ^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './distro/2023.9/core/staged/seed-environment-conda.yml'
Error: Process completed with exit code 1.

@gregcaporaso
Copy link
Member

@ChrisKeefe, we're going to close this PR, but it's referenced from the framework where we'll ultimately integrate this. We'll need to work on migrating this to a framework PR on our end.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants