Add tests for Scheduler job and job definition creation with input folder, refactor execution manager test

[andrii-i]

• Add tests for Scheduler job and job definition creation with input folder
• Refactor DefaultExecutionManager test to use db fixtures

[dlqqq]

Thank you for contributing this! I left some feedback below. The feedback mostly centers around using fixtures instead of hard-coded paths, so these fixtures can be easily re-used in future tests.

[dlqqq]

1. os.path.join() should be used to construct file paths, as the directory delimiter can be platform-dependent.

   • Windows uses backslashes instead of forward slashes, e.g. C:\some_dir\child_dir.

2. When using directories in tests, we should not commit the test dirs into the source repo, as this doesn't scale well when more tests are added. Instead, you should use the tmp_path fixture provided by pytest. When used, this will create a new temporary directory (under /tmp on Linux) and return the path to that temporary directory.

   So here you would want to replace TEST_ROOT_DIR with something like:

   @pytest.fixture
   def scheduler_root_dir(tmp_path):
       return os.path.join(tmp_path, 'workspace_root')

For DB_FILE_PATH, you should use the jp_data_dir fixture provided by jupyter_server.pytest_plugin. This fixture is already under tmp_path.

[dlqqq]

We should have a fixture for the staging directory which uses the default staging directory name in Jupyter Scheduler:

@pytest.fixture
def staging_dir(jp_data_dir):
    return os.path.join(jp_data_dir, "scheduler_staging_area")

For the side effect test:

1. Move output_side_effect.txt and side_effects.ipynb under a new directory containing static files used in tests, e.g. jupyter_scheduler/tests/static.
2. Write a new staging_dir_with_side_effects fixture that copies these files to staging_dir, and returns a two-tuple of the input URI and the side effect file. So this fixture should return ('side_effects.ipynb', 'output_side_effect.txt')

[dlqqq]

Instead of hard-coding the job ID, you should modify this fixture to return the ID of the job record it created. This is more similar to how jobs are created by the Scheduler backend.

[dlqqq]

Similar to my previous comment, you can replace this with a input_file_with_input_dir fixture that moves import-helloworld.ipynb and a/b/helloworld.txt to scheduler_root_dir, and then return the tuple ('import-helloworld.ipynb', 'a/b/helloworld.txt').

[dlqqq]

In a 1:1 conversation, @andrii-i raised a good point that my feedback falls outside the scope of this PR, and that it would more appropriate to address this feedback for all tests (not just these 2 test files) in a separate PR. I'm tracking my previous feedback in a new issue: #514.

Awesome work!

[andrii-i]

Thank you for looking into this @dlqqq