-
Notifications
You must be signed in to change notification settings - Fork 6
/
TODO
185 lines (126 loc) · 9.14 KB
/
TODO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
Imminent: run high-level non-automatic tests. DONE
Imminent: /bin/seamless --local feature, pilot test. DONE
Imminent: seamless-jupyter not working because of jsonschema mismatch (conda AND Docker image).
Re-run imperative-nested-jupyter.sh, as well as other jupyter tests (check for "jupyter" in test files). DONE
Imminent: seamless-fingertip doesn't work if you remove the result but not the deepcell. DONE
Imminent: imperative-delegated fails, fix. DONE
Imminent: rename seamless-bash, seamless-XXX-trusted. DONE
Imminent:
/seamless/bin/seamless already works, but other tools in /seamless/bin do not,
since they require SEAMLESS_TOOLS_DIR to be defined (devel Conda only)
Harmonize /seamless/bin/seamless with /seamless-tools/seamless-cli
(seamless-upload-XXX is already in both)
DONE
TO DOCUMENT:
1. /seamless-tools/scripts
2. /seamless-tools/scripts/utils
3. /seamless/bin
4. /seamless-tools/seamless-cli
1.
- Python scripts (and bash runners) that need to import Seamless
2. Bash scripts and file-centered command line tools that need to run inside a Seamless environment
3.
- bash front to 1. and 2. (using SEAMLESS_SCRIPTS_DIR),
- /bin/seamless, with _seamlessify and activate-seamless-mode.sh
4.
- Unix command line tools that fire up a Seamless Docker image
- Python scripts that do not need to import Seamless
- Seamless assistants and their Docker files
- bash front to 1. and 2. (using Docker).
- bash front to /bin/seamless (using Docker)
- Add seamless-buffer-length DONE
- Add --index and --directory options to seamless-download. DONE
- Add --file argument to /bin/seamless. Mapping will be the same as for other files. Works for directories, too. DONE
- /bin/seamless: --dry, DONE
- /bin/seamless: --write-job, requires --dry. To write:
- outputfile.JOB, corresponds to transformation checksum, needs to be uploaded (and add outputfile.JOB.CHECKSUM support to run-transformation )
- outputfile.COMMAND, corresponds to the generated bash code (needs to be uploaded since its checksum is referenced)
- outputfile.DUNDER (if not empty)
- outputfile.ENV (needs to be uploaded since its checksum is referenced in the dunder file)
DONE
- run-transformation: Add --from-job support:
- If transformation.JOB.CHECKSUM present: load its checksum and run
- Else: uploads .JOB, .COMMAND and (if present) .ENV, calculate .JOB.CHECKSUM and run.
Uses transformation.DUNDER as dunderfile if present.
DONE
- /bin/seamless: add --docker, --conda-env support, to build env/dunder, make a test. DONE
- /bin/seamless: --upload, requires --dry. Uploads all input files, as well as .JOB, .COMMAND and .ENV. If --write-job, outputfile.JOB, outputfile.COMMAND and outputfile.ENV are not written to disk; instead write outputfile.XXX.CHECKSUM files.
Make a test involving --upload, seamless-buffer-length and seamless-download --directoty --index.
- Add seamless-download --directory option. DONE
- Try to revert https://github.com/sjdv1982/seamless/commit/a95b55a68bc747ca8aa1661037497ef4981d695e
Instead, bytes pins in bash transformers are now .inp-bypassed ("deep" pins) or from .inp (auth) depending on inchannel presence.
To test: re-run headify, re-run all tests/highlevel with "bytes" in the source
DONE
To document: embedding matrix
=============================
workflow style embeds workflow style: yes, with a special API (stdlib.map)
workflow style embeds Python style: yes
workflow style embeds cmd style:
for pure bash: yes, but requires work (canonical interface for federation).
for nested invocation (scripts containing /bin/seamless commands): see "cmd style embeds cmd style".
Python style embeds workflow style: yes, with a special API
Python style embeds Python style: yes
Python style embeds cmd style: no
cmd style embeds workflow style: no
cmd style embeds Python style: no
cmd style embeds cmd style: yes (for local mini assistant, not remote; see also https://github.com/sjdv1982/seamless/issues/217)
=============================
TODO before 0.12 release:
- Add singularity support for bashdocker jobs. DONE
- Fix https://github.com/sjdv1982/seamless/issues/246 (both issues). DONE
- Add local fairserver test. Rip the RPBS fairserver, create add_fairserver API. Adapt reproducible PDB viewer and deepcell-pdb / deepfolder-pdb. DONE
- Add bufferdir-deploy-deepfolder tool (checksum => ./deployed/<folder>, either hardlinks or softlinks into bufferdir). DONE.
- Make sure that bash jobs with a buffer read folder have get_filename working directory. Rip filezones. Use bufferdir_deploy_deepfolder, then test get_directory feature. Test with /bin/seamless as well! DONE.
- Make sure that save_vault respects scratch (i.e. make it behave the same as buffer_remote.write). Also for save_zip. DONE
- Clean up seamless-tools/database-* tools: DONE
- revive database-run-actions. DONE.
- Re-run pin-filesystem.sh tests. DONE
- Make sure the examples can function without RPBS buffer server, or add it in the notebook. DONE
- Document the vault + database path as a valid (advanced-use) setup. Rip the warning in init_database_from_env, refactor/document API. DONE
- Overhaul serve-graph by adding delegation. Test load-vault/add-zip as well. Run tests/examples requiring serve-graph. DONE
- Re-review new-project. DONE
======================
- Fix issue #249, re-run cmd/headify test. Issue seems to prevent federation (last line of headify/test.out is too slow) (DONE for now)
- Clean up and re-run all tests.
* low-level (DONE)
* fairserver (DONE)
* cmd (DONE)
* high-level (DONE)
- Make sure to add the correct scripts and wrappers from /seamless/bin and /seamless-tools/utils into the Docker images (rpbs/seamless and seamless-devel). Currently, /seamless-tools/utils is not mounted at all, therefore seamless-upload fails. /seamless-tools/seamless-cli/seamless-upload-vault also doesn't seem to work. As tests, run /lowlevel/pin-filesystem.sh, /highlevel/pin-filesystem.sh, /highlevel/get-file-and-directory.sh, /cmd/highlevel/get-file-and-directory.sh
DONE
- Run the examples once more. DONE.
=====================
Release
=====================
- Maintain a Conda environment file for development that updates seamless-minimal-dependencies. DONE
- Rip seamless-minimal. DONE
- Before distribution, create a seamless-unify-webserver-ports tool, unifying using canonical port values. Include this value in the seamless.js client. DONE
- Build and upload the Seamless Docker image. DONE
- seamless-cli:for seamless-delegate, set up non-devel assistants. docker compose --build only for devel assistants. DONE
- For seamless-cli, be sure to add hashserver to the cli directory. No deps since they are inside the Dockerfile. In addition, there is no more Silk dependency, update install instructions accordingly. However, there is now a dependency on psutil, sqlite and peewee. DONE
- Distribute seamless-cli. Make sure activate-seamless-mode works well in distro for seamless-cli. DONE
- Update the seamless-framework Conda environments. Here, *also* install seamless-cli so that tools and seamless Docker container can be fired up at will (DONE). *Mark all Conda environments with
SEAMLESS_CONDA_ENVIRONMENT=1*, this is so that seamless-bash will work from within the Conda environment DONE.
- Finalize the seamless-framework Conda environments. Make sure that bin/seamless is included, and that seamless-mode is activated. This is done using two packages `seamless-cli-bin` and `seamless-cli-complement`, which are aimed at Conda-only installations. DONE
- Commit seamless-delegate-remote fix, rebuild seamless-cli and seamless-cli-complement. DONE
- Port dask-deployment build-environment-devel to build-environment. Take seamless-exact environment as a basis where to install Dask libraries into. DONE
- Port run-db-hashserver-devel to non-devel environment for distribution. DONE
- Adapt the seamless-binder-demo. Note that it is based on the Conda package. DONE
=====================
Documentation
=====================
- Development guidelines: set __version__ to "development" during development, then give a version number just before release.
- Document: buffer read servers/folders fail silently, the buffer write server must succeed
- Document: seamless-mode (works inside Docker image).
- General documentation, describing the three styles. Old TODO entries have been added to the Seamless Documentation issue.
- Document the various packages/installation methods.
* seamless-minimal has been ripped
* The Conda environment will have co-equal status with the Seamless Docker image
* Note that even the Conda environment still requires Docker to make delegation work. Mini and micro assistants require Seamless Docker image as well.
If you don't have Docker, either use no delegation (bad!), or set it up via HPC/Dask-oriented tools (run-db-hashserver.sh + dask-deployment tools). The latter option is advanced use case. Document delegation and why!
- Write a small tutorial with two methods for port forwarding: ngrok and vscode, and show that it works for seamless-serve-graph.
=====================
Post-release
=====================
- After release: fix cloudless, rip communion-related parameters, always invoke delegation, rip proxy.
- Assistant robustness (create issue)