NEWS

New in Version 2.0.19
=====================
* created a number of tests exercising CLI functionality
  (for performance reasons, these require pytest-console-scripts to run)
* minor pipelines (LSQ12, NLIN): fixed some argument handling issues
* check_pipeline_status.py: update to Pyro5
* created a GitHub action to run the tests on push to master/develop
* updated the doctests
* make the linter happier
* add pyproject.toml and requirements.txt
* updated example config files to reflect Singularity and Makeflow options
* fixed a bug --no-execute was not respected with --backend=makeflow
* setup.py now uses the modern `entry_points = { 'console_scripts' : ...` idiom
  + the entry point for each pipeline is called `application`
* various minor bugfixes to Python code and templates, code cleanup, and dead code removal

New in Version 2.0.18
=====================
* add support for using Makeflow (https://ccl.cse.nd.edu/software/makeflow) as a backend
  - this support more queue types than the `qbatch`+`Pyro` backend
  - by default jobs are submitted individually
  - for server-executor execution use --queue_type=wq to use Makeflow's Workqueue backend
    (currently requires starting work_queue_factory yourself)
  - known issues:
    + since directories are included in the dependencies, you may see harmless warnings that directories have been modified
    + our dependency specification may not include all output files;
      this may be insufficiently strict for some Makeflow backends
* Pyro4 updated to Pyro5

New in Version 2.0.17
=====================
* add Singularity support via --use-singularity, --container-path, --container-args (easy to extend to Apptainer, Docker, Podman, ...)
* minor executor internal changes and improved logging
* fix a bug in the default minctracc template created for 2.0.16

New in Version 2.0.16
=====================
* user-editable shell command templates with Jinja2 (see README.md for usage)
* fix a bug where files could be created in incorrect subdirectory (but in correct row of generated .csv files)

New in Version 2.0.15
=====================
* when MAGeT is enabled in MBM-type pipelines, nonlinear averages are automatically segmented via MAGeT by default
* disable some choices for --registration-method that are not currently supported (DRAMMS, Elastix, Demons)
* add --use-robust-averaging flag to nonlinear model building pipelines to downweight subjects with poor registrations
  (uses the --robust flag of mincbigaverage; otherwise is a no-op)
* internals: add and use a `defer_map` method on `Stages` in preference to iterated use of `defer` (currently equivalent,
  but may gain improved distributed computing semantics in future)
* MAGeT masking improvements to match recommendations from e.g. ANTsX developers:
  - added a --mask-dilation which dilates input masks by a specified number of voxels
  - added --hard-mask which hard-masks input images by multiplying them with MAGeT masks
  - added a --hard-mask-dilation flag which dilates the MAGeT masks by a specified number of voxels before applying them
  - by default these are not used (i.e. the results will be the same as before) until we make recommendations
* fix a couple crashes in less popular pipelines

New in Version 2.0.14
=====================
* fix a file naming bug when minctracc is used for average building
* fixes to reduce memory/vmem usage due to Numpy/OpenBLAS
* rotational_minctracc.py now uses both source and target masks if present
* bug fixes to output CSV file generation
* improved help documentation for --restart and --smart-restart
* create fewer intermediate resamplings
* minimum Python version is now correctly specified as 3.6 due to use of format strings
* typing module is removed as a dependency since it's present in 3.6 and incompatible with 3.8
* various other small bugfixes and internals changes

New in Version 2.0.13
=====================
* in the two-level pipeline, the inputs specified via --csv-file are now by default relative
  to the csv file rather than the current working directory; this matches the behaviour of MBM.py.
  To get the opposite behaviour, MBM and twolevel now support a new flag, --csv-paths-relative-to-wd
* switched from xfmavg to xfmavg_scipy.py; this adds a dependency on minc2-simple (see INSTALL)
* fixed a bug causing installation failure with Python > 3.6.4
* fixed crashes at startup in certain flag combinations
* pipeline correctly handles undefined PYDPIPER_CONFIG_FILE environment variable
* minor changes to montage generation
* added the -label flag to (most/all) mincresample commands
* changed some internals to avoid Pandas deprecation warnings
* internal changes to LSQ6 and MBM option parsing to support tissuevision pipeline
* add a (somewhat unrepresentative) test script to the repo for now;
  better ergonomics and documentation to follow

New in Version 2.0.12
=====================
* improvements to output CSVs for MBM and twolevel model pipelines: use analysis.csv for most cases
* additional outputs to analysis.csv: masks, segmentations
* --resolution flag works correctly (i.e., not ignored)
* CmdStage API no longer needs you to manually string-ify arguments
* cleanup module imports to save memory on executors (see #378)
* various fixes to directory structure and file naming
* fixes to nii2mnc, mnc2nii
* fix to determinants_at_fwhms when no inverse transforms supplied (previously crashed)
* option added to disable normalization in mincaverage
* internal cleanup of minctracc configuration objects to make rotational_minctracc
  more similar to other LSQ6 modes; more improvements coming
* intensity and labelling fixes to QC images, which are now produced as JPG instead of PNG for space
* autocrop command added
* internal functionality added to allow passing environment variables to shell commands
* an error is signalled if PYDPIPER_CONFIG_FILE doesn't point to a readable file
* --smart-restart flag added to allow manually modifying files and have Pydpiper automatically
  recompute all succeeding stages
* --time-to-seppuku flag changed to --max-idle-time
* various other small internal changes and fixes

New in Version 2.0.11
=====================
* add preliminary support for antsRegistration (only with MICe default parameters currently)
* new modular framework for combining image and transform file formats, registration algorithms,
  and registration topologies.  The code for this is somewhat experimental and rather messy
  at the moment; it uses the Typing and ABC modules but the latter could be dropped,
  resulting in much simplification, if we dropped support for Python 3.5 and lower.
* added the --registration-strategy=<...> flag to model building pipelines to control
  the registration topology as mentioned above; current options include build_model (default),
  tournament (currently unweighted), pairwise (similar to LSQ12 topology; not really recommended
  for producing final registrations), tournament_and_build_model, and pairwise_and_build_model
  (the most useful addition).  An --nlin-pairs flag has been added to control the complexity
  of pairwise nonlinear model building with some small default intended for producing an
  intermediate average prior to 'standard' model building.
* added the 'tamarack'-style to the repo.  This acts sort of like a 'transposed' registration
  chain in which between-timepoint registrations are obtained (via composition) from
  average-to-average registrations, which may improve things in low-SNR regimes.
  (See recent papers by Darren Fernandes and friends).  Needs more validation ...
  after filtering to a fixed blurring kernel (please be careful to do this).
* begin adding support (not yet functioning, mostly due to ITK<->MINC/xfm file conversion issues)
  for other registration tools based on ITK/NIfTI: Elastix, Demons, DRAMMS, and some
  auxiliary tools (e.g., c3d).
* increase maximum memory requests in the MICe/SLURM config file
* add option to defer file directory creation until runtime in the hope of starting a pipeline
  sooner.  Currently quite (time/memory) inefficient so not recommended but much room for improvement.
* output files are now optionally checked to ensure they exist when a stage finishes.
  Flags --{,no-}check-outputs (default: False) and --fs-delay control this.
* write out more/better output CSV files; for MBM.py, see analysis.csv, which is in tidy format
  ready for modelling as soon as you filter by your chosen blurring kernel width (`fwhm` column)
  (please remember to do this!)
* MBM.py doesn't do common space registration by default
* configuration objects now track the flags which can be used to control them
  and this is used in some error messages to provide more useful suggestions
* added stage_embryos_in_4D_atlas.py; this may move to mouse-imaging-centre/MICe-lab
* better/more automated checking of input files from pipeline graph
* use networkx 2.x instead of 1.x API
* log file locations can now be specified when creating a CmdStage
* delete obsolete test_registration_fh.py
* many miscellaneous internal changes and bugfixes

New in Version 2.0.10
=====================
* fix average misnaming (#348)
* correctly set jobname (#349) and log file locations
* add nocheck_dimensions option (default: false) to mincaverage

New in Version 2.0.9
====================
* Pydpiper now depends on `qbatch` (and hence a recent `gnuparallel`).
  One can configure queueing options through either qbatch or Pydpiper; see the INSTALL file.
  The 'headless' (`--submit-server`) mode hasn't been ported and still uses `qsub` directly.
* the above change adds SLURM scheduler support
* modifications to the supplied config files due to the above
* backport mincaverage copy_header fix from develop

New in Version 2.0.8
====================
* add option to perform atlas-to-atlas registration of the study model to a 'common space model' and resample images
  and determinants into this common space (MBM.py only).  The relevant flags are --(no-)common-space-registration,
  --common-space-model, --common-space-mask.  At the moment the registration parameters are automatically taken from the
  model-building LSQ12 and (final generation of) nonlinear registration settings
* add extra columns to output CSVs: common space images/transforms/determinants if created; input images in various 'spaces'

New in Version 2.0.7
====================
* this release fixes only one issue: minctracc-based model building didn't use previously computed transforms
  as initial transforms in the next level of registrations

New in Version 2.0.6
====================
* fix MAGeT extra resampling of labels along transform from original onto new templates (#304)
* MAGeT internals rewrite/simplification
* MBM and MAGeT can read files and masks from csv file containing `file` and optionally `mask_file` columns
* improved minctracc parameters and defaults for stiffness, weight, and lattice_diameter;
  max_def_magnitude has been removed (it no longer appears in minctracc's -help)
* additional determinism to allow restarting pipelines without losing stages
* --mem-request-attribute can now take a comma-separated list of attributes (e.g., "vmem,mem"),
  all of which currently get the same values
* --max-templates now includes the templates from the original library; this might be reverted soon
* the default blur level for the determinants is now 0.2mm only;
  specify additional/different levels via --stats-kernels, e.g., '0.1,0.2,0.5' (the old default)
* switched from mincANTS to ANTS (but symlinking to mincANTS is possible)
* the installation instructions now require minc-toolkit-v2 rather than minc-toolkit due to use of ANTS
* ANTS now uses the source ("fixed") instead of target ("moving") mask
* added some progress information in "verbose" mode
* improved (too few) filenames and locations
* some improved error messages
* numerous internal changes

New in Version 2.0.5
====================
* pairwise lsq12 now (correctly) includes a registration to "itself" for each subject.

New in Version 2.0.4
====================
* execution bugfixes

New in Version 2.0.3
====================
* better montage file locations/naming
* bugfix in "pride of models" association of models to timepoints
* execution bugfixes

New in Version 2.0.2
====================
* automatically set MAGeT's protocols based off model building LSQ12/nlin protocols when possible
* two-level pipeline supports multiple initial models
* when LSQ6 is disabled, input files are resampled to a common grid so various programs (e.g., mincaverage) will work
  (currently implemented in MBM, hence also in two-level, rather than in LSQ6 code itself)
* zip_safe flag disabled in `setup` so that config files will be installed in unzipped form
* more information is printed about stages unable to run due to memory limits
* fix hang running executors locally
* allow specification of objective function for linear minctracc stages
* slight fix to ANTS stage formatting allowing compatibility with minc-toolkit v2
* update SickKids HPF config file
* various internal code improvements

New in Version 2.0.1
====================
* made the lsq12 protocols required in order to provide proper error messages when not specified
* allowed the specification of a mincANTS protocol with a single level in lsq12_nlin which is used for MAGeT (among others)

New in Version 2.0
==================
* switch from Python2 to Python3.5 or higher
* PydPiper2.0 is a rewrite of PydPiper to facilitate easier creation of pipelines
* we've made significant changes in how the communication between server and executors work, and most network related issues appear to be resolved.
* new pipelines: asymmetry.py which is a pipeline that runs a twolevel_model_building on the input brains and their flipped versions
* MBM and twolevel_model_building have a build-in MAGeT option
* registration_chain code allows you to specify a "pride of models"; this is necessary for instance when there are large size differences between your timepoints
* you can provide your csv that you'll use for your entire project (and stats) and feed that into: MBM, registration_chain, twolevel_model_building
* usage differences:
  - must specify --files for MBM.py
  - some protocol defaults not specified; you need to specify these


New in Version 1.18.1
=====================
* point to ConfigArgParse 0.11.0 on github until its maintainer makes a new release and uploads to PyPI

New in Version 1.18
===================
* bug fix to minctracc memory config
* added a `--use-dir-names` flag to two-level code to create groups based on directory names
  rather than first input files
* update the dependencies; in particular, specify minimum ConfigArgParse version due to bug in that library
* many changes supporting operation on the Sick Kids CCM grid (and other PBS/Torque 'capability'
  systems) - mostly slowly implementing a HPC queue submission library inside Pydpiper ...
* new config file for CCM cluster
* added a --submit-server flag so that use of this mode of operations isn't determined by queue type
  (at the moment this only works for PBS/Torque grids)
* added some logic to executor `mainLoop` procedure to detect Pyro4 timeouts ... this doesn't ever
  seem to be triggered, so should probably be removed again in future
* fixed a bug where only --lsq12-max-pairs images were used to generate the lsq12 average;
  doesn't seem to have made much difference
* runnable-time hooks are now cached
  (and are now added via a method, rather than by appending to runnable_hooks directly)
* very hackish memory estimates for pmincaverage (for these to agree with SGE's estimates,
  probably need minc-stuffs 0.1.14 or later)
* fewer executors are launched in some situations when more aren't needed
* qsub return codes are checked
* job arrays are used to submit executors to avoid choking the grid engine and to reduce
  vmem usage of the server when submitting; query these with "qsub -t <jid>[]"
* Pyro knows how to serialize more exceptions
* removed the executor heartbeat thread; may be restored in future
* use subprocess32 instead of subprocess for (hopefully) improved correctness


New in Version 1.15
===================
* some of the python dependencies are now installed automatically (from PyPI): 
  ConfigArgParse, networkx, pygraphviz, Pyro4, pytest, pydot
* default value for the lsq12-max-pairs is set to 25
* the LSQ6.py module now has the ability to copy the header information of the 
  first input into the lsq6-average file
* the non-linear modules are hard coded to use pmincaverage instead of mincaverage 
  (for the time being)
* updates to the code to speed up the server process (less disk access when creating 
  directories, and directories are crated only once)
* new SciNet default (in the configuration file): latency tolerance is set to 10 
  minutes (delay between executor-server pings), executor-start-delay is set to 
  10 minutes: the local executor starts 10 minutes after the server to make sure 
  that the server is ready/created the uri, otherwise the executor dies right away, 
  and no processing will be done on the compute node that runs the server.
* the server logs its own memory usage, and uses this information when launching 
  local executors
* changed the graph representation to save memory/time when computing newly runnable 
  stages after a stage completes
* opened up an argument for the executors to specify which pe (parallel environment) 
  is used on SGE
* executors shutdown when they are unable to communicate with the server. This is to 
  avoid having them sit idle when the server is gone
* removed the deprecated options --queue and --sge-queue-opts
* added the option --mem-request-variable which allows you to change the memory request 
  variable to use for sge/ogs


New in Version 1.14
===================
* the build model and registration chain applications now both create a
  composite image of the input files and print a message to the user,
  allowing them to check that the orientation is sufficiently similar
  that the LSQ6 registration method chosen is likely to succeed
* memory requirements for mincblur and nonlinear minctracc stages are now
  estimated dynamically before such a stage runs.  The nlin minctracc protocol
  now accepts a "memory" parameter allowing you to override the per-voxel
  default (based on the native-resolution call) with a fixed amount in GB
  per minctracc generation, e.g.: '"memory";0.5;1;1;1.5;2' dependending
  on the particularities of your images or to get better estimates than those
  based on the finest resolution
* due to poor performance on SciNet, blur calls running on PBS queues now
  use pmincaverage instead of mincaverage.  This adds a dependency on pyminc;
  see the INSTALL file.
* PydPiper's understanding of the "gradient" variable in the minctracc
  nonlinear protocols and the internal `useGradient` variable have changed:
  "FALSE"/False (resp.) causes the blurring to be applied to the image intensities;
  "TRUE"/True causes it to be applied to the image gradients, vs. the previous
  behaviour (True implied blurring of *both* the intensities and the gradient)
* changes to the supplied minctracc nonlinear protocol, partly to reflect the
  above gradient-related changes
* the registration chain now allows you to specify --average-time-point=-1
  to use the last set of scans for inter-subject registration
* additional checking of input files
* a test script which exercises the MBM and MAGeT applications is
  now provided with instructions in pydpiper_testing/
* SGE executors now request memory dynamically based on the requirements of
  the runnable stages.  In this configuration, the --mem option now gives
  a maximum amount allowed per executor.  The system exits if jobs require
  more memory than this maximum.
* on PBS, the remote server now writes its stdout to a text file in the working
  directory, so the user needn't dig into, e.g., SciNet's /var to examine
  it while the pipeline is still running
* a few changes to defaults: default memory is now 1.75G/job rather than 2;
  PBS executors wait for up to 3 minutes for the server to start instead of 50s;
  a few changes to the supplied config files
* better handling of csv files with extra commas in the registration chain
* various bugfixes, in particular to the registration chain
* some improved logic w.r.t. updating state of stages after failure of a
  dependency
* the internal queue has been changed to a set to support additional operations
  related to scheduling and resource management (particularly the dynamic
  executor memory requests; see above)
* miscellaneous code improvements
* *** *** BETA 2 *** ***
* fixed bug in determining the max filename length for mincblur
* added the "alternate input" functionality (using a separate data set for
  the LSQ6 stage, then apply the transformations to the original data set
  and continue with that)
* better handling of options for qsub
* added ability to set stage memory default
* added the --greedy flag to force executors to use the alloted memory
* fixed pmincaverage call


New in Version 1.13
===================
* a composite image is now created after the LSQ6 stage and a message output,
  allowing the user to abort the pipeline if the registration has failed
* reverted back to launching executors without syncwith (which didn't work)
* memory usage for mincANTS commands is determined by the code based on total
  number of voxels in the files (can be overridden using the non linear protocol)
* the command stage type now has runnable-time and completion-time hooks, allowing
  arbitrary Python code to be run (this is used in the implementation of the memory
  usage and composite image features above)
* a verification image is created after the LSQ6 stage
* the registration chain works (but has not been tested on its full functionality)
* fixed a bug that removed part of the filename for *final-nlin* files if the 
  original files contained a dot 

  (NOTE: this breaks compatibility with version 
  1.12, because the output filenames are now changed)

* heartbeat counts are stored in order to debug latency issues between the server
  and executors

New in Version 1.12
===================
* LSQ6 (MBM) fixes to the combination of --no-nuc and --inormalize. This would incorrectly apply the lsq6 transformation to the inormalized file twice
* MAGeT -- now uses the pipeline_name argument -- has better error messages when there are issues with the atlases/labels/masks -- if no atlases are found in the atlas library, the program exists -- has an explicit way to specify --pairwise
* improvements to error handling in the code
* most pipeline related files and directories are now prefixed with the pipeline name
* fixes to how the initial model is used. Now there are two possible scenarios:

  image.mnc
  image_mask.mnc

  or

  image.mnc
  image_mask.mnc
  image_native.mnc
  image_native_mask.mnc
  image_native_to_standard.xfm

* debug statements added in order to inspect heartbeat issues -- heartbeat checking can also be turned off at the moment
* on SciNet, as many executors as possible will start with the server using syncwith rather than after (in qsub)

New in Version 1.11
===================
(in addition to the changes in the alpha version)
* MAGeT fix: the argument max-templates was incorrectly used, and meant in 
  practice that atlases/segmentations were only created for the first 25 input 
  files (default for max-templates). This is now fixed.
* added the bootstrap option to MBM: use the first input file to the pipeline as 
  the target for the LSQ6 stage
* fixed the lsq6-target option for MBM: specify a single external file that will 
  be used as the target for the LSQ6 stage
* The parameters for some stages (mincblur/minctracc) are based on the resolution 
  of the input files. When the stage graph is created prior to running the pipeline, 
  this resolution is not always known and we used the resolution of the original input 
  files. At run time, the resolution sometimes changed, creating discrepancies between 
  stages at graph creation and run time. This would cause these stages to be rerun. 
  This is now fixed. The resolution of the files are based predominantly on the initial 
  model files or the target that is used for the LSQ6 stage.
* fixed a masking issue in MAGeT: it would produce output files with an image range of 
  unsigned short 0 to 1. The masked files now have a proper image range.
* mincblur has a hardcoded limit in terms of its output file. At graph creation time the 
  pipeline now checks to see whether mincblur can actually be run. If not, the pipeline 
  won't start and advice you to either shorten the input file names or change the path 
  you are running the pipeline in.
* Added standard configurations for pipelines (MICe, SciNet), these are accessed through 
  the environment variable PYDPIPER_CONFIG_FILE
* file containing the command that was run and the version of pydpiper being used is written out
* instead of pickling the pipeline status (which wasn't happening anymore), hashes of 
  finished jobs are written out to a finished stages file
* the default mincANTS parameters have changed from:
  [SyN 0.5, Gauss 5 1, 100x100x100x0], [SyN 0.4, Gauss 5 1, 100x100x100x20], [SyN 0.4, Gauss 5 1, 100x100x100x50]
  to:
  [SyN 0.1, Gauss 2 1, 100x100x100x0], [SyN 0.1, Gauss 2 1, 100x100x100x20], [SyN 0.1, Gauss 2 1, 100x100x100x100]
* lsq12-max-pairs is now implemented

New in Version 1.11-alpha
=========================
* Moved from optparse to ConfigArgParse, which must be installed.
  See README for usage details.
* Improve server logic for handling failed executors and cases where
  executors have insufficient resources to run available jobs.
* Improved logic for handling walltime restrictions on PBS queues.
  New flags are available to set some PBS-related variables.
* Load correct modules on SciNet, with better error reporting
  from module load failures and xfmavg stages.
* To replace the old pickling of the current system state, there's
  a new mechanism for logging completed stages to disk
  (in pydpiper-backups/finished-stages) and correctly restarting the pipeline.
* Fixed a bug where executor threads spawned from the server could hang.
* Various small bugfixes, internal improvements, changes to logging and
  usage messages, and updates to comments.
* Added a .gitignore.

New in Version 1.10
===================
* PYRO environment variables are handled inside the code. The SERVERTYPE is 
  set statically to multiplex (there are potential issues with the thread option)
  and the LOGLEVEL and DETAILED_TRACEBACK can be overridden by environment 
  variables (see the README)

New in Version 1.9
==================
* MAGeT uses the originally optimized defaults again. Hardcoded paths to the 
  linear and non linear protocol for now. With a help message of where to find
  them when not running the code at MICe.

New in Version 1.8
==================
* major code restructuring on the server and executors. The executors now send
  out a heartbeat signal to the server, which enables the server to notice 
  an executor having died. Also, all executors now properly die when the server
  is done (or gets killed).
* pickling of the pipeline has been disabled, because it turned out to cause
  huge communication delays (minutes in large pipelines: > 30000 stages)
* command line tool added to query the server as to what's happening:
  check_pipeline_status.py
* logging is separated out again (each executor has its own log file)
* Pyro4 environment variables should be set as:
   
  PYRO_DETAILED_TRACEBACK=True
  PYRO_SERVERTYPE=multiplex
  PYRO_LOGLEVEL=INFO


New in Version 1.7
==================
* the communication between the server and the executors has been upgraded
  from Pyro3 to Pyro4. Currently the following environment variables should
  be set:
  
  PYRO_DETAILED_TRACEBACK=True
  PYRO_COMMTIMEOUT=2
  PYRO_SERVERTYPE=multiplex
  PYRO_LOGLEVEL=DEBUG
  
* the stiffness, weight and similarity parameters for minctracc can now be
  set through the non linear protocol in MBM.py 
* all output from the server/executors will be stored in the current 
  working directory


New in Version 1.6
====================
* fix to MAGeT.py. Solves issues with the voxel_vote command. 

New in Version 1.5
====================
* added retrying/resubmitting of failed files to the server. At this moment
  each stage is resubmitted twice before considering it truly failed. This solves
  an apparent nfs race condition issue.

New in Version 1.4
====================
* added a warning statement when the server is finished running and shutting
  down. There is still a communication issue, and sometimes executors do not
  shut down correctly.
* added verbosity option to the server. By default is now less verbose
* changed the output of the server to the user. Now displays more meaningful
  information: total number of stages to be run, and the number of stages that
  have already been processed.
* bug fix in the pipeline executor: can now be launched from the command line
  again

New in Version 1.3
====================
* huge amounts of code restructuring
* server now launches and relaunches executors (keeps track of executors that
  are registered with the server. However, for now does not check whether the 
  registered executors are alive once in a while)
* executors can kill themselves when they've been idle for too long. They can also
  "drain" themselves (not accept any new jobs) after a certain time and then 
  shutdown when the running jobs have finished.
* added an interrupt handler for ctr+c to the server and executors
* executors on shutdown kill the running processes by using their process IDs (pid)
* a whole bunch of changes to the twolevel_model_building code... (see GitHub)
* a whole bunch of changes to the registration_chain code... (see GitHub)