Bpipe 0.9.13

• CHANGE: Supported version of Java is now JDK11 and version of
  Groovy is 3.0.10

• FEATURE: Many improvements to agents including:

  • a single Bpipe agent can now listen to
    multiple queues
  • support for JMS selectors so that different
    commands can be handled with different settings

• FIX: bpipe.$LAUNCHER_PID.run.pid race condition

• FEATURE: support for a "bootstrap" config file
  (bpipe.bootstrap.config) that is loaded first and
  can perform meta-configuration (configuration
  of downstream configuration files)

• FEATURE (experimental): quiet mode to cause stages to output
  messages only if they actually execute something

• FEATURE: improved scalability to very large numbers of
  input files and output directories

• FEATURE: allow AMQ connection config to come from file

• FEATURE: Improvements to AWS support, including:

  • retry AWS commands for api rate limit exceeded, and insufficient
    capacity errors with customisable settings
  • ability to provide initScript
  • improved "transfer" mode that relies only on SSH to transfer files
    to the instance
  • support for stopping rather than terminating instances, either to make
    them reusable or to aid in troubleshooting
  • improved support for running docker containers on AWS EC2 instances
  • support for reading AWS credentials file and specifying profile
    to provide credentials

• FEATURE: support for stage level configuration in bpipe.config
  to configure commands

• FEATURE: support for specifying Anaconda environments to run
  commands in, where Bpipe handles activating the environment so
  that use of the environment is not hard coded in the pipeline

• FEATURE: Support setting of uid to current user in docker commands

• FIX: "bpipe stop" killing non-bpipe processes when using containers on Mac

• FEATURE: many improvements to "dev mode" where bpipe steps through
  the pipeline interactively and shows highlighted commands

• FIX: various file handle leaks related to output directories

• FEATURE: Add new execOptions for Singularity container config

• FEATURE: Add bpipe pipeline id and command id to Name tag attached to EC2 instances
  because: useful for dev / test where maintaining ongoing

• FEATURE: add support to read .netrc file to obtain API authentication credentials

• FEATURE: Allow configuration of custom options to Docker command

• FEATURE: support for durable replies in JMS agent : even if
  Bpipe errors and is retried, the replyTo will be respected

• FEATURE: support to specify non default platform for container images

Bpipe Version 0.9.12

Change Notes

The most important change in this release is support for Groovy 3.0 and JDK11+. This ensures that Bpipe keeps using well supported versions of Groovy and Java, and also paves the way to longer term use of advanced JDK versions (21+), where some significant new JVM features should allow resolution of some long standing issues with scalability of Bpipe pipelines.

Note: for this release, both Groovy 3.0 and Groovy 2.5.13 version will be provided as downloads. Future releases may
transition to only supporting Groovy 3.0 - all users are encouraged to use the Groovy 3.0 version unless
issues are experienced. Groovy 2.5.x can not support recent Java versions so this will be a necessary migration
at some point.

Other notable changes include a convenient "dev" mode that operates similarly to using bpipe test, but
allows you to see colorised commands showing which parts Bpipe recognises as input or output variables, as well
as showing other contextual information about the command that is to execute such as variables,
assigned resources and container settings. A key element of this mode is that it can dynamically update when
you edit the source code to allow fast iteration to get the command looking right.

A new syntax also allows convenient creation of "named" branches by multiplying a Groovy Map literal. As an example:

'%.csv' * [  
  quality_control: check_format + check values,
  analysis: clean_data + train_model
] + finalise_results

This will create two branches named quality_control and analysis. As always, Bpipe allows for
dynamic construction of these inside your run method to give lots of flexibility.

Apart from the listed features and fixes, there are a huge number of optimisations and minor bug fixes
included in this release. Bpipe should run faster and more reliably than before.

---

• FEATURE: support for Groovy 3.0 and JDK11+

• FEATURE: experimental 'dev' mode - use bpipe dev ...
  to enter test mode that highlights
  input and outputs in commands, hot updates with changes
  and shows branch/command context

• FEATURE: support for optional outputs via .optional pseudo
  extension

• FEATURE: named branches - use multiply syntax with
  groovy Map literal as alternative to list, to create named
  parallel branches

• FEATURE: dynamic splits. Use forwardSplit(...) to create
  pipeline branches that are dynamically determined by result
  of a previous stage. (see docs)

• FEATURE: Command configuration can now be done based on stage name rather
  than command name. This significantly relieves the need to add dedicated
  config names to commands in order to adjust their configuration.

• FEATURE: duplicate output files are now explicitly detected and warned about

• FEATURE: enhanced warnings and messages when transform / produce / filter
  do not create expected files

• FEATURE: add display of cores into stats table

• FIX: Slurm executor polling not operating correctly

• FIX: Don't create stray empty .bpipe folders

• FIX: Autoarchive leaving behind empty dirs on some file systems

• FIX: wrong input may be used in from(multiple) transform() to() construct

• DEPRECATED: the Multi command is no deprecated as there are no scenarios where
  this is preferred over paralleising using branching structures

Bpipe Version 0.9.11

• FEATURE: support for required attribute on notification
  channels to ensure pipeline failure if channel config fails

• FEATURE: experimental support for virtual file extensions / file type
  mappings , defined using: filetype <virtual ext> : [ 'ext.one', 'ext.two',...]

• FEATURE: agent will wait for commands to finish if it is asked to run
  multiple in the same directory at the same time

• FEATURE: support for "queue" command : schedules pipeline to run after
  another one finishes in a directory

• FIX: only match process owned by user when checking for running bpipe process

• FEATURE(experimental): from clause can now be added in pipeline definition to
  cause inputs to prioritise from a given stage, with branch matching

• FEATURE: update aws sdk to support latest instance types

• FEATURE: EC2 commands now show name of stage via name tag, eg. in aws console

• FEAUTURE: accept aws creds from environment if not defined in bpipe.config
  using standard AWS env variables for token / secret

• FEATURE: support direct transfer of files via scp for AWSEC2 executor.
  If storage config setting is not defined and 'transfer' setting is true
  then Bpipe will utilise scp to transfer input/output files to EC2 instances

• FEATURE: allow queue to be overridden within pipeline when sending
  to JMS channel via explicit 'queue' property

• FEATURE: support replyTo as property for replying to JMS messages
  (rather than JMSReplyTo)

• FEATURE: improved support for archiving: autoarchive and ability to run
  archive command from outside bpipe dir

• FEATURE: implement support for 'required' and 'unique' as options on
  gitlab messages : ensures pipeline reports error status if
  gitlab calls fail

• FEATURE: support for experimental "when" feature to make stages conditional

• FEATURE: mask 'token' config property as sensitive info from log files

• FEATURE: support for setting R_LIBS_USER with R { r_libs="..." }
  setting in bpipe.config

• FEATURE: add checks to result message returned in reply-to on
  JMS messages

• FEATURE: allow specification of gitlab projects by id when
  sending to gitlab

• FEATURE: show custom title in bpipe startup when configured
  using title : '...' or title '...'

• FEATURE: support to set env variables for commands via bpipe.config,
  by added env = : , ... to command config

• FIX: http send not retried on bpipe retry if initial attempt failed

• FIX: errors printed by trying to send JSON via SMTP

• FIX: adjust bpipe agent jvm options to reduce footprint on large servers

• FIX: fix transform of wildcard (eg: *.txt) to multiple outputs
  resulting in regex error

• FIX: undefined pipeline variable somtimes shows bpipe internal stack trace

• FIX: possible concurrentmodification errors when user-specificed config
  referenced concurrently in stages / reports

• FIX: fix multilevel branch names not reflected in file names

• FIX: error thrown when creating diagram using 'bpipe diagram ...'

• Support for reservation of gpus with torque (set gpus=N in command config)

• FIX: body of message relayed in JMS message header (causing parse fail in
  some clients / protocols)

• FIX: fix regex in transform matching multi inputs only resolving single

• FIX: fix error when literal regex used as transform input specifier

• FIX: previous dirty state carrying over causing unexpected
  cleanup of good files

• FIX: put retry around output dir watcher to handle race conditions
  / file system hiccups better

• FIX: Explicitly set pwd for singularity as this seems not always
  to occur by default

• FIX:: fix downstream reference to output created by check failing
  on retry

• FIX: inability to create gitlab issue with empty content

• FIX: json output double encoded in .bpipe/sent files

• FIX: some commands not running with set -e by default

• FIX:: fix warning printed for dirty files even though
  they don't exist, already notified

• FIX: remove duplicates from printing of test outputs

• FIX: remove printout of shebang stripped statement

Bpipe Version 0.9.10

This release includes a range of significant new features, including upgrade to
Groovy 2.5.13 as default groovy version, initial support for running commands
in docker or singularity containers, experimental cross-branch dependency support,
across the board improvements in accuracy and clarity of error messages,
robustness of error checking and efficiency, and improvements in support for
glcoud and AWS execution. Also included are many smaller features (see release
notes).

Release Notes

• FEATURE: default groovy now changed to 2.5.13. To move back to
  old groovy, compile using -PGROOVY_VERSION=2.4.8

• FEATURE: many improvments to efficiency, decrease load created by bpipe

• FEATURE: initial singularity and docker container support

• FIX: reference to $output in check causing incorrect output not
  found error

• FIX: various fixes to AWS EC2 support

• FEATURE: support for auth and headers in http send, and
  support to config http as notification channel

• FIX: undefined variables referenced in stages resolving to
  value instead of throwing error

• FIX: output.dir unexpectedly uses absolute path when inside run dir

• FIX: branch outputs not resolved downstream when user terminates branch
  with success

• FIX: error generating reports with latest groovy 2.5.x

• FIX: ugly error if stage returns w/o output from annotated
  transform/filter

• FIX: null pointer exception thrown for pipeline with no outputs
  for some pipelines

• FIX: support custom project for gcloud executor

• FEATURE: more flexible forwarding of local files

• FIX: reports potentially overwriting each other inside
  transform with fixed outputs

• FIX: storage system printed out even when local for not found files

• FIX: legacy / manual output poller failing to see files outside
  root dir

• FEATURE: config entries in bpipe.config defined as closure
  now receive optional arg containing config, eg: to configure
  memory based on procs

• Initial implementation of channels to enable controlled async cross
  branch dependencies, using from(..., crossBranch:true) { ... }

• FIX: in-progress files cleaned up when bpipe stats executed

• FIX: resolving outputs across sibling branches now fails unless
  config option set

• FIX: failure to resolve full path in from when duplicated
  in args

• FIX: regression causing resolution of pre-existing files to fail

• FIX: failure checking dependencies when only input is directory

• FEATURE: support for environments in bpipe.config, selected via
  -e option, embed multiple configs for specific env inside named
  sections of bpipe.config

• FEATURE: send can override template by supplying 'template' attribute
  after send, eg: send text(....) to template: 'report_template.html'

• FIX: outputs sometimes not recognised as generated in printed output after
  pipeline finished

• FEATURE: Support fo file attachments in gitlab notification / send

• FIX: fix content of saved sent file null when sending json

• FIX: crash / fail to send notification if pipeline input/output/file
  included in send of json

• FEATURE: print queuing system job id in command status output

• FEATURE: check for dirty files now enforced by markers in .bpipe/inprogress -
  error printed if present on retry.

Bpipe Version 0.9.9.9

Summary

This release includes a number of smaller features and improvements and
some important fixes.

NOTE: this version of Bpipe requires Java 1.8 or higher

Features

• Ability to filter gitlab issues identified
  by title search for updating

• Feature: support for 'debug' command that launches
  groovy shell. Add debug() to pipeline stage to launch
  interactive shell.

• Revamp monitoring of command status for PBS/Torque
  based pipelines, by pooling calls to qstat between
  commands.

• Feature: archive command to clean up .bpipe directory into
  restorable zip file

• Support referencing params set on command line in
  bpipe.config

• Bpipe pipelines can now accept the first line as a "shebang"
  so that bpipe pipelines become executable scripts. ie: make
  the first line of your pipeline: #!/path/to/bpipe

Fixes

• Fix not completely robust reading of exit code for
  local commands (sometimes command succeeds when shouldn't)

• Fix local commands not always terminating on explicit stop

• Add setsid by default to local jobs on linux -
  prevents some cases where jobs abort when precessing
  ctrl-c after starting bpipe

• Fix new gitlab issues with large content failing to send

• Fix possible runaway killing of processes when
  local preallocated jobs are explicitly stopped with
  bpipe stop

• Fix file notifications not always incrementing and
  therefore overwriting previous notification

• Fix notification manager not respecting timeout to allow for
  setup of channels

• Fix possible error due to race condition
  reading read PID with local command executor

• Fix file handle leak on some file systems with large scale
  highly parallelised, long running pipelines

• Fix the default file notification channel overwriting files from
  other send channels

• Fix possible error if input is referenced in details of send

Bpipe Version 0.9.9.8

Changes for Bpipe Version 0.9.9.8

• Feature: options statement for easy parsing of script specific
  command line arguments

• Add workaround to fix possible failure on old pipelines checking
  file existence

• BUG: Fix spurious error printed if broken symbolic link present

• BUG: prevent agent launched commands hanging due to if bpipe prompts
  for information

• Feature: add fileSystemSyncTimeMs which allows for customizable delay in
  file system sync before failing pipeline on missing output

• BUG: fix regression - transform using regex does not substitute groups

  eg: transform('(.*)_R1.fastq') to("$1.txt") { ... }

• Many fixes / optimisations to GCloud usage:

  • make workingDirectory specified for stage automatically
    placed within mounted storage

  • inherit region from main config inside
    filesystems, so it does not need to be double specified

  • support preemptible flag passed to google cloud
    instances

  • add sha1 hash of pipeline path / host to
    instanceid to make it unique b/w pipelines

  • fix region configured
    for cloud executor overriding region specified with -L

  • search for root input files in all filesystems, not only
    those configured as storages for stages

  • fix zone flag missing from gcloud commands, don't
    die mounting storage if instance not created

  • support using split / partitioned regions in cloud provider storage

• BUG: Don't start executor pools when user is only testing the pipeline

• BUG: ensure that executors clean up even if they error in stop
  command

• Fix table output format so it shows correctly in GitHub markdown

• Better behavior for bpipe test with send, and send for missing outputs

• BUG: input patterns with dir in the pattern causing error / exception

• Feature: support for reply -to in JMS bpipe agent

  If the reply-to or JMSReplyTo header is set, the bpipe agent will now
  send a message containing the original command and pipeline success / fail
  to specified reply-to address (allows eg: camel inOut routing)

• BUG: Fix inverted logic causing incorrect detection of file existence
  when file is out of sync, causing file systems with high latency
  to fail sporadically

• Better tolerance of file system sync issues for files referenced
  using file(...)

• BUG: from sometimes failing to ensure specified input is resolved in
  preference to other available inputs preceding it in the
  pipeline

• BUG: from sometimes selecting incorrect input in complex scenarios

• BUG: fix file system watchers still created even if poller watcher specified,
  hence still resulting in file watcher exhaustion errors

• BUG: -d option not respected in some situations, incorrect output dir may
  be used if default output already referenced

• Feature: Support for Groovy 2.5.6 via optional gradle.properties setting

• Feature: support for template dirs created by bpipe agent with incrementing
  values. Directories containing {inc} will be named with auto-incrementing
  counter.

• Feature: 'terminateOnError' flag in send attributes causes
  notification failure as to be treated as error

• BUG: improve unfriendly / incorrect behavior when duplicate mappings exist in transform

• BUG: checks sometimes re-executing when not necessary

• Feature: bpipe touch command updates files from current run
  of pipeline to new timestamps so they do not re-run on bpipe run.

Bpipe Version 0.9.9.7

Summary

This release includes several major new features including prelimary support for
running Bpipe pipelines on cloud providers (Google Cloud, Amazon Web Services), a
new merge point operator for making it easier to
construct parallel pipelines using scatter-gather parallelism. In addition to these,
significant work has been done to dramatically improve performance and reduce
resource consumption on very highly parallel pipelines with large numbers of
input / output files.

Features

• Preliminary support for executing pipelines on Google Cloud Services
  (Compute Engine) and mounting storage for pipelines from Google Cloud
  Storage

• Preliminary support for executing pipelines on Amazon Web Services
  using EC2 and mounting storage for pipelines from S3

• The 'groovy' command can now run embedded groovy (executed outside
  Bpipe) using the groovy runtime bundled with Bpipe

• Support aliasing to string values in addition to outputs

• Experimental support for beforeRun hook in command config: execute
  arbitrary groovy code before a command executes

• Many performance improvements, esp. for large, highly
  parallel pipelines

• Support configuration for number of retries for status
  polling of HPC jobs (statusPollRetries setting)

• Support for 'optional' inputs in pipelines: to make input optional,
  suffix with 'optional'. Also can add 'flag' to add flags
  in commands eg: ${input.csv.optional.flag('--csv')}

• New operator: merge point operator (>>>) automatically configures a stage
  to merge outputs from a previous parallel split

• Add region.bedFlag(flag) method for convenience when passing
  regions to commands

• 'var' expressions may now be added in the main pipeline script,
  not just pipeline stages. These define optional
  variables, and provide a default.

• JMS support now responds to 'ping' message with 'pong' reply
  if JMS 'Reply-To' is set to allow for status monitoring

Fixes

• Fix incorrect "abnormal termination" messages
  printed to console when pipeline stopped with 'bpipe stop'

• Fix incorrect 'pre-existing' printed for outputs that were
  created by pipeline

• Fix genome not accessible in pipeline the first time downloaded,
  printing error

• Re-execute checks if a commmand in the same stage has executed

• synchronize initialization of dir watcher to fix sporadic
  ConcurrentModificationExceptions

• Fix empty embedded parallel stage list causing resolution of incorrect
  downstream input

• Fix leak of 'var' variables across branches when 'using' applied to
  pipeline stage

• Fix error if 4 or more arguments passed to "to" in transform

• Fix bpipe complaining spurious outputs not created on retry,
  but not original run

• Fix some bugs where branch names were not being observed

• Fix branch name sometimes inserted without separating period for transforms

• Avoid redundantly putting branch name into files

• Improved detail in error / log messages in a few places

• Fix missing branch and '..' in filenames

• Change: globally defined variables must now be held constant
  once pipeline starts

• Fix split regions not stable between runs, set region id as branch
  name

• Fix bed.split producing different splits if run repeatedly on same bed

• Fix errors output if SLF4J referenced in user loaded libraries

• Fix npe / improve error message when filter used with mismatching output ext

• Fix error in stage body resulting in confusing 'no associated storage'
  assertion failure

• Add 'allowForeign' option to 'from' to let it process non-outputs

• Lessen the retries and retry interval when file cannot be cleaned
  up

Release 0.9.9.6

Bpipe Version 0.9.9.6

Summary

This release includes some bug fixes that could cause confusing behahavior,
clearer and better messages in various error scenarios, and a couple of key
features such as support for gitlab, setting config parameters from within
your pipeline script, setting pipeline parameters from bpipe.config, and
others.

Changes

• Fix: a send command encountered during bpipe test would not send the
  message but would record it as sent so that it would not be sent if
  the pipeline is then subsequently executed.

• Fix: various issues with bpipe stats printing bad dates / times

• you can now run bpipe stats while a pipeline is runnign to get realtime
  info

• The bpipe agent now supports a ping command that will respond to confirm
  the agent is alive (via ActiveMQ).

• You can now set parameters (variables) from within bpipe.config via
  a parameters section. This sets variables the same as if you set them
  via -p name=value when running bpipe.

• You can now set pipeline limits (memory,concurrency,etc) from a config section
  within a pipeline script

• Bpipe now supports gitlab through its send functionality. Issues can be created
  or updated added with syntax like: send issue(title: ...., assignee: ...) to gitlab

Release 0.9.9.5

Release 0.9.9.5

Features:

• Support for Bpipe 'agent' which can listen to requests via
  JMS (activemq for now) to run pipelines

• Support sending json to JMS/ActiveMQ via send command

• Utilitity function 'asQuotedList' formats lists of inputs
  as syntax compatible with js,python,groovy scripts

• Experimental support for "sequential" or power operator

• Print out the outputs to be created when run in test mode,
  in addition to the command to run (existing behavior)

• Support for post-command hook to execute a script after every
  command

• Add command start and stop time to output from 'bpipe query'

• Add start and stop time to display in 'bpipe errors'

• You can now specify java libraries as list in bpipe.config,
  in addition to the existing string form

• Support for check review states, and improved check display

Fixes:

• Fix exception/error if double input extension overlaps
  output with sng ext, eg. $input.foo.bar happens to match
  upstream $output.foo

• Fix completed event sometimes not sent for failed stages

• Fix failed checks written to std out at end of run

• Fix situations where checks are re-executed when not
  required

• Support supplying input files explicitly to run in
  run([...])

• Support for JMS notifications of pipeline events

• Fix error displayed by 'bpipe log' if no pipeline has ever
  been executed in dir

• Prevent recursive load caused defining a segment inside a
  loaded file

• Add segment structure to JSON pipeline representation
  Previously segments were expanded into their constituent stages
  in the JSON representation, and thus the higher level pipeline
  structure was lost. Now each node and edge in the group
  is given a type, and both the segment and its intenral
  structure are output as nodes / edges.

• Fix sometimes getting wrong last run pid (thinks still
  running)

• Change to make error in loaded script terminate pipeline
  instead of just printing error and continuing

• Fix bpipe log stops working after some commands until bpipe run
  again

• Avoid printing out cleanup message if zero files to cleanup

• Move some of the docs out of Language and into Guides

• Prevent exception in bpipe errors if command state not saved

• Fix 'bpipe errors' not showing log output if hyphen in branch

• Fix status command, add regression test

• Groovy command in script now handles embedded quotes,
  which previously caused syntax errors

• Fix error when nested segment made of single stage defined

Bpipe 0.9.9.4

Bpipe Version 0.9.9.4

This is a quick update that bundles a few new features with some critical fixes
to 0.9.9.3 based on feedback after it was released.

• Fix issue #217
• Add bpipe stats command to show runtimes of stages
• Support config of concurrency limit in bpipe.config,
  via concurrency=<n>
• Support for embedding config inside pipeline scripts via 'config'
  statement
• Fix command status sometimes incorrect after stop preallocated
• Fix pool commands in WAITING status not recognised as active
• Fix trailing output from commands sometimes missing when output
  appears very fast at point where command terminates