Need running tasks to kill their jobs on error

[stevekm]

If a tasks is creating qsub jobs for every sample in an analysis, and one sample causes an exception to be raised, the remaining qsub jobs might not be killed if they are not in the background jobs list yet. Need to look into method for handling this

[stevekm]

maybe give each task object its own internal job queue to put jobs into and monitor from, and wrap a try ... except around the task running itself, catch if exception is raised and run a task.kill_jobs() method similar to the function used in job_management

[stevekm]

confirmed that raising an exception while a task is monitoring its own jobs leads to the jobs being left running after the program exits

[stevekm]

need to modularize the 'job_management' and maybe add it as a task method

[stevekm]

convert the job_management module into a JobManager object class, use one for monitoring the pipeline background jobs and add them to each task to monitor jobs