Voting Stage Walltime and Restart Problems

[gdevenyi]

The voting stage before it votes, runs stage_register_templates to register all templates to the subjects. This is the longest step and frequently has issues with both memory and walltime.

Currently we store these results in self.temp_dir which means that every time the job fails, we lose the results and have to restart from scratch.

The key line is:

reg_dir, transform_file = self.xfm_path(template, subject, self.temp_dir, check_reg_dir = True)

We could just remove self.temp_dir, which would cause the files to be generated in registrations/ or perhaps a subdir, such as voting

@raihaan Can you help me with a play dataset to test such a change?