master HTCondor workflow breaks submission of seemingly random jobs

[HerrHorizontal]

Bug description

With the latest commit on the master branch the HTCondorWorkflow execution breaks for random jobs with an output like:

running htcondor_wrapper_2701241706.sh for job number 211
empty htcondor job arguments for LAW_HTCONDOR_JOB_NUMBER 211

I noticed that the reported LAW_HTCONDOR_JOB_NUMBER 211 does not correspond to the job number I would expect from the Error, Log, Output, and stdall files, that share for this particular job above the suffix _863To864.txt. This might be related.

[HerrHorizontal]

Checking out an earlier commit that doesn't include the HTCondor group submission introduced in PR #176, e.g. commit 1848c57 seems to run fine. I expect the bug has been introduced in PR #176 .

[riga]

Hi @HerrHorizontal ,

odd indeed. Are you sure your workflow didn't pick up a submission json file that was generated with the previous submission mode?

Btw, for the time being, you can also set job::htcondor_job_grouping_submit to False without the need to switch to an older version.

[HerrHorizontal]

I have removed the submission json before I ran the test. So I am pretty sure that it didn't.

I will try this out. Where do I set the job::htcondor_job_grouping_submit configuration for a certain workflow?

[riga]

You can set this value globally in the config, or you put this into your htcondor workflow

def htcondor_create_job_manager(self, **kwargs):
    job_manager = super().htcondor_create_job_manager(**kwargs)
    job_manager.job_grouping_submit = True
    job_manager.chunk_size_submit = 0  # all in one
    return job_manager

Regarding the issue you're seeing, I could not spot anything obviously wrong. Could you sent me the content of the submission directory, including the main job files? This would help. Thank you!

[harrypuuter]

Hi all,

i am currently observing the same issue as reported by @HerrHorizontal - when I look at the submission jdl and the arguments in the htconodor_wrapper_xxxx.sh file, the first submission looks fine. Problems arise, as soon as a single job fails, and jobs have to be resubmitted.

I have not looked into the implementation in more detail, but I would suspect something like

• running jobs fail and will be resubmitted
• an updated arguments file for the resumission is created, that does not contain all jobs anymore, but only the ones to be resubmitted
• scheduled jobs, that have not started yet, are now able to start but since there is only a single arguments file (which was modified due to the resubmission) the error mentioned above can occur, resulting in a cascade of failing jobs.

Could this be the reason for the errors ?

[riga]

@harrypuuter

Thanks for confirming and the suggestion! I think you're onto something. I'm going to create a reproducer this week to debug this further.

[HerrHorizontal]

I, and also independently @harrypuuter, have encountered another issue with the new HTCondor group submission. When turning the group submission off as suggested previously, with

def htcondor_create_job_manager(self, **kwargs):
    job_manager = super().htcondor_create_job_manager(**kwargs)
    job_manager.job_grouping_submit = True
    job_manager.chunk_size_submit = 0  # all in one
    return job_manager

the rendering of the values enclosed by double curly brackets, for instance {{law_job_base}}, in the job script fails. Also, it seems that the 'XToY' suffix is not consistently attached to the files. This leads to the jobs failing already at the setup stage with File or Directory Not Found Errors.