fsaverage being deleted when running multiple subjects in parallel

[oricon]

What happened?

I'm re-running fmriprep 23.2.0 on my data in order to take advantage of the added msmsulc registration so I'm submitting multiple subjects in parallel. However I'm seeing crashes during the freesurfer step because fsaverage is being deleted and re-written. The error is below. In the log I see there is an option for "overwrite_fsaverage" that is set to False. Perhaps if it is set to True the files would be overwritten fast enough to prevent this from occurring.

Node inputs:

derivatives = /scratch/group/congalab/vts_reward_mri/derivatives/fmriprep
freesurfer_home = /opt/freesurfer
minimum_fs_version = 7.0.0
overwrite_fsaverage = False
spaces = []
subjects_dir = /scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer

What command did you use?

singularity run -B ${TEMPLATEFLOW_HOME:-/scratch/group/congalab/software/templateflow}:/templateflow --cleanenv --home $SCRATCH -B /scratch/user/joseph.orr/:/work /scratch/group/congalab/software/fmriprep_23.2.0.sif /scratch/group/congalab/vts_reward_mri/bids/ /scratch/group/congalab/vts_reward_mri/derivatives/fmriprep participant --participant-label ${subject} --fs-license-file $FS_LICENSE --me-output-echos --me-t2s-fit-method curvefit --nthreads 16 --omp-nthreads 16 --mem 116000

What version of fMRIPrep are you running?

23.2.0

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

shutil.Error: [('/opt/freesurfer/subjects/fsaverage/label/lh.BA1_exvivo.label', '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/label/lh.BA1_exvivo.label', "[Errno 2] No such file or directory: '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/label/lh.BA1_exvivo.label'"), ('/opt/freesurfer/subjects/fsaverage/label/lh.BA1_exvivo.thresh.label', '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/label/lh.BA1_exvivo.thresh.label', "[Errno 2] No such file or directory: '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/label/lh.BA1_exvivo.thresh.label'"), ('/opt/freesurfer/subjects/fsaverage/label/lh.BA2_exvivo.label', '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/label/lh.BA2_exvivo.label', "[Errno 2] No such file or directory: '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/label/lh.BA2_exvivo.label'"), ('/opt/freesurfer/subjects/fsaverage/label/lh.BA2_exvivo.thresh.label', '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/label/lh.BA2_exvivo.thresh.label', "[Errno 2] No such file or directory: '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/label/lh.BA2_exvivo.thresh.label'"), ('/opt/freesurfer/subjects/fsaverage/mri', '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/mri', "[Errno 17] File exists: '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/mri'"), ('/opt/freesurfer/subjects/fsaverage/xhemi', '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/xhemi', "[Errno 17] File exists: '/scratch/group/congalab/vts_reward_mri/derivatives/fmriprep/sourcedata/freesurfer/fsaverage/xhemi'")]

Additional information / screenshots

No response

[effigies]

This is an interaction of a "feature" (nipreps/niworkflows#739) and running in parallel (#2505). If you start one subject and let it completely replace the fsaverage directory, then you can start the rest, and they should not attempt to delete.

[oricon]

Got it, thanks!

[effigies]

Ah, sorry, not #2505 but nipreps/niworkflows#804. Anyway, closing as a duplicate of that. Dealing with that race is a long-standing ideal, but handling it robustly for different filesystems (e.g., network and Windows filesystems, proxied through Docker's VFS) has been a challenge.

[effigies]

PRs and testing welcome, though!