Traing issue:“torch.autograd.set_detect_anomaly(True)”

[Luke-Huang-Github]

Environment： GPU： single RTX4090,
Dataset: T1 <-> T2 in IXI dataset
Number of images: 2674 pairs
In "train.py" file, when I deleted this line of code：“torch.autograd.set_detect_anomaly(True)” , the speed of training one eopch is faster , but the result of psnr and ssim get worse sometimes.
Time consumption in one epoch: [original: about 1.5hour ; deleted the code: about 36 minutes]
PSNR: [original: 29.5 ; deleted the code: 25.5]

Could you please tell me how to solve this problem?