-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regarding training duration #12
Comments
@Kadakol Two 2080Ti GPUs are used when training with patch size of 256 x 256. It costs about 45s for every 100 iterations and costs 125s for validation every 5000 iterations. |
Thank you for your response! I understand. I've included a small section of the training logs. Looks like training every 100 iterations varies from 39 seconds to 138 seconds and the validation phase is taking about 8 minutes.
|
hi,i think time is not problem,if ur computer have some other processes,your gpt can't use all of it power,i met this question ..but i want to know which lossfunction you used?why the loss in my computer always between e-2 to e-1..wish your respoend |
Thank you so much for releasing this code!
I had a question regarding the amount of time it takes to complete training. From the paper. I found the following information:
All models are built on the PyTorch framework and trained with NVIDIA 2080Ti GPU. When the patch size of input is set to 256 x 256, the total training time is about 5 days.
I have started training on my system with NVIDIA RTX 3090 GPU. However, after training for 6 days, I notice that only 410k iterations are completed and ~600k iterations are pending. I would expect training to be faster using the 3090 GPU card compared to the 2080Ti GPU card, but this seems to be extremely slow. I am using the code from this repository as is with no modifications apart from the training data paths.
Could you please let me know if I am missing something? Is some additional step required in order to accelerate the training?
Thank you.
The text was updated successfully, but these errors were encountered: