[Fix] fix growing mem usage pytorch crnn

[felixdittrich92]

This PR:

• fix growing mem usage
• training still works

[codecov]

Codecov Report

Merging #1357 (c9ae4a9) into main (e83c3ab) will increase coverage by 0.01%.
Report is 1 commits behind head on main.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1357      +/-   ##
==========================================
+ Coverage   95.78%   95.80%   +0.01%     
==========================================
  Files         154      154              
  Lines        6910     6909       -1     
==========================================
  Hits         6619     6619              
+ Misses        291      290       -1     

Flag	Coverage Δ
unittests	95.80% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
doctr/datasets/vocabs.py	100.00% <100.00%> (ø)
doctr/models/recognition/crnn/pytorch.py	94.56% <100.00%> (-0.12%)	⬇️

... and 1 file with indirect coverage changes

[odulcy-mindee]

Ok, funny to see it fixed this issue. Thanks !