cudaFree failed with error an illegal memory access was encountered

[ShaYito]

I am frequently encountering CUDA-related memory issues while using NeRFshop. This issue could appear at any time when running the window (once it happened when the window just appeared and the training just began), it mainly happens after using NeRFshop for a while though.
I am using CUDA 11.6 on Windows 10.
Could you help me with it? Thank you in advance.

PS C:\_\nerfshop> ./build/nerfshop --scene data/nerfshop_scenes/Ficus/transforms.json
14:00:34 INFO     Loading NeRF dataset from
14:00:34 INFO       data\nerfshop_scenes\Ficus\transforms.json
14:00:35 SUCCESS  Loaded 400 images of size 800x800 after 0s
14:00:35 INFO       cam_aabb=[min=[-1.15679,-1.03824,-1.13484], max=[2.19975,2.29833,2.20784]
14:00:35 INFO     Loading network config from: configs\nerf\base.json
14:00:35 INFO     GridEncoding:  Nmin=16 b=1.38191 F=2 T=2^19 L=16
14:00:35 INFO     Color model:   3
14:00:35 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=1)]-->1
14:00:35 INFO       total_encoding_params=12196240 total_network_params=10240
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:454 cudaDeviceSynchronize() failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:454 cudaDeviceSynchronize() failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:454 cudaDeviceSynchronize() failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:454 cudaDeviceSynchronize() failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:454 cudaDeviceSynchronize() failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error an illegal memory access was encountered
C:\_\nerfshop\dependencies\tiny-cuda-nn\src\fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered
14:00:38 ERROR    Uncaught exception: Could not copy to host: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:285 cudaMemcpy(host_data, data(), num_elements * sizeof(T), cudaMemcpyDeviceToHost) failed with error an illegal memory access was encountered
Could not free memory: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:454 cudaDeviceSynchronize() failed with error an illegal memory access was encountered

The same error occurs multiple times at different places as below

15:36:11 ERROR    Uncaught exception: C:\_\nerfshop\src\testbed_nerf.cu:2932 cudaStreamSynchronize(stream) failed with error unspecified launch failure

15:32:02 ERROR    Uncaught exception: C:\_\nerfshop\src\testbed_nerf.cu:2819 cudaMemsetAsync(m_alive_counter.data(), 0, sizeof(uint32_t), stream) failed with error unspecified launch failure

15:23:36 ERROR    Uncaught exception: Could not copy to host: C:\_\nerfshop\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:285 cudaMemcpy(host_data, data(), num_elements * sizeof(T), cudaMemcpyDeviceToHost) failed with error an illegal memory access was encountered

15:21:50 ERROR    Uncaught exception: C:\_\nerfshop\src\testbed_nerf.cu:2827 cudaMemcpyAsync(&n_alive, m_alive_counter.data(), sizeof(uint32_t), cudaMemcpyDeviceToHost, stream) failed with error an illegal memory access was encountered