Some Issues with Replication

[wuwangzhuanwan]

(1) I used mmcv.imread to read the data, because directly using mmcv.load as in your code would result in an error, indicating unsupported color.
(2) Using the configuration hop_bevdet4d-r50-depth.py and reading the data with mmcv.imread shouldn't cause any issues. However, the replicated accuracy is only around 0.36+ mAP.

Can you please provide more details about your configuration and setup? It's possible that there might be some misconfigurations or parameters that need adjustment. Additionally, double-checking the dataset, model settings, and any preprocessing steps could help identify any potential sources of the lower accuracy.

[CaraJ7]

Hi @wuwangzhuanwan , thanks for your feedback.

We fix the bug for loading data by disk. It should work fine now.

Specifically, I find that mmcv.imread is not equal to the original code Image.open from BEVDet. This may probably be the reason for the low accuracy.

[xu19971109]

(1) I used mmcv.imread to read the data, because directly using mmcv.load as in your code would result in an error, indicating unsupported color. (2) Using the configuration hop_bevdet4d-r50-depth.py and reading the data with mmcv.imread shouldn't cause any issues. However, the replicated accuracy is only around 0.36+ mAP.

Can you please provide more details about your configuration and setup? It's possible that there might be some misconfigurations or parameters that need adjustment. Additionally, double-checking the dataset, model settings, and any preprocessing steps could help identify any potential sources of the lower accuracy.

Have you tried python tools/test.py? My training log is 0.36+ mAP but 0.39+ mAP in testing, which is almost same as the author released. It doesn't need to change configs or anything, just run python tools/test.py $config $ checkpoint. I guess there may be some TTA diff between val and test.

NDS=0.5066/0.5099,mAP=0.3979/0.3990

[CaraJ7]

Hi, the performance in the log file is based on epoch_24.pth. However, we test the model with epoch_24_ema.pth. The ema checkpoint is empirically better than the original model. This should account for the difference of mAP.