Some Difference in Reproduced Results

[JaydenLyh]

The paper and the code are both very well done and have been very helpful for my research. I particularly appreciated the detailed explanations and the thoroughness of the experiments.
I cloned this repository and trained the dyffusion, but I got some difference in reproduced results.
Steps Followed
I followed the steps provided in the README to train the model:
First stage: Train the interpolator network
python run.py experiment=spring_mesh_interpolation
Second stage: Train the forecaster network
Encountered some issues, so I commented out lines 229-231 in wandb_api.py:
if os.path.exists(ckpt_path) and ckpt_path != best_model_fname: os.remove(ckpt_path) # remove if one exists from before os.rename(best_model_fname, ckpt_path)
Then ran:
python run.py experiment=spring_mesh_dyffusion diffusion.interpolator_run_id=<WANDB_RUN_ID>
Testing：
python run.py mode=test logger.wandb.project=DYffusion-spring-mesh logger.wandb.id=<run_id>

Actual Results
test/50ens_mems/avg/crps = 1.54671955108642
test/50ens_mems/avg/mse = 36.7373008728027
test/50ens_mems/avg/ssr = 2.02742147445678

Results on Navier Stokes Dataset
On the Navier Stokes dataset, I was able to reproduce results similar to those reported in your paper.
Could you please provide guidance on what might be causing these discrepancies? Any help or suggestions would be greatly appreciated. Thank you

[salvaRC]

Hi and thank you for the feedback! I'm glad that you found our work and code helpful and happy to hear that the navier stokes results are as expected.

I am sorry to hear about the issue with the spring mesh dataset. To better understand it, could you share with me the configs (i.e. hydra_config.yaml files) for both the interpolation and dyffusion training stages that you use?

[JaydenLyh]

springmesh_configs.zip
Thank you so much for your response!
I've uploaded the files you requested. Please let me know if there's anything else you need or if there are any additional details I can provide.
Thanks again for your support!

[salvaRC]

Thank you for the configs. I had to dive deeper into why this mismatch was occurring. I apologize for not getting back to you sooner. I have figured out the reasons and fixed them in the latest version of this repo. I had overlooked some details when open-sourcing the repo, which are minor but impactful for the spring-mesh experiments. They are fixed now.

The issues should only have affected the training of the forecaster. So, you can just try repeating the second stage only.

If it helps, I have published the correctly reproduced interpolator and forecaster runs to a public weights & biases project: https://wandb.ai/salv47/DYffusion-spring-mesh-public/workspace