rerun always begins playback at 30fps, even when the dataset prescribes a different fps.

[RussTedrake]

System Info

- `lerobot` version: 0.1.0 (building from source: commit: 461d5472d329a6a461f56a85175d4744b8e78e61)
- Platform: macOS-14.5-arm64-arm-64bit
- Python version: 3.12.3
- Huggingface_hub version: 0.23.4
- Dataset version: 2.20.0
- Numpy version: 1.26.4
- PyTorch version (GPU?): 2.3.1 (False)
- Cuda version: N/A
- Using GPU in script?: no

Information

• One of the scripts in the examples/ folder of LeRobot
• My own task or dataset (give details below)

Reproduction

python lerobot/scripts/visualize_dataset.py \
        --repo-id lerobot/xarm_push_medium \
        --episode-index 1

or with my dataset, which was the original motivation:

 python lerobot/scripts/visualize_dataset.py \
        --repo-id russtedrake/tri-small-BimanualStackPlatesOnTableFromTable \
        --episode-index 1        

Expected behavior

The rerun.io viewer should open, and start playing the data back at 15 fps (as specified in the dataset). But instead it always defaults to 30 fps playback.

For my dataset, I hoped for 10 fps playback, but still get 30 fps by default. I've also confirmed that my mp4s are encoded at the proper rate, and playback at the correct speed e.g. here: https://huggingface.co/datasets/RussTedrake/tri-small-BimanualStackPlatesOnTableFromTable/blob/main/videos/scene_left_0_episode_000002.mp4

I wonder if a call to this might help: https://docs.rs/re_viewer_context/0.16.1/re_viewer_context/struct.TimeControl.html#method.set_fps

fwiw -- i've also blown away my local cache and confirmed that the dataset which is getting loaded does indeed report dataset.fps does indeed equal 15 (or 10) respectively.

cc @notmahi who I was talking to about this last week.

[Cadene]

@RussTedrake Sorry for the trouble. We reached out to rerun developers for help.

Out of curiosity, did you try set_fps?

What do you mean by "i've also blown away my local cache"?
Could you be more specific? It would greatly help us.

We have another option to visualize datasets. It's still not on master yet: #188
It will create a local web server similar to: https://huggingface.co/spaces/lerobot/visualize_all_datasets

[nikolausWest]

Playback speed is not yet settable from code (blueprint) in Rerun (here is the issue to follow rerun-io/rerun#5577). When that is done and released, the Rerun visualizer in LeRobot would need to use match the dataset fps to that blueprint setting. In the meantime, it's possible to work around that by using a timeline with timestaps instead of frame numbers which will make it play back at the correct speed by default

[RussTedrake]

Thank you both. @nikolausWest -- I did observer that playing back with the timeline works, but can I open the rerun browser in that mode automatically? or do I have instruct everyone to either manually change the FPS or manually change to timeline?

Out of curiosity, did you try set_fps?

I tried for a moment, but couldn't see how to call it with the python API. Sounds like it's not possible yet.

What do you mean by "i've also blown away my local cache"?

I had an early revision of my dataset that accidentally chose 30 fps instead of 10 fps. So I was just double-checking to make sure that the huggingface dataloader was not using any piece of the local cache, instead of downloading the entirely new revision.

I know it probably doesn't seem like a big deal, but my team works very hard to make sure our simulator has beautiful physics. If the default first experience people have when viewing our dataset is 3x playback, which makes the physics look more toy, then that's sad for me.

Thanks for the fast responses!

[nikolausWest]

I did observer that playing back with the timeline works, but can I open the rerun browser in that mode automatically? or do I have instruct everyone to either manually change the FPS or manually change to timeline?

@RussTedrake, unfortunately there is no programatic control over which timeline is shown first yet in (that will come when we blueprintify the time controls at the same time as allowing you to set playback speed). In the meantime, the starting timeline is chosen by sorting the user defined timelines alphabetically and then choosing the first. If you can control the name of the timelines, you can use that to hack around this limitation in the meantime (although admittedly hacky).

We haven't received a lot of requests for this feature yet. Are you open to sharing what project this is?

[RussTedrake]

Of course. To be clear, resolving rerun-io/rerun#5577 would be the best solution, from my perspective (and then having lerobot communicate the fps it already knows about to rerun in visualize_dataset.py).

I just described the project (we call it "lbm_eval") in a tutorial at RSS last week. Here is the slide deck. And here is a link to the specific slide where I was trying to use rerun (via lerobot) to let people step through a few sample teleop demonstrations in in the sim eval that we hope to make public soon.