Skip to content

Dataset formats

Dataset Studio exports two formats directly, and converts to two more for the model you want to train. Choose by what you train with:

You train with Format How
LeRobot: ACT, Diffusion Policy, SmolVLA, pi0 LeRobot v3.0 Export › LeRobot v3.0
Your own pipeline Raw folder: MP4s and CSVs Export › Raw folder
ACT, ALOHA, act-plus-plus HDF5, a file per episode the raw folder, converted with raw-to-hdf5.py
robomimic HDF5, one file the raw folder, converted with raw-to-hdf5.py
OpenVLA, Octo, Open X-Embodiment RLDS the raw folder, converted with raw-to-rlds.py

The converters are Python scripts you run on your own computer: nothing is uploaded. Both read the raw folder, so tick Raw folder in Export (it can go in the same download as LeRobot). Each decodes every camera's video into frames, so the result needs no video library to train on.

A trimmed episode's videos hold the whole take, and the raw folder says where its kept part starts (video_start_frame in dataset.json). The scripts here read it. A copy downloaded before 2 October 2026 stops with "has N frames, samples.csv M" on a trimmed episode: download it again.

HDF5 for ACT, act-plus-plus and robomimic

Download raw-to-hdf5.py, then:

python -m pip install h5py numpy av
python raw-to-hdf5.py sim2bot-dataset/raw hdf5 --layout act
python raw-to-hdf5.py sim2bot-dataset/raw dataset.hdf5 --layout robomimic

ACT and ALOHA (--layout act) get one episode_<n>.hdf5 per episode: /observations/qpos and /observations/qvel, /observations/images/<camera> and /action, marked sim. qpos and action are the joints, then the gripper's opening (0 closed to 1 open); the gripper's velocity is not recorded and reads 0.

  • The original ACT needs every episode the same length. Add --pad to hold each episode's last frame until it is as long as the longest.
  • act-plus-plus takes episodes as they are. It adds two columns to every action for Mobile ALOHA's base, zero here, so its actions are two wider than qpos.

robomimic (--layout robomimic) gets one file with a demo per episode: actions, rewards (0) and dones (1 on the last step), and these observations:

  • robot0_joint_pos, robot0_joint_vel and robot0_gripper_qpos;
  • robot0_eef_pos and robot0_eef_quat, the tool point, when the robot has one;
  • <camera>_image for each camera.

mask/train and mask/valid hold out the last tenth of the demos for validation (--valid-fraction).

Images are compressed inside the file; every HDF5 reader undoes that without being asked. --no-compress stores them raw instead.

RLDS for OpenVLA and Octo

Download raw-to-rlds.py, then:

python -m pip install tensorflow tensorflow-datasets av
python raw-to-rlds.py sim2bot-dataset/raw rlds

It writes a TensorFlow Datasets dataset, rlds/sim2bot_dataset/1.0.0, laid out as the Open X-Embodiment datasets are. Each episode is a list of steps:

  • observation/image: the first camera; each other camera is observation/image_<camera>;
  • observation/state: the joints, then the gripper's opening;
  • observation/joint_velocity, and observation/tcp (the tool point's position and orientation, when the robot has one);
  • action: the commanded joints and gripper;
  • language_instruction: the episode's task;
  • is_first, is_last, is_terminal, reward and discount, as RLDS defines them.

--val-fraction 0.1 also writes a val split. Open it as any RLDS dataset:

import tensorflow_datasets as tfds

builder = tfds.builder_from_directory("rlds/sim2bot_dataset/1.0.0")
for episode in builder.as_dataset(split="train").take(1):
    for step in episode["steps"]:
        print(step["language_instruction"], step["action"])

OpenVLA and Octo name each dataset's image, state and action keys in their own configuration: here they are image, state and action.

How the converters were checked

Each output was loaded with the reader that trains on it, and every value compared with the raw folder it came from:

  • ACT's and act-plus-plus's own data loaders;
  • robomimic 0.3.0's SequenceDataset, with every demo and each filter key;
  • TensorFlow Datasets 4.9.10, reading every step of every episode.

States and actions came back exactly. HDF5 images were identical; RLDS stores JPEG, so its images are within JPEG's small loss.