Python SDK concepts¶

The sim2bot Python package makes the browser simulator look like a robot endpoint. This guide explains how its discovery, addressing, timing, telemetry, and camera concepts fit together. Use the API reference for exact signatures and Python examples for complete programs.

This is the recommended integration surface for Python projects. Use the lower level WebSocket, TCP, or UDP protocol only when you need another language or a custom transport.

What runs where

Your Python script and the bridge run on your computer (or, with deliberate LAN configuration, another PC on your network). The MuJoCo simulation still runs in the browser tab. Keep that tab visible for consistent real-time behaviour.

Install and first connection¶

From the repository:

cd services/bridge
pip install -e ./sim2bot          # SDK, control, telemetry
pip install -e "./sim2bot[cv2]"  # optional: decode camera frames to numpy/OpenCV

The easiest first script starts the bridge if needed and waits for the browser simulator to connect:

from sim2bot import Robot

with Robot(auto_bridge=True, wait_for_sim=True) as sim:
    robots = sim.describe()
    for arm in robots:
        print(f"[{arm.index}] {arm.name}: {arm.dof} DOF")

Open Sim2Bot, then use Tools → Bridge → Connect. wait_for_sim=True blocks until the browser announces a scene; use wait_for_sim_timeout=30 to fail after 30 seconds instead of waiting indefinitely.

`Robot(...)`¶

Robot(
    url="ws://localhost:8765/ws",
    video_url=None,
    connect_timeout=5.0,
    transport="ws",
    udp_host=None,
    udp_port=8771,
    auto_bridge=False,
    bridge_timeout=10.0,
    wait_for_sim=False,
    wait_for_sim_timeout=None,
    wait_for_sim_poll_interval=0.5,
    api_key=None,
    room=None,
)

Argument	Meaning
`url`	Control + telemetry WebSocket. The local default is correct for most use.
`video_url`	Optional binary camera WebSocket. By default it is derived from `url` (`/ws` → `/video`).
`transport`	`"ws"` (default, reliable WebSocket) or `"udp"` (latest-value control/telemetry over UDP). Camera video always uses WebSocket.
`udp_host`, `udp_port`	UDP bridge destination; default port is `8771`.
`auto_bridge`	Starts a local bridge process if one is not already running.
`wait_for_sim`	Waits for a browser scene to announce during `connect()`.
`api_key`	Required only for a bridge deliberately exposed to another PC on the LAN. Same-device loopback use needs no key.
`room`	Optional pairing name. Only a browser simulator and controllers with the same room communicate.

Use it as a context manager (recommended), or call connect() and close():

sim = Robot(auto_bridge=True).connect()
try:
    print(sim.describe())
finally:
    sim.close()

Discover the current scene¶

Never hard-code the robot DOF, joint order, camera IDs, or home pose. Ask the scene what it contains.

with Robot(auto_bridge=True, wait_for_sim=True) as sim:
    arms = sim.describe()
    for arm in arms:
        print(arm.index, arm.id, arm.name)
        print(arm.joint_names, arm.joint_limits, arm.home)

    for camera in sim.cameras():
        print(camera["id"], camera["label"], camera["mount"])

`RobotInfo`¶

describe() and wait_for_sim() return list[RobotInfo].

Field	Meaning
`index`	Current bridge command address. Pass this as `robot=...`.
`id`	Stable scene identity while the scene is loaded; useful for labels and correlation.
`name`, `dof`	Human-readable robot name and arm joint count.
`joint_names`, `joint_limits`, `home`	Model-derived command order, limits, and home joint positions.
`has_gripper`	Whether `gripper()` has a robot gripper to drive.
`locomotion`, `base_dof`	`arm`, mobile manipulator, or aerial capability metadata.

Scene and wait methods¶

Method	Result
`describe(timeout=2.0)`	Refreshes and returns the available robots.
`wait_for_sim(timeout=None, poll_interval=0.5)`	Blocks until at least one browser robot is announced.
`cameras(timeout=2.0)`	Returns camera discovery dictionaries.
`room_devices(timeout=2.0)`	Returns authored doors/windows, if present.

Robot motion API¶

Every robot-specific method has robot=0. Index 0 addresses the first robot; use the RobotInfo.index discovered above for another arm.

with Robot(auto_bridge=True, wait_for_sim=True) as sim:
    left, right = sim.describe()

    sim.move_to(left.home, robot=left.index)
    sim.move_to(right.home, robot=right.index)
    sim.wait_until_reached(left.home, robot=left.index)

Method	Purpose
`move_to(q, robot=0)`	Send joint-position targets in the discovered joint order.
`set_velocity(qd, robot=0)`	Send joint velocities in rad/s. Refresh at least every 500 ms; the safety watchdog expires stale velocity commands.
`move_to_pose(position, orientation=None, robot=0)`	Send a world-frame TCP target in metres. The browser solves IK; quaternion order is `[x, y, z, w]`.
`gripper(fraction, robot=0)`	Set gripper openness: `0.0` closed, `1.0` open.
`move_trajectory(points, robot=0, loop=False)`	Play timed `(seconds, joint_targets)` waypoints.
`stop_trajectory(robot=0)`	Cancel trajectory playback and hold position.
`stop(robot=0)`	Freeze the addressed robot at its current pose.
`reset(robot=0)`	Reset the simulation and release the addressed external command. At present the simulation reset itself is scene-wide.
`base_velocity(vx, vy, vz, omega, robot=0)`	Drive a supported mobile/aerial base. Multi-base targeting is planned; currently the scene's primary base is driven.

Cartesian control¶

# Position only: the solver chooses a feasible wrist orientation.
sim.move_to_pose([0.45, 0.0, 0.35])

# Full pose: position in metres, orientation as xyzw quaternion.
sim.move_to_pose([0.45, 0.0, 0.35], [0.0, 0.0, 0.0, 1.0])

An unreachable pose settles at the closest configuration the in-browser IK can find. For predictable programs, verify the resulting state().tcp and joint state before continuing.

Timed trajectory¶

home = sim.state().q
wave = [q + 0.25 for q in home]

sim.move_trajectory([
    (0.0, home),
    (1.0, wave),
    (2.0, home),
])

The times are seconds from the trajectory start and must be strictly increasing. App motion-recording exports use milliseconds, so divide recorded t by 1000 before passing it to move_trajectory().

Telemetry and robot state¶

state() is deliberately latest-value, not a queue. This prevents a slow Python process from falling seconds behind the simulation.

state = sim.state(robot=0)
if state is not None:
    print(state.q)                       # joint position, radians
    print(state.qd, state.qdd, state.qddd)  # velocity, acceleration, jerk
    print(state.tcp)                     # world TCP XYZ, metres
    print(state.tcp_orientation)         # xyzw quaternion
    print(state.latency)                 # seconds from browser capture to now

`RobotState` fields¶

Field	Units / meaning
`q`, `target`	Joint position and latest position target, radians.
`qd`, `qdd`, `qddd`	Joint velocity, acceleration, and jerk: rad/s, rad/s², rad/s³.
`tcp`, `tcp_orientation`	TCP world position (m) and orientation quaternion `[x,y,z,w]`.
`tcp_linear_velocity`, `tcp_angular_velocity`	TCP linear m/s and angular rad/s velocity.
`tcp_linear_acceleration`, `tcp_angular_acceleration`	TCP acceleration.
`tcp_linear_jerk`, `tcp_angular_jerk`	TCP jerk.
`gripper`	Commanded fraction, or `None` if unavailable.
`sensors`	Authored sensor readings attached to this robot.
`samples`	High-rate `{t, q, qd}` physics-substep snapshots since the prior packet.
`capture_ts`, `latency`	Browser wall-clock capture time and current age in seconds.

Use states(robot=0) for just the high-rate sample batch. Use on_telemetry(callback) for an event-style integration; it runs on the SDK reader thread, so hand data to your own queue if the callback does real work.

def report(state):
    print(state.robot, state.tcp)

sim.on_telemetry(report)

Camera API¶

Cameras are global scene resources, independent of the robot index used by motion commands. One overhead camera may watch two arms; it appears once in cameras() and either arm can be commanded while its stream is open.

cameras = sim.cameras()
overhead = next(c for c in cameras if c["mount"]["kind"] == "world")

with sim.camera(overhead["id"], fps=24, width=640, height=480) as feed:
    frame = feed.read(timeout=2.0)
    if frame:
        print(frame.camera_id, frame.width, frame.height, frame.age)
        image_bgr = frame.image()  # needs: pip install "sim2bot[cv2]"

`camera(camera_id, ...)`¶

Argument	Meaning
`camera_id`	Required ID from `cameras()`: `model:<index>`, `custom:<id>`, or `sensor:<id>`.
`fps`	1–30 FPS; omitted uses the camera's GUI stream default.
`width`, `height`	Requested pixels, currently clamped to 16–1920 × 16–1080.
`quality`	JPEG quality from 0.1 to 1.0.
`codec`	`"jpeg"` (default, compact) or `"raw"` (RGBA, lowest localhost latency). H.264 is reserved for later.

CameraStream.read(timeout=5.0) blocks for a newer frame or returns None. Iterating it yields only fresh frames. Close the stream (or use with) to stop the subscription. CameraFrame.image() returns a BGR numpy image when the optional cv2 extra is installed.

Debug markers¶

Markers are visual-only helpers for a policy target, grasp point, planned path, or annotation. They never alter physics and do not appear in camera feeds.

sim.marker(
    "target",
    "sphere",
    position=[0.45, 0.0, 0.35],
    scale=0.06,
    color=[0.36, 0.54, 0.92, 1.0],
)
sim.delete_marker("target")
sim.clear_markers()

Supported shapes are sphere, box, arrow, line, text, axes, and points. See Python control (bridge) for shape-specific arguments.

Timing and safety¶

Current telemetry packet rate is configurable up to 60 Hz per robot.
samples preserves physics-substep resolution (commonly 200–500 Hz, according to the MJCF timestep) inside those packets.
New command application is paced by the visible browser simulation loop, normally around 60 Hz; sending faster is safe but becomes latest-command-wins.
Camera streams are subscription-gated and capped at 30 FPS.
GUI plot smoothing is display-only. SDK telemetry and exports retain raw derivatives.
The local bridge binds to loopback by default. LAN use is intentional and requires a token; do not expose it directly to the public internet.

For protocol-level JSON/WebSocket/TCP/UDP details, see Python control (bridge) and the repository's docs/bridge-protocol.md.