Python control (the bridge)¶

Browsers can't listen on raw UDP/TCP sockets, so a small local bridge relays between your controller code and the browser simulator.

controller (Python)  --WS / TCP / UDP-->  bridge  --WS-->  browser simulator
browser simulator    --telemetry-------->  bridge  ------>  controller

Run the bridge¶

One command (sets up the venv + deps on first run, then starts):

npm run bridge            # from sim2bot/   (or ./run.sh from services/bridge)

Or manage the venv yourself:

cd sim2bot/services/bridge
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --host 0.0.0.0 --port 8765

Then in the app open Tools → Bridge and click Connect. Integration tests for the relay: npm run bridge:test.

Transports¶

Transport	Port	Framing
WebSocket	8765 (`/ws`)	one JSON message
TCP	8770	newline-delimited JSON (`{...}\n`)
UDP	8771	one JSON object per datagram

All carry the same schema.

Addressing a robot (multi-robot)¶

Every command may include a robot index — 0 = the first robot, 1.. = the rest in scene order; omit it to address robot 0. The simulator announces the available robots in its hello (robots: [{index, name, dof}]), and telemetry is tagged with each robot's robot index and name.

{"type": "joint_position", "robot": 1, "q": [0, -1.57, 1.57, -1.57, -1.57, 0]}
{"type": "telemetry", "robot": 1, "name": "Universal Robots UR5e", "dof": 6, "q": [...]}

One telemetry packet per robot¶

The simulator sends a separate, self-contained telemetry packet for each robot — one after another every tick — rather than bundling all robots into a single message. This mirrors the command direction (commands are per-robot addressed too), maps cleanly to the SDK's per-Robot model (a consumer that cares about one robot just filters by robot index), keeps each message small, and is robust to the robot set changing.

All packets from the same tick share the same t (the sim clock, stamped once per tick), so to assemble a synchronized multi-robot snapshot, group incoming packets by t:

# pseudo: collect a full multi-robot frame
frame = {}
while len(frame) < n_robots:
    pkt = recv()
    frame[pkt["robot"]] = pkt   # all share pkt["t"] for this tick

In the GUI, the Bridge panel shows this as Telemetry: 59 Hz · 3 robots — the per-robot rate (each robot streams at the rate you set) and the robot count. Total messages/sec on the wire = rate × robots.

Commands¶

Command	Payload	Effect
`joint_position`	`q: number[]`	Joint position targets (PD-tracked).
`joint_velocity`	`qd: number[]`	Joint velocity targets (500 ms watchdog).
`gripper`	`fraction` (0..1)	Open/close the gripper (no-op without one).
`base_velocity`	`vx, vy, vz, omega`	Drive a mobile/aerial base (robot frame; `vz` aerial only).
`tcp_pose`	`position: [x,y,z]`, optional `orientation: [x,y,z,w]`	Cartesian target — the sim solves IK in-browser and drives the arm there (position-only, or full 6-DOF with the quaternion).
`joint_trajectory`	`points: [{t, q}, ...]`, optional `loop`	Play a timed joint trajectory (simplified FollowJointTrajectory) — see below.
`joint_trajectory_stop`	—	Cancel a playing trajectory and freeze at the current pose.
`marker`	`marker: {id, shape, ...}`	Draw/update a debug marker by id (RViz-style).
`marker_delete` / `marker_clear`	`id` / —	Remove one marker / clear all.
`reset` / `stop`	—	Reset + release / freeze at current pose.

{"type": "gripper", "fraction": 0.5}
{"type": "base_velocity", "vx": 0.3, "vy": 0.0, "vz": 0.0, "omega": 0.4}
{"type": "tcp_pose", "position": [0.45, 0.0, 0.4]}
{"type": "marker", "marker": {"id": "target", "shape": "sphere", "position": [0.4, 0, 0.5], "scale": 0.08, "color": [1, 0.3, 0.3, 1]}}

Debug markers (RViz Marker style) let controller/policy code draw primitives into the scene for debugging — a target, predicted grasp, planned path, labels. shape ∈ sphere | box | arrow | line | text | axes | points; optional position [x,y,z], orientation quat [x,y,z,w], scale (number or [x,y,z]), color rgba (0..1), and shape-specific points (a polyline for line, or a point cloud for points — scale = point size), from/to (arrow), text. Re-send an id to update it. SDK: robot.marker(id, shape, ...), robot.delete_marker(id), robot.clear_markers() (see examples/markers_example.py). Markers are visual only and don't appear in camera feeds.

The SDK wraps this as Robot.move_to_pose(position, orientation=None) — Cartesian control without computing joint angles yourself:

from sim2bot import Robot
with Robot() as robot:
    robot.move_to_pose([0.45, 0.0, 0.4])                 # position-only IK
    robot.move_to_pose([0.45, 0.0, 0.4], [0, 0, 0, 1])   # full 6-DOF pose

Joint trajectory (a simplified FollowJointTrajectory) plays a timed sequence of joint waypoints — record a motion by hand (Motion panel -> Record & replay -> Export JSON), or generate one programmatically, then play it back over the bridge instead of streaming joint_position yourself:

{"type": "joint_trajectory", "points": [{"t": 0.0, "q": [0,0,0,0,0,0,0]}, {"t": 2.0, "q": [0.5,0,0,0,0,0,0]}], "loop": false}

points[].t is SECONDS from the start of the trajectory (strictly increasing); q is the per-joint target array. The sim interpolates between waypoints by wall-clock time and drives the arm through the same external-target path as joint_position — servoed in Physics mode, exact in Kinematic. Playback holds the final point once it ends (or loops with loop: true) until a new command — joint_position, joint_velocity, tcp_pose, reset, stop, or joint_trajectory_stop — is sent for that robot. SDK:

from sim2bot import Robot
with Robot() as robot:
    home = robot.state().q
    bent = [v + 0.5 for v in home]
    robot.move_trajectory([(0.0, home), (1.5, bent), (3.0, home)])  # (t, q) pairs, t in seconds
    robot.stop_trajectory()  # cancel early, freeze in place

A recording exported from the app has t in MILLISECONDS — divide by 1000: points = [(f["t"] / 1000, f["q"]) for f in recording["trajectory"]]. See examples/joint_trajectory_example.py.

Telemetry packets carry q, target, qd (joint velocity), tcp (end-effector world position, per robot), and gripper (commanded fraction) when available, plus captureTs and a high-rate samples batch ([{t, q, qd}, …], one per physics substep since the last packet) so a controller gets ~500 Hz–1 kHz resolution at the packet cadence (robot.states() in the SDK).

Rates & limits¶

Telemetry data resolution: up to the model's physics rate (~500 Hz Franka, ~200 Hz SO-101), as the timestamped samples batch inside each packet.
Telemetry packet rate: configurable up to 60 Hz (default 60; Tools → Bridge → Telemetry rate). Evenly-spaced 100–500 Hz packets aren't possible from a browser tab.
Command application: commands are accepted instantly at any rate (last-writer-wins) but applied at the ~60 Hz control-loop tick.
Both ceilings come from physics running in the browser's render loop (~60 Hz). Full rationale and the path past 60 Hz (Worker-based physics): see docs/realtime-rate-limits.md in the repo.

Keep the Sim2Bot window visible

The whole sim — physics, rendering, and camera/telemetry streaming — runs in the browser tab's requestAnimationFrame loop. Browsers throttle a tab that's hidden or occluded (covered by another window) to ~1 fps, so the sim slows to a crawl and your stream/command rates collapse. Keep the Sim2Bot window visible (split-screen or a second monitor) while your controller runs — this applies in real use, not just testing. The only way to remove the dependency is a server-side/headless sim.

Discovery (`describe`)¶

Rather than hard-coding DOF/home, ask the bridge what's loaded:

--> {"type": "describe"}
<-- {"type": "scene", "robots": [
      {"index": 0, "name": "Franka Emika Panda", "dof": 7,
       "home": [0, 0, 0, -1.57, 0, 1.57, -0.79],
       "jointLimits": [[-2.9, 2.9], ...], "hasGripper": true,
       "locomotion": "arm", "baseDof": 0}
    ]}

Works over WS, TCP, and UDP. The simulator re-announces when the scene's robot set changes, so the cache stays current. jointLimits entries are [lo, hi] or null for an unlimited joint.

Sensors¶

Authored scene sensors (Tools → Scene → Sensors — IMU, force, torque, touch, rangefinder, jointpos, jointvel, GPS) are streamed inside the telemetry packet of the robot they're attached to, under a sensors array. The field is omitted when a robot has no sensors.

{"type": "telemetry", "robot": 0, "name": "Franka Emika Panda", "dof": 9,
 "q": [...], "target": [...], "tcp": [x, y, z],
 "sensors": [
   {"id": "sensor_imu_ab12", "name": "IMU 1", "type": "imu",
    "values": [ax, ay, az, gx, gy, gz]},
   {"id": "sensor_force_cd34", "name": "Force 1", "type": "force",
    "values": [fx, fy, fz]}
 ]}

A sensor belongs to the robot its anchor link/joint/site sits on; sensors on scene objects (or in a single-robot scene) ride robot 0's packet. values length depends on the type (IMU = 6, force/torque/gps = 3, jointpos/jointvel = 1).

Camera streaming¶

Camera feeds stream on a separate binary WebSocket (/video on port 8765), kept apart from control/telemetry so video can never delay a command. A controller subscribes on the control socket; the simulator only renders + encodes cameras that someone is watching (re-send to keep a feed alive — it expires a few seconds after the last subscribe):

{"type": "camera_subscribe", "camera": "model:0", "fps": 30, "width": 640, "height": 480}
{"type": "camera_unsubscribe", "camera": "model:0"}

camera ids come from describe (its cameras list): model:<index>, custom:<id>, sensor:<id>. Each /video frame is a binary packet: version, codec (0=JPEG, 1=H264, 2=RAW), flags (keyframe), sequence, captureTs, width, height, cameraId, frame bytes. captureTs is the sim clock at render, so consumers measure latency and skip stale frames. Two codecs ship: JPEG (codec:"jpeg") and RAW uncompressed RGBA (codec:"raw") — RAW is the lowest-latency choice on localhost (no encode/decode; bandwidth is free on loopback). Frames are read back asynchronously (no GPU stall). H.264 is a future option for bandwidth-limited links.

Python SDK¶

The sim2bot package wraps control, telemetry, discovery, and camera feeds:

pip install -e ./sim2bot           # or "./sim2bot[cv2]" to decode camera frames

from sim2bot import Robot

with Robot() as robot:
    info = robot.describe()[0]            # dof, home, limits, gripper, ...
    robot.move_to(info.home)
    print(robot.state().q)                # telemetry
    for frame in robot.camera("model:0", fps=30):
        img = frame.image()               # numpy BGR (needs [cv2])
        break

Examples¶

services/bridge/examples/controller_example_sdk.py — the SDK (recommended).
services/bridge/examples/controller_example.py — raw WebSocket; robot index arg.
services/bridge/examples/controller_example_udp.py — raw UDP / TCP.
services/bridge/examples/joint_trajectory_example.py — play + cancel a timed joint trajectory (move_trajectory / stop_trajectory, incl. looping).

python examples/controller_example_sdk.py   # discover + drive + camera (SDK)
python examples/controller_example.py 1     # drive robot 1 over WebSocket
python examples/controller_example_udp.py udp 0

Commands are validated and clamped to each actuator's range, and velocity commands carry a 500 ms watchdog, so a crashed controller can't run a joint away.