Teleoperation · Physical Intelligence — Perit
What gets captured, and why it is worth having.
Environment: Small robotics workshop, LED work lamp
Captured with: Bimanual leader–follower arms, two camera views
Task: Stack three bowls
Teleoperation is the embodiment-matched end of the spectrum: the demonstration is performed on the robot the policy will run on, so there is no transfer gap to close. It is slower and dearer per episode than human video, which is why the protocol matters — every episode has to count.
Operators pass the same calibration gate as the rest of the bench, on the task and the rig, before a paid episode is recorded. Interventions and failed takes are logged, not discarded, because they are training data too.
What it trains
- Behaviour cloning and diffusion policies
- Policy evaluation and intervention data
- Recovery-from-failure demonstrations
- Embodiment-matched fine-tuning
How it is captured, and what you receive.
How it is captured
Primary rig: Bimanual leader–follower arms (ALOHA-class)
VR rig: Headset and controllers driving a single or dual arm
Your robot: Bring-your-own-stack, driven by our trained operators
Cameras: Wrist, front and overhead RGB, 2–4 synchronised streams
Logging: Joint state, gripper state and leader commands per step
Operators: Calibrated on the task and the rig before a paid episode
What you receive
- Cameras: 2–4 synchronised RGB streams, 480×640 to 1080p
- Frame rate: 30 fps
- State: Joint positions, velocities and gripper state per step
- Actions: Leader-arm or controller commands per step
- Format: LeRobot- or RLDS-style layout on request; HDF5 or Parquet
- Episode: 20 seconds – 3 minutes
Where it is recorded, and what people are asked to do.
Tasks are recorded where they actually happen — not in a studio dressed to look like a kitchen. Hover a room to see the kind of setting the protocol calls for.
Kitchens
- Indian, European and Brazilian home kitchens, lived-in and cluttered, plus small commercial kitchens and pantries.
Cooking preparation:- Washing up
- Cupboards and drawers
- Appliances
- Pouring and scooping
Living rooms and bedrooms
- Real Western homes in Spain, Germany, France and Italy, and apartments in India, Brazil, Japan and Korea. Sofas, wardrobes, shelves, laundry.
Tidying and folding:- Making beds
- Shelving and storage
- Laundry
- Furniture interaction
Offices
- Open-plan and small offices: desks, pantries, meeting rooms, filing and cable clutter.
Desk tasks:- Document handling
- Cables and peripherals
- Pantry tasks
- Drawers and storage
Retail and small shops
- Kirana stores, groceries, pharmacies and supermarket aisles, with real stock and real customers out of focus.
Shelf restocking:- Picking to a list
- Bagging and counters
- Inventory checks
- Crates and boxes
Warehouses and light industrial
- Racking aisles, packing benches, courier depots and small production lines with pallets, trolleys and parts bins.
Tasks:- Pallet to shelf
- Parcel sorting
- Packing and taping
- Parts movement
- Inspection
Workshops and labs
- Mechanical and electronics workshops, repair benches and robotics labs where a follower rig can be set up.
Tasks:- Tool handling
- Assembly and fastening
- Pegboards and racks
- Bench tidying
- Teleoperation setups
Agriculture and outdoor
- Polytunnels, packing sheds, market stalls and yards, in daylight that changes from one episode to the next.
Tasks:- Sorting produce
- Crates and trays
- Pruning and picking
- Weighing and bagging
- Loading
Client-defined and confidential
- Your site, or a space recreated to your drawings, run by an NDA'd crew with limited-visibility workflows and no clip leaving your region.
Tasks:- Proprietary workflows
- Recreated spaces
- Stealth programs
- On-site with your rig
- Your robot, our operators
Deliverables, and the labels you can add.
Deliverables
- Synchronised camera streams per episode
- State and action logs per step
- Task and environment metadata per episode
- QC status and reviewer chain per episode
- Consent reference on every file
- Quality report per batch: accepted, rejected, and why
- Weekly drops to your bucket, region pinned at kickoff
Annotation add-ons
- Success and failure labels per episode
- Sub-task boundaries
- Language instruction per episode
- Operator interventions flagged
Labelled on the same bench as our speech work, to a written guide, behind the same calibration test.
How a pilot runs
- Brief: The task family, the environments, the rig, the sensors, the episode count. One page, agreed before anything is bought or anyone is recruited.
- SOP and kit: A written capture protocol — framing, lighting, where a task starts and ends, what counts as a failed take — and the kit list. Operators are trained and tested on it before a single episode is paid for.
- Pilot episodes: A small batch, ten to thirty episodes, delivered with QC status and a quality report. This is where the protocol breaks — on purpose, and cheaply.
- Review and scale: You review the pilot, we fix the protocol, then the run scales on a weekly delivery cadence with the same report attached to every batch.
What QC checks, every time.
- Framing: The task stays in frame; hands and workspace inside the protocol's bounds for the whole episode.
- Exposure and focus: No blown highlights or focus hunting beyond the tolerance set for the room.
- Motion: No camera whip, no dropped frames; IMU continuous where the rig has one.
- Sync: Every stream aligned within one frame; timestamps monotonic across cameras, state and action logs.
- Task compliance: Start and end states match the SOP; no skipped or reordered steps unless the protocol allows it.
- Duration: Inside the episode window for the task family; overruns are trimmed to the protocol, not the clip.
- Metadata: Environment, task, rig, operator id, session and consent reference present and valid on every episode.
- Privacy: No bystander faces, screens or documents without consent; blurred where the protocol flags it.
- Calibration: Intrinsics and extrinsics on file per rig and session for depth, stereo and multi-camera capture.
- Labels: Where annotation is ordered, agreement against the gold set above the threshold before the batch ships.
Samples and the other lines
2 teleoperation episodes to watch and download.
Each with the rig, the room, the task and the file list of the full download.