Depth · Physical Intelligence — Perit
What it is
What gets captured, and why it is worth having.
Environment: Living room, afternoon light
Captured with: Stereo passthrough headset, RGB + aligned depth
Task: Pack a mug into a box and close the flaps
The same tasks and rooms as egocentric capture, with a second stream: a per-frame depth map aligned to the RGB image, metric where the sensor supports it. Hand–object distance, container interiors, the gap between a plate and the shelf it is going onto — the geometry a monocular camera has to guess.
Rigs range from a consumer stereo headset to a LiDAR phone to a calibrated stereo pair; calibration files ship with every episode so the depth can be trusted downstream.
What it trains
- Depth estimation and 3D scene understanding
- 3D hand and object pose
- Spatial reasoning for manipulation
- Sim-to-real gap measurement
The spec
How it is captured, and what you receive.
How it is captured
- Primary rig: Stereo passthrough headset (Quest-class)
- Phone rig: LiDAR phone (iPhone Pro-class), RGB plus metric depth
- Head mount: RealSense-class depth camera on a low-profile strap
- Custom: Calibrated stereo pair on a head strap or cap brim
- Calibration: Intrinsics and extrinsics per rig, checked against a target every session
- Sync: Depth aligned to RGB per frame, one clock across streams
What you receive
- Video: H.264 in MP4, RGB stream
- Depth: Per-frame depth map aligned to RGB, sensor-native resolution
- Resolution: 1080p RGB, 4K on request
- Frame rate: 30 fps
- Calibration: Intrinsics and extrinsics per rig, per session
- Episode: 2–10 minutes
Environments and tasks
Where it is recorded, and what people are asked to do.
Tasks are recorded where they actually happen — not in a studio dressed to look like a kitchen. Hover a room to see the kind of setting the protocol calls for.
Kitchens
- Indian, European and Brazilian home kitchens, lived-in and cluttered, plus small commercial kitchens and pantries.
- Cooking preparation
- Washing up
- Cupboards and drawers
- Appliances
- Pouring and scooping
Living rooms and bedrooms
- Real Western homes in Spain, Germany, France and Italy, and apartments in India, Brazil, Japan and Korea. Sofas, wardrobes, shelves, laundry.
- Tidying and folding
- Making beds
- Shelving and storage
- Laundry
- Furniture interaction
Offices
- Open-plan and small offices: desks, pantries, meeting rooms, filing and cable clutter.
- Desk tasks
- Document handling
- Cables and peripherals
- Pantry tasks
- Drawers and storage
Retail and small shops
- Kirana stores, groceries, pharmacies and supermarket aisles, with real stock and real customers out of focus.
- Shelf restocking
- Picking to a list
- Bagging and counters
- Inventory checks
- Crates and boxes
Warehouses and light industrial
- Racking aisles, packing benches, courier depots and small production lines with pallets, trolleys and parts bins.
- Pallet to shelf
- Parcel sorting
- Packing and taping
- Parts movement
- Inspection
Workshops and labs
- Mechanical and electronics workshops, repair benches and robotics labs where a follower rig can be set up.
- Tool handling
- Assembly and fastening
- Pegboards and racks
- Bench tidying
- Teleoperation setups
Agriculture and outdoor
- Polytunnels, packing sheds, market stalls and yards, in daylight that changes from one episode to the next.
- Sorting produce
- Crates and trays
- Pruning and picking
- Weighing and bagging
- Loading
Client-defined and confidential
- Your site, or a space recreated to your drawings, run by an NDA'd crew with limited-visibility workflows and no clip leaving your region.
- Proprietary workflows
- Recreated spaces
- Stealth programs
- On-site with your rig
- Your robot, our operators
What ships
Deliverables, and the labels you can add.
Deliverables
- RGB video and aligned depth per episode
- Calibration files per rig
- Task and environment metadata per episode
- QC status and reviewer chain per episode
- Consent reference on every file
- Quality report per batch: accepted, rejected, and why
- Weekly drops to your bucket, region pinned at kickoff
Annotation add-ons
- 3D hand pose ground truth
- Object masks with depth
- Scene graphs
- Captions
Labelled on the same bench as our speech work, to a written guide, behind the same calibration test.
How a pilot runs
Ten to thirty episodes, then the volume.
01 day 0
Brief: The task family, the environments, the rig, the sensors, the episode count. One page, agreed before anything is bought or anyone is recruited.02 week 1
SOP and kit: A written capture protocol — framing, lighting, where a task starts and ends, what counts as a failed take — and the kit list. Operators are trained and tested on it before a single episode is paid for.03 week 2
Pilot episodes: A small batch, ten to thirty episodes, delivered with QC status and a quality report. This is where the protocol breaks — on purpose, and cheaply.04 ongoing
Review and scale: You review the pilot, we fix the protocol, then the run scales on a weekly delivery cadence with the same report attached to every batch.
Turnaround
- 48 hrs: Written spec and quote
- 5 days: Sample episode in your format
- 2 weeks: Pilot batch with its QA report
- Weekly: Delivery cadence once the pilot clears
Before an episode counts
What QC checks, every time.
- Framing: The task stays in frame; hands and workspace inside the protocol's bounds for the whole episode.
- Exposure and focus: No blown highlights or focus hunting beyond the tolerance set for the room.
- Motion: No camera whip, no dropped frames; IMU continuous where the rig has one.
- Sync: Every stream aligned within one frame; timestamps monotonic across cameras, state and action logs.
- Task compliance: Start and end states match the SOP; no skipped or reordered steps unless the protocol allows it.
- Duration: Inside the episode window for the task family; overruns are trimmed to the protocol, not the clip.
- Metadata: Environment, task, rig, operator id, session and consent reference present and valid on every episode.
- Privacy: No bystander faces, screens or documents without consent; blurred where the protocol flags it.
- Calibration: Intrinsics and extrinsics on file per rig and session for depth, stereo and multi-camera capture.
- Labels: Where annotation is ordered, agreement against the gold set above the threshold before the batch ships.
Samples and the other lines
3 depth episodes to watch and download.
Each with the rig, the room, the task and the file list of the full download.
- Open the samples
- All samples and hardware
- Egocentric: First-person RGB video of real people doing real tasks.
- Teleoperation: Human-guided robot demonstrations, one episode at a time.
- UMI Gripper: In-the-wild wrist-gripper demos for grasp generalisation.
- Hand Pose: Absolute 3D hand pose ground truth, occlusions included.
- Tactile: Contact and pressure streams for force-aware manipulation.
- Bring Your Own Rig: Your hardware and protocol, our crews, rooms and QA.
- Video Annotation: Segmentation, captions, masks and success labels on Foundry.
- Samples and hardware: Watch and download sample episodes, and see what every line is recorded with.