# What it is

## What gets captured, and why it is worth having.

**Environment**: Living room, afternoon light  
**Captured with**: Stereo passthrough headset, RGB + aligned depth  
**Task**: Pack a mug into a box and close the flaps

The same tasks and rooms as egocentric capture, with a second stream: a per-frame depth map aligned to the RGB image, metric where the sensor supports it. Hand–object distance, container interiors, the gap between a plate and the shelf it is going onto — the geometry a monocular camera has to guess.

Rigs range from a consumer stereo headset to a LiDAR phone to a calibrated stereo pair; calibration files ship with every episode so the depth can be trusted downstream.

## What it trains

- Depth estimation and 3D scene understanding  
- 3D hand and object pose  
- Spatial reasoning for manipulation  
- Sim-to-real gap measurement

# The spec

## How it is captured, and what you receive.

**How it is captured**
- **Primary rig**: Stereo passthrough headset (Quest-class)  
- **Phone rig**: LiDAR phone (iPhone Pro-class), RGB plus metric depth  
- **Head mount**: RealSense-class depth camera on a low-profile strap  
- **Custom**: Calibrated stereo pair on a head strap or cap brim  
- **Calibration**: Intrinsics and extrinsics per rig, checked against a target every session  
- **Sync**: Depth aligned to RGB per frame, one clock across streams

**What you receive**  
- **Video**: H.264 in MP4, RGB stream  
- **Depth**: Per-frame depth map aligned to RGB, sensor-native resolution  
- **Resolution**: 1080p RGB, 4K on request  
- **Frame rate**: 30 fps  
- **Calibration**: Intrinsics and extrinsics per rig, per session  
- **Episode**: 2–10 minutes

# Environments and tasks

## Where it is recorded, and what people are asked to do.

Tasks are recorded where they actually happen — not in a studio dressed to look like a kitchen. Hover a room to see the kind of setting the protocol calls for.

### Kitchens
- Indian, European and Brazilian home kitchens, lived-in and cluttered, plus small commercial kitchens and pantries.  
- **Cooking preparation**  
- **Washing up**  
- **Cupboards and drawers**  
- **Appliances**  
- **Pouring and scooping**

### Living rooms and bedrooms
- Real Western homes in Spain, Germany, France and Italy, and apartments in India, Brazil, Japan and Korea. Sofas, wardrobes, shelves, laundry.
- **Tidying and folding**  
- **Making beds**  
- **Shelving and storage**  
- **Laundry**  
- **Furniture interaction**

### Offices
- Open-plan and small offices: desks, pantries, meeting rooms, filing and cable clutter.  
- **Desk tasks**  
- **Document handling**  
- **Cables and peripherals**  
- **Pantry tasks**  
- **Drawers and storage**

### Retail and small shops
- Kirana stores, groceries, pharmacies and supermarket aisles, with real stock and real customers out of focus.  
- **Shelf restocking**  
- **Picking to a list**  
- **Bagging and counters**  
- **Inventory checks**  
- **Crates and boxes**

### Warehouses and light industrial
- Racking aisles, packing benches, courier depots and small production lines with pallets, trolleys and parts bins.  
- **Pallet to shelf**  
- **Parcel sorting**  
- **Packing and taping**  
- **Parts movement**  
- **Inspection**

### Workshops and labs
- Mechanical and electronics workshops, repair benches and robotics labs where a follower rig can be set up.  
- **Tool handling**  
- **Assembly and fastening**  
- **Pegboards and racks**  
- **Bench tidying**  
- **Teleoperation setups**

### Agriculture and outdoor
- Polytunnels, packing sheds, market stalls and yards, in daylight that changes from one episode to the next.  
- **Sorting produce**  
- **Crates and trays**  
- **Pruning and picking**  
- **Weighing and bagging**  
- **Loading**

### Client-defined and confidential
- Your site, or a space recreated to your drawings, run by an NDA'd crew with limited-visibility workflows and no clip leaving your region.  
- **Proprietary workflows**  
- **Recreated spaces**  
- **Stealth programs**  
- **On-site with your rig**  
- **Your robot, our operators**

# What ships

## Deliverables, and the labels you can add.

### Deliverables
- RGB video and aligned depth per episode  
- Calibration files per rig  
- Task and environment metadata per episode  
- QC status and reviewer chain per episode  
- Consent reference on every file  
- Quality report per batch: accepted, rejected, and why  
- Weekly drops to your bucket, region pinned at kickoff

### Annotation add-ons
- 3D hand pose ground truth  
- Object masks with depth  
- Scene graphs  
- Captions

Labelled on the same bench as our speech work, to a written guide, behind the same calibration test.

# How a pilot runs

## Ten to thirty episodes, then the volume.

1. **01 day 0**  
   Brief: The task family, the environments, the rig, the sensors, the episode count. One page, agreed before anything is bought or anyone is recruited.

2. **02 week 1**  
   SOP and kit: A written capture protocol — framing, lighting, where a task starts and ends, what counts as a failed take — and the kit list. Operators are trained and tested on it before a single episode is paid for.

3. **03 week 2**  
   Pilot episodes: A small batch, ten to thirty episodes, delivered with QC status and a quality report. This is where the protocol breaks — on purpose, and cheaply.

4. **04 ongoing**  
   Review and scale: You review the pilot, we fix the protocol, then the run scales on a weekly delivery cadence with the same report attached to every batch.

# Turnaround

- **48 hrs**: Written spec and quote  
- **5 days**: Sample episode in your format  
- **2 weeks**: Pilot batch with its QA report  
- **Weekly**: Delivery cadence once the pilot clears

# Before an episode counts

## What QC checks, every time.

- **Framing**: The task stays in frame; hands and workspace inside the protocol's bounds for the whole episode.  
- **Exposure and focus**: No blown highlights or focus hunting beyond the tolerance set for the room.  
- **Motion**: No camera whip, no dropped frames; IMU continuous where the rig has one.  
- **Sync**: Every stream aligned within one frame; timestamps monotonic across cameras, state and action logs.  
- **Task compliance**: Start and end states match the SOP; no skipped or reordered steps unless the protocol allows it.  
- **Duration**: Inside the episode window for the task family; overruns are trimmed to the protocol, not the clip.  
- **Metadata**: Environment, task, rig, operator id, session and consent reference present and valid on every episode.  
- **Privacy**: No bystander faces, screens or documents without consent; blurred where the protocol flags it.  
- **Calibration**: Intrinsics and extrinsics on file per rig and session for depth, stereo and multi-camera capture.  
- **Labels**: Where annotation is ordered, agreement against the gold set above the threshold before the batch ships.

# Samples and the other lines

## 3 depth episodes to watch and download.

Each with the rig, the room, the task and the file list of the full download.
- [Open the samples](/content/samples/depth/index.html)  
- [All samples and hardware](/content/samples/index.html)  
- **Egocentric**: First-person RGB video of real people doing real tasks.  
- **Teleoperation**: Human-guided robot demonstrations, one episode at a time.  
- **UMI Gripper**: In-the-wild wrist-gripper demos for grasp generalisation.  
- **Hand Pose**: Absolute 3D hand pose ground truth, occlusions included.  
- **Tactile**: Contact and pressure streams for force-aware manipulation.  
- **Bring Your Own Rig**: Your hardware and protocol, our crews, rooms and QA.  
- **Video Annotation**: Segmentation, captions, masks and success labels on Foundry.  
- **Samples and hardware**: Watch and download sample episodes, and see what every line is recorded with.
