Data annotation — Perit

Two crafts, both running today

Transcription Overview

Transcription

Word Alignment

Data Types

Any data type, one calibration gate.

The guide, the training module and the test are the same shape whatever the file is. Every project gets a gold set, a calibration pass before paid work, and QA sampling scored against the guide.

Audio

3,000+ hours transcribed · 1,000+ hours aligned · 95%+ QA-sample accuracy

Image

Gold set per project · agreement and rework rate reported per batch

Video

Episode-level labels · reviewers stream the clip, never download it See Physical Intelligence

Text

Practitioner graders · rubric written with an operator from the sector

Before anyone is paid

Nobody starts on production work.

Every annotator clears the same gate, on every project, however long they have been on the bench.

  1. Read the guide
    One written convention per project — casing, tags, how a number is written out, what counts as an edge case. Versioned; the version ships with the batch.
  2. Training module
    About thirty samples with the answers visible, so the guide stops being abstract before anyone is paid for it.
  3. Testing module
    A held-back set, graded against the guide. No pass, no production access — the test can be retaken after re-reading, not after arguing.
  4. Production, sampled
    Work goes out with QA sampling behind it. Below threshold routes back to calibration, and rejected work is not billed to the customer.

What arrives in your bucket.

How a rubric gets built.

A rubric is not a list of nice qualities.

  1. Capture the standard
    We sit with an operator and write down what they check before they call a call handled. That list is the rubric's first draft.
  2. Make it applicable
    Every criterion is rewritten until two strangers grading the same call land on the same score.
  3. Build the verifier
    The mechanical criteria become a function returning a score in [0,1]. Judgment criteria stay with people, and we say which is which.
  4. Ship with the quality report
    Nothing is delivered without the numbers it was produced at: QA-sample accuracy, WER against the adjudicated reference, inter-annotator agreement, and what was reworked.

Sample rubric · Banking Card dispute · caller reports two charges they do not recognise

Two strangers, same call, same score — or the criterion gets rewritten. Agreement on every criterion is measured and reported per batch.

What keeps the number honest.

  1. A guide before a keystroke
    Every project starts from one written convention, agreed with the customer and versioned. Disagreement about what a tag means is a guide bug, and it gets fixed in the guide.
  2. Calibrate, then produce
    Training on about 30 samples, then a held-back test graded against the guide. Nobody reaches paid production work without clearing it.
  3. Three measures, all reported
    Delivered batches carry accuracy against the guide on a QA sample, WER against an adjudicated reference, and inter-annotator agreement on the double-passed portion. All three ship with the batch, whether they flatter us or not.
  4. Agreement is a schema signal
    When IAA drops on a task, we treat it as the guide being ambiguous before we treat it as an annotator being wrong. Agreement is measured per batch and per reviewer, so drift shows up as a trend rather than a surprise.
  5. Rework is ours, not yours
    Work below threshold is redone before delivery and is not billed. The rework rate goes in the batch report because hiding it would make the accuracy number meaningless.
  6. Benchmark references are double-passed
    Customer delivery is QA-sampled. Benchmark reference transcripts are stricter: written twice, independently, with a senior reviewer adjudicating every disagreement — nothing else is defensible as a scoring baseline.

The bench

Who annotates, and what they had to clear.

Nobody touches production data before passing a calibration set. Reviewers are re-checked against that same set every week — graders drift too, and measuring it is the only way to catch it.

Role Bar to enter What they do
Annotator Sector experience, passes a calibration set Transcription, entity spans, turn boundaries, preference judgments
Reviewer Sustained QA-sample accuracy above threshold Second pass, adjudication on disagreement, agreement reporting, batch sign-off
Rubric author Practising operator — the person who would sign off Writes what a good answer is, in criteria a stranger can apply
Calibration lead Reviewer who has held threshold while running a guide revision Weekly reviewer checks, gold-set maintenance, guide revisions

Questions

Before you send a batch.

Send a sample, get it back labelled.

Ten minutes of audio, a hundred images or a page of text, and your guide if you have one. It comes back annotated within five working days with the agreement report, so you can judge the bench on your own data before anything is scoped.