## The six rules

### Every benchmark on this site follows these.

1. **01 Held-out audio stays held out**  
   The scored set is never published. A 60-clip development split per sector is open so anyone can calibrate against the same conventions without seeing the test.

2. **02 Two passes, then adjudication**  
   Every reference transcript and every rubric score is produced twice, independently. A senior reviewer resolves disagreements and the resolution is recorded with the item.

3. **03 Runs, not a run**  
   A single pass hides how much a score moves. Intervals are bootstrapped at 95% and printed next to the headline number wherever a run produces them.

4. **04 A human control on every board**  
   Someone who does that job takes the same set, blind and timed, at market rate. It is a working standard, not a ceiling — the human line carries its own error bar.

5. **05 Rubrics come from operators**  
   Each criterion is written by a practitioner and then rewritten until two strangers grading the same call land on the same score.

6. **06 Provenance travels with the artifact**  
   Annotator, reviewer and adjudicator are attached to every item, so a disputed score can be traced to the people who set it.

## Disclosure

### What we publish, and what we will not.

The line is drawn so the boards stay useful without the test set leaking.

**We publish**  
- Leaderboards with confidence intervals, per sector and overall  
- The development split, and the conventions the references follow  
- The metric definitions, including what counts as an entity error  
- Notes on what failed and why, when a run is complete

**We do not**  
- The held-out audio, or transcripts of it  
- Contributor and reviewer identities  
- Customer-commissioned sets, ever — those belong to the customer  
- Scores for a named model we have not run ourselves

## How a run is executed

### Two tracks, one discipline.

The same rules, applied to what a model heard and to when it heard it.

### PERIT-STT

#### Running speech-to-text.

Scored on word error rate — errors per hundred words against a double-passed human transcript, adjudicated by a third reviewer on disagreement.

1. **01 Audio is held out and never published.** A 60-clip development split per sector is open for calibration.
2. **02 Reference transcripts are written by two annotators independently;** a senior reviewer adjudicates every disagreement.
3. **03 Each clip is run 3 times per model.** Reported intervals are bootstrapped at 95%.
4. **04 Human control is a working transcriptionist from the same sector, timed and paid at market rate.**

[See the speech-to-text boards](/content/research/benchmark/speech-to-text/index.html)

### PERIT-ALIGN

#### Running word alignment.

Scored on timing accuracy @100 ms — share of words whose start and end both land within 100 ms of the reference timing.

1. **01 Every model gets the same audio.** Speech-to-text models return word timestamps with their transcript; forced aligners place a known transcript in time.
2. **02 Returned words are matched to the reference words.** Timing is scored on matched words, and coverage reports how many matched.
3. **03 Each boundary is compared on its own** — start against start, end against end — and reported at 20, 50, 100 and 200 ms.
4. **04 Ranked by timing accuracy at 100 ms;** ties are broken by mean boundary error.
5. **05 Latency is measured under concurrent load and is indicative only.**

[See the word alignment boards](/content/research/benchmark/word-alignment/index.html)

## The protocol comes first.

It is published in full so the format can be argued with before any score is read, not after it flatters someone. Held-out sets stay held out; the development split is open to every entrant.

[Evaluate your model](/content/contact?topic=Evaluate%20my%20model/index.html) [Request the development split](/content/contact?topic=Development%20split%20request/index.html)
