Method — Perit
The six rules
Every benchmark on this site follows these.
01 Held-out audio stays held out
The scored set is never published. A 60-clip development split per sector is open so anyone can calibrate against the same conventions without seeing the test.02 Two passes, then adjudication
Every reference transcript and every rubric score is produced twice, independently. A senior reviewer resolves disagreements and the resolution is recorded with the item.03 Runs, not a run
A single pass hides how much a score moves. Intervals are bootstrapped at 95% and printed next to the headline number wherever a run produces them.04 A human control on every board
Someone who does that job takes the same set, blind and timed, at market rate. It is a working standard, not a ceiling — the human line carries its own error bar.05 Rubrics come from operators
Each criterion is written by a practitioner and then rewritten until two strangers grading the same call land on the same score.06 Provenance travels with the artifact
Annotator, reviewer and adjudicator are attached to every item, so a disputed score can be traced to the people who set it.
Disclosure
What we publish, and what we will not.
The line is drawn so the boards stay useful without the test set leaking.
We publish
- Leaderboards with confidence intervals, per sector and overall
- The development split, and the conventions the references follow
- The metric definitions, including what counts as an entity error
- Notes on what failed and why, when a run is complete
We do not
- The held-out audio, or transcripts of it
- Contributor and reviewer identities
- Customer-commissioned sets, ever — those belong to the customer
- Scores for a named model we have not run ourselves
How a run is executed
Two tracks, one discipline.
The same rules, applied to what a model heard and to when it heard it.
PERIT-STT
Running speech-to-text.
Scored on word error rate — errors per hundred words against a double-passed human transcript, adjudicated by a third reviewer on disagreement.
- 01 Audio is held out and never published. A 60-clip development split per sector is open for calibration.
- 02 Reference transcripts are written by two annotators independently; a senior reviewer adjudicates every disagreement.
- 03 Each clip is run 3 times per model. Reported intervals are bootstrapped at 95%.
- 04 Human control is a working transcriptionist from the same sector, timed and paid at market rate.
PERIT-ALIGN
Running word alignment.
Scored on timing accuracy @100 ms — share of words whose start and end both land within 100 ms of the reference timing.
- 01 Every model gets the same audio. Speech-to-text models return word timestamps with their transcript; forced aligners place a known transcript in time.
- 02 Returned words are matched to the reference words. Timing is scored on matched words, and coverage reports how many matched.
- 03 Each boundary is compared on its own — start against start, end against end — and reported at 20, 50, 100 and 200 ms.
- 04 Ranked by timing accuracy at 100 ms; ties are broken by mean boundary error.
- 05 Latency is measured under concurrent load and is indicative only.
The protocol comes first.
It is published in full so the format can be argued with before any score is read, not after it flatters someone. Held-out sets stay held out; the development split is open to every entrant.