Method — Perit

The six rules

Every benchmark on this site follows these.

  1. 01 Held-out audio stays held out
    The scored set is never published. A 60-clip development split per sector is open so anyone can calibrate against the same conventions without seeing the test.

  2. 02 Two passes, then adjudication
    Every reference transcript and every rubric score is produced twice, independently. A senior reviewer resolves disagreements and the resolution is recorded with the item.

  3. 03 Runs, not a run
    A single pass hides how much a score moves. Intervals are bootstrapped at 95% and printed next to the headline number wherever a run produces them.

  4. 04 A human control on every board
    Someone who does that job takes the same set, blind and timed, at market rate. It is a working standard, not a ceiling — the human line carries its own error bar.

  5. 05 Rubrics come from operators
    Each criterion is written by a practitioner and then rewritten until two strangers grading the same call land on the same score.

  6. 06 Provenance travels with the artifact
    Annotator, reviewer and adjudicator are attached to every item, so a disputed score can be traced to the people who set it.

Disclosure

What we publish, and what we will not.

The line is drawn so the boards stay useful without the test set leaking.

We publish

We do not

How a run is executed

Two tracks, one discipline.

The same rules, applied to what a model heard and to when it heard it.

PERIT-STT

Running speech-to-text.

Scored on word error rate — errors per hundred words against a double-passed human transcript, adjudicated by a third reviewer on disagreement.

  1. 01 Audio is held out and never published. A 60-clip development split per sector is open for calibration.
  2. 02 Reference transcripts are written by two annotators independently; a senior reviewer adjudicates every disagreement.
  3. 03 Each clip is run 3 times per model. Reported intervals are bootstrapped at 95%.
  4. 04 Human control is a working transcriptionist from the same sector, timed and paid at market rate.

See the speech-to-text boards

PERIT-ALIGN

Running word alignment.

Scored on timing accuracy @100 ms — share of words whose start and end both land within 100 ms of the reference timing.

  1. 01 Every model gets the same audio. Speech-to-text models return word timestamps with their transcript; forced aligners place a known transcript in time.
  2. 02 Returned words are matched to the reference words. Timing is scored on matched words, and coverage reports how many matched.
  3. 03 Each boundary is compared on its own — start against start, end against end — and reported at 20, 50, 100 and 200 ms.
  4. 04 Ranked by timing accuracy at 100 ms; ties are broken by mean boundary error.
  5. 05 Latency is measured under concurrent load and is indicative only.

See the word alignment boards

The protocol comes first.

It is published in full so the format can be argued with before any score is read, not after it flatters someone. Held-out sets stay held out; the development split is open to every entrant.

Evaluate your model Request the development split