Scoring · Perit Expert Tasks

How a trial runs

Four steps, and all four leave something you can open afterwards.

  1. An isolated container
    Built rootless on a dedicated pool and pinned by digest. Network egress is denied by default, so a trial cannot reach anything it was not granted.

  2. The agent attempts it
    It gets the instruction and a working directory. The test suite is not in its image — a submission that copies tests into the agent is refused before it ever runs.

  3. A verifier scores it
    A separate image runs the checks and writes a reward to /logs/verifier/reward.txt. The number comes out of the container; no model judges another model.

  4. The trial is kept
    Reward, tokens, cost, wall time and the whole trajectory are stored as a row. A score you cannot open is a score you cannot check.

pass@k and reward

Two numbers, each with its direction stated, because a reader who has to infer it will sometimes infer it backwards.

The model matrix

12 trials decide whether a task is hard enough — gate-5 policy, read from the platform rather than written into this page. 27 models across 3 providers, read from the registry rather than written into this page. These are the models a difficulty probe runs against today.

The six gates

Five run on their own. The sixth is a person.

  1. It builds
    Agent and verifier images build inside the category timeout and land under the size ceiling. (automatic, no trials)

  2. The answer works
    The reference solution scores reward ≥ 1.0 on all 5 of 5 attempts. One errored trial fails the gate. (5 trials)

  3. Doing nothing fails
    An agent that does nothing scores at most 0. Catches a task the environment already answers. (1 trial)

  4. It is not memorised
    Inputs are regenerated. The oracle still scores 1.0; replaying the old answer still scores 0. (2 trials)

  5. It is hard enough, and not too hard
    A model probe. It must finish before review. The pass rate is recorded and does not block accept. (12 trials)

  6. A person read the transcript
    A reviewer who is not the author opens at least 2 failing trajectories before they may accept. (manual)