Perit Expert Tasks: a task is not data until it has been graded

A task is not data until it has been graded

We build containerised tasks for frontier labs and put every one through six gates before it counts as a datapoint. The reference solution has to work. An agent that does nothing has to fail. The pass rate has to land in a band. And a person has to have read the transcript.

Gates and Stages

Seventeen capabilities, measured in containers

Each category carries its own artifact contract, base image and resource ceiling, so a scientific-computing task is not graded by the rules that suit a payroll one.

  1. Software engineering: Evaluates whether an agent can change a real codebase and keep its tests passing.
  2. Debugging: Measures whether an agent can localise a fault from a failing run and repair it.
  3. Security: Evaluates whether an agent finds and fixes a vulnerability without weakening the rest.
  4. Cybersecurity: Measures a defensive source audit end to end.
  5. System administration: Measures command-line competence against a machine that must end in a given state.
  6. Data and infrastructure operations: Evaluates whether an agent can operate running infrastructure rather than describe it.
  7. Build and dependency management: Measures whether an agent can make a broken build resolve and compile again.
  8. Data processing: Evaluates transformation of real files into an exactly specified output.
  9. Scientific computing: Measures numerical work where the answer is checked to a tolerance, not by eye.
  10. Machine learning: Evaluates whether an agent can train, evaluate and ship a model that clears a bar.
  11. Data science and reporting: Measures whether an analyst deliverable still holds on data it has never seen.
  12. Payroll and compliance: Measures arithmetic and rule-following where one wrong cell fails the task.
  13. Finance and taxation: Measures whether an agent can apply statutory arithmetic that is dated, jurisdictional and held somewhere other than the repository.
  14. Audio and media data engineering: Evaluates handling of binary media formats against a verifiable artifact.
  15. Games: Measures planning over long horizons where a score, not a diff, is the outcome.
  16. Hardware: Evaluates work against physical or device-level constraints.
  17. Local cloud emulation: Evaluates an agent driving a local cloud-shaped control plane.

The tasks, and the model each one already beat

Each of these was run under gate 5 before it was listed, and carries the pass rate that round measured. Every one opens onto the per-model breakdown behind its figure.

  1. Merge a coding agent's queued messages into one turn: 33% over 6 trials

    • Difficulty: Hard Tier, Python
    • Models: 2
    • Task link
  2. Plan which MCP tools a turn can offer: 50% over 6 trials

    • Difficulty: Medium Tier, Python
    • Models: 2
    • Task link
  3. Rebuild a settlement ledger against withholding mints: 58% over 12 trials

    • Difficulty: Medium Tier, Python
    • Models: 3
    • Task link

... (additional tasks omitted for brevity)

Scoring

A score only counts if the task was hard enough

Difficulty is measured on the model matrix before a task is admitted — not estimated by whoever wrote it. Land outside the band and the task is rejected, however interesting it looks.

One measurement

  1. An isolated container: Built rootless on a dedicated pool and pinned by digest.
  2. The agent attempts it: It gets the instruction and working directory.
  3. A verifier scores it: A separate image runs the checks.
  4. The trial is kept: Reward, tokens, cost are stored as a row.

Intake

Nothing runs until the sixth stage: An uploaded bundle is a stranger’s code. The first five stages read it, measure it and refuse it without executing a single byte.

In the matrix

Every trial runs an agent against a named model from the registry. A model is eligible only while it is marked ready.

Conclusion

A benchmark a model passes on the first try measured nothing. The useful task is the one that fails honestly — and proving that takes more work than writing it.