Perit Expert Tasks: a task is not data until it has been graded
A task is not data until it has been graded
We build containerised tasks for frontier labs and put every one through six gates before it counts as a datapoint. The reference solution has to work. An agent that does nothing has to fail. The pass rate has to land in a band. And a person has to have read the transcript.
Gates and Stages
- 6 gates: 6 intake stages
- 17 task categories: Each with its own artifact contract, base image and size ceiling rather than one global guess.
- 6 gates before a score counts: Build, oracle, do-nothing, new-seed, difficulty, then a person. Five are automatic.
- 6 intake stages before a byte runs: Checksum, unpack, structure, normalise and lint all complete before stage 6, which is the first that executes anything uploaded.
Seventeen capabilities, measured in containers
Each category carries its own artifact contract, base image and resource ceiling, so a scientific-computing task is not graded by the rules that suit a payroll one.
- Software engineering: Evaluates whether an agent can change a real codebase and keep its tests passing.
- Debugging: Measures whether an agent can localise a fault from a failing run and repair it.
- Security: Evaluates whether an agent finds and fixes a vulnerability without weakening the rest.
- Cybersecurity: Measures a defensive source audit end to end.
- System administration: Measures command-line competence against a machine that must end in a given state.
- Data and infrastructure operations: Evaluates whether an agent can operate running infrastructure rather than describe it.
- Build and dependency management: Measures whether an agent can make a broken build resolve and compile again.
- Data processing: Evaluates transformation of real files into an exactly specified output.
- Scientific computing: Measures numerical work where the answer is checked to a tolerance, not by eye.
- Machine learning: Evaluates whether an agent can train, evaluate and ship a model that clears a bar.
- Data science and reporting: Measures whether an analyst deliverable still holds on data it has never seen.
- Payroll and compliance: Measures arithmetic and rule-following where one wrong cell fails the task.
- Finance and taxation: Measures whether an agent can apply statutory arithmetic that is dated, jurisdictional and held somewhere other than the repository.
- Audio and media data engineering: Evaluates handling of binary media formats against a verifiable artifact.
- Games: Measures planning over long horizons where a score, not a diff, is the outcome.
- Hardware: Evaluates work against physical or device-level constraints.
- Local cloud emulation: Evaluates an agent driving a local cloud-shaped control plane.
The tasks, and the model each one already beat
Each of these was run under gate 5 before it was listed, and carries the pass rate that round measured. Every one opens onto the per-model breakdown behind its figure.
Merge a coding agent's queued messages into one turn: 33% over 6 trials
- Difficulty: Hard Tier, Python
- Models: 2
- Task link
Plan which MCP tools a turn can offer: 50% over 6 trials
- Difficulty: Medium Tier, Python
- Models: 2
- Task link
Rebuild a settlement ledger against withholding mints: 58% over 12 trials
- Difficulty: Medium Tier, Python
- Models: 3
- Task link
... (additional tasks omitted for brevity)
Scoring
A score only counts if the task was hard enough
Difficulty is measured on the model matrix before a task is admitted — not estimated by whoever wrote it. Land outside the band and the task is rejected, however interesting it looks.
One measurement
- An isolated container: Built rootless on a dedicated pool and pinned by digest.
- The agent attempts it: It gets the instruction and working directory.
- A verifier scores it: A separate image runs the checks.
- The trial is kept: Reward, tokens, cost are stored as a row.
Intake
Nothing runs until the sixth stage: An uploaded bundle is a stranger’s code. The first five stages read it, measure it and refuse it without executing a single byte.
In the matrix
Every trial runs an agent against a named model from the registry. A model is eligible only while it is marked ready.
Conclusion
A benchmark a model passes on the first try measured nothing. The useful task is the one that fails honestly — and proving that takes more work than writing it.