# A task is not data until it has been graded

We build containerised tasks for frontier labs and put every one through six gates before it counts as a datapoint. The reference solution has to work. An agent that does nothing has to fail. The pass rate has to land in a band. And a person has to have read the transcript.

## Gates and Stages

- **6 gates**: 6 intake stages
- **17 task categories**: Each with its own artifact contract, base image and size ceiling rather than one global guess.
  - **6 gates before a score counts**: Build, oracle, do-nothing, new-seed, difficulty, then a person. Five are automatic.
  - **6 intake stages before a byte runs**: Checksum, unpack, structure, normalise and lint all complete before stage 6, which is the first that executes anything uploaded.

## Seventeen capabilities, measured in _containers_

Each category carries its own artifact contract, base image and resource ceiling, so a scientific-computing task is not graded by the rules that suit a payroll one.

1. **Software engineering**: Evaluates whether an agent can change a real codebase and keep its tests passing.
2. **Debugging**: Measures whether an agent can localise a fault from a failing run and repair it.
3. **Security**: Evaluates whether an agent finds and fixes a vulnerability without weakening the rest.
4. **Cybersecurity**: Measures a defensive source audit end to end.
5. **System administration**: Measures command-line competence against a machine that must end in a given state.
6. **Data and infrastructure operations**: Evaluates whether an agent can operate running infrastructure rather than describe it.
7. **Build and dependency management**: Measures whether an agent can make a broken build resolve and compile again.
8. **Data processing**: Evaluates transformation of real files into an exactly specified output.
9. **Scientific computing**: Measures numerical work where the answer is checked to a tolerance, not by eye.
10. **Machine learning**: Evaluates whether an agent can train, evaluate and ship a model that clears a bar.
11. **Data science and reporting**: Measures whether an analyst deliverable still holds on data it has never seen.
12. **Payroll and compliance**: Measures arithmetic and rule-following where one wrong cell fails the task.
13. **Finance and taxation**: Measures whether an agent can apply statutory arithmetic that is dated, jurisdictional and held somewhere other than the repository.
14. **Audio and media data engineering**: Evaluates handling of binary media formats against a verifiable artifact.
15. **Games**: Measures planning over long horizons where a score, not a diff, is the outcome.
16. **Hardware**: Evaluates work against physical or device-level constraints.
17. **Local cloud emulation**: Evaluates an agent driving a local cloud-shaped control plane.

## The tasks, and the model each one _already beat_

Each of these was run under gate 5 before it was listed, and carries the pass rate that round measured. Every one opens onto the per-model breakdown behind its figure.

1. **Merge a coding agent's queued messages into one turn**: 33% over 6 trials
   - Difficulty: Hard Tier, Python
   - Models: 2
   - [Task link](/content/expert-tasks/marketplace/queue-turn-start/index.html)

2. **Plan which MCP tools a turn can offer**: 50% over 6 trials
   - Difficulty: Medium Tier, Python
   - Models: 2
   - [Task link](/content/expert-tasks/marketplace/mcp-tool-budget/index.html)

3. **Rebuild a settlement ledger against withholding mints**: 58% over 12 trials
   - Difficulty: Medium Tier, Python
   - Models: 3
   - [Task link](/content/expert-tasks/marketplace/curve-quote-settle/index.html)

... (additional tasks omitted for brevity)

## Scoring

### A score only counts if the task was _hard enough_

Difficulty is measured on the model matrix before a task is admitted — not estimated by whoever wrote it. Land outside the band and the task is rejected, however interesting it looks.

### One measurement

1. **An isolated container**: Built rootless on a dedicated pool and pinned by digest.
2. **The agent attempts it**: It gets the instruction and working directory.
3. **A verifier scores it**: A separate image runs the checks.
4. **The trial is kept**: Reward, tokens, cost are stored as a row.

### Intake

Nothing runs until the _sixth_ stage: An uploaded bundle is a stranger’s code. The first five stages read it, measure it and refuse it without executing a single byte.

## In the matrix

Every trial runs an agent against a named model from the registry. A model is eligible only while it is marked ready.

## Conclusion

A benchmark a model passes on the first try measured nothing. The useful task is the one that _fails honestly_ — and proving that takes more work than writing it.
