MOOT-Bench
Does topology beat hierarchy?
MOOT-Bench is an open question stated as a falsifiable hypothesis — not a leaderboard. It asks whether a gated peer graph of agents can beat a star orchestrator at the same task, on the same model. Below is the method, the metrics, and the task classes where we expect MootDeck to lose.
01Why
Benchmark the wiring, not the model.
Most agent benchmarks measure the model. Swap in a stronger model and the score moves — SWE-bench, Terminal-Bench, and the rest are, at heart, model leaderboards. That is not what MootDeck changes.
MootDeck changes exactly one thing: how agents are wired to each other. Same model, same task, different topology. So the only honest question we can ask is narrow and answerable — hold the model fixed, and see whether the shape of a multi-agent session changes cost, speed, and success.
If MOOT-Bench ever shows MootDeck ahead, it must be because the wiring helped — never because we quietly picked a smarter model for our own arm. The model is the constant. The topology is the variable.
02Topologies
Three ways to wire a team.
The three arms are not three products competing on quality. They are three graph shapes for the same coordination problem, each already common in the wild.
| Arm | Shape | Parallel? | Lateral visibility |
|---|---|---|---|
| Star — Claude Code subagents | Orchestrator spawns isolated children; each returns one result upward. Siblings never talk. | yes | none |
| Serial — Codex CLI | One agent, one context, step after step. No division of labour. | no | n/a |
| Graph — MootDeck ropes | Peers see direct neighbours; ask, check and note run laterally, gated and non-transitive. | yes | gated |
The star already parallelises. The interesting difference is the last column: in a star, a worker that needs a sibling's half-done result cannot get it without routing back through the parent. In a graph, it asks the neighbour directly — if, and only if, you drew the rope.
03Hypothesis
Where a graph should win — and where it shouldn't.
Stated so it can fail:
For tasks that decompose into interdependent subtasks — where a worker needs another worker's partial result — a gated peer graph reaches a passing result with fewer total tokens and less wall-clock than a star orchestrator, at equal model.
For embarrassingly-parallel work (N independent chores) or purely sequential work (one long refactor), the graph should not win, and may cost more — coordination is overhead when there is nothing to coordinate. A hypothesis that can't lose isn't research.
04Metrics
Five numbers, measured per task.
- Tokens to green — total tokens across every agent to a passing result. The efficiency number.
- Wall-clock to green — real elapsed time. If parallelism pays, it pays here or nowhere.
- Rework rate — share of work discarded or duplicated, e.g. two workers solving the same thing because neither could see the other.
- Context load — how much irrelevant context each worker carries. A direct proxy for what gating is supposed to buy.
- Human touches — interventions needed before completion. Autonomy is a cost too.
Everything is reported at a fixed budget cap, so a run that never finishes counts as a loss rather than an infinite score.
05Tasks
A small, honest task matrix.
The task set is chosen up front and includes cases we expect to lose. Predicting the winner before the run is the whole point.
| Task class | Predicted winner | Why |
|---|---|---|
| Contract-coupled build — frontend needs the backend's live API shape | Graph | Workers must see each other's partial output; routing through a parent adds a round-trip. |
| Specialist pull-in — ask a DB expert mid-task | Graph | A lateral ask beats re-spawning and re-briefing a child. |
| Fan-out review — one change, N reviewers | Tie | Parallel but independent; a star already handles this cleanly. |
| Wide independent chores | Star | Clean context isolation, zero cross-talk — the graph's channel is dead weight. |
| Long linear refactor | Serial | Nothing to coordinate; every extra agent is overhead. |
06Method
The rules that keep it honest.
- Same model and version in every arm. No cherry-picking a stronger model for our own.
- Same repo snapshot, same prompts, same budget cap across all three shapes.
- Report ties and losses as prominently as wins. A benchmark that only publishes its wins is marketing.
- Open harness and task definitions, so any result is reproducible by someone who does not trust us.
- Null-egress by default — any run data is opt-in and local-first, the same promise the product makes.
07Status
No numbers yet. That's on purpose.
This page is the method, published before the results. A benchmark you design after you already know who wins is not a benchmark — so the hypotheses, metrics and task matrix are on the record first, and the leaderboard waits until the harness is open and the runs reproduce.
The rest of this site is built on saying the awkward thing plainly. Posting invented numbers here would undo all of it. When there is a result, it will arrive with the harness that produced it.
Which task classes belong in the first run? Where have you watched lateral coordination pay off — or fall flat? That is exactly the evidence MOOT-Bench needs. Bring it to Discord.
NEXTKeep reading