Incident Investigator

/Overview

When production breaks, the first plausible explanation is usually the one that sticks. This project asks a narrow question and tries to answer it honestly: does splitting root-cause analysis across specialist agents actually beat one capable agent with all the tools, and at what price?

Both systems share everything except their orchestration: the same tool harness (with simulated faults), budgets, report schema and domain guidance. They run against 16 seeded incidents. Scoring uses answer keys held out from the agent code, deterministic evidence checks and an LLM judge.

/Architecture

1. PlannerLLM

   Sees topology, every recent change and each specialist's toolset; proposes hypotheses and tasks with dependencies

2. Orchestratorcode

   Deterministic manager and the only writer of shared state: applies outcomes, budgets, routing

3. || Send(task) · specialists in parallel

   Observability

   metrics, logs, traces, infra events

   Dependency

   topology, blast radius, health

   Change history

   always opens with an unfiltered change sweep

4. Reasoningo4-mini

   Ranks competing hypotheses against supporting and contradicting evidence

5. Critico4-mini

   accept · need more evidence · inconclusive (must name the blocking gap)

6. Evidence gatecode

   Top hypotheses need evidence and every specialist must have run

7. Reportout

   Ranked causes, cited facts vs assumptions, confidence, corrective actions

↺ The critic can send targeted tasks back through the orchestrator (max 3 rounds). Hitting any budget triggers one closing pass instead of a blind stop.

/Key decisions

• One writer. An LLM planner proposes; a deterministic manager owns shared state. Each agent returns one typed Outcome, applied in task order, so parallel fan-out via Send is race-free.
• Grounded by construction. Every tool call gets a call_id. Evidence that cites a call the agent didn't make is dropped, and reports separate cited facts from assumptions.
• Structure where models are unreliable, judgment where it matters. Code guarantees coverage: every specialist runs, the unfiltered change sweep always happens, and an evidence gate blocks the report until the top hypotheses have support or contradiction. Only the critic, an LLM, decides when the evidence is enough.
• Every loop is bounded. Critic rounds, tool calls, tokens, wall clock, per-task calls, retries and worker attempts all have limits. A failing tool becomes a recorded gap, not a crash.
• A fair fight. The baseline gets the same harness, tools, faults, budgets, metering and guidance. A test keeps the answer keys isolated from agent code, and key evidence only counts if the returned data actually contains it.

/Results

Metric	Multi-agent	Single agent
Correct conclusive vs inconclusive call	16/16	16/16
Root cause fully correct	13/16 (+2 partial)	15/16 (+1 partial)
Premature-closure traps solved	3/3	2/3
Claims citing an executed tool call	1.00	0.99
Red herrings adopted / forbidden actions	0 / 0	0 / 0
Key evidence retrieved and cited	0.88	0.93
Tool calls / tokens per run	29 / 152K	19 / 128K
Latency / cost per run	118 s / $0.073	55 s / $0.010

/What I learned

• Multi-agent isn't automatically more accurate. On straightforward incidents the single agent actually named more root causes fully (15 vs 13). The multi-agent system's wins are coverage and discipline: it never closes early, it grounds every claim, and it stays on track when one agent's context would fill up.
• The critic's model matters most. With a small model as critic, the system declared correct investigations inconclusive. A reasoning model in the reasoner and critic seats removed those errors, while cheap models do the gathering.
• Context is the real ceiling. In a matched-model check on three hard cases, the single o3 agent hit the 400K-token budget on 2 of 3 runs. The multi-agent system used fewer tokens (234K vs 346K) and cited all key evidence (1.00 vs 0.61).
• The price is real. About 7× the cost and 2× the latency. That's the trade, stated plainly.