Study design and proposed evaluation protocol

1. Rationale

There is a practical difference between giving an agent access to multiple models and coordinating multiple agents. Routing may lower cost or improve task-model fit. Coordination can help separate subtasks, check decisions, and provide redundancy; it can also consume time and introduce new failure points. Treating those two features as one intervention would prevent a useful interpretation of the results.

2. Comparison conditions

Arm	Planned configuration
A	One agent using a fixed, prespecified model.
B	One agent with a routing policy that can select among prespecified models.
C	Several agents, heterogeneous model routing, and a documented delegation policy.
D	Condition C with defined human approval gates and intervention logs.

The same task prompts, available tools, maximum budget, timeout policy, and evaluation rubric should apply to all conditions unless a deviation is explicitly justified. Separate the effect of routing from the effect of additional agents. If a condition receives more total compute, label the comparison accordingly rather than implying matched resources.

3. Tasks and allocation

Start with a small pilot composed of tasks that vary in decomposability, ambiguity, and external-state dependence. Candidate families include repository maintenance, constrained bug fixes, documentation consistency, and structured operational planning. Use licensed or locally authored tasks; do not expose confidential repositories, private employment information, student records, or API keys.

Before data collection, define a task manifest, difficulty strata, a randomization or matched-pair plan, exclusion criteria, maximum retries, and a stopping rule. Pilot runs should be used to repair the harness, not silently pooled with the confirmatory dataset.

4. Outcome definitions

Outcome	Proposed recording procedure
Task success	Binary or graded rubric independently applied to the final artifact; report denominators and disagreements.
Total cost	Model, tool, execution and human-review expenditure including failed attempts and retries.
Latency	Elapsed time from released task to verified output; separately report waiting and review delays.
Reliability	Crash, timeout, policy block, unsupported claim, recovery, and hidden-state mismatch categories.
Oversight	Count and severity of human corrections, actions blocked, review burden, and remaining errors.

Primary comparisons should use verified task success under a matched budget. The eventual statistical procedure must account for repeated tasks and paired observations. Report descriptive counts even when sample sizes are too small for inference.

5. Evidence and reproducibility

A valid run should preserve its task ID; condition; configuration digest; code revision; runtime and model versions; cost record; redacted event log; final artifact; rubric; evaluator; and completion state. The outcome evaluator should not rely solely on an agent's own report.

Available scaffolding: study-spec.json, scoring.mjs, and synthetic-fixture tests. The scorer calculates descriptive statistics from input supplied to it. It cannot independently verify that the supplied “verified” flag is correct, and no research findings are bundled.

6. Analysis plan

Predefine the primary comparison and any secondary comparisons before examining confirmatory outcomes. Analyze task-matched differences in completion and the distribution of total cost. Present uncertainty intervals where appropriate, individual failure classes, and sensitivity to task difficulty and budget constraints. If too few runs are completed, describe the findings as exploratory.

7. Validity threats

Provider drift, nondeterminism, prompt leakage between runs, unequal tool permissions, varying model context limits, human-evaluator disagreement, and task selection bias could change the apparent result. The project must also consider whether extra coordination improves measured success only by making more attempts or spending more resources. Failing to count unsuccessful trials would invalidate cost-per-success comparisons.

8. Next experiments

1. Freeze the benchmark task manifest and evaluator rubric.
2. Verify condition-specific execution and recording against synthetic tasks.
3. Run a pilot with complete traces and independent scoring.
4. Revise and freeze a final protocol before confirmatory runs.
5. Only then publish comparative results, including unsuccessful and inconclusive trials.

For the broader research question, return to the working note. For implementation roles and provenance, see field notes.