How I Structure an Agent Evaluation Note

Start with the estimand

Before choosing a metric, state what the experiment is meant to estimate. For example, an average over all events can answer a different question from an average that gives each agent equal weight.

Keep controls structurally relevant

A useful control preserves the properties that could explain the observed gain. In pseudocode:

for seed in paired_seeds:
    baseline = evaluate(policy="reference", seed=seed)
    candidate = evaluate(policy="candidate", seed=seed)
    paired_differences.append(candidate - baseline)

Report what the evidence does not establish

An evaluation note should distinguish a measured property from broader claims. A replay check may establish deterministic reconstruction under a defined journal contract; it does not by itself establish fairness or human realism.