How accurate can complexity estimation get?

Starting point

Yesod estimates complexity only after a Note has been grounded into an executable plan. The current scale—Easy, Medium, or Advanced—is intentionally coarse. It is a routing signal, not a promise about duration or cost.

The open question is whether that signal can become reliably calibrated. Can the Note, acceptance criteria, Bead tree, affected code, and repository history predict how difficult execution will actually be? And can the estimate become accurate enough to change which model Yesod chooses without increasing retries or review burden?

What accuracy means

• Calibration — work labeled Easy should usually finish like other Easy work, not merely sound easy to an estimator.
• Discrimination — the estimator should separate tasks that need stronger reasoning from tasks a cheaper model can complete reliably.
• Stability — independent estimators should reach compatible conclusions from the same grounded plan.
• Decision value — using the estimate should improve cost, latency, or reliability compared with a simple default routing policy.

What to track

• The estimate and confidence recorded before dispatch.
• Task kind, repository, plan shape, dependency depth, coupling, ambiguity, and blast radius.
• The selected model and execution environment.
• Tokens, wall time, retries, rework, gate failures, review burden, and final outcome.
• False-low estimates that under-provision work versus false-high estimates that waste expensive capacity.
• How calibration changes by repository, task type, model family, and planner quality.

Experiments

Freeze every estimate before execution, then compare it with the complete run record. Score the same task corpus with rules, inexpensive models, and frontier models. Test whether repository-specific history improves calibration over a universal rubric. Most importantly, separate intrinsic task complexity from a weak plan, an unsuitable model, or a broken environment—four causes that can produce the same expensive run but require different corrections.

Working hypothesis

A three-tier estimate can become useful enough for routing before it becomes precise enough for forecasting. The highest value may come from knowing which work is unsafe to under-estimate, while learning when apparently complex work has been planned well enough to move down the cost curve.

Living record

Estimator versions, calibration reports, routing experiments, and cases where predicted and observed difficulty diverge belong here.