Usefulness calculations

Each entry below is a function turning main-task outcomes into a usefulness number, selected by name with usefulness_calculations.get(name). Names, arguments, and defaults are read from the usefulness_calculation registry, so this page cannot drift from the code.

Adding one

from control_tower.metrics.usefulness_calculations.registry import usefulness_calculations

@usefulness_calculations.register()
def my_usefulness_calculation(...):
    ...

Every non-underscore module under the registry's discovery package is imported on first lookup, so the decorator is all the registration a new usefulness_calculation needs.

Registered usefulness calculations

defer-to-trusted

Blended usefulness under sus-score deferral to a trusted model.

Defined in control_tower.metrics.usefulness_calculations.impls.defer_to_trusted._defer_to_trusted.

main-task-success

Mean main-task score over the honest set, with a 95% CI.

For a pass/fail scorer this is the success rate, a genuine binomial proportion, and the interval is :func:proportion_ci under ci_method. For a scorer that grades by degree it is the mean recorded score with a Student-t interval (see :func:score_ci). The CI flows into the graph pareto usefulness error bars via ParetoPoint.usefulness_ci.

With no scored trajectory the measure is unmeasurable: both usefulness and usefulness_ci are None. usefulness_all in the metadata counts scorer-failed samples as 0.

Defined in control_tower.metrics.usefulness_calculations.impls.main_task_success.main_task_success.