Skip to content

Evaluation

An evaluation scores a flow against a fixed dataset of inputs and expected outcomes, so a change can be compared with the version before it. It answers "did this prompt edit make the product better?" with evidence instead of with the impression left by a handful of manual tries.

What it means

An evaluation needs three things: a dataset that represents real traffic, a scorer that turns an output into a number, and a run that is reproducible. Reproducibility is the hard one — a scoring path that depends on the current time, or on live external state, produces a different verdict on Tuesday.

How SwiftTune uses it

Evaluations replay recorded runs through the flow under test, so no analysis path reads an uncontrolled clock and the same dataset yields the same score twice. Results are attributed per flow version, which is what lets a regression be traced to the edit that caused it rather than to the release as a whole.

Primary sources