Before scaling agent evaluation, establish what good output means. This small experiment explores how to turn that quality bar into checks and test whether judges apply it consistently.
The example uses a public earnings-call skill to make the decisions concrete: what must be sourced, which gaps are acceptable, and when a summary should fail. MLflow provides the setup; the approach can be used with other evaluation tools.
What should the stock research skill produce?
A source-linked earnings-call brief: what changed, key financial metrics, guidance comparisons, management explanations, Q&A themes, and missing evidence. For this example, the brief compares four completed fiscal calls.
For this example, each trace links the company and target calls, retrieved transcripts and official releases, the skill version, and the saved summary. The evaluation harness records its checks separately.
A guidance comparison should preserve the metric, fiscal period, accounting basis and range. A narrowed range is not automatically a raise. Missing calls or unsupported comparisons should be visible gaps.
1. Set up the experiment
Run the research skill and capture its outputs and evidence as traces in MLflow. Record the skill, rubric, model and judge versions with the experiment.
1 Runs are traced. 2 The job scores new traces once a day. 3 Labels and traces are read for alignment. 4 Judges call the model. 5 The gate reads the golden dataset.
2. Turn the quality bar into checks
The owner of the output standard agrees the rubric with reviewers. The left loop uses it to improve the skill; the right branch tests whether judges apply it consistently. Reviewer disagreements expose decisions the rubric still needs. Select a node to inspect either path.
Method and reading guide
Freeze the review criteria before comparing skill versions. Use held-out examples to compare judge candidates. The alignment branch is the focus of this setup experiment.
3. Use earnings-summary failures to define the rubric
The earnings-call summary and its execution trace answer different questions. Start by checking both: was the note supported, and did the required review actually run? This first pass turns a missing QC result into an explicit output requirement.
Method and reading guide
The walkthrough starts with 30 runs, groups their failure reasons and changes the skill instructions. Select a step or use Back / Next. Treat the displayed reasons as illustrative until original traces are supplied.
Inspect the seven failure modes
Failure modes found in the 30 baseline runs
Each mode is a group of judge reasons that describe the same problem. Select a mode to see the runs and the words the judge used, and which scorer catches it today.
4. Trace the experiment, step by step
Follow the skill changes, judge alignment and monitoring as separate lanes. Select a step or use Back / Next to see what changed and which version was used.
Method and reading guide
The top lane changes the skill; the middle lane changes the judges. The bands record their versions. Live scoring is a later extension of the offline setup.
5. Compare skill outputs with fixed scorers
Hold the evaluator fixed while comparing the research skill. Select a version for its changes or a score for its runs. Improvement is mixed: figure provenance drops from 53/60 in skill v5 to 47/60 in v6.
Method and reading guide
Each version has 60 recorded runs: 20 company call bundles × 3. The table uses the same v2.1 scorer set. Counts are fixture results, with illustrative reasons; they are not independently verified exports.
Selected version or score details
6. Replay a retrospective regression baseline
The walkthrough sets a floor from the accepted v6 results, then replays earlier versions. That makes this a retrospective regression baseline, not an independent test proving v6 is ready to deploy.
Method and reading guide
Select a version to replay the recorded checks. A reproducible gate must also enforce stock coverage, the expected run count and complete scores. The demo replay does not enforce all of those conditions.
| Check | Rule | If it fails |
|---|---|---|
| 5 process checks: skill invocation, approved tools and sites, required flow, and file saving | Every run passes | FAIL |
Fixed in v4 to v6: qc_result_present, qc_not_in_note, sections_complete, inline_citations | Every run passes, so the fix can't come back | FAIL |
Still open: figure_provenance, internal_consistency | At least 47 of 60 runs pass (the v6 level) | FAIL |
| Coverage | At least one run for each golden stock | FAIL |
7. Compare judge alignment candidates
Compare three candidates on the same held-out traces: the original judge, raw MemAlign and a separate judge containing manually approved rules. Here the reference labels are synthetic, so agreement does not establish expert validity.
Method and reading guide
The setup uses 40 alignment traces and 20 held-out traces. Four judges × 20 traces gives 80 verdicts. Keep related runs of the same stock on one side of the split, and freeze labels before tuning.
Agreement with synthetic labels, held-out traces
The walkthrough selects the reviewed-rule judge even though raw MemAlign has higher agreement with the synthetic labels. That is a policy choice: some learned rules were overruled. If that changes the rubric, revise the labels and retest on untouched examples.
What changed in the judges
MLflow setup and evidence to retain
Judge assessments and reference feedback must share the judge's assessment name on the same trace. Record verdicts and reasons, and keep synthetic labels distinguishable from human review. MemAlign learns guidelines and retrieves examples; it is currently documented as experimental.
Keep the output contract, trace split, rubric and labels, candidate judge versions, and per-criterion held-out results. Five queue questions on 60 traces produce 300 labels; only four judge criteria enter the 80-verdict comparison. Eighty verdicts on twenty traces are not eighty independent company examples.
8. Extend the setup to monitoring
After offline validation, the same checks can monitor fresh traces. This diagram shows the walkthrough's Databricks-specific extension; a daily managed job is not required to test MLflow judge alignment.
Method and reading guide
Keep monitoring inputs consistent with offline evaluation. A note-only judge cannot verify source facts it has not been given.
All 11 live scorers and sampling rates
9. Milestones
required_flow, judges v2.1, +2 rules.10. Known limits
- Sizes follow the recommended minimums: 30 runs for error analysis, 20 inputs × 3 runs per version, 60 labelled traces (20 held out). Go larger for higher-stakes agents.
- v6 still has 2 open misses: company-reported (non-GAAP) figures in sections meant for release figures, and a guidance change explained two ways when release and transcript figures disagree.
11. Recommended next steps
The technical direction to test is a shared, versioned quality bar: agree the output contract and review criteria, make policy decisions explicit, then validate judges against reviewer labels before relying on their scores. A person owns changes to the standard. This illustrative experiment shows how to explore that direction; real reviewer evidence is still needed to adopt it.
- Watch live scores weekly. Triage failures; send borderline runs to the SME queue.
- Validate the gate implementation. Enforce coverage, complete scores and the expected run count before using it for an acceptance decision. Predefine the next quality bar rather than deriving it from the candidate.
- Re-align the judges after about 50 new SME labels. Swap only if held-out agreement improves.
- Optional v7: state any release vs transcript (GAAP vs non-GAAP) difference once, in Guidance, labelled.
- For a real rollout: real SMEs, real golden facts, batch runs in the runner.
Rules in force: no change to live scorers or the job without the owner's OK. Other teams' scorers are never touched.