Before scaling agent evaluation, establish what good output means. This small experiment explores how to turn that quality bar into checks and test whether judges apply it consistently.

The example uses a public earnings-call skill to make the decisions concrete: what must be sourced, which gaps are acceptable, and when a summary should fail. MLflow provides the setup; the approach can be used with other evaluation tools.

What should the stock research skill produce?

A source-linked earnings-call brief: what changed, key financial metrics, guidance comparisons, management explanations, Q&A themes, and missing evidence. For this example, the brief compares four completed fiscal calls.

For this example, each trace links the company and target calls, retrieved transcripts and official releases, the skill version, and the saved summary. The evaluation harness records its checks separately.

A guidance comparison should preserve the metric, fiscal period, accounting basis and range. A narrowed range is not automatically a raise. Missing calls or unsupported comparisons should be visible gaps.

Explore the experiment

1. Set up the experiment

Run the research skill and capture its outputs and evidence as traces in MLflow. Record the skill, rubric, model and judge versions with the experiment.

Agent runner earnings-call-summary -v3 (lap 1) -v4 -v5 -v6 (accepted) 20 company call bundles × 3 runs per version Your tracing platform Traces Tool calls, inputs and outputs Scoring job (daily) 7 rules at 100%, 4 judges at 10% SME review queue 5 questions, 60 traces Golden test dataset earnings_summary_golden_test 20 records: facts + regression gate Judge model: claude-sonnet-5-5 used by the 4 LLM judges, MemAlign and fact_coverage Local evaluation Evaluation code Find and score runs Align judges Run the quality gate 1 2 3 4 5

1 Runs are traced. 2 The job scores new traces once a day. 3 Labels and traces are read for alignment. 4 Judges call the model. 5 The gate reads the golden dataset.

2. Turn the quality bar into checks

The owner of the output standard agrees the rubric with reviewers. The left loop uses it to improve the skill; the right branch tests whether judges apply it consistently. Reviewer disagreements expose decisions the rubric still needs. Select a node to inspect either path.

Method and reading guide

Freeze the review criteria before comparing skill versions. Use held-out examples to compare judge candidates. The alignment branch is the focus of this setup experiment.

Automated improvement loop repeat until the quality bar is met 1. Find errorsread runs, group failure modesL1 · B1 · B2 2. Define checksrules + one judge per criterionL4 · B3 3. Fix the agentsmallest change, new versionL2 · B4 4. Re-run + scoresame inputs, same checksL3 · B5 · B6 5. Quality gatebar met? lock the fixesB7 6. Ship + live scoringrules on all, judges sampledP1 · P2 · P3 not met:failing reasons arethe next errors met new live traces Judge alignment experiment synthetic reference labels in this demo a. Reference labelsone answer per judgeA1 b. MemAlignlearns guidelines from labelsA2 c. Review the ruleskeep, conflict, move, dropA3 d. Test + swap inonly if agreement improvesA4 · A5 · A6 judges aligned judges

3. Use earnings-summary failures to define the rubric

The earnings-call summary and its execution trace answer different questions. Start by checking both: was the note supported, and did the required review actually run? This first pass turns a missing QC result into an explicit output requirement.

Method and reading guide

The walkthrough starts with 30 runs, groups their failure reasons and changes the skill instructions. Select a step or use Back / Next. Treat the displayed reasons as illustrative until original traces are supplied.

Inspect the seven failure modes

Failure modes found in the 30 baseline runs

Each mode is a group of judge reasons that describe the same problem. Select a mode to see the runs and the words the judge used, and which scorer catches it today.

How the modes were found. We read all 30 judge reasons and the traces they point to, wrote a short note for each problem, then grouped the notes. This is the recommended size: read 20 to 50 traces before writing any scorer. Lap 1 started from the skill's rules and a reported complaint, then let the reasons show the rest.

4. Trace the experiment, step by step

Follow the skill changes, judge alignment and monitoring as separate lanes. Select a step or use Back / Next to see what changed and which version was used.

Method and reading guide

The top lane changes the skill; the middle lane changes the judges. The bands record their versions. Live scoring is a later extension of the offline setup.

5. Compare skill outputs with fixed scorers

Hold the evaluator fixed while comparing the research skill. Select a version for its changes or a score for its runs. Improvement is mixed: figure provenance drops from 53/60 in skill v5 to 47/60 in v6.

Method and reading guide

Each version has 60 recorded runs: 20 company call bundles × 3. The table uses the same v2.1 scorer set. Counts are fixture results, with illustrative reasons; they are not independently verified exports.

Selected version or score details

6. Replay a retrospective regression baseline

The walkthrough sets a floor from the accepted v6 results, then replays earlier versions. That makes this a retrospective regression baseline, not an independent test proving v6 is ready to deploy.

Method and reading guide

Select a version to replay the recorded checks. A reproducible gate must also enforce stock coverage, the expected run count and complete scores. The demo replay does not enforce all of those conditions.

CheckRuleIf it fails
5 process checks: skill invocation, approved tools and sites, required flow, and file savingEvery run passesFAIL
Fixed in v4 to v6: qc_result_present, qc_not_in_note, sections_complete, inline_citationsEvery run passes, so the fix can't come backFAIL
Still open: figure_provenance, internal_consistencyAt least 47 of 60 runs pass (the v6 level)FAIL
CoverageAt least one run for each golden stockFAIL

7. Compare judge alignment candidates

Compare three candidates on the same held-out traces: the original judge, raw MemAlign and a separate judge containing manually approved rules. Here the reference labels are synthetic, so agreement does not establish expert validity.

Method and reading guide

The setup uses 40 alignment traces and 20 held-out traces. Four judges × 20 traces gives 80 verdicts. Keep related runs of the same stock on one side of the split, and freeze labels before tuning.

Agreement with synthetic labels, held-out traces

v1 (original)46/80
v2 (approved rules)66/80
MemAlign raw74/80

The walkthrough selects the reviewed-rule judge even though raw MemAlign has higher agreement with the synthetic labels. That is a policy choice: some learned rules were overruled. If that changes the rubric, revise the labels and retest on untouched examples.

What changed in the judges

v1
Base judge split into 4, one criterion each. 4 edge-case rules agreed.
v2
+19 approved rules from MemAlign.
v2.1
Allows attributed management guidance in Guidance. Same verdicts as v2 on 60 runs.
live
Reads only the saved note. Agrees with v2.1 on 447 of 480 verdicts.
MLflow setup and evidence to retain

Judge assessments and reference feedback must share the judge's assessment name on the same trace. Record verdicts and reasons, and keep synthetic labels distinguishable from human review. MemAlign learns guidelines and retrieves examples; it is currently documented as experimental.

Keep the output contract, trace split, rubric and labels, candidate judge versions, and per-criterion held-out results. Five queue questions on 60 traces produce 300 labels; only four judge criteria enter the 80-verdict comparison. Eighty verdicts on twenty traces are not eighty independent company examples.

8. Extend the setup to monitoring

After offline validation, the same checks can monitor fresh traces. This diagram shows the walkthrough's Databricks-specific extension; a daily managed job is not required to test MLflow judge alignment.

Method and reading guide

Keep monitoring inputs consistent with offline evaluation. A note-only judge cannot verify source facts it has not been given.

Skill runin the runnertrace saved to UC Daily jobnew traces sincethe last run Filteragent.workflow.nameLIKE earnings-summary-note-v% 7 rulesevery trace, code 4 judges10% sample, LLM Feedbackon each trace,yes/no + reason Last check (run on demand): 60 new v6 runs. 7 rules: 60/60 each. 6 runs sampled for the judges. The judges now read only the saved note. A whole trace hit 1.93M tokens, above the 1M model limit.
All 11 live scorers and sampling rates

9. Milestones

Lap 1
Lap 1. Baseline on 30 runs, Step 5 fix, 60 fresh runs. Checklist read 8/30 → 60/60; quality judge 0/30 → 47/60.
Lap 1
Judge split into 4 criteria + a QC rule. 4 edge-case rules agreed.
Loop A
Live scoring on (5 rules, 4 judges). SME queue, 300 synthetic labels, MemAlign test, test golden dataset.
Loop B
Judge v2 / v2.1. Failures grouped into 7 causes. Skill v4, v5, v6 written, run and scored.
Gate
v6 accepted. Regression gate built. Live scorers updated: fixed required_flow, judges v2.1, +2 rules.
Live
Judges made note-only after a 1M-token error. Job reset to STANDARD.

10. Known limits

Demo caveat: the SME labels are synthetic. A model acting as a synthetic senior analyst with planted views labelled 60 traces, and wrote the facts for the 20 golden records. The judges' extra rules come from those labels. This illustrates the pipeline; it does not establish agreement with real experts.
  • Sizes follow the recommended minimums: 30 runs for error analysis, 20 inputs × 3 runs per version, 60 labelled traces (20 held out). Go larger for higher-stakes agents.
  • v6 still has 2 open misses: company-reported (non-GAAP) figures in sections meant for release figures, and a guidance change explained two ways when release and transcript figures disagree.

11. Recommended next steps

The technical direction to test is a shared, versioned quality bar: agree the output contract and review criteria, make policy decisions explicit, then validate judges against reviewer labels before relying on their scores. A person owns changes to the standard. This illustrative experiment shows how to explore that direction; real reviewer evidence is still needed to adopt it.

  1. Watch live scores weekly. Triage failures; send borderline runs to the SME queue.
  2. Validate the gate implementation. Enforce coverage, complete scores and the expected run count before using it for an acceptance decision. Predefine the next quality bar rather than deriving it from the candidate.
  3. Re-align the judges after about 50 new SME labels. Swap only if held-out agreement improves.
  4. Optional v7: state any release vs transcript (GAAP vs non-GAAP) difference once, in Guidance, labelled.
  5. For a real rollout: real SMEs, real golden facts, batch runs in the runner.

Rules in force: no change to live scorers or the job without the owner's OK. Other teams' scorers are never touched.