A system improves when evidence changes its future behavior and that change produces better outcomes. Automatically rewriting prompts is only one possible mechanism, and it can make results worse. This independent lesson closes the Doc Steward example with a controlled learning loop: observe failures, create useful evaluations, propose a change, compare outcomes, and govern adoption. It draws on public sources rather than MEGA's paid lessons.
Learning goals
- Turn feedback into a reproducible evaluation case.
- Distinguish a proven fix from a candidate that passes existing checks.
- Explain distillation at the workflow and model levels.
- Optimize useful outcomes under explicit governance and rollback rules.
Feedback must point to an outcome
“Your patch was confusing” is valuable feedback, but it is not yet an executable requirement. Ask what the reader needed to do, what the patch said, and where the mismatch occurred. Preserve the input revision, generated artifact, checks, and reviewer observation. Without those facts, the next change may address a different problem.
John Lindquist's teaching post recommends choosing a concrete skill and its context, then demonstrating steps with observable before-and-after states. We apply that idea to feedback: identify the behavior the system should make possible, then define evidence that the behavior happened.
For Doc Steward, a reviewer reports that a parameter rename was documented but its default value disappeared. The actual defect is specific: a request example remains valid, yet readers can no longer predict how many records an omitted parameter returns. Feedback should link to that patch and the API evidence establishing the default.
Classify the cause before rewriting instructions. The reference collector may have omitted relevant text. The draft may have ignored available evidence. The verifier may have checked only syntax. A retrieval failure needs a knowledge-delivery change; a missing acceptance check needs a verifier change. Adding “be more careful” to every prompt obscures ownership.
Evaluation converts complaints into constraints
Anthropic's evaluation article distinguishes a task, repeated trials, graders, traces, and the final outcome. It also separates capability evaluations from regression evaluations. Those distinctions help us measure whether a change fixes a weakness while preserving established behavior.
Our evaluation fixture pins the old guide, API diff, and authoritative default. A deterministic check confirms that the renamed parameter is used. Another checks that the output contains the correct default in a machine-readable claim record. A human or calibrated semantic grader checks that the prose explains omission clearly. Exact sentence matching would reject valid alternative wording.
Grade the produced artifact, then inspect traces when needed to diagnose failures. A worker can say it preserved every default while the patch omits one. Conversely, a different sequence of tool calls can produce a fully correct result. Evaluation should reward the required behavior rather than one preferred implementation path.
Include negative cases. A change that adds default values everywhere may invent defaults for parameters whose behavior is unspecified. Add a fixture where the reference explicitly leaves the default undefined. Also retain earlier pagination, stale-revision, and forbidden-path cases. Improvement in one dimension cannot compensate for crossing an authority boundary.
Distillation makes lessons reusable
Distillation has several meanings. In this workflow, it means compressing a failure and its evidence into a reusable rule, fixture, or focused instruction. It does not mean the model has been trained. A model-training project might use teacher outputs to train another model, but that is a separate process with different data and validation requirements.
Prefer the smallest durable representation. If a parser can extract parameter defaults reliably, store that structured field rather than repeatedly asking a model to rediscover it. If a judgment remains necessary, provide a short example showing both what to preserve and when to mark information as unknown.
John's mdflow evolution documentation describes a proposal-first loop with feedback-linked evaluation and separate application. It distinguishes a candidate that demonstrates a fix from one that merely passes declared guardrails. That distinction is useful even when implementing the loop with other tools.
Distilled feedback is still untrusted until reviewed. A complaint may be mistaken, malicious, or specific to one release. An automatically generated test can encode the wrong expected answer. Keep its evidence visible and have a maintainer approve the assertion before treating it as a trusted gate.
Optimization needs a stable objective
Choose what the system should improve: correct proposals per reviewed task, reviewer minutes, completion latency, or operating cost. Measure the complete workflow. A cheaper route decision can be irrelevant if incorrect routing causes expensive drafting retries. A faster draft can increase total time if reviewers must repair omitted facts.
Use a constrained objective: reduce reviewer repair time while preserving factual correctness, required coverage, and permission boundaries. Hard constraints remain gates. Do not combine unauthorized writes with prose quality into a weighted average that can still pass.
Compare baseline and candidate on the same pinned fixtures, with fresh task state and repeated trials where outputs vary. Record the model, prompt, adapters, source revisions, and limits. If the environment changed at the same time as the prompt, you cannot confidently attribute a difference to either one.
A confidence threshold is also a configuration change. TypeSafe's confidence guidance recommends validating thresholds with domain data and considering action risk. Lowering a threshold may increase automation coverage while increasing review escapes. Track both instead of declaring success from fewer escalations alone.
Worked example
This is an original evaluation-and-promotion design, expressed as conceptual pseudocode rather than a tested mdflow API.
feedback = capture("default omitted", artifact="patch-42")
case = reviewed_fixture(feedback, expected_default=20)
baseline = evaluate(current_bundle, case, trials=3)
candidate = propose_change(
target="reference extraction and claim checks",
permission_delta="none"
)
result = evaluate(candidate, all_fixtures, trials=3)
if baseline.fails(case) and result.passes(case):
label = "demonstrated fix on these fixtures"
else:
label = "fix not established"
if result.all_required_gates_pass and reviewer.approves(candidate):
apply_if_current_hash_matches(candidate.base_hash)
retain_rollback_bundle(current_bundle)Suppose the baseline omits the default in all three trials. The candidate adds a structured default_behavior field to the evidence packet and requires the patch checker to account for it. It preserves the default in three trials, keeps unspecified defaults marked unknown, and passes prior boundary cases. Expected evidence supports a fix on these fixtures, not universal reliability.
If the candidate also requests publication access, reject that capability change even if content quality improves. If another maintainer edits the baseline before application, the hash comparison prevents overwriting their work. Re-evaluate against the new baseline before promotion.
After adoption, sample actual proposals and monitor omission rate, review effort, and escalations. If repair time rises, roll back the versioned bundle and retain the new evidence. A rollback should restore the relevant prompt, extraction, and checker versions together, because their contracts may have changed together.
Practice
Create a failure fixture for Doc Steward and two candidate changes. One should address the cause; the other should improve a visible metric while creating a regression. No live model is required: recorded artifacts can demonstrate the reasoning.
Acceptance criteria:
- Feedback links to an input revision and an inspectable artifact.
- A reviewed case fails on the baseline for the stated reason.
- The selected candidate passes the new case and prior required gates.
- The rejected candidate has a documented regression or capability expansion.
- Promotion records a version and a rollback target.
A strong answer identifies why each grader exists and which valid wording variations it accepts. The rubric rejects self-scored success summaries, tests that only mirror a preferred sentence, and claims of universal improvement from a tiny fixture set. Include one honest limitation of the evidence and one production signal that could reveal a new failure.
Check your understanding
- Is passing the old suite proof that feedback was fixed? Answer: No. The old suite may never exercise that failure; show the targeted case failing before and passing after.
- Does distilling feedback necessarily train a model? Answer: No. It can produce a rule, fixture, structured field, or instruction without changing model weights.
- Why retain a complete rollback bundle? Answer: Prompts, extractors, and checkers may depend on matching versions, so reverting only one can leave incompatible behavior.
Sources
- John's teaching method: concrete outcomes and observable steps.
- Demystifying evals for AI agents: trials, graders, outcomes, and regression coverage.
- mdflow evolution protocol: proposal-first improvement and explicit application.
- TypeSafe confidence: task-specific thresholds and risk.