Did the work actually improve?
Acceptance criteria met · wrong/missing facts · full-path behavior · evidence completeness.
Field guide / experiment protocol
Field Guide Lab Method
Do not merely try a better pattern. Make the change measurable enough that you can tell whether it actually helped.
Field GuideLab MethodExperiment protocol
Shared lab protocol
What you'll learn
Run field-guide experiments in a way that separates observation from belief and preserves evidence someone else can inspect.
Bring / required
One task/system, a baseline you can rerun, and a place to save evidence.
What you'll do
Capture the current behavior, change one structural element, rerun under comparable conditions, measure what changed and deliberately test the new pattern.
Take away
A reusable lab record containing baseline, hypothesis, intervention, measurements, counterexample, conclusion, artifact and remaining limitation.
The eight-step protocol
01
Baseline
Capture the weaker/current system before teaching the answer.
02
Hypothesis
State the single change you believe will improve one observable defect.
03
Intervene
Change one structural element where practical.
04
Rerun
Repeat under reasonably comparable conditions.
05
Measure
Compare the smallest useful before/after evidence.
06
Break it
Remove the change or inject a counterexample.
07
Conclude
State what the evidence supports — and what it does not.
08
Submit / keep
Preserve the baseline, artifact, rerun, counterexample and limitation.
What to measure
Do not turn every lab into a dashboard. Pick the smallest set of signals that exposes the lesson.
Did babysitting decrease?
Human interventions · retries · clarifications · manual restarts · manual cleanup.
What did reliability cost?
Elapsed time · token/tool cost · review time · orchestration overhead.
Did it survive failure?
Restart success · state loss · repeated failure · wrong-scope changes · safe stop/escalation.
Common ways to fool yourself
- Changing five things at once. The output improves but you do not know why.
- Teaching before the baseline. You never see the failure the lesson was supposed to solve.
- Picking only successful runs. Reliability disappears from view.
- Moving the acceptance criteria after the result arrives. The system always seems to pass.
- Comparing incomparable tasks. A harder second input is interpreted as a worse harness or prompt.
- Measuring only model output. Human rescue, review and cleanup remain invisible.
- No counterexample. You know the pattern can work, but not where it stops working.
Aha
A good AI lab is not “I tried this and it looked better.” It is a small argument supported by a before/after run and an explicit limitation.
Use this now
LAB RECORD
Copy this into a working note for any Prompt, Context, Harness, Loop or Graph experiment.
LAB ______________________________________________________ PARENT GUIDE Work With AI / Give AI Work / Run AI Prompt / Context / Harness / Loop / Graph BASELINE Task: Current setup: Observed failure: Evidence saved: HYPOTHESIS If I change: ______________________________________________________ I expect: ______________________________________________________ Because: ______________________________________________________ CONTROLLED INTERVENTION One thing changed: ______________________________________________________ RERUN What stayed the same: ______________________________________________________ What necessarily changed: ______________________________________________________ MEASURE Metric / evidence 1: Before: After: Metric / evidence 2: Before: After: BREAK IT / COUNTEREXAMPLE What I removed or injected: ______________________________________________________ What happened: ______________________________________________________ CONCLUSION What the evidence supports: ______________________________________________________ What it does NOT prove: ______________________________________________________ REMAINING LIMITATION ______________________________________________________ SUBMIT / KEEP [ ] Baseline record [ ] Changed artifact [ ] Rerun evidence [ ] Counterexample / failure [ ] Conclusion [ ] Remaining limitation NEXT EXPERIMENT ______________________________________________________
How this fits the recordings
The recording should show the same evidence trail: baseline first, then the hypothesis and intervention, then the rerun. Preserve useful mistakes. End by opening the lab record and stating what changed, what did not, and what you would test next.