CHAPTER 06 / 8 MIN READ
Change the machinery
Move from a fixed workflow to a workflow you can evaluate and revise.
START HERE
You replace a model inside a carefully tuned workflow and performance falls. Is the model weaker for this task, or did the surrounding prompts, tools, and runtime stop fitting it?
A self-improvement proposal becomes useful only when an evaluation can distinguish a better system from a different one.
1.Make the configuration an object.
DSPy expresses language-model pipelines as modules with signatures and compiles their prompting behaviour against a metric. The object being optimised is a program around the model, rather than one isolated instruction edited by intuition.
OpenJarvis makes several parts of a personal AI stack independently editable through a typed specification. Its search considers changes across that specification. This widens the space of possible interventions.
2.Code changes need selection pressure.
The Darwin Gödel Machine explores agents that modify their own code and evaluates the variants empirically. An archive retains alternative agents that may be useful starting points for further changes.
This is a search process over implementations. A proposed edit is not inherently an improvement; the evaluation and acceptance process are what give “improvement” a meaning.
3.The metric defines what the search rewards.
A benchmark may reward successful outputs while missing fragility, excessive resource use, or unsafe behaviour. Optimising against it can preserve those blind spots.
Keep separate development feedback from held-out evaluation. Ask whether the same task examples guided the change and then supplied the apparent proof that it generalised.
An optimised research workflow
BEFORE
Change the prompt until a few familiar questions look good.
AFTER
Define the output contract, use a development set to select changes, and check a separate set for generalisation and cost.
A systematic search can still overfit. Better evaluation is part of the mechanism, not paperwork after it.
PAUSE & PREDICT
An edited harness improves on the examples used to select it. What remains unknown?
These papers do not all optimise the same object or use the same evaluation. Their shared theme is editable machinery, not interchangeable results.
GO TO THE SOURCE
DSPy2023 ↗Darwin Godel Machine2025 ↗OpenJarvis2026 ↗This chapter is an original teaching synthesis. Open each research note for source links, findings, and classification caveats.
TAKE IT INTO YOUR OWN WORK
Choose one harness component you would optimise. State the metric, a held-out check, and a behaviour the metric might accidentally reward.