Self-Improving Prompt Loops Fail When Nothing Reads the Output
September 18, 2026
A self-improving prompt loop only counts if its result reaches the user. Mine generated, scored, compared, and committed repairs for six weeks, but nothing read the column it wrote to, so users saw no improvement. Every state transition needs a reader before you ship it.
What went wrong with my self-improving prompt system?
I built a closed-loop repair system for my prompt optimizer: score a prompt, generate a candidate, compare it against the original, and ship the better one. The pitch was that the user watches the system get smarter.
For six weeks the system was improving, technically. A background task generated repair candidates, scored them, compared them with the original, and committed the winner. The commit wrote the corrected prompt into a database column on the existing history row, and a flag flipped to "repaired."
That column was read by nothing. The history endpoint didn't include it, the version service didn't read it, and the UI panel didn't show it. Users saw zero improvement, because it all happened inside a sealed box.
Why didn't my tests catch it?
Every state transition in the loop had a test: generate, score, compare, commit. What nobody tested was whether the commit had a downstream consumer. A value was written, and nobody read it.
This is the gap between "each step works" and "the feature works for someone using it." Unit and integration tests verify the first. Only a check that starts from the user's side verifies the second.
How do I check whether my loop is closed?
Trigger a repair on a prompt you know is failing, then look at what the user sees next.
If you have to call a separate endpoint, open a different panel, or dig through raw database rows to find what the repair wrote, the loop has not closed.
What makes a repair loop real?
Three conditions had to hold in my system.
| Requirement | The bug I had | The fix |
|---|---|---|
| The repair appears in the same response | The repair was written to a row nobody fetched | Return the corrected prompt alongside the original, before and after, at the moment the user cares |
| Both scores use the same ruler | The repair and the original were scored by different evaluators on different scales | The same evaluator scores both on the same rubric, so "better" means better |
| The evaluator's grade reaches the decision | The decision path read a stale placeholder instead of the live score | The computed grade is the grade used |
The second row is like weighing yourself on two different scales and celebrating the lower number.
What is the general principle?
Map every state transition to a reader before you ship it. If you generate a value, someone has to consume it. If you score, compare, and commit, the commit has to reach the user, in the response, in the UI, or in the artifact they carry forward.
Otherwise you don't have a feature. You have a logging tax: compute running in a sealed box, producing output nobody sees.
This is one instance of a broader pattern. In the system around the prompt is the prompt, I audit five surfaces of a prompt tool for the same class of silent failure.
FAQ
What is a self-improving prompt loop?
A loop that scores a prompt, generates a repaired candidate, compares the two, and ships the better one. It only works if the user actually receives the result.
How do I tell if my loop is a "logging tax"?
Follow the output. If nothing in a user-facing response, panel, or artifact consumes the value your loop writes, it is a logging tax.
Why must the same evaluator score both prompts?
Different evaluators use different scales. Comparing scores from two of them tells you nothing about which prompt is better.
Is passing CI enough to trust an improvement loop?
No. CI can confirm every step ran. It cannot confirm a user ever saw the result.
This post was first shared on r/PromptEngineering by u/Parking-Kangaroo-63.