In Prompt Engineering, the System Around the Prompt Is the Prompt
September 18, 2026
Prompts are artifacts with histories, versions, evaluations, and repairs, and none of that matters if the surrounding system fails silently. Passing tests do not prove a feature works for a real user. Audit five surfaces (history, model catalog, eval repair, template renderer, and cache) to find gaps CI misses.
Why doesn't passing CI mean the system works?
I work on a prompt-optimizer stack: an MCP server, a client panel, and a small web product, all talking to one backend. Each piece passed its own tests, and the end-to-end stack passed CI. It took me a year to ask the question I had been avoiding: does this work for someone using it today? Not "did the test pass." Does it work.
In modern prompt engineering, prompts are artifacts. They have histories, they version, and they get evaluated, repaired, and rolled forward. That machinery only matters if the system around the prompts works:
- If History returns empty, prompt evolution isn't reproducible.
- If a self-improvement loop computes the improvement but never surfaces it, eval-driven iteration is theatre.
- If the model dropdown feeds the optimizer retired model IDs, the workflow has stopped being portable.
None of those gaps show up in CI.
What did the audit find?
I spent a week reading my own source like a stranger. I traced every advertised endpoint, every "auto-improvement" flow, and every user-visible affordance, and asked whether it worked for someone using the product today.
I expected edge-case bugs. I found surfaces that had never worked for users at all: affordances shipped to empty backends, 404s returned as "the feature isn't loaded yet," and values computed into columns nobody read.
One concrete example: my spend endpoint returned zero for every user, with no error. Tests, CI, and prod all passed. A query referenced a column the table didn't have, and tenancy was enforced through a JOIN the query never used. The database migration surfaced the bug loudly, but the Python fallback would have surfaced it silently. The same shape existed in two places, both masked by passing tests.
How do I audit the system around my prompts?
Run these five checks on your own stack.
| Surface | What to do | Healthy looks like | A gap looks like |
|---|---|---|---|
| History | Hit the history endpoint for a recent job | Rows returned, capped per row, scoped to the requesting user | Empty response when rows clearly exist |
| Model catalog | Open any surface with a model dropdown | Live items fetched on mount, plus a static fallback that is never empty | The optimizer can be fed a retired model ID |
| Eval-driven iteration | Trigger an eval on a prompt you know fails | A repair, returned alongside the failing prompt | Only the score returns |
| Template renderer | Submit a template with a hostile construct (recursion, attribute lookup) | Rejected at parse time, in microseconds, at bounded cost | Rejected at execution time, so you can pin compute |
| Cache | Change the model mid-flow | The cache key contains the model name | The same prompt on two models returns the same cached body |
Why does this matter for prompt engineering specifically?
Each failure removes one of the properties that make prompts-as-artifacts worth having. A tool that loses its history loses its reproducibility story. A self-improvement loop that hides its output loses its credibility. Cached responses that aren't keyed to the active model lose their cost story.
The lesson isn't "audit more." The system around the prompt is the prompt, and the principle of prompts-as-artifacts only holds if you can audit, retrieve, and iterate the actual artifacts the user sees.
For a deeper look at one of these failures, read self-improving prompt loops fail when nothing reads the output.
What should I ask before I ship?
If you ship anything prompt-related (a tool, an MCP server, an extension, a hosted endpoint), ask the question I avoided for a year: does this work for someone using it today? Not: did CI pass today.
FAQ
What does "the system around the prompt is the prompt" mean?
The prompt text is only one part of what a user experiences. History, versioning, model selection, evaluation, and caching decide whether the prompt behaves reliably, so they are part of the prompt's real behavior.
Why can a feature pass every test and still not work?
Tests verify that each step runs. They rarely verify that the output reaches a user, so a value can be computed correctly and never consumed.
What is the quickest audit to run first?
Hit your history endpoint for a recent job. If it returns nothing when rows clearly exist, you have found a real gap in minutes.
Why key the cache on the model name?
Two models produce different outputs for the same prompt. Without the model in the key, a cache hit returns the wrong model's answer and skews your cost numbers.
This post was first shared on r/PromptEngineering by u/Parking-Kangaroo-63.