This article is not a survey or a research contribution. It is a synthesis of the recurring ideas that surfaced while I read recent literature on software engineering for production AI systems.
After finishing my master's, as I described in an earlier post, I started thinking more seriously about a long-term research direction. My earlier work on performance anomaly detection in large-scale athletics data had already pulled me toward evaluation, reliability, and system design, and it left me with a worry I could not shake: an intelligent system is only worth anything if it keeps behaving reliably once it leaves the laboratory.
The problem was never a lack of interest but a lack of focus. Software engineering for AI, reliability, evaluation, and production system architecture all felt important, yet I could never quite say how they fit together. What eventually helped was changing the question I was asking, from "What should I research?" to "What problems do researchers keep struggling with?" That small shift changed how I read: I started reading papers for the assumptions they leaned on and the difficulties they admitted were unresolved.
I organised my reading into two layers of notes: paper notes, one per paper, recording its contributions, limitations, and open questions; and concept notes, cross-paper notes tracing how an idea, among them production AI systems, continuous evaluation, telemetry, distribution shift, and nondeterminism, grew as more work touched it. After a few weeks I noticed I had stopped thinking in terms of individual papers and started to see the field as a whole, concepts converging not because I forced them together but because the literature kept drawing the connections. What follows keeps separate the observations the work broadly agrees on, the trends that are emerging without being settled, and the questions that are still open, tying each claim to a specific source.
The story the literature is telling
The papers I read were aimed at different problems, but lined up, they told a surprisingly continuous story. Classical software testing could assume it was dealing with deterministic programs whose behavior was meant to match a specification, so testing was a matter of checking the program against that specification. The catch was always deciding whether a given output was in fact correct, a difficulty known as the test oracle problem, and the survey literature on that problem is candid that when specifications and automated oracles fall short, the final oracle is simply a human being who knows what the answer should look like [1].
Machine learning made that catch much worse. The influential framing of ML systems as hidden technical debt pointed out that the model is a small fraction of a much larger system, with the real cost in data dependencies, configuration, and feedback loops [2]. A case study of teams building AI features found these systems qualitatively unlike conventional software [3]. A systematic mapping of ML-testing research makes the modern version concrete: behavior is defined jointly by code and data, and retraining can change outputs without anything being broken [4]. LLM agents push this further, since a system is rarely one model answering a prompt but a coordination of retrieval, tools, memory, and users. The through-line is a slow change in the unit of evaluation, from the model toward the system as a whole.
Offline performance is not enough to tell you how the system will behave
The most consistent starting point across this reading is a negative one, and it recurs often enough that I treat it as established: a model that scores well on a held-out benchmark is not, for that reason alone, a system that behaves well in production.
A practitioner-facing discussion of agent evaluation puts it bluntly: you cannot anticipate what a user will do, so the failures that hurt are the ones nobody wrote a test for, and the only way to find them is to keep looking after launch [5]. A survey of agent evaluation reaches the same conclusion from the benchmark side: static benchmarks saturate quickly, and the field is shifting toward realistic, continuously updated evaluations [6]. Deployment case studies find trouble spread across every stage of the workflow, not only training [7]. None of this is new to language models: a production-readiness rubric from nearly a decade ago paired testing with monitoring because prediction behavior is hard to specify in advance [8], and testing surveys show that testing has had to reach past the learned function to the data around it [9]. The same instinct reappears as behavioral testing that probes capabilities directly [10], and as holistic evaluation across many scenarios rather than a single number [11]. Offline accuracy describes a model against a fixed distribution; acceptable behavior describes a system living among users, data, and an environment that all move.
What people mean by system-level evaluation
If the system is the thing worth evaluating, evaluation has to be rescoped to fit it, and here I would call the situation a convergence in progress rather than a settled result. The system is a pipeline: it takes in input, filters or guards it, retrieves information or calls tools, invokes one or more models, post-processes the result, and sits on an operational layer that logs, routes, and serves. Work on reliable AI systems argues that reliability is a property of the whole invocation chain rather than of any single model, since faults propagate through the chain and emerge from how components are coordinated [12]. A large study of teams running agents in production reports the same empirically: reliability is the top development challenge, addressed mainly through systems-level design rather than better models [13]. The ML-testing mapping had already framed it this way [4], interview work with ML engineers distilled velocity, validation, and versioning as what governs success [14], and definitions of MLOps place the model as one component inside a continuously operated workflow [15].
Component quality does not add up cleanly. A retrieval step and a generation step can each look reasonable in isolation and still combine into a wrong answer.
Figure 1: The production AI system as the unit of evaluation.
Evaluation does not end at deployment; it begins again there
The next idea follows almost inevitably: if the system is the unit and it lives in an environment that keeps changing, a single evaluation performed before release cannot vouch for how the system behaves afterward, so evaluation has to continue while the system runs. Continuous evaluation assesses outputs against some notion of quality repeatedly over time. Runtime monitoring watches live behavior against thresholds. Telemetry is the raw material, the instrumented signals a system emits about latencies, inputs, and outputs. Observability is the most demanding of the four, and asks not merely whether you can detect that something changed but whether you can explain why. Monitoring tells you a number moved; observability lets you say why.
A framework for enterprise agents treats continuous evaluation as an operational capability built from telemetry, drift detection, and business-outcome correlation [16]. The practitioner discussion describes the same progression from logging to monitoring to analytics [5]; the MLOps literature folds monitoring and validation into the lifecycle [14], [15]; and the production study finds teams instrumenting heavily so the system produces its own evidence [13]. Increasingly that evidence includes model-based evaluators, convenient but carrying their own biases [17]. This has become expensive enough that evaluation is now an engineering problem in its own right [6].
Figure 2: The production AI system lifecycle as a loop.
┌────────────────┐
┌──────►│ Data collection│─────┐
│ └────────────────┘ │
│ ▼
┌───────────┐ ┌──────────┐
│ Update │ │ Training │
└───────────┘ └──────────┘
▲ │
│ ▼
┌─────────────────┐ ┌────────────────────┐
│ Drift detection │ │ Offline evaluation │
└─────────────────┘ └────────────────────┘
▲ │
│ ▼
┌───────────────────┐ ┌────────────┐
│Runtime evaluation │ ◄───────── │ Deployment │
└───────────────────┘ └────────────┘
▲ │
│ ▼
│ ┌───────────┐
└────────────────────── │ Telemetry │
└───────────┘
Every trip around the loop can change the system.
Why the loop never quite closes: the system keeps moving
Production AI systems change along several axes at once. The input distribution moves as the world and the user base move. The relationship between inputs and the outputs you want moves too, so a system can face familiar inputs and still need to respond differently. And the system itself will not sit still, as models are swapped, prompts are edited, and retrieval corpora are reindexed.
The ML-testing mapping covers part of this: inputs at deployment can fall outside the training distribution, and a behavior change is not enough to conclude a regression, because the same change can appear from retraining alone, which is why statistical comparison rather than exact matching is needed [4]. A unifying taxonomy separates covariate, prior-probability, and concept shift [18], and concept-drift surveys frame the response as detection, understanding, and adaptation while stressing that unmanaged drift erodes performance [19], [20]. Conventional software tends to fail loudly on unexpected input while ML systems fail silently [21]. The enterprise framework adds semantic drift, a change in meaning from evolving prompts or intent even when the model is unchanged [16]. The production study treats agent behavior as nondeterministic by default [13], and the technical-debt framing had already named a changing external world as a risk [2]. Underneath all of it sit two properties this machinery protects: realism, how faithfully evaluation reflects real conditions, and reliability, whether the system keeps behaving acceptably as everything around it changes.
Where the ideas converge
Flattening the concept notes into their common ground, the reading converges on a single composite claim that no one source states in full:
Production AI systems are composed software systems whose behavior emerges from interactions between models, prompts, retrieval, users, and their operating environment. Because these components continue to evolve after deployment, offline benchmark performance alone cannot determine how the system will behave in practice. Consequently, evaluation shifts from measuring an isolated model to continuously assessing the behavior of the entire system using evidence collected during operation.
Each part of this is backed by more than one source: the model-versus-system framing by the technical-debt, testing, and reliability lines [2], [4], [12], [13]; the insufficiency of offline scores by the testing surveys and behavioral evaluation [8], [9], [10], [11]; the continuous, instrumented character of evaluation [5], [16]; the moving-target problem by the mapping and the shift and drift literature [4], [16], [18], [19], [20]; and the older recognition that deciding correctness is the hard part [1]. The observations are not in dispute; what remains unsettled is almost everything about how to act on them.
The seam: "acceptable" is never defined, and the human is still the oracle
Here the synthesis turns from description to something more pointed. There is no shortage of solutions in this literature: evaluation frameworks, drift detectors, observability architectures, model-based judges, and structured production-readiness practices. What kept showing up in my concept notes was subtler than neglect: what it means for a production AI system to be behaving acceptably is almost always assumed rather than defined.
The oldest source I read is also the clearest. The oracle survey defines the difficulty as distinguishing correct from incorrect behavior and concludes that when automated oracles are inadequate, the human remains the final judge [1]. Two decades later the production study reports exactly that: most deployed teams still rely on human evaluation, with no consensus on how to identify what counts as an error [13]. Readiness rubrics and enterprise frameworks define evaluation as gathering evidence about whether the system behaves acceptably, yet neither pins down what acceptable is [8], [16]. Drift detection sharpens the tension rather than resolving it, since a change is not the same as an unacceptable behavior [18], [21]. The sharpest evidence that this is not merely missing tooling comes from a study of people building LLM evaluators, which names a phenomenon it calls criteria drift: users need criteria to grade outputs, but grading is what helps them discover their criteria [22]. Even model-based judges inherit the biases of the models doing the scoring [17]. The field has agreed on the machinery of evaluation after deployment while leaving underspecified the very thing that machinery is meant to certify. Verification assumes a specification to verify against, and most of this work measures in the absence of one.
The problems that keep surviving
Reading for recurrence rather than for contribution, a small set of specific problems reappears across the sources: some have engineering responses that reduce the pain without closing the question, some are recognized by the research community with active work under way, and at least one has barely been framed as a research problem at all.
1. The production oracle problem: This is the deepest of the set. The production study reports that 74% of deployed teams still rely primarily on human evaluation, with no consensus on how to identify what counts as an error [13]. LLM-as-a-judge is now dominant in practice and can reach agreement with human preferences above 80% [17], but it is not a solution: the judge still needs a specification to judge against, and it carries systematic biases [17], [22]. What has not been seriously attempted is a principled, system-level account of what "acceptable" means for a composed, nondeterministic system, as opposed to a heuristic score on a single output [1], [4], [13], [16].
2. Telling a real regression apart from nondeterministic variation: Inference-level nondeterminism is well understood: accuracy can still shift by up to 9% across runs for reasoning models from batch configuration alone, so most deployments accept some variance. The harder problem is at the system level: distinguishing a genuine regression from acceptable variance as the composed pipeline changes over time requires statistical framing few teams have formalized [4], [13].
3. Discovering failures nobody specified, at the level of the system: Observability platforms now cluster production traces to surface failure categories that were not explicitly defined [5]. Still open: how to prioritize undiscovered failure modes, distinguish a system-level failure from a model-level one, and generalize a failure into a test for the next instance of the same class [2], [4], [12], [13].
4. Multiple, interacting forms of drift, semantic drift included: Covariate shift and concept drift are mature, with established detectors [18], [19], [20], [21], but these transfer imperfectly to LLM systems, where the input space is unstructured text. Semantic drift, where user intent shifts even though the distribution appears stable, is being researched through behavioral anchoring. What is not solved is cleanly separating semantic drift from nondeterminism, prompt sensitivity, and retrieval-corpus changes when all three operate at once [4], [16].
5. Realism, and benchmarks that go stale: Dynamic benchmark construction is an active and growing area [6]; the production analog is building golden datasets from real failures. Open: how to maintain an evaluation that tracks a moving system without itself drifting in validity, and how to avoid a benchmark being contaminated through the evaluations it generates [4], [9].
6. How much evidence, at what cost: Sampling and cost-optimization features are now standard in observability platforms, and a growing majority of teams with deployed agents use automated LLM-as-a-judge evaluation [6]. Unresolved: given a constrained budget, what sampling strategy provides the tightest bound on confidence in acceptable behavior? An active-learning problem adapted to semantic evaluation, it has not been formally posed [5], [13], [16].
7. Co-evolution of production AI systems and their evaluation: The least addressed of the seven. There is growing work on self-evolving agent systems, but how evaluation criteria should be revised as the system changes has barely been framed as a research question. Criteria drift is the sharpest evidence that this is real: if users' own sense of what is acceptable co-evolves with the outputs they observe, any fixed evaluation apparatus progressively misrepresents the system it is meant to certify [22], [5], [6].
What I now file under established, emerging, and open
Some things I am comfortable calling established, because each rests on more than one source: offline benchmark performance does not by itself determine production behavior [4], [5], [6], [8], [9], [11]; production AI systems are composed systems whose interactions matter as much as any model [2], [3], [4], [12]; and deployed systems are non-stationary in their inputs, concepts, and configuration [2], [4], [16], [18], [19], [20].
Other things look like trends that are converging without having settled: evaluation as continuous and system-level rather than one-shot [5], [13], [15], [16]; evidence triangulated across telemetry, monitoring, and operational signals [5], [16]; observability increasingly held apart from monitoring [13], [16]; model-based evaluation adopted quickly even as its own reliability is still being characterized [17], [22]; and benchmarks moving toward realistic, continuously updated environments [6].
The rest is open, and the seven problems above are my attempt to name it precisely. What strikes me is how tightly they cluster around one node, the definition of acceptable behavior, while the established and emerging material clusters around machinery that quietly presupposes it. Nondeterminism and drift make that node acute, turning "acceptable" into a moving, statistical target rather than a fixed one, and that is the direction I find myself heading: less toward adding another measurement tool, more toward asking what a production AI system is being measured against in the first place. Whether that becomes a dissertation problem is a decision for another day, but as a reading of where the field stands, it is where my notes keep pointing.
References
[1] E. T. Barr et al., "The Oracle Problem in Software Testing: A Survey," IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507-525, 2015.
[2] D. Sculley et al., "Hidden Technical Debt in Machine Learning Systems," in Advances in Neural Information Processing Systems (NeurIPS), vol. 28, 2015, pp. 2503-2511.
[3] S. Amershi et al., "Software Engineering for Machine Learning: A Case Study," in Proc. IEEE/ACM 41st Int. Conf. Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019, pp. 291-300.
[4] V. Riccio et al., "Testing Machine Learning Based Systems: A Systematic Mapping," Empirical Software Engineering, vol. 25, no. 6, pp. 5193-5254, 2020.
[5] S. Clark, "How to Find the Agent Failures Your Evals Miss," The TWIML AI Podcast, episode 767, 2025.
[6] A. Yehudai et al., "Survey on Evaluation of LLM-based Agents," arXiv:2503.16416, 2025.
[7] A. Paleyes, R.-G. Urma, and N. D. Lawrence, "Challenges in Deploying Machine Learning: A Survey of Case Studies," ACM Computing Surveys, vol. 55, no. 6, art. 114, 2022.
[8] E. Breck et al., "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction," in Proc. IEEE Int. Conf. Big Data, 2017, pp. 1123-1132.
[9] J. M. Zhang et al., "Machine Learning Testing: Survey, Landscapes and Horizons," IEEE Transactions on Software Engineering, vol. 48, no. 1, pp. 1-36, 2022.
[10] M. T. Ribeiro et al., "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList," in Proc. 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 4902-4912.
[11] P. Liang et al., "Holistic Evaluation of Language Models," arXiv:2211.09110, 2022. (Also Transactions on Machine Learning Research, 2023.)
[12] C. Qi and X. Qiao, "Building Reliable AI Systems: A Framework That Bridges Architecture and Operations," SSRN working paper, 2026.
[13] M. Z. Pan et al., "Measuring Agents in Production," arXiv:2512.04123, 2025.
[14] S. Shankar et al., "Operationalizing Machine Learning: An Interview Study," arXiv:2209.09125, 2022.
[15] D. Kreuzberger, N. Kühl, and S. Hirschl, "Machine Learning Operations (MLOps): Overview, Definition, and Architecture," IEEE Access, vol. 11, pp. 31866-31879, 2023.
[16] K. Anitha Raja, "Continuous Evaluation and Observability for Enterprise AI Agents: A Unified Framework for LLM and ML Systems," International Journal of Computational and Experimental Science and Engineering, vol. 12, no. 1, 2026.
[17] L. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, Datasets and Benchmarks Track, 2023.
[18] J. G. Moreno-Torres et al., "A Unifying View on Dataset Shift in Classification," Pattern Recognition, vol. 45, no. 1, pp. 521-530, 2012.
[19] J. Gama et al., "A Survey on Concept Drift Adaptation," ACM Computing Surveys, vol. 46, no. 4, art. 44, pp. 1-37, 2014.
[20] J. Lu et al., "Learning under Concept Drift: A Review," IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 12, pp. 2346-2363, 2019.
[21] S. Rabanser, S. Günnemann, and Z. C. Lipton, "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift," in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019, pp. 1394-1406.
[22] S. Shankar et al., "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences," in Proc. 37th Annual ACM Symposium on User Interface Software and Technology (UIST), 2024, art. 131.