Observability Is Not Evaluation: What LLM Tracing Cannot Tell You
Your team installed Langfuse or LangSmith, the dashboards are green, and everyone feels covered. Tracing tells you what the model did. It cannot tell you whether the answer was right, because it has no idea what right looks like. Where the two tools differ, why the gap is structural, and how to use your traces to close it.
A team ships an LLM feature, wires up tracing, and watches the dashboards fill in. Latency is fine. Cost per call is under budget. Error rate is near zero. Every span is green.
Six weeks later a customer points out that the system has been putting the wrong figure in one field since the model was updated. Nobody can say when it started, because nothing in the stack was ever measuring that.
The dashboards were not lying. They were answering a different question than the one anybody cared about.
The two questions
Observability answers: what did the system do? It records the call, the prompt version, the retrieved chunks, the tokens, the latency, the cost, the exception if one was thrown. Modern LLM observability platforms do this well, and any serious production system should have it.
Evaluation answers: was the output correct? That question requires something observability never has: a known-correct answer to compare against.
This is not a feature gap that a vendor will close in the next release. It is structural. A trace is a recording of what happened. Correctness is a judgment about what should have happened. No amount of recording produces the judgment.
What tracing genuinely gives you
Being fair about this matters, because the tools are good and the answer is not "rip it out."
| Question | Observability | Evaluation |
|---|---|---|
| Which prompt version ran? | Yes | No |
| How long did it take, and what did it cost? | Yes | No |
| Which documents were retrieved? | Yes | No |
| Did the call throw an error? | Yes | No |
| Was the extracted date correct? | No | Yes |
| Did the answer drop a required field? | No | Yes |
| Is this model better than the one we ran last month? | No | Yes |
| Which failure class costs us the most? | No | Yes |
Tracing tells you the system is running. Evaluation tells you the system is right. Teams conflate them because both produce charts.
Three things mistaken for a baseline
Dashboards. Green means nothing crashed. An LLM fails without crashing. It returns a well-formed, confident, wrong answer, with a 200 status code and unremarkable latency. Every silent failure looks exactly like a success to the transport layer.
User feedback. Thumbs-down data tells you when something went wrong badly enough that a person noticed and bothered to report it. That is a small and heavily biased sample. The failures that matter most in regulated work are the quiet ones: a plausible value in a field nobody checks, a dropped requirement, a number off by enough to matter and not enough to look odd.
An aggregate accuracy score. A single number averaged across everything hides its own worst case. 94% overall reads as healthy right up to the moment you learn the missing 6% sits in the one field a decision depends on. In systems where a wrong answer carries real consequences, the average is the least interesting statistic you can compute.
Why the gap keeps going unnoticed
Three reasons, in the order we usually find them.
Buying a tool feels like solving a problem. The platform is installed, it produces sophisticated views, and the question "are we measuring quality" gets a yes in the status meeting. Nobody asks quality of what, against what.
The failure mode is silent by construction. A wrong extraction does not page anyone. It flows downstream into a report, a decision, or a customer-facing summary, and surfaces weeks later through a complaint rather than an alert.
Nobody owns correctness. Platform teams own uptime and cost. Product owns the roadmap. Nobody is on the hook for accuracy per field, so no one builds the thing that would measure it.
Your traces are the raw material
Here is the constructive part, and the reason not to treat these tools as opposites.
The single hardest part of evaluation is assembling a golden set that reflects reality rather than someone's imagination of it. Your traces are already the best possible source for that set. They are real inputs, at real frequency, including the malformed and the adversarial ones your team would never have thought to write down.
The path from one to the other:
Sample from production traffic, not from your head. Pull a few hundred real cases out of your traces. Stratify them so the rare and expensive categories are over-represented relative to their natural frequency, because that is where a regression hurts.
Label the correct output, field by field. This is the work, and it cannot be skipped or fully automated. A domain expert says what the right answer was for each case. That labeled set is the asset. Models are rentals; the golden set is property.
Score per field, not per response. One blended number tells you nothing actionable. Field-level scores tell you exactly which part of the output degraded, which is the difference between "quality dropped" and "the effective date extraction broke on multi-page documents."
Run it on every change. Prompt edits, retrieval changes, model swaps. Once the harness runs in CI, upgrading a model becomes a config change instead of a gamble.
Feed failures back into the set. Every incident that reaches a customer becomes a permanent test case. The set gets more valuable every time something goes wrong.
What this looks like in practice
A team with tracing alone can tell you their system handled 40,000 calls last month at an average of 1.2 seconds and $0.004 per call, with a 0.3% error rate.
A team with both can tell you the same, and add: extraction accuracy on effective dates is 97.8%, on counterparty names 99.1%, and on payment terms 89.4%, which is down from 94% since the model upgrade three weeks ago, and payment terms is the field the downstream approval depends on.
Only the second team can make a decision.
Where to start this week
You do not need a project to find out where you stand. Take twenty real cases from your traces. Have someone who knows the domain write down the correct output for each, field by field. Run your current system against them and count.
The number you get will either reassure you or explain a problem you have been unable to name. Both are worth the afternoon.
If it turns out to be the second, the fix is a measurement layer, not a better model. We build that layer as a fixed-scope reliability assessment: a golden set from your real traffic, a harness that runs in your CI, and accuracy measured per field. Your team keeps all of it.
Talk to the team behind this
Building something like this in production?
Our senior engineers ship this kind of work for real teams. 45-minute call, no pitch deck — just architecture, trade-offs, and whether we're the right fit for your problem.
