Agent Evaluation: Tracing is table stakes, the loop is not.

Every serious LLM engineering platform can now trace an agent, run a judge over a dataset, and chart the results. If you compare feature checklists, the top of the funnel looks identical: OpenTelemetry ingestion, sessions, LLM-as-a-judge, datasets, playgrounds, CI hooks. We build one of these platforms, and we would be the first to tell you that choosing on those rows is choosing on noise.

Three philosophies

Langfuse treats the job as observability. Its SDKs are built directly on OpenTelemetry, its agent-graph views reconstruct what a multi-step agent actually did, and its prompt management is the richest in the category: version labels, composable prompts, client-side caching with fallbacks, webhook sync. If your question is "what is my agent doing in production, and can my whole team see it," Langfuse answers it with more integrations than anyone.

Braintrust treats the job as experimentation. Its Eval() framework is the best offline-eval developer experience shipping today: run every case N times and average the trials, hill-climb against your previous best experiment, compare runs head-to-head with human pairwise judgments, and iterate in playgrounds that behave like collaborative documents. Its storage engine and in-product assistant both exist to make that iteration loop faster.

AgentX treat the job as governance: closing the loop. Observing production and experimenting before release are both halves of something larger. Production evidence should become test cases. Judges should be measured against reality and improved when they disagree with it. Prompts and tool definitions should change through proposals that arrive with proof. That loop, running end to end, is what AgentX Trace & Eval is built around.

None of these philosophies is wrong. They lead to different products.

Where the divergence actually matters

Setting aside the rows where the category has converged (tracing, sessions, offline evals, code scorers, online scoring, CI gates, human review: all three platforms have them), here are the rows where the checklists genuinely split:

Feature status verified against each platform's public documentation, changelog, and roadmap, August 2026. Each platform also has strengths outside this table; the philosophy section above and the sections below credit them.

1. Judges: running one, measuring one, fixing one

Everyone runs LLM judges on live traffic. The divergence starts after that.

Langfuse ships real judge measurement: score analytics that compare judge verdicts against human annotations with agreement statistics and confusion matrices. That is genuinely useful, and more than most platforms offer. But when the numbers say your judge is misaligned, fixing it is on you.

Our position is that a judge you cannot recalibrate is a liability that compounds quietly. So AgentX treats judge quality as a loop: verdicts are continuously compared against three kinds of ground truth (human re-scores, reported real-world outcomes, and end-user votes), the disagreements are used to generate a rewrite of the judge's own grading criteria, and the rewrite is validated by exact re-judging, proving it fixes the cases it got wrong while preserving a control set it got right, before a human publishes it.

The judges get judged, and then they get better, with evidence.

2. Coverage: what production contains vs. whether your tests match it

Topic clustering, grouping what your agents are actually being asked, is becoming standard, and Braintrust's implementation of it is strong. But clustering answers "what does traffic contain?" We think the question engineers actually lose sleep over is one step further: does my eval suite look anything like my traffic?

AgentX joins production topics to your datasets and reports the gap as three numbers: traffic-weighted coverage, topic breadth, and risk-weighted coverage. The distance between the first and the third is usually the finding: teams test what is common, not what is dangerous. Coverage is computed as a facility-location value over the topic's real traces, so generating fifty near-duplicate test cases moves the number by approximately nothing, which is the point. And a probe endpoint lets you ask about one specific query, or paste an entire launch spec, and get back "covered" only if the dataset would literally reject that query as a duplicate.

3. Testing conversations that have never happened

Every platform can evaluate traffic you already have. Before launch, there is no traffic. AgentX ships persona-driven conversation simulation: describe a user, their goal, and their temperament, and the simulated conversation runs against your real prompt and tool schema, landing as an ordinary session that your entire monitoring stack (session judges, pattern detection, signals) already understands. The impatient customer who demands proof and gives up is a test case you can run on Tuesday, not an incident you discover after launch.

4. Governing tools, not just prompts

Prompt registries are universal now. But for tool-calling agents, the tool schema is prompt-adjacent surface area with the same failure modes, and it mostly escapes governance. AgentX runs prompts and tool schemas through the identical lifecycle: proposals generated from real failure evidence, validated baseline-vs-candidate against golden cases, published only by a human. Tools that fail in traffic without ever being registered get surfaced for one-click registration, drafted from the observed calls. The improvement loop even runs itself: a background sweep notices accumulating failure evidence, generates the proposal, runs its validation, and queues it with the measured verdict attached. Humans keep the only pen.

5. The shape of the deployment

This one is a genuine philosophical split, so we will state it neutrally.

Braintrust runs as a managed platform, with an option to run the data plane inside your own infrastructure. Langfuse self-hosts fully, as a multi-service stack: web and worker containers, Postgres, ClickHouse, Redis, and object storage. Both shapes are rational for what those products optimize.

AgentX is one binary that scales by configuration instead of by architecture. The default is a single process on SQLite: no containers, no external dependencies, running in seconds on a laptop. Set one environment variable and the same binary runs on Postgres with a natively partitioned trace store. Set a second and spans move to ClickHouse while the control plane stays relational. Same binary, same wire API, all three tiers. Every span read and write in the engine goes through one storage port, and a contract test suite runs identical assertions against all three backends on every merge, which is how one codebase stays honest across that range.

Engineering for trust, not just throughput

A governance platform is the thing you use to decide whether your AI is behaving. It earns that role only if its own engineering is more disciplined than what it monitors. Some of the practices we hold ourselves to, because they are the parts we would want to inspect if we were evaluating us:

• Backpressure never lies. Ingestion uses a bounded queue with micro-batching, and a span is acknowledged only after its batch commits. A full queue answers 429 with Retry-After. A failed write answers 503, is counted on a public metrics endpoint, and is never silently dropped or falsely deduplicated.

• Failure modes are rehearsed, not documented. A chaos suite runs weekly in CI: it kills the telemetry store mid-traffic and asserts the API answers honestly while the control plane stays up, SIGTERMs the engine mid-burst and asserts every acknowledged span survives restart, and round-trips a backup asserting byte equality. These drills have caught real bugs that every unit test missed, which is exactly why they exist.

• The architecture defends itself. A boundary test fails the build if any module touches trace storage without going through the port. The wire API is pinned by strict schema contracts, so a response field cannot appear or vanish without the same commit updating the contract. Nightly benchmarks are a merge gate with regression floors, not a dashboard.

Choosing

If your primary question is "what is happening in production across my whole org, with maximum ecosystem compatibility," Langfuse is a very good answer. If it is "how fast can my team iterate on evals before release," Braintrust is a very good answer.

If your question is "how does production evidence systematically make my agents, my tests, and my judges better, on infrastructure I control," that is the question we built AgentX Trace & Eval to answer, and the loop is open source, one curl | bash away.