Evaluating LLM Outputs with Orthogonal Judges
Introduction
As large language models (LLMs) move from research prototypes into production systems, a recurring challenge emerges: how do you evaluate the quality of LLM-generated outputs when ground truth annotations are unavailable? In traditional machine learning, evaluation is straightforward, compare predictions against labeled data and compute a metric. But for many LLM applications, particularly those deployed in specialized or internal domains, curated ground truth simply does not exist. There are no human-annotated “correct” outputs to compare against, no gold standard to measure distance from.
This gap is not merely an inconvenience. Without a principled evaluation framework, teams resort to ad hoc quality checks, anecdotal assessments, or, most commonly a single LLM-as-judge that produces a scalar quality score on an ordinal scale. While this approach is easy to implement, it obscures more than it reveals, collapsing distinct failure modes into a single number that offers little diagnostic value.
This article presents an alternative: a framework of orthogonal binary judges, where each judge evaluates a single, independent dimension of output quality. Rather than asking “how good is this output?” and receiving a score of 3 out of 5, the framework asks a series of targeted yes/no questions, each corresponding to a specific way the output can fail. The result is not a single quality score but a failure fingerprint: a multi-dimensional profile that reveals not just whether an output is good, but precisely where and how it falls short.
The limitations of composite scoring
The most common approach to LLM evaluation without ground truth is to prompt a judge model with the input-output pair and ask it to assign a quality score on a numerical scale, typically 1 to 5. This pattern has become widespread due to its simplicity, but it suffers from several fundamental limitations.
First, a single score conflates independent failure modes. Consider a code-generation assistant that produces an output referencing an API that does not exist. This is a hallucination, a fundamentally different failure than an output that references only real APIs but omits a critical parameter. Both might receive a score of 2 out of 5, but they require entirely different interventions to fix. The composite score erases this distinction.
Second, ordinal scales introduce noise and subjectivity. The difference between a 3 and a 4 is poorly defined and inconsistent, both across judge models and across evaluation instances. This ambiguity compounds when scores are averaged over a dataset, producing aggregate metrics that appear precise but are built on unreliable foundations.
Third, and perhaps most importantly, composite scores are not actionable. When an evaluation reveals that average quality has dropped from 3.8 to 3.4, the natural follow-up question is “why?”, and the score provides no answer. The team is left re-examining individual outputs manually, which defeats the purpose of automated evaluation in the first place.
Orthogonal judges as an alternative framework
The core idea is to replace a single composite judge with multiple independent judges, each targeting a specific, well-defined failure mode. These judges are designed to be orthogonal — meaning each one captures a dimension of quality that the others do not, and each can pass or fail independently of the rest.
The number of judges is not fixed. It depends on the application, the types of outputs being generated, and the failure modes that matter most. A simple question-answering system might require only two or three judges. A complex code-generation pipeline might require more. The governing principle is decomposition: every meaningful way the system can fail should be represented by its own judge.
To illustrate the approach, consider a code-generation assistant that maps natural language queries to code. The judges for such a system can be organized into two levels: those that evaluate the quality of the input query, and those that evaluate the quality of the generated output.
Query-level judges
At the query level, two dimensions are worth evaluating independently.
The first is answerability: whether the query can be answered using the available context alone, without relying on external assumptions or missing information. This judge identifies queries that are fundamentally unanswerable given the system’s access to source material. If the information required to answer a query is not present in the retrieved context, then no amount of generation quality can compensate. Flagging these cases separately prevents them from contaminating the evaluation of the generation pipeline itself.
The second is usefulness: whether the query is phrased as something a real user would plausibly ask in practice. This is particularly important when evaluation datasets are synthetically generated. Queries that are technically valid but unrealistic — overly generic, unnaturally phrased, or disconnected from actual workflows — can inflate or deflate quality metrics in misleading ways. A usefulness judge acts as a filter to ensure the evaluation set reflects genuine usage patterns.
Output-level judges
At the output level, two further dimensions emerge.
Faithfulness asks whether the output accurately represents what the source material says, without hallucinating entities, mischaracterizing behaviors, or making unsupported claims. This is the broadest and arguably most critical quality check on any LLM output. It encompasses both outright fabrication — referencing functions, APIs, or parameters that do not exist — and subtler misrepresentation, such as describing a function as sorting by date when it actually sorts by name. Both are violations of faithfulness, and grouping them under a single judge reflects the fact that they stem from the same underlying failure: the model generating content that diverges from the provided source material.
Completeness asks whether the output covers the important details present in the source material — key parameters, constraints, conditions, and edge cases. An output can be entirely faithful and still be inadequate if it omits critical information. For example, if a function silently changes behavior based on a default parameter value, an output that correctly describes the function’s primary behavior but fails to mention this parameter is incomplete. It is not wrong, but it is insufficient. This distinction matters because faithfulness and completeness require different interventions: faithfulness failures suggest the model is generating beyond its evidence, while completeness failures suggest the model is not attending to enough of it.
The orthogonality property
The critical design property of this framework is that each judge operates independently. A query can be answerable yet produce an unfaithful output. An output can be faithful yet incomplete. These are distinct failure modes with distinct causes and distinct fixes. When two judges consistently agree — always passing or failing together — it suggests that they are measuring the same underlying dimension and should be merged into one.
This independence is what makes the framework diagnostic rather than merely evaluative. A composite score tells you that something is wrong. Orthogonal judges tell you what is wrong.
Design principles for judge construction
While the specific judges described above are tailored to a code-generation use case, the design principles behind them generalize to any LLM application.
The first principle is one axis per judge. Each judge should evaluate exactly one dimension of quality. If a judge is checking two things — for example, whether an output is both correct and complete — it should be split into two separate judges. Combining multiple criteria into a single judge reintroduces the conflation problem that the framework is designed to avoid.
The second principle is binary outputs. Each judge should return a yes or no, not a score on a scale. Binary outputs force precision in the judge’s definition: the question must be specific enough to admit a clear answer. Scales, by contrast, invite ambiguity. A judge that returns 3 out of 5 is a judge that is not sure — and that uncertainty gets silently absorbed when scores are aggregated, producing false confidence in the resulting metrics.
The third principle is independence. Each judge should be capable of passing while others fail. If two judges always agree, one of them is redundant and should be removed or merged with the other. Testing for independence can be done empirically by examining the correlation between judge outputs across a sufficiently large evaluation set.
The fourth principle is domain grounding. Judges should be anchored in specific, observed failure modes, not generic notions of quality. A judge defined as “Is this a good response?” is too vague to be useful. A judge defined as “Does this output reference only APIs that exist in the source code?” is specific enough to produce consistent, actionable results. The best judges come from studying actual system failures and formalizing each distinct failure pattern into its own evaluation criterion.
Applying the framework in practice
The practical workflow for evaluation without ground truth follows a straightforward pipeline. It begins with collecting real queries from target users or, when real queries are unavailable, generating synthetic queries and filtering them through query-level judges such as usefulness and answerability. The system under evaluation then produces outputs for these queries. Each judge is applied independently to every query-output pair, producing a binary pass/fail for each dimension. The results are aggregated per judge to produce a quality profile — a set of pass rates that reveals the system’s strengths and weaknesses across each dimension.
This profile is inherently more informative than a single aggregate score. A system with 95% faithfulness but 60% completeness tells a very different story than one with 80% on both dimensions, even if their composite scores might be similar. The former needs better attention to detail; the latter has a broader generation quality problem. These are different diagnoses that lead to different interventions.
Iteration then targets the weakest dimension first. If faithfulness is the primary failure mode, interventions might include tighter grounding in retrieved context, more constrained generation, or better retrieval. If completeness is the bottleneck, interventions might include richer context windows, explicit instruction to enumerate constraints, or the inclusion of function signatures in the prompt. Each dimension has its own set of levers, and the orthogonal framework makes it clear which levers to pull.
Conclusion
Ground truth is a luxury that most production LLM applications do not have. Waiting for it means not evaluating at all — or worse, evaluating with a single composite score that obscures more than it reveals.
Orthogonal binary judges offer a practical alternative. By decomposing quality into independent, well-defined dimensions, they produce a failure fingerprint rather than a quality score. This fingerprint is diagnostic, actionable, and extensible — new judges can be added as new failure modes are discovered, without disrupting existing evaluations.
The framework requires no human annotations, no gold standard outputs, and no labeled data. What it does require is a clear understanding of the ways your system can fail — and the discipline to evaluate each one separately.