Building a Deterministic Offline Evaluation Engine for LLM Systems

llm-eval-kit's pipeline originally lacked a deterministic, offline evaluation harness, forcing teams to rely on non-deterministic "LLM-as-a-judge" cloud APIs or slow manual reviews. This created severe latency bottlenecks, incurred recurring per-token costs, introduced data privacy risks, and produced flaky scores that prevented continuous evaluation within automated CI/CD pipelines. This project set out to close that gap.

What changed. The pipeline moved from monolithic, ad-hoc checks to a decoupled, layered evaluation engine: a decorator-based registry isolates criteria definitions, a fail-fast orchestrator validates inputs before invoking dense embeddings, a hybrid verification engine checks numeric consistency alongside semantic similarity, and an aggregation layer excludes ungrounded checks rather than penalizing missing context. A zero-overhead CLI, multilingual Urdu heuristics, and wheel packaging rules were added alongside this core pipeline.

The criteria registry and hybrid verification model. To scale across diverse evaluation checks without creating a fragile orchestrator, we adapted a micro-kernel registry pattern where criteria functions self-register via @register_criterion into a centralized dispatch table. The central Evaluator interacts solely with this registry interface, ensuring new criteria require no orchestrator modifications. Alongside this dispatch mechanism, we paired dense sentence embeddings with an orthogonal symbolic parser in numeric_utils.py to overcome a critical failure mode: standard semantic similarity treats syntactically identical sentences with conflicting numbers (such as "water boils at 50°C" versus "100°C") as near-identical (~0.90 similarity). The hybrid parser extracts numeric claims across scales, canonicalizes units through synonym mappings, and validates response claims against the context's numeric union, capping unsupported claims at 0.30. This design combines dense representations with fast symbolic sanity checks, acting as an offline consistency heuristic rather than open-world factual verification, which remained outside this project's initial scope.

Engineering outcomes. Automated test coverage grew from initial prototype checks to 88 tests across nine files, raising overall test coverage to 93 percent. These tests caught three real defects before review: a backward-compatibility failure on Python 3.9 caused by Python 3.10 union syntax (|), omitted static configuration JSONs in wheel archives resolved via package-data rules in pyproject.toml, and non-zero cosine similarity artifacts on empty inputs caused by BERT [CLS] and [SEP] tokens, resolved with pre-encoding guards. A CI matrix workflow now runs the full suite across Python 3.9 through 3.12 on every push and pull request.

What we learned. Working through review cycles reinforced concrete engineering habits: enforcing strict fail-fast validation before computing expensive embeddings, maintaining explicit interface contracts across criteria boundaries, and shipping small, single-purpose pull requests with targeted fixtures rather than bundling orchestration and criteria changes together.

Graceful degradation changed how we approached the evaluation problem itself. A naive aggregation treats an unexecuted check as a zero, but penalizing a model when reference context is absent misrepresents its actual quality. Implementing a deliberate three-valued scoring model—where inapplicable criteria return score: None with an explanation and are cleanly excluded from the composite average—was a reminder that an evaluation pipeline must distinguish between an LLM failing a check and the pipeline lacking the context to evaluate it.

Architecting an extensible, zero-network evaluation engine with strict interface boundaries, while balancing deterministic offline heuristics against the real-world constraints of continuous integration, was the central engineering contribution of this work.