Building a Reproducible NLP Dataset Quality Auditor

During the INFERENCE Lab Engineering Fellowship, we built inference-audit, a Python library and command line tool designed to audit NLP datasets systematically and produce reproducible, citable quality evidence. The project targets common dataset problems that can affect the validity of downstream experiments, particularly in low-resource and social-media NLP. The project specification defines five checks as the core quality signal of the tool, label distribution, near-duplicate detection, language contamination, missing values, and annotation consistency, each returning a scored CheckResult that feeds into an overall weighted quality score.

Implementation of the Checks

The Research and Implementation Engineer implemented all five audit checks along with their test suites, synthetic fixtures, and initial performance evaluation.

Label distribution measures class imbalance using the majority-to-minority class ratio, normalized around a 1:1 baseline so a perfectly balanced dataset scores 100 while increasing imbalance lowers the score toward zero. During testing, the Research and Implementation Engineer found that the initial formula incorrectly penalized a balanced 50/50 dataset, corrected the normalization, and added explicit handling for empty datasets, missing columns, single-class data, invalid ratio limits, and null labels, which are now reported separately rather than silently excluded.

The missing-values check distinguishes null entries, whitespace-only strings, and text shorter than a configurable minimum length, and rejects non-string values outright rather than silently converting them, preventing unrelated values such as numeric identifiers from being misread as short text.

For near-duplicate detection, the Research and Implementation Engineer implemented MinHash with Locality-Sensitive Hashing over character 3-grams, avoiding the cost of comparing every pair of samples directly. Testing exposed false matches for text under three characters, since such text produces no meaningful 3-grams; filtering and explicit reporting for skipped rows were added. A further regression test, raised during Lead Engineer review, caught a case where every row was too short to compare: the check had been returning a perfect score even though nothing had actually been measured, and was corrected to return score=None instead.

The most research-intensive component was language contamination. Early testing showed that selecting the majority language predicted by langdetect as a baseline was unreliable for Roman Urdu; on the real target corpus, this approach could select an incorrect language as the baseline and produce an unstable, uninterpretable score. Following review and a joint discussion between both engineers, the majority-vote baseline was replaced with a configurable concern-language list, deterministic detection, and confidence thresholding, an approach first proposed by the Lead Engineer after independently reproducing the instability on the real corpus. The Research and Implementation Engineer then added exact-text caching so repeated social-media messages are detected once rather than once per row, measuring an 85.7x speedup on repetitive text, and documented the remaining, bounded false-positive behavior of langdetect on genuine Roman Urdu rather than presenting the check as fully solved.

Annotation consistency flags low-confidence annotations against a configurable threshold, validates that threshold, handles missing or null confidence data, rejects non-numeric values, and skips gracefully when no usable confidence column is available.

Architecture, Review, and Design Decisions

The Lead Engineer owned the project's core architecture: the CheckResult and AuditReport schema that every check returns data into, the weighted overall scoring formula, the data loading module responsible for validating a dataset before any check runs, the CLI, and the HTML report renderer, which embeds charts as base64 images so output remains a single, self-contained file. The Lead Engineer also led review on every pull request submitted to the project.

Early in the project, we considered generating audit reports or scores using an external large language model API instead of the deterministic pipeline. We evaluated this seriously before rejecting it: model output is not guaranteed deterministic even at low temperature, which conflicts with the project's central requirement of a reproducible, citable score; it would introduce a network dependency into a report required to remain self-contained; and it would put the project's CPU performance budget at risk. This decision was documented rather than left as an unspoken assumption.

What Broke, and How It Was Fixed

Several issues surfaced only once the system was tested end to end rather than in isolation. The label distribution scoring bug and the near-duplicate all-short-text bug, described above, were both caught through this kind of testing rather than by inspection alone.

A more significant issue emerged once the full pipeline was tested against the real, full-scale target corpus rather than smaller fixtures. Near-duplicate detection and language contamination, though they passed the team's standard 100K-row performance benchmark at 26.294 seconds, comfortably under the project's 60-second requirement, were separately found to degrade sharply on a genuinely high-uniqueness, real-world dataset; a dedicated worst-case benchmark had already flagged this as a documented risk rather than a hidden one. The Lead Engineer diagnosed each check's bottleneck independently rather than treating both as one problem: near-duplicate detection was rebuilt to group rows by exact text before running any similarity comparison, avoiding redundant MinHash computation for identical rows, while language contamination was parallelized across available CPU cores, since each row's detection has no dependency on any other. Both changes were verified against the original implementations on every existing fixture before being adopted, to confirm identical output rather than only faster output.

Separately, the project's continuous integration matrix began failing on Python 3.9 specifically, while 3.10 through 3.12 passed. The Lead Engineer traced this to an import of packages_distributions from importlib.metadata, a function only available from Python 3.10 onward; a guard had already been placed around the function call, but the import statement itself sat outside that guard and failed before the fallback logic could ever run. Moving the import itself inside the guarded block, rather than only the call, resolved the failure without affecting the versions that already passed.

Testing, Evaluation, and Lessons

Testing was treated as part of the implementation process throughout, not a separate phase at the end. We generated reproducible fixtures, tested both normal and adversarial cases, and added regression tests for every issue surfaced during review. Since the team consisted of only two members, the fixture generation, test coverage, and performance benchmarking responsibilities that would ordinarily belong to a dedicated Integration and Evaluation Engineer were split between us, with the Lead Engineer performing a final, independent verification pass rather than relying solely on self-review.

The central lesson from this project was that a quality signal is only useful when its assumptions, failure modes, and limitations are made explicit. Several early implementations produced plausible results that turned out to be misleading under edge cases or on real Roman Urdu data, and it was iterative testing and direct code review, not any single design session, that surfaced these problems. The finished project reflects that approach: measurable checks, explicit failure states, reproducible tests, and documented limitations, rather than results that only look convincing.