Building a Reproducible NLP Dataset Quality Auditor
We built inference-audit, a reproducible Python toolkit for systematically auditing NLP dataset quality across five core dimensions.
The pipeline combines deterministic checks for label distribution, missing values, near-duplicates, language contamination, and annotation consistency.
Extensive testing and real-corpus evaluation exposed performance and reliability issues, leading to targeted fixes and measurable optimizations.
The final system emphasizes reproducibility, explicit failure states, citable evidence, and transparent limitations.