Engineering a Relational Data Layer and Consistency-Aware Synthetic Data Generation for faker-pk
Engineering faker-pk into a consistency-aware synthetic data layer where every generated field agrees with the same real-world context.
Lab Note & Technical Documentation
Problem
faker-pk v1.x embedded all reference data as hardcoded Python lists. This produced three failures: data and code were tightly coupled, fields were sampled independently (a Peshawar institution could be paired with a Lahore postal code), and filtered queries (e.g., "a university in Punjab") were impossible.
Design Decision
The data layer was migrated to a normalized SQLite database bundled inside the wheel via pyproject.toml's package-data. A post-install hook was rejected — it introduces first-import latency and a write-permission dependency on site-packages. The pre-built .db (120 KB) keeps pip install deterministic. ensure_db_exists() in utils.py supports lazy regeneration if the file is absent.
Implementation
The database contains eleven normalized tables: names, locations, sim_providers, sim_prefixes, castes, sects, banks, industries, companies, job_titles, and institutions. Foreign keys connect institutions.city → locations.city and companies/job_titles.industry_code → industries.code.
utils.py exposes four parameterized query helpers (query_value, query_row, query_list, query_rows) wrapping a single _run() function. All domain modules (personal.py, address.py, company.py) were refactored to use these helpers, removing all random.choice() calls on in-memory lists.
education.py was authored from scratch. student_profile() selects an institution row first via a JOIN between institutions and locations; every other field derives from that anchor:
| Constraint | Mechanism |
|---|---|
| City/province matches institution | Both read from the same row |
| DOB age-range matches level | student_dob(level) called with anchor level |
| CNIC encodes gender | Digit 13 odd → male; even → female (NADRA convention) |
| Name matches gender | male_name() / female_name() called after gender is fixed |
Evaluation
| Metric | Value |
|---|---|
| Tests | 42 passing, 0 failures |
| Overall coverage | 94% |
address.py / personal.py / provider.py | 100% |
| CI fail-under threshold | 90% (enforced policy) |
Constraint validation runs 20–50 iterations per test. No violations observed across the full suite.
Defects Resolved
CI reported 0% coverage despite passing tests. pytest-cov instrumented the wrong root, and utils.py was imported before the trace hook activated. Fix: coverage now starts at collection time; --cov=faker_pk --cov-fail-under=90 is enforced in CI.
salary() returned fractional PKR values. round(x / 1000) * 1000 left float residuals. Fix: granularity changed to 500 PKR — the conventional negotiation unit for Pakistani monthly salaries — producing clean integers (e.g., 47,500).
Key Finding
Relational integrity is harder than generation. A random name or CNIC is trivial; making them agree — female name, even-digit CNIC, institution in the right city, DOB in the correct age window — required a single query-then-derive pipeline anchored to one database row, not independent generators.
Limitations
Institution coverage is sparse (42 institutions); missing level/city combinations raise ValueError. Salary bands are static and require a rebuild to update.
Project Repositories & Artifacts
Data Engineer
Building a Reproducible NLP Dataset Quality Auditor
We built `inference-audit`, a reproducible Python toolkit for systematically auditing NLP dataset quality across five core dimensions. The pipeline combines deterministic checks for label distribution, missing values, near-duplicates, language contamination, and annotation consistency. Extensive testing and real-corpus evaluation exposed performance and reliability issues, leading to targeted fixes and measurable optimizations. The final system emphasizes reproducibility, explicit failure states, citable evidence, and transparent limitations.
Read Report llm-eval-kitBuilding a Deterministic Offline Evaluation Engine for LLM Systems
We redesigned llm-eval-kit into a deterministic, zero-network evaluation pipeline built for reliable continuous evaluation in CI/CD environments. The new architecture introduces a decoupled criteria registry, fail-fast orchestration, hybrid semantic and symbolic verification, and graceful handling of inapplicable evaluation criteria. The implementation reached 93% test coverage across 88 tests, with cross-version CI validation from Python 3.9 to 3.12. The work demonstrates how carefully defined evaluation contracts and offline heuristics can provide reproducible, privacy-preserving model assessment without relying on external LLM-as-a-judge APIs.
Read Report