Project
Data Quality Testing Pipeline
Data quality testing pipeline that loads CSV batches, executes SQL rules on DuckDB and generates reproducible evidence and reports. It includes a public demo for comparing a clean dataset with another containing controlled defects.
Overview
QA pipeline built with Python and DuckDB that validates customer, order and payment batches through 19 SQL rules, automated tests and JSON and HTML reports.
Stack: Python 3.13, DuckDB, SQL, Pytest, HTML, CSS, JavaScript, Docker, GitHub Actions, GitHub Pages
What it does
The project represents a data quality control process performed before information is consumed. It verifies that customer, order and payment data is complete, valid, unique and consistent, preventing an incorrect batch from continuing through an automated process.
How it is built
The solution is built with Python 3.13 and loads CSV files into an in-memory DuckDB database. An engine executes 19 SQL rules and collects their results, violation counts and evidence. The application generates JSON reports and HTML dashboards, provides a command-line interface, runs with Docker and includes 10 automated tests with Pytest.
What it validates or automates
The pipeline approves the clean batch and rejects the defective batch through exit codes that can be used as quality gates. GitHub Actions runs the tests, validates both scenarios, checks the Docker image and preserves reports as artifacts. The demo allows users to select either controlled dataset and inspect the real reports produced by the engine.
Key points
- Quality engine with 19 SQL rules for customers, orders and payments, covering completeness, uniqueness, validity, integrity and consistency.
- Reproducible JSON and HTML reports with summaries, individual rule status and evidence for incorrect records.
- CLI and Docker execution, 10 automated tests and quality gates integrated with GitHub Actions.