Why I built this (well — had an agent build it)
Open datasets from government portals, NGOs, and research teams are almost always messier than they look: dates written five different ways, blank cells that say "N/A" in one row and "-" in the next, duplicate rows nobody noticed, typos in city names. Enterprise data-quality tools exist (Informatica, Alation, Great Expectations), but they're built for teams with infrastructure budgets — not for the one person who just has a CSV and needs to know what's wrong with it.
I didn't go looking for this problem myself, though. I set up an AI agent to research what a solo builder could realistically ship, gave it a few guardrails, and let it pick. This — a rule-based data quality checker, nothing fancy — is what it came back with, and what I approved.
How the agent flow worked
I didn't just prompt an agent once and hope. I set up an "orchestrator" pattern: one agent plays the coordinator, delegating research, building, and testing to itself in separate passes, with everything logged to a repo as a paper trail. Four points where I had to explicitly say yes before anything moved forward:
- Gate 1 — approve the direction (which idea to build)
- Gate 2 — approve starting the build
- Gate 3 — approve anything going public (publishing, distribution)
- Gate 4 — approve anything that costs money (budget cap started at zero and stayed there)
Everything in between — writing the rules, writing tests, opening pull requests, fixing its own mistakes — the agent did on its own. I reviewed and merged each pull request; nothing landed on the main branch without me looking at the diff first.
What happened when it hit real data
The agent built 10 validation rules and 18 tests against data it made up itself — clean, convenient test fixtures. Before calling anything done, I made it run against a real public dataset (NYC's building permit filings, 500 rows, 95 columns). First honest result: 16,454 flagged issues.
That number was a red flag by itself — too many to be real. Turned out three of the ten rules were broken in ways the tidy test data never exposed: a "missing value" rule that reported a column empty in all 500 rows as 500 separate problems instead of one; a "unit consistency" rule that read a street address like "123 Main Street" as if "Street" were a unit of measurement, like kilograms; and an "outlier" rule flagging ID numbers as statistically abnormal values, even though an ID has no normal range to begin with.
The agent fixed all three, re-ran the same dataset, and the count dropped to 3,842 — a much more honest number, with no loss of genuine detections. That loop (build → test on real data → find your own false positives → fix → re-verify) is the part of this whole process I trust the most, because it's the part that can't be faked with a demo.
How to use dqagent
It's a small Python CLI. Point it at a CSV, get a report. It never auto-deletes or auto-edits anything — it tells you what's wrong and you decide what to do about it.
# Quick console summary
dqagent mydata.csv
# Full issue list as JSON
dqagent mydata.csv --json report.json
# Interactive HTML review — open in any browser, no server needed
dqagent mydata.csv --html review.html
# Flag rows missing a value in columns YOU say are required
dqagent mydata.csv --required name,date_of_birth
The --html flag is the part I like most: it generates
one self-contained HTML file — a table of every issue, filter
buttons per rule, and Accept/Reject buttons per row. Your decisions
save to the browser's own storage, nothing is sent anywhere. An
export button downloads your decisions as JSON when you're done.
What it checks (10 rules)
- Ambiguous or inconsistent date formats
- Missing values (including the five different ways people write "blank")
- Exact duplicate rows
- Columns that are mostly numbers but have stray text
- Encoding artifacts (mojibake)
- Inconsistent whitespace/casing
- Statistical outliers (skips ID-like columns, after the bug above)
- Likely typos in region/city names
- Missing required fields (only when you tell it which ones)
- Mixed units in one column (kg vs. gram, etc.)
What the agent did vs. what I did
The agent wrote every line of code and every test, and found and fixed its own bugs once I pointed it at real data. What it didn't do: decide whether this idea was worth building in the first place, merge its own pull requests, or declare "this is good enough, ship it." Those calls stayed mine, on purpose — I spent my time on the one-sentence decisions instead of the hours of implementation in between.
PS: dqagent is still early — it's purely rule-based, no AI/LLM calls involved in checking your data (on purpose: keeps it free, fast, and usable offline). If you've got a messy public dataset lying around, I'd love to know if it actually helps or just yells at you about typos you already knew about.