Every data team I’ve worked with has had quality checks. Most of them were useless.

The pattern is always the same. Someone adds a not-null check to a critical column. It fires at 3am. The on-call engineer looks at it, discovers it’s a known upstream issue that self-resolves by 6am, and adds a Slack message to the #data-alerts channel saying “ignore this.” Within a month, nobody looks at #data-alerts at all.

The problem isn’t the check. It’s the architecture of the gate.

Three types of gates, three different jobs

Quality gates aren’t one thing. They’re three things pretending to be one, and conflating them is why most implementations fail.

Schema gates validate structure. Did the expected columns arrive? Are the types right? Is the file format what we agreed on? These should be hard stops — if the schema is wrong, nothing downstream will work anyway. No tolerance, no warnings. Stop the pipeline.

Statistical gates validate distribution. Is today’s row count within two standard deviations of the trailing 30-day average? Has the null rate on a key column jumped? These need baselines and tolerance bands, not fixed thresholds. A table that normally has 50,000 rows doesn’t need an alert when it has 49,800. It needs an alert when it has 5,000.

Business rule gates validate semantics. Does revenue equal quantity times unit price? Do all customer IDs in the orders table exist in the customers table? Are there future-dated transactions? These are the hardest to get right because they require domain knowledge that engineers often don’t have and analysts often don’t encode.

The score matters more than the gate

Here’s what changed my thinking on quality gates: stop treating them as pass/fail and start treating them as a measurement system.

Every gate should produce two outputs. First, a binary signal that controls pipeline continuation — stop or proceed. Second, a quality score that feeds a time-series dashboard. The score is a number between 0 and 1 for each dataset at each boundary.

The binary signal catches catastrophic failures. The score catches drift.

A column that passes its null check every day but whose null rate has crept from 0.1% to 2.8% over six weeks is telling you something. The gate won’t catch it — the threshold is 5%. But the trend line will, if anyone’s watching. The dashboard makes someone watch.

Where to put them

I’ve seen teams put quality checks only at ingestion — validating source data on arrival. That catches source problems but misses transformation bugs. I’ve seen teams put them only in dbt tests — catching modelling errors but missing ingestion failures.

Gates belong at every handoff:

  • Source to landing: schema gates. Did we get what we expected?
  • Landing to staging: statistical gates. Does this batch look reasonable compared to history?
  • Staging to serving: business rule gates. Does the transformed data make semantic sense?

Each boundary has a different failure mode and needs a different type of check. Stacking all your validation at one boundary is like putting all your smoke detectors in the kitchen.

The alerting problem

The real engineering challenge isn’t writing checks. It’s deciding what to do when they fire.

My rule: if a gate fires more than twice in a month without requiring action, it’s miscalibrated. Either tighten the threshold until it only fires on real problems, or remove it entirely. A noisy gate is worse than no gate — it trains people to ignore alerts.

For statistical gates, I use adaptive thresholds that recalculate weekly from trailing data. For business rule gates, I maintain an exception registry — known, accepted deviations that won’t fire an alert but will show on the dashboard with a reason attached.

The goal isn’t zero alerts. It’s zero ignored alerts.

What this looks like in practice

In Python, I build quality gate frameworks as thin wrappers around assertion functions that produce structured output — a result object with the check name, the measured value, the threshold, the pass/fail flag, and a timestamp. Every result gets written to a quality metrics table before the pipeline decides whether to continue.

In dbt, I use a combination of built-in tests for schema validation and custom generic tests for statistical and business rule checks. The key is severity levels: warn for things that should show on the dashboard, error for things that should stop the build.

The unsexy truth about data quality is that the tooling matters less than the discipline. Great Expectations, dbt tests, Monte Carlo, custom Python — they all work. What doesn’t work is bolting on quality checks after the platform is built and expecting them to catch problems that were baked in from the start.

Quality is an architecture decision, not an afterthought.