CDC has a reputation it doesn’t quite deserve: because it’s “always on”, people assume it’s always right. It isn’t. None of the four failure modes below throw an error. Every one of them shows up as green on the pipeline’s own dashboard right up until someone else finds the damage, on the source disk, in a finance report, in a schema mismatch nobody flagged. Smart Data Frameworks (SDF) was built to make each of these visible before it costs you.
| Silent killer | What’s actually happening | How it costs you |
|---|---|---|
| A replication slot nobody drops | WAL keeps accumulating for a consumer that no longer exists | The source database’s disk fills, not the pipeline’s |
| Lag that grows in silence | Throughput trails the change rate and nobody’s watching the trend | The gap never closes, it just gets normalised |
| Schema drift corrupting the stream | A column changes upstream and the stream decodes against the wrong schema | Values shift or miscast with no error thrown |
| A snapshot that never reconciles | The snapshot-to-stream handoff isn’t watermarked precisely | Rows duplicate or vanish, found only when the numbers don’t match |
Batch jobs fail loudly. They time out, they throw a stack trace, someone gets paged. CDC pipelines mostly don’t. They keep running, keep reporting “healthy”, and keep quietly drifting from the truth in the background, which is exactly why the failures below are so expensive: nobody’s looking for them because nothing told them to.
Here are the four we see most often.
A Replication Slot Nobody Ever Drops
If a CDC pipeline is decommissioned or replaced without dropping its replication slot, the source database starts accumulating WAL for a consumer that no longer exists, and it won’t stop on its own.
This is how logical replication slots work, by design: the source keeps every write-ahead log segment behind the slot’s confirmed position until a consumer reads and acknowledges it. That’s correct behaviour when a consumer is temporarily offline. It’s a slow-motion outage when the consumer is gone for good and nobody told the source about it. WAL keeps growing, disk keeps filling, and the primary database (not the CDC tool) is what eventually falls over.
You can see this coming with one query:
-- Postgres: WAL retained behind each replication slot
SELECT slot_name,
active,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;The failure mode we see most often: a pipeline gets rebuilt or migrated to a new tool, the old slot is left in place “just in case”, and six weeks later the on-call DBA is paged for a full disk with no idea a CDC project from two months ago is the cause. Nothing about the incident points at the pipeline, because the pipeline’s own dashboard never saw a problem. It was watching the target, not the source.
What a healthy setup looks like: the slot’s lifecycle is owned by the same definition that owns the pipeline. Decommission the pipeline, drop the slot, automatically, not as a manual step on a runbook someone might skip.
SDF ties replication slot lifecycle to the pipeline definition, so retiring a CDC pipeline retires its slot with it.
Lag That Grows Because Nobody’s Watching the Trend
If your CDC pipeline’s throughput doesn’t clear the source’s peak change rate with margin to spare, lag doesn’t fluctuate, it accumulates, and every day nobody looks is a day further behind.
There’s a difference between lag that spikes and lag that grows, and most monitoring only catches the first one. A month-end batch job on the source triples the change rate for six hours. A properly sized pipeline absorbs that and catches up by evening. An undersized one doesn’t quite catch up, carries a small deficit into tomorrow, and repeats the pattern at the next month-end. Nobody notices for months because the absolute lag number still looks tolerable, right up until it isn’t.
The number worth watching isn’t lag itself, it’s the trend: lag(today) minus lag(a week ago). One spike that recovers is normal operation. A week of consistently positive deltas is a capacity ceiling, and it will not fix itself by waiting.
What a healthy setup looks like: lag and its week-over-week trend are both on a dashboard someone actually looks at, with an alert on the trend, not just the absolute value. Capacity planning is sized to the change rate you’ve actually observed at peak, not the one estimated on day one of the project.
SDF surfaces lag as a live number with history, so a trend that’s quietly getting worse is a fact on a screen, not something you reconstruct after someone asks why a report looks stale.
Schema Drift That Corrupts the Stream Without Ever Failing
A CDC connector that decodes row changes against whatever schema is current right now, rather than the schema that was active when the change was captured, will silently shift or miscast values the moment a column is added, dropped, or retyped upstream, and it will not throw an error while doing it.
This is the failure mode people find hardest to believe until they’ve seen it. Add a column in the middle of a table upstream, and a capture mechanism that isn’t schema-version-aware can shift every value one column to the left for every row processed afterwards. Drop a column, and downstream fields quietly go null instead of raising a “column not found.” Nothing crashes. The pipeline keeps reporting success, because as far as it’s concerned, nothing went wrong.
The tools that get this right (Debezium’s schema history is the reference example) version the schema against the log position, so a change captured at a given LSN or binlog offset is always decoded against the schema that was live at that exact point, not whatever the schema looks like today. That distinction, decode against the schema at capture time versus decode against the schema right now, is the entire difference between correct CDC and a slow leak of bad data.
What a healthy setup looks like: DDL changes are captured as events in their own right, versioned alongside the data changes, and applied to the mapping before the next row is decoded, not discovered afterwards during a data quality review.
SDF decodes every CDC event against the schema version active at capture time and flags drift as it happens, instead of quietly reinterpreting old rows against today’s schema.
A Snapshot That Never Gets Reconciled Against the Source
Every CDC pipeline starts with a full snapshot and hands off to streaming at some boundary, an LSN, an SCN, a timestamp, and if that boundary is off by even one transaction, you get a duplicate row or a silent gap, and neither one announces itself.
The handoff is the part people underestimate. If the snapshot is taken with one tool (a database export, a bulk copy) and streaming picks up with a different tool that wasn’t watching the same consistency point, a transaction committed during the transition can land in both, or in neither. One row out of position on day one is invisible. It’s still invisible on day ninety, when it’s not one row anymore, it’s a few thousand, spread across enough tables that nobody’s going to eyeball their way to finding it.
The only way this gets caught early is a reconciliation job that actually runs: row counts and checksums, source against target, on a schedule, with an alert on divergence past a small tolerance. Not a one-off validation at go-live. A recurring one, because drift from a handoff bug compounds quietly for as long as nobody’s counting.
What a healthy setup looks like: the snapshot and the stream share one engine and one consistency watermark, so there’s no seam for a transaction to fall through, and reconciliation is a scheduled job, not a favour someone does before a big meeting.
SDF’s snapshot and CDC stream share a single consistency watermark, and reconciliation runs on a schedule, not when someone finally asks whether the numbers match.
The Pattern: CDC Fails Quietly, Not Loudly
None of these four throw an error. That’s the thread connecting all of them, and it’s also why CDC earns more trust than it’s owed: a pipeline that’s “always on” looks healthy by default, and proving otherwise takes someone actively going looking, on the source database, in a reconciliation report, in a schema diff nobody ran. Most teams don’t go looking until something downstream is visibly wrong, which is usually months after the drift started.
The same discipline applies whether you’re running CDC in steady state or relying on lag numbers to plan a migration cutover: if lag, drift, and reconciliation aren’t first-class numbers someone can quote on demand, you don’t have observability, you have a pipeline that hasn’t failed yet.
SDF (Smart Data Frameworks) treats each of these as a solved problem rather than an operational risk: slot lifecycle tied to the pipeline definition, lag and its trend surfaced live, schema drift decoded correctly at capture time, and reconciliation running on a schedule against a shared consistency watermark. Batch and CDC run on the same engine, so none of this is a bolt-on.
If any of these four look familiar from your own environment, talk to the team, the earlier a silent failure is found, the cheaper it is to fix.
