Data migrations rarely miss their deadlines because of data volume. They miss because of process failures that were visible months earlier — and all five of the signs below can be spotted (and fixed) before they cost you the date. Smart Data Frameworks (SDF) was built to remove every one of them.
| Warning sign | What it really means | How it kills the deadline |
|---|---|---|
| Hand-written COPY scripts per table | Effort scales linearly with table count | Weeks of scripting before any data moves |
| Schema changes surprise you in prod | No drift detection | Every upstream change costs a re-run |
| Nobody knows the real replication lag | No pipeline observability | Cutover window is a guess |
| Every source = a new pipeline | Nothing is reusable | Scope changes add weeks, not days |
| Cutover plan is a Slack thread | Cutover has never been rehearsed | The last weekend fails publicly |
Most migration post-mortems tell the same story. The project didn’t die in month one — it died slowly, from process debt that everyone could see and nobody priced in. The good news is that the warning signs show up early, and they’re specific.
Here are the five we see most often.
You’re Hand-Writing COPY Scripts for Every Table
If every table needs its own hand-written extraction script, your timeline scales linearly with table count — and your estimate almost certainly didn’t. A 2,000-table warehouse at even half an hour per script (writing, testing, fixing the quoting) is a thousand engineer-hours before a single row has moved.
Per-table scripts carry three costs beyond the raw hours:
- They hard-code dialect quirks. Every source database does extraction differently — Netezza doesn’t even have a COPY command — so scripts written against Postgres assumptions fail outright, and the workarounds get pasted into every script that follows.
- They’re write-once artifacts. The moment a column is added or retyped upstream, the script is wrong — and nothing tells you.
- They have no consistent restart logic. When script 1,400 of 2,000 dies mid-run, can you resume from the failure point, or does the whole batch re-run? If each engineer answered that question their own way, you have 2,000 different answers.
What a healthy project looks like: the migration is defined once — source, target, mappings, types — and the per-dialect extraction is generated from that definition. Adding a table is a config change, not an engineering task.
SDF works exactly this way: one definition, every dialect quirk handled at the framework layer.
Schema Changes Keep Surprising You in Production
If you learn about upstream schema changes when a load fails — or worse, when the numbers look wrong — you have no drift detection, and every change costs you a full re-run.
Schema drift during a migration isn’t an edge case; it’s a certainty. Source systems don’t freeze because you’re migrating them. A six-month migration will absorb dozens of upstream releases, and some of them will rename a column, change a type, or add a field mid-run.
Drift shows up in three ways, in decreasing order of luck:
- Hard failure. The load errors immediately. Annoying, but honest.
- Silent misload. Columns shift or coerce, the load “succeeds”, and the corruption is found weeks later during validation — after everything downstream of it has been built.
- Partial drift. Some tables picked up the change, some didn’t. Reconciliation becomes archaeology.
The deadline cost isn’t the fix — it’s the re-run. Discovering drift at load time means re-extracting, re-loading, and re-validating everything the change touched. Do that three or four times across a large estate and there goes your quarter.
What a healthy project looks like: drift is detected at the source, at definition level, before data moves. The pipeline pauses or adapts, the mapping is corrected once, and downstream stays clean.
SDF treats schema-drift detection as a first-class feature, not a post-incident review item.
Nobody Can Tell You the Real Replication Lag
If nobody on the team can quote your replication lag in seconds and rows — right now, without going away to write a query — your cutover date is a guess.
Cutover planning is delta arithmetic: how far behind is the replica, how fast does it catch up, and how long must the freeze window be for the final sync to converge? All three depend on lag being a number someone actually watches. In practice, lag grows silently. A replication slot backs up, throughput drops below the source’s change rate, and nobody notices because nothing fails — the pipeline is running, it’s just losing ground every hour.
You can get the raw number from the database if you know where to look:
-- Postgres: bytes of WAL each replication slot is holding back
SELECT slot_name,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag
FROM pg_replication_slots;But a query an engineer could run is not observability. Observability is when lag, throughput, and rows-moved are on a screen someone looks at daily, with history — so “lag doubled this week” is a fact you act on, not a surprise you excavate on cutover weekend when the “final sync” turns out to need three more days.
The standup test: ask what the lag is right now and what it was yesterday. If the answer starts with “let me check”, you don’t have it.
SDF puts lag, rows moved, and pipeline throughput up front — real numbers, on demand.
Every New Source Means Building Another Pipeline From Scratch
If adding a source system means building a new pipeline rather than adding a definition, any scope change will blow up the plan — and migration scope always changes.
Somewhere around month two, someone discovers the CRM database also needs to come across. Or a regional instance. Or the finance system that “wasn’t in scope” until an auditor said otherwise. In a healthy project that’s days of work; in a pipeline-per-source project it’s weeks, because each new pipeline re-implements the same machinery from scratch: type mapping, quoting, chunked extraction, retry handling, monitoring.
This is the classic N×M problem — N sources, M targets, and hand-built plumbing for every pair. The engineering cost isn’t even the worst part; the worst part is inconsistency. Each pipeline handles NULLs, booleans, and timestamps slightly differently, and those differences surface as reconciliation failures at exactly the wrong moment.
What a healthy project looks like: sources and targets are interchangeable endpoints of one framework. The dialect layer handles each system’s quirks; the pipeline logic is shared. Source #7 costs roughly what source #2 did.
SDF is multi-dialect by design — Netezza, Postgres, Snowflake, Databricks and more — so “define the migration once” isn’t a slogan, it’s the architecture.
Your Cutover Plan Is a Slack Thread
If the cutover plan lives in a Slack thread, it hasn’t been rehearsed — and cutover is the one stage of a migration you can’t quietly retry.
A migration can run smoothly for six months and still fail in public on one weekend, because cutover concentrates all the risk: the freeze window, the final delta sync, the reconciliation, the switch, and every stakeholder watching. A real cutover plan is a document with:
- A freeze window with an owner and a communicated start time
- A final-sync estimate derived from measured lag (see sign #3), not optimism
- Reconciliation criteria — row counts and checksums per table, with named sign-off
- A rollback trigger — the specific condition under which you abort, decided before the weekend, not negotiated at 3am
- A rehearsal — the plan has been run end-to-end against a staging target at least once
If any of those live only in chat scrollback, they don’t exist under pressure. The teams that cut over cleanly are the ones for whom the production run is the boring second performance.
SDF’s reconciliation and sync verification make the checklist executable rather than aspirational — counts, sync state, and lag are read off the platform, not assembled by hand at midnight.
The Pattern: These Are Tooling Symptoms, Not People Problems
All five signs have the same root cause: the migration is being run as a pile of scripts rather than as a system. Scripts don’t scale with table count, don’t notice drift, don’t report lag, don’t generalise across sources, and don’t rehearse cutovers. Systems do.
SDF (Smart Data Frameworks) was built by engineers who have spent 20+ years running large-scale migrations. One definition drives every dialect; schema drift is detected before it corrupts a load; lag and throughput are first-class numbers; batch and CDC run on the same engine; and reconciliation is built in — so the cutover plan is a report, not a Slack thread.
If any of these five signs look familiar, talk to the team — the earlier they’re fixed, the cheaper the fix.
