Now in development. Join the early access waitlist.
SDF is a self-hosted, Docker-native platform for moving data between databases, warehouses, and cloud stores. Define a job once. Run it on a schedule. Community edition is free.
The original SDF was a professional services engagement. The new SDF is a product you run yourself: pull a Docker image, connect your databases, and start moving data in minutes. No account is required for the Community edition. Enterprise support for Netezza, Oracle, Change Data Capture, and bulk transfer is available by licence.
PostgreSQL, MySQL, MSSQL, Oracle, IBM Netezza, Databricks, Redshift, Snowflake, S3, GCS, and CSV/Parquet/Avro files. Community covers the essentials; Pro and Enterprise add cloud stores and legacy systems.
Diffs source against a stored baseline before every run. Classifies changes as safe or destructive and applies them automatically per a configurable policy, with no manual ALTER TABLE scripts required.
Available across SDF’s supported databases, rolling out per engine. Detects inserts, updates, and deletes automatically, with no watermark column required. Netezza shipping now; additional engines on the roadmap. Enterprise edition.
Native protocols per engine: PostgreSQL binary COPY, Netezza external table unload, Oracle array binding, MySQL LOAD DATA. Orders of magnitude faster than row-by-row INSERT. Enterprise edition.
Pick from predefined frequencies (every 5/15/30 minutes, hourly, every 6h/12h, daily) or enter a custom cron-format schedule. No external scheduler required. Pro and Enterprise editions.
Per-table row counts, progress bars, step-level logs, and a complete run history. Know exactly what moved, when it moved, and why it stopped. All editions.
Every edition is self-hosted. Compare the capabilities included with Community, Pro, and Enterprise.
| Capability | Community | Pro | Enterprise |
|---|---|---|---|
| PostgreSQL, MySQL, MSSQL | ✓ | ✓ | ✓ |
| CSV, Parquet, Avro files | ✓ | ✓ | ✓ |
| Full execution history | ✓ | ✓ | ✓ |
| Schema drift detection | ✗ | ✓ | ✓ |
| AWS S3 / Google Cloud Storage | ✗ | ✓ | ✓ |
| Databricks, Redshift, Snowflake | ✗ | ✓ | ✓ |
| Incremental sync (watermark) | ✗ | ✓ | ✓ |
| Built-in job scheduling | ✗ | ✓ | ✓ |
| Oracle / IBM Netezza | ✗ | ✗ | ✓ |
| Change Data Capture | ✗ | ✗ | ✓ |
| Bulk transfer (native protocols) | ✗ | ✗ | ✓ |
| Cloud-staged transfers (S3/GCS, Snowflake via S3) | ✗ | ✗ | ✓ |
| Licence required | No | Yes | Yes |
All editions are self-hosted. No data leaves your infrastructure.
A completely new, self-hosted SDF is in development: visual schema discovery, mapping and conversion, high-performance load and unload, log-based Change Data Capture, and a full pipeline builder with scheduling and monitoring, across every supported platform. Join the waitlist to get early access and help decide which capabilities ship first.
Takes under a minute – just your name and email; everything else is optional. You will get an immediate confirmation, and we will notify you the moment the new SDF is available for your platform.
Your data is processed by Smart Associates Limited in accordance with our Privacy Policy and will never be shared with third parties.
Want to discuss editions or pricing?
Frequently Asked Questions
SDF (Smart Data Frameworks) is a self-hosted, Docker-native data migration platform. You define a job – source, target, tables, and an optional schedule – and SDF handles the rest: schema creation, incremental sync, schema drift detection, and bulk transfer. It runs entirely inside your infrastructure with no data leaving your environment.
Community edition covers PostgreSQL, MySQL, MSSQL, CSV, Parquet, and Avro files. Pro adds AWS S3, Google Cloud Storage, Databricks, Redshift, and Snowflake. Enterprise adds Oracle and IBM Netezza, plus native bulk transfer protocols and Change Data Capture.
Community is free with no licence required and covers the most common open databases. Pro adds cloud warehouses (Databricks, Redshift, Snowflake), S3/GCS, schema drift detection, incremental sync, and built-in job scheduling. Enterprise adds Oracle, IBM Netezza, Change Data Capture, bulk transfer via native protocols, and cloud-staged transfers including Snowflake via S3. Pro and Enterprise require a licence key.
Yes, fully self-hosted. SDF runs as a Docker container in your own infrastructure. Your data moves directly between your source and target systems; it never passes through Smart Associates servers or any third-party cloud. The licence validation call is the only outbound network request SDF makes, and that transmits no data.
SDF has a built-in scheduler, with no external orchestrator, Airflow, or cron daemon required. You pick from predefined frequencies (every 5, 15, or 30 minutes; hourly; every 6 or 12 hours; daily) or enter a custom cron-format expression for precise control. Scheduling is a Pro and Enterprise feature.
Schema drift detection diffs your source schema against a stored baseline before every run. If columns have been added, removed, or had their types changed, SDF classifies each change as safe or destructive and applies them automatically according to your configured policy, without manual ALTER TABLE scripts. This prevents silent failures when upstream teams change source tables. Schema drift detection is a Pro and Enterprise feature.
Change Data Capture (CDC) captures only the rows that have changed since the last run – inserts, updates, and deletes – without requiring a watermark column. It is an Enterprise edition feature available across SDF’s supported databases, rolling out per engine. Netezza CDC is shipping now; additional engines are on the roadmap.
SDF is self-hosted and data-sovereign: your data never leaves your infrastructure. Airbyte and Fivetran are cloud-hosted SaaS connectors; Matillion requires a cloud data warehouse. SDF is built for organisations with on-premises or private-cloud databases, strict data residency requirements, or legacy engines (Netezza, Oracle) that cloud ETL tools do not support well.
SDF ships as a Docker image. Pull it, supply a configuration file with your connection strings, and start the container. There is no agent to install on source or target databases, no firewall exceptions beyond normal database port access, and no cloud account to create. Community edition requires no licence key at all.
Join the waitlist on this page to be notified when SDF is available. Once it launches, signing up is entirely self-service – no sales call required.
Yes. Community edition requires no licence key, no account registration, and no usage reporting. You pull the Docker image and run it. The only limitation is the connector set: Community covers PostgreSQL, MySQL, MSSQL, and flat files. Pro and Enterprise connectors (cloud warehouses, Oracle, Netezza, CDC) require a licence.
Community edition is supported via public documentation and community channels. Pro licences include email support with a business-day response SLA. Enterprise licences include priority support and access to Smart Associates engineers who have run SDF in production environments. Contact us to discuss your requirements.
