The dataframe world went columnar and never looked back. Polars, DuckDB, Arrow Flight, the whole modern analytics stack moves data in Apache Arrow’s columnar format and processes whole columns at once. Netezza, an appliance built for exactly this kind of set-based analytics, has been oddly hard to feed into it. The standard path is a DB-API cursor handing back one Python tuple per row, which you then reassemble into a dataframe: millions of throwaway Python objects, built and torn down, purely as a translation tax between two systems that both think in columns.
netezza-arrow removes the tax. It runs your query and materialises the result directly into an Apache Arrow table, typed per column, ready for the columnar tools that expect it.
IBM does publish an Arrow ADBC driver for Netezza (IBM/nz-adbc), and if you already run the IBM client stack it is worth a look. It links libnzpq from IBM’s client kit and ships as a C++ project you build from source with submodules, so adopting it means installing the IBM client on every machine that needs it and adding a compile step to your build. netezza-arrow takes the other route: pure Python on our own wire-protocol driver, nothing from IBM to install, and no build step.
Typed, not stringly
The point of Arrow is types, so the package maps Netezza’s types to Arrow’s properly: integers to sized integer columns, doubles to float columns, and crucially NUMERIC to Arrow’s exact decimal128 rather than a lossy float. Booleans, dates and timestamps become their native Arrow types; character data stays UTF-8. A SQL NULL becomes an Arrow null in any column, which our live test checks with a null in a numeric column. From there the data is genuinely columnar: to_polars hands it to Polars with no copy, and it feeds Arrow Flight or any Arrow-native tool directly.
DuckDB over Netezza, in-process
The combination we are most pleased with is DuckDB. Because Arrow and DuckDB share memory with no copy, netezza-arrow can pull a query into Arrow and register it into a DuckDB connection, so you then run DuckDB SQL – joins, window functions, aggregations, the lot – against Netezza data locally, in-process, at DuckDB’s speed. Our live test pulls a sales table from both a classic NPS 7.2 rack and a current NPS 11.2 system and runs a grouped aggregation in DuckDB over it, getting exact decimal sums with nulls correctly excluded. For the “pull a working set out of the warehouse and slice it locally” pattern that data engineers run all day, this is the whole workflow in three lines.
Built on the async driver
None of this needed a new protocol implementation. netezza-arrow sits on aionetezza, our pure-stdlib async driver, and adds only the Arrow typing layer. That is the dividend of having built the foundation first: each new integration is a thin, well-scoped layer, not a fresh implementation of the wire protocol.
Honest scope
The roadmap calls for a full native DuckDB scanner extension in C++ that speaks the wire protocol from inside DuckDB itself; that is a larger piece of work and remains ahead of us. What ships today reaches the same practical destination – DuckDB SQL over Netezza data – by way of Arrow registration, with nothing to compile. Polars is an optional dependency and the bridge to it is a single Arrow conversion. Numeric columns default to a wide decimal scale and fall back to lossless text on the rare value that overflows it, a refinement we will tighten by reading the column modifier. PyPI publication is coming soon.
Two systems that both think in columns should not translate through rows.
Where you get it
We built this for the Netezza community rather than as a product of our own. If your Polars or DuckDB pipeline currently starts with a slow row-by-row Netezza fetch, this is the columnar shortcut. Talk to us if you want it in your own estate.
Built from the published open source Netezza drivers (IBM/nzpy and IBM/nzalchemy under Apache-2.0, IBM/nzgo under MIT), verified live against both NPS 7.2 and NPS 11.2, and produced without decompiling, disassembling or inspecting any IBM proprietary binary.
Running Netezza? Let’s talk. Smart Associates has been engineering on IBM Netezza for two decades: disaster recovery, migrations, performance and the deep platform work in between. If any of the above sounds familiar, why not get in touch.
