If you connect to IBM Netezza from Python, there is a good chance every euro sign in your CHAR and VARCHAR columns has been coming back as ¤. Not on some queries. On every single read, since the day you first installed the driver.
We found this while hardening the Netezza integration inside our SMF platform, and it turned out to be one of four serious defects in the standard open-source Python driver (nzpy). We have fixed all of them in an enhanced build, and we validated the result against three different appliance generations, including a physical multi-rack system. This post explains what was broken, why nobody noticed, and what the fixed driver now does that the standard one cannot.
The character set problem
Netezza stores CHAR and VARCHAR columns in LATIN9 (ISO 8859-15). The standard Python driver decodes them as Latin-1. Those two encodings agree on almost everything, which is exactly why this bug survived for years: plain English text round-trips perfectly. But eight code points differ, and they are not obscure ones. The euro sign is among them.
stored on the appliance: €uro Š ž œ Ÿ
returned to Python: ¤uro ¦ ¸ ½ ¾
Every application reading that data received the corrupted version. Reports, exports, downstream loads: all silently wrong, with no error raised anywhere. Anyone who has run a data platform through a merger, a migration or a regulatory audit knows what silent corruption of currency symbols and European names eventually costs.
Character set conversion is a subject we have history with. Our engineers solved exactly this class of problem for a major European bank years ago, back when cross-code-page loading was one of the hardest parts of getting data onto an appliance, and it was tested to destruction with Cyrillic, Arabic, Hebrew and East Asian scripts. So when we rebuilt the Python path, we held it to that standard. The enhanced driver decodes CHAR and VARCHAR as the LATIN9 the appliance actually stores, and we verified byte-exact round-trips for Cyrillic, Arabic, Hebrew, Japanese, Chinese (simplified and traditional), Korean, Greek, Danish, mixed right-to-left text, combining accents and even four-byte emoji, in both directions, through every path an application can take: single-row inserts, parameterised queries, lookups and high-volume bulk load and unload. We also proved the results are independent of the appliance’s host locale by changing it and restarting the system between test runs.
Two appliance behaviours worth knowing about, whichever driver you use: Netezza silently replaces any character that does not fit LATIN9 with
?when it lands in a CHAR/VARCHAR column (use NVARCHAR for international data), and it rejects UTF-8 that is not NFC-normalised. Both are server-side rules, and both are the kind of thing you want to learn from a blog post rather than from production.
Bulk data movement that actually works everywhere
Netezza’s external tables are the fastest way to move serious data volumes in and out of an appliance, and the Python driver is supposed to support them without any ODBC or JDBC installation. In practice, it did not work at all on the NPS 7.x systems that still make up a large share of the installed base: the appliance rejects the syntax the driver requires. Rewrite your SQL for the old syntax and it breaks on NPS 11. There was no single statement that worked on both.
The enhanced driver resolves this the way a driver should: it detects the appliance generation once per connection and speaks whichever dialect that system accepts. Application code is written once and runs unmodified across a mixed estate of classic TwinFin-era racks and current CP4D deployments. We verified full round-trips (unload to a client-side file, reload, compare byte-for-byte) on both generations, in delimited text and in Netezza’s compressed binary format, with the full international character matrix above.
Parameterised queries: broken, and unsafe
Two more defects sat in the same code path. First, parameterised queries (the standard ? placeholder style every Python developer uses) simply crash against NPS 7.x systems. Second, and worse: the standard driver performs no quote escaping when it substitutes parameters. A value as ordinary as O’Brien produced broken SQL, and the same gap is a textbook SQL injection vector in any application that passes user-supplied values as parameters, which is precisely what parameters are for.
The enhanced build fixes both. Parameterised queries now run against every appliance generation, and values are escaped correctly, verified with hostile inputs as well as ordinary names. If your Python code touches Netezza with user-facing input anywhere upstream, this fix alone justifies the swap.
And the things you only hit at 2am
A handful of smaller fixes round out the build, each one first reported by users of the standard driver and left unresolved upstream: connections failing outright inside OpenShift and Kubernetes containers (the driver assumed the process user always has a passwd entry, which is not true under arbitrary-UID security policies), catalog timestamp columns quietly reporting a different column’s timestamp (reported upstream in March 2022 and still open four years later), and the driver being uninstallable on the Python versions that enterprise Linux distributions actually ship. All fixed. All boring. All exactly the kind of thing that takes down a scheduled job at the worst possible moment.
Performance: measured, not assumed
A driver fix that slows down your nightly loads is not a fix. We benchmarked the enhanced build against the standard one with interleaved test rounds on both a virtualised NPS 7 system and a physical appliance, across short-query loops, hundred-kilobyte SQL statements and multi-thousand-row fetches. The differences were within measurement noise in every workload, in both directions. Meanwhile the enhanced driver completes an entire class of work (parameterised queries on NPS 7, bulk movement everywhere) that the standard driver cannot run at all.
Write your SQL once, in one dialect, with correct international data and safe parameters, and let the driver handle the appliance generation. That is the whole idea.
Where you get it
The enhanced driver now powers everything our software does against Netezza: disaster recovery, data curation, query history capture and metadata scanning across mixed appliance estates. If you need it in your own estate, talk to us.
If you are running Netezza, and especially if you are running more than one generation of it, your Python integrations are almost certainly affected by at least one of the defects above. The euro-sign test takes thirty seconds: insert '€' into a VARCHAR column, read it back, and see what you get.
Built from the published open source Netezza drivers (IBM/nzpy and IBM/nzalchemy under Apache-2.0, IBM/nzgo under MIT), verified live against both NPS 7.2 and NPS 11.2, and produced without decompiling, disassembling or inspecting any IBM proprietary binary.
Running Netezza? Let’s talk. Smart Associates has been engineering on IBM Netezza for two decades: disaster recovery, migrations, performance and the deep platform work in between. If any of the above sounds familiar, why not get in touch.
