How to Optimally Resource your Netezza Team
IBM’s Netezza Performance Server systems are incredibly fault tolerant and are known to require far less effort to administer than a traditional RDBMS. This can, however, lead to a false sense of security, and so when DBA resourcing levels are reviewed, some businesses will attempt to reduce costs by handing over the day to day running of Netezza to a non-specialist team. The problem with doing this is that when unforeseen hardware, software, systemic, or environmental problems occur, a lot of head scratching takes place and what was once the well-behaved kid in the corner becomes the problem child that needs a lot of attention.
Why is this? Well, Netezza is a complex integrated system containing many distinct pre-configured parts which traverse traditional departmental boundaries. It is an appliance that uses a proprietary architecture called Asymmetric Massively Parallel Processing (AMPP) that combines open blade-based servers and disk storage with a proprietary data filtering process using field-programmable gate arrays (FPGAs). The key word there is proprietary, and most DBAs can’t be expected to know the first thing about what settings and standards and procedures they need to follow to ensure the system operates at maximum performance.
It’s not just the proprietary nature of Netezza that requires specialist knowledge, but also the requirement to keep the system at high availability. Netezza systems are usually mission critical for businesses, often with Board level visibility. They are responsible for managing large volumes of commercially sensitive data, subject to a variety of different concurrent users and workloads, some of which are extremely performance dependent, and all of which are contending for a finite set of resources. So, making sure you have the right skills on hand is pretty important as downtime is not acceptable.
A Case of Not Knowing What You Don’t Know
An age-old conundrum when managing a sophisticated appliance is knowing what you don’t know, and that is where augmenting your Netezza resources with on-demand skilled ones is paramount. It may sound obvious and even outdated to bring this up, but we know of a client who handed their Netezza administration over to their dedicated Unix team who took very little time to bring the system to its knees. 
Shortly after the handover their Unix team looked at the Netezza box and said something along the lines of ‘this is terrible, it’s running a very old version of Red Hat, it's not patched against all the latest security vulnerabilities. We’ll just upgrade it to the latest version of Oracle Linux so that all our platforms are standardised”. The problem is they didn't realise that Netezza is an appliance and had been set up that way with a Postgres database acting as a repository for all the metadata. So, they steamrolled over the whole thing and broke it. The Unix team did not know that the only way to upgrade Netezza is by using IBM authorised patches and upgrades that they have tested to make sure don’t compromise the system in any way.
Operating system aside, danger lurks when non-Netezza DBAs assume responsibility for Netezza. The problem here is that in the day to day running of Netezza, Oracle DBAs, for example, will do Oracle-type things and they won’t know to do the Netezza-type things to keep the system running at peak performance. Over time, because critical Netezza tasks are not performed, the performance deteriorates. So, the problem is actually magnified and you could argue that the more Oracle DBAs you throw at the problem, the worse it can become and eventually you have to call in the experts anyway.
A Broad Spectrum of Team Structures
There are many different resourcing models out there that will determine how much Netezza expertise exists on-site. At one end of the spectrum are companies whose operational structure is for highly compartmentalized teams that assume responsibility for different aspects of the system e.g. Linux, Network, Database, Operations, Security, Backup and Recovery. With this setup, you will have teams of people who are responsible for very discreet areas, none of which can be expected to be Netezza experts, but they may still all get involved to solve a Netezza problem which comes at a cost, especially if they are not actually able to contribute anything to the solution.
At the opposite end of the spectrum are companies who have a dedicated Netezza team. However, providing a whole team of FTEs is a costly exercise for a system that is 99% plus available. And as soon as you start diluting that such as relying on just one or two Netezza specialists, those guys have to sleep, have to go on holiday, do training, and sometimes gets sick. So, when any of those things occur basically the whole system is put at risk.
Our Recommendations for Ensuring Sufficient Netezza Expertise
Clearly neither extreme is optimal. They have their own relative advantages and disadvantages and so therefore our recommended approach is to have a single team responsible for the Netezza platform overall, and this team should consult with other internal functional teams for best practices and in-house standards to be implemented, as well as define communication and workflow interfaces between the teams.
We suggest automating as many systems management functions as possible - e.g. backups, groom tables/versions, genstats, manual vacuums, AD/LDAP synchronization for users, groups, and permissions and monitoring and alerting, etc.
Furthermore, to overcome the problem of needing 24x7 availability of your mission critical system, we suggest augmenting your in-house resources with external ‘experience on- demand’ type services to provide 24x7 cover and assist with diagnosing/resolving any issues that arise.
The Pitfalls of DIY Automation
Quite a lot of the tasks that are routinely done by Netezza DBAs can be automated and many companies have done just that. Herein lies another problem, though. If the automation tasks are written to perform on the Netezza host, as they usually are, then as soon as the customer wants to upgrade to the NPS Cloud Pak for Data system, any process they have running on the host currently will have to be migrated off to somewhere else. So that is a pretty significant effort that may have to be repeated each time a major system upgrade occurs. Our solution to that is Smart Management Frameworks because all the automations and housekeeping tasks have already been taken off the host so regardless of what changes are made to the architecture, nothing is going to fall over.
Of course, automation can't do everything and if for example you've got a performance problem you will need someone to investigate what the underlying cause of the problem is. It could be badly written SQL, a poor database design or workload management settings that need to be tweaked to alter the priority of different queries being run by different users within the organization. That type of investigative work is almost impossible to automate, as is formulating a plan on how to fix it. For that reason, we have our Virtual DBA offering which provides performance tuning, workload management, query optimization and so forth on a 24 x 7 basis. So rather than paying round the clock for Netezza resources, this service allows customers to augment their in-house team for such occurrences and/or when their dedicated resources are rostered off.
Reaching the Best Possible Outcome
To summarize, Netezza is a complicated system that requires very niche skills to keep running at peak performance, especially now that it is hosted on IBM’s Cloud Pak for Data System with its containerized, modularized architecture. Retaining highly trained resources in a niche field has always been expensive, which is why we have developed a suite of products and services that allow companies to augment their Netezza skills with automation tools and on-demand expertise.
Furthermore, if you want to migrate to NPS on your own private cloud, we can supply you with a bundled solution that includes not just the NPS software all installed and configured, but also migration to the new platform; safety, security, and surety automation via Smart Management Frameworks; and Virtual DBA support all included.
If you’d like to find out more about this, or how to optimize your Netezza team structure, please get in touch.