In today's ever-evolving cybersecurity landscape, Database Activity Monitoring (DAM) systems are growing in popularity as a means to help ensure proper governance of sensitive data and compliance with data protection legislation - like for example the EU’s General Data Protection Regulation (GDPR) and the US Health Insurance Portability and Accountability Act (HIPAA).

Failure to comply with such stringent regulations can potentially result in substantial financial penalties (not to mention reputational damage and class action lawsuits), so DAM tools typically offer a variety of features like real-time monitoring, anomaly detection, verifiable audit trails, and comprehensive reporting - all of which help mitigate security risks and ensure regulatory compliance.

Database Activity Monitoring Challenges

Two of the main challenges DAM software vendors face are firstly how to determine the legitimacy of each individual database query, given the sheer number of users, volume of data processed, and variety of objects being accessed for different business purposes at any given moment; and secondly what to do about it when potentially suspicious, illegal, fraudulent, or otherwise undesirable behaviour is actually detected.

Key to solving the first problem is the ability to differentiate between permitted and suspect behavioural patterns. Note that this isn’t simply a permission issue – for example, customer service agents will regularly need to access customer details to perform their duties, and so must therefore have read permissions granted against customer data structures. Those same permissions though could potentially be used to try and exfiltrate a subset of the most valuable customer records- which most definitely should be regarded as suspicious.

So how best to go about ignoring the former but flagging the latter? One way is to construct a series of rules for detecting anomalous behaviour, and then to alert relevant stakeholders when such rules are breached. The downfall of such an approach is that it is predicated on knowing in advance all the possible behaviours that should be considered suspect – but in reality this is an impossible task that will inevitably result in waiting for data breaches to occur before retrospectively trying to plug them (metaphorically locking the stable door after the horse has already bolted).

A far better and more proactive approach could simply be to automatically keep track of the average duration of queries executed by different groups of users, and the average number of records such queries return, and then to flag for further investigation and report any queries by any users within those groups that are a couple of standard deviations away from the mean.

Of course, such a crude statistical analysis technique isn’t perfect either - because setting the standard deviation selection criteria too small could result in too many false positives, and conversely setting it too high could result in too few truly suspicious queries ever being detected.

A slightly more sophisticated approach could therefore be to train a machine learning model (e.g. using an isolation forest algorithm) to learn what constitutes ‘normal’ runtimes and numbers of affected records, and then have it flag any and all queries that are anomalous.

But even then, such an approach doesn’t take into account the sheer variety of query types, joins, predicates etc that are routinely executed against a typical production data warehouse during an average day. To cater for this a yet more sophisticated approach would be required – parsing queries into their constituent parts and representing them in such a way that they can be meaningfully compared against each other using a similarity metric (e.g., Cosine similarity , Levenshtein distance ). That way, if a query is significantly different in terms of structure, content, or meaning from the norm, then it could be flagged as an anomaly.

In truth, a combination of all the approaches described above can be helpful when it comes to effective Database Activity Monitoring – but the thing they all have in common is the requirement to be able to monitor query activity to spot anomalous behaviour. The question then becomes, what is the best way of achieving this? Broadly speaking, there are two competing approaches for monitoring database activity: either via an external component (agent-based) or by using the database system’s own audit mechanism (native logging).

There are pros and cons with either approach. Agent-based DAM requires additional components that have to be separately installed, maintained, and monitored that may conflict with or otherwise impact regular database activities. Conversely, native logging has relatively little additional administration effort, operating overhead, or infrastructural requirements (assuming such logging is routinely enabled in the database environment itself already).

In addition, Agent-based DAM systems tend to be significantly more expensive than native approaches – due to the licensing and maintenance of those additional components – but can be more sophisticated in terms of their ability to proactively and pre-emptively intercept and disrupt database communications when they detect what appears to be potentially malign behaviour. This can often manifest the unfortunate side effect of preventing key business users from being able to run legitimate queries and effectively thwarting them from doing their job.

Our Approach

Smart Management Frameworks (SMF) from Smart Associates on the other hand provides a balanced approach to DAM, data governance, and identity access management (IAM) – leveraging native capabilities where these already exist (for simplicity, efficiency, and low cost), but also providing:

• A separate secure enclave for the tamper-proof persistent storage of operational metadata (which provides an audit log of suspicious database activity for subsequent review),
• The ability to create, maintain, and manage flexible and customisable business rules (including those statistical, machine learning, and syntax-based techniques described above) used to determine what patterns of behaviour could be indicative of a potential breach,
• A configurable alert mechanism to inform key stakeholders of security events warranting further investigation, as well as potentially automating remedial action e.g. forced disconnection, restricting future database logins, and/or limiting query executions for particularly egregious behaviour

The advantages of this approach are:

• No need for the installation and maintenance of costly separate Agent software to intercept all database communications
• No additional bottleneck or overhead that could impact database availability, concurrent throughput, or individual query performance
• No possibility that the DAM can misinterpret and disrupt legitimate benign database access requests
• Protection from a malign actor attempting to ‘cover their tracks’ and remove evidence of unauthorised behaviour after the event by deleting native database log entries
• Supports separation of duties, as database users won’t have access to SMF’s secure enclave
• Leverages native database capabilities, rather than attempting to reproduce them elsewhere

Furthermore, in addition to the SuperGovernance module capabilities described above, Smart Management Frameworks can also optionally simplify and automate granular Database Control Language (DCL) user and group administration and permission synchronisation between one or more Entra-ID (Formerly Active Directory) and/or Lightweight Directory Access Protocol (LDAP) domains across an entire estate of databases using the Smart Access Control IAM module.

The beauty of this approach is that it doesn’t rely on manual processes, which are prone to delays, inconsistencies, and human error. Smart Access Control can also automatically detect and revoke superfluous or invalid permissions that may have accidentally or maliciously been granted to (or by) rogue users, as well as grant additional permissions where necessary if these are missing at the database level.

Executive Summary

• Database Activity Monitoring (DAM) systems are essential for ensuring data security and regulatory compliance.
• DAM tools offer real-time monitoring, anomaly detection, audit trails, and reporting, helping organizations safeguard sensitive information, but they can be expensive and if not properly implemented can cause disruption
• Businesses using platforms like Netezza, Piranha, or Yellowbrick can achieve robust DAM and data governance with minimal investment, simplifying their security and compliance efforts.
• Key challenges for DAM include differentiating between legitimate and suspicious behavior and effectively responding to detected threats.
• An approach that combines statistical methods, machine learning models, and query pattern analysis is recommended for detecting anomalies, but careful consideration on methods and calibration is needed
• Smart Associates’ Smart Management Framework (SMF) offers a cost-effective and balanced approach by leveraging native database capabilities while providing secure, tamper-proof audit logs and flexible business rules for identifying potential breaches.
• SMF reduces the need for expensive agent-based solutions and prevents system disruptions, all while enhancing data governance and identity access management.

It might also interest potential SMF customers to discover they don’t need to invest in separate elaborate and costly backup and recovery solutions either, as these important features are also available in SMF. We’ll discuss this in our next blog.

If you want more information, why not get in touch.