Honing our blue team skills

Cybersecurity is a never-ending game of cat and mouse that is played between attackers (who seek to damage or otherwise misuse information assets) and defenders (who seek to ensure that those assets are only available for use as intended). In this post we’ll summarise some of the ways we are constantly honing our blue team skills.

The yin and the yang, the ebb and the flow

On one side, the attackers constantly work on their Tactics, Techniques and Procedures (TTPs), looking for new ways to identify and exploit vulnerabilities. They want to gain access to their target assets, generally while attracting as little attention as possible. On the other side are the target defenders: they work to keep the information assets secure or at worst, to respond quickly when defences fail and the attackers find a way in.

In this cat-and-mouse contest, the defenders rely on a range of systems and techniques, and one of the main systems in their arsenal is a Managed Detection and Response (MDR) service which often includes or utilises a Managed Security Information and Event Management (MSIEM) service. Everyone has their own ideas regarding the exact meaning of MDR, SIEM, MSIEM and SOC but as far was we’re concerned, MDR/MSIEM provides a complete description of what we’re talking about today: A service that is able to manage information (which is derived from log-event data that is collected from a system) and events (which are observable changes in a system) and in so doing, support reporting and alerting, response-to and containment of threatening and anomalous system behaviour.

Fire and forget... and fail!

Alas, an MDR/SIEM service is not a fire and forget service and in fact it needs constant care and love, making it much less like this and much more like this.

Analogies and cute dogs aside, it’s important to note that it’s not enough to just set up a MDR/SIEM service that imports some previously-identified data and reports on some previously-conceived security issues, and then leave it to continue in that state forever. Why? Because, as we noted above, the attackers are constantly working on revised and new TTPs and so the MDR/SIEM service needs to evolve along-side the new kinds of attacks. Furthermore, the SIEM service also needs to be working with the best and most up-to-date information possible: A monitored environment will (ideally) include people, processes and technology, but people, processes and technology change, come and go over time. It is therefore extremely important that the MDR/MSIEM is “made aware” of those changes as they take place, if the MDR/SIEM service is to have any hope of continuing to operate correctly and optimally over time.

Ultimately therefore, an effective MDR/SIEM service is not something that you buy and then put in some corner of the cloud to magically operate for ever after. If only! No, an effective MDR/SIEM must be tended by experienced security specialists that understand attacker TTPs and who can respond as new TTPs are discovered; experts who are trained in SIEM service deployment, design and implementation; operators who are able to prioritise threats, capture actionable intelligence, create and update detections and alerts, and respond appropriately to notifiable events that are generated by the SIEM service.

And if you’ve read this far, you know what we’re going to say next: DotSec has 12 years of MDR/SIEM experience; we are very aware of what it takes to run an MDR/SIEM service and our security specialists work continuously to ensure that our MDR/SIEM deployments evolve and remain effective, even as the monitored environments (people, processes and technology) change over time. Is this some kind of magic? No, it’s just hard work! Let’s look at some of the work that our MDR/SIEM specialists undertake to keep our clients aware of what’s going on across their computing environments:

Task 1: Devise detections

DotSec’s MDR/SIEM professionals take on a defensive role and so form part of what is commonly referred to as the “blue team”. Of course, the best defenders also know about attacks, so our MDR/SIEM team works with the DotSec “red (or offensive) team” to identify and test TTPs that are commonly used by attackers. DotSec’s red team emulates adversary techniques in a controlled environment, enabling our blue team specialists to develop techniques that will identify threatening and anomalous behaviour in real-world environments in an efficient and timely manner.

In a world where the average time to detect a breach is in excess of two hundred days, the quality of these detections is integral to providing effective SIEM. Writing detections is easy – writing good detections is not. Consider, for example, a recent attacker-emulation exercise that we coordinated with one of our MDR/MSIEM clients. The red team was able to craft and deploy an enumeration tool that the blue team was not able to detect when it was used as part of the exercise. Of course, the customer is part of this whole exercise and so they were well-aware that the MDR/MSIEM system was bypassed by the red team, and they were also aware of the MDR/MSIEM detection-mechanism changes that we made in order to detect such an attack in the future.

But that’s the whole point of collaboration: DotSec works collaboratively with our clients who in turn get to see our work, warts and all. Both parties get to see how each other works and improve their effectiveness as a result.

Task 2: Gather intelligence

DotSec collects threat intelligence from various sources – including security industry partners, vendor vulnerability notifications, and industry news.

An MDR/MSIEM is able to look at events coming in from relevant and in-scope systems, but it also needs to then be told how to look at those events. With thousands (or even millions) of events being processed each day, the MDR/MSIEM system must perform a balancing act, picking out what the system considers to be “interesting” or “unusual” activity without causing undue load. And load is an important factor: It can be load both on the server and on the operators. The wheels will fall off real quick if we bog the system down with searches it cannot complete, and/or if we drown the operator in a sea of false positives.

Identifying what is/isn’t useful information can often be difficult, and not something that can be done effectively alone. DotSec uses numerous sources for this process including industry researchers, partners (such as Splunk’s own research information), vendor notifications and even news, twitter and chat. With the relevant information to hand, DotSec develops detections and alerts that look through those thousands/millions of events for what comprises interesting/anomalous/threatening behaviour… things like attempts to access a certain address on a website, programs being run a particular way, or even one program attempting to interact with another in a specific way. The detection
methods are varied, but the goal is the same: find behaviour/events of interest to the MDR/MSIEM system and its operators.

Additionally, DotSec’s blue team researches, develops, tests and implements new detection techniques, with the goal of identifying the any activity of interest which may have gone unnoticed before. This gives the customer confidence that they have not been historically compromised by any new or ‘zero-day’ attacks. By way of example, DotSec was able to define mechanisms to detect and alert on the exploitation of the vulnerability known as Print Nightmare, before detections were made available by our clients’ SIEM vendors. DotSec’s experts figured out how to do this by investigating threat intelligence sources, and our newly-created detection was able to alert on exploitation of the Print Nightmare vulnerability in our client’s systems.

Task 3: Proactively improve

Customer environments continuously evolve, and those changes needs to be reflected in changes to the MDR/MSIEM configuration. What are some kinds of common changes? Things like:

Asset and identity registers containing the list of each user and host used by the organisation.
Changes in criticality of assets that are owned by the organisation.
Changes to third party services such as email providers and cloud-based applications.

Proactive improvement also provides other benefits such as:

Improving DotSec’s understanding of customer environments. DotSec collaborates with clients to actually understand how their organisation operates. Improving understanding also goes beyond updating lists and scanning for new devices – it includes collaboration with client stakeholders to discover systems that may not have been originally considered, looking at current systems design now and how that design will change in the future. Without this understanding, vital logs could be missed, and detections reliant on the priority of a system or user will not trigger an appropriate alert.
Identifying security risks and opportunities for improvement: An effective MDR/MSIEM turns raw logs into deep insights, helping to uncover subtle trends and misconfiguration that can go unnoticed and undetected. As an example (from our experience with one of our clients) while threat hunting, our specialists detected an anomaly in which users were able to bypass MFA while authenticating with a popular IaaS solution. With the help of the MDR, the root cause was identified as a misconfiguration in the access policies, and was subsequently rectified by our client.
Identifying a baseline for activity within an environment. An action that is completely benign in one environment may be unauthorised activity in another. For example, we expect users in a software development company to be running the occasional script and using system tools, but such activity would be atypical in an accounting firm. Identification of this information needs to be done carefully so as to not dismiss suspicious and potentially-threatening activity.
Uncovering previously unknown potential attack vectors. A good MDR/SIEM can not only detect attacks after they have happened, it can also detect the presence of an attack vector that the organization was previously unaware of. A good example of this (another client this time!) was when the DotSec MDR/MSIEM was able to detect the presence of brute-force attack vector in a client systems that was exposed to the internet; the MDR/MSIEM identified large numbers of failed authentication attempts (probably generated from a script like this one https://github.com/dejanlevaja/owa-brute-forcer-2) and from there we could determine that MFA was not in place and could advise the client accordingly.

The detection mechanisms used by SIEM also undergo regular testing. This testing helps to ensure that detections are accurate and that as much activity of interest is detected, while also preventing alert fatigue for both the customers and DotSec.

A quick case study!

Another example to summarise what we wrote about above! Some time ago, we responded to a security incident for a large national retailer with a large on-line presence and over 70 physical stores across the country.

Our first goal was to determine the cause and extent of the compromise, and to then contain it as quickly as possible. In order to achieve these goals, we needed to get a holistic picture of what was going on by collecting as many logs from as many systems as possible. We therefore set up a new MDR/MSIEM instance (something we automate with Ansible and CloudFormation; the whole setup takes about 3 hours) and collected what evidence we could from the affected systems

Although we were originally tasked with investigating reports of credit card fraud, our MDR/MSIEM analysis soon indicated that in addition to the credit card-related compromise, the logs presented evidence that indicated that the organisation had also been compromised on two occasions prior to the current incident. It therefore became necessary to broaden the scope of the MDR/MSIEM across the organisation’s wider environment, in order to be able to respond to the current attack while also monitoring the systems for any more signs of activity from the previous attacks.

In summary, the containment and recovery work was successful. We maintained our monitoring over a period of time to reduce the likelihood that an attacker was still hiding (silently waiting) within the organisation’s systems. Once both DotSec and the customer were satisfied, we wrapped up the incident response exercise but were asked to continue to manage the MDR/MSIEM as part of a new, separate managed services agreement. Continuing the MDR/MSIEM as a managed service turned out to be a great idea: A few months after the credit card incident, our MDR/MSIEM operators picked up activity that was indicative of an attacker uploading a shell to one of the organisation’s monitored servers. A P1 incident was raised, further investigation took place, the details were promptly passed on to the customer, and the customer used those details as part of their incident-response plan and successfully mitigated the problem before further exploitation was able to take place.

So there you have it!

A fully-managed service provides full management of log collection, aggregation indexing, searching, alerting and reporting – services that will address your organisation’s security and compliance needs with only a minimum of input on your part.

As the previous points have illustrated, a MDR/MSIEM system is best operated by a team of security specialists. While it’s possible to operate a MDR/SIEM system with in-house staff, it is a challenge to employ, train and retain a team of skilled and experienced professionals. The alternative is to utilise the services of an external team dedicated to the various aspects of looking after a MDR/MSIEM system, such as:

Operating the MDR/MSIEM infrastructure according to a Service Level Agreement that includes 24×7 prioritised alerting.
Ensuring constant system review and improvement so that the capabilities of the managed SIEM environment are being improved pro-actively.
Actively hunting for Indicators Of Compromise (IOCs) and newly-discovered threats.
Collecting actionable intelligence, prioritising threats, and identifying what actually is activity of interest.
Maintaining the MDR/MSIEM system itself, including updates, security, backups, storage, and data retention.

Having a dedicated team also takes the concern of staff retention out of the equation, as a dedicated team will always have the necessary staff on-hand.

Whether you’re looking to implement or improve your business’ SIEM services for insurance, compliance or incident response reasons, DotSec’s expert team is ready to help.