If you’re in the security operations industry, chances are you spend a significant part of your workday interacting with a SIEM. Over the past two decades, SIEMs have evolved from a young security tool into a core element of almost every SOC. This year, Gartner’s Security Operations Hype Cycle placed SIEM just before its “Plateau of Productivity”, implying that it’s been widely established as a mature and effective tool in the industry, and is quickly approaching mainstream adoption.
But ubiquitous as they are, SIEMs are more often than not considered a necessary evil by the analysts who actually work with them. Tedious maintenance, alert fatigue, and monotonous manual investigations are just a few of the complaints commonly voiced by analysts. And in an industry that’s notorious for its rapid turnover and high levels of burnout, analyst satisfaction has never been more critical.
While SIEM may have been the best tool for the job when it was first introduced, it simply hasn’t been able to keep up with the rapidly evolving needs of the modern SOC. In the meantime, a variety of cloud-based security platforms have entered the market, built to serve the SOCs of the current day.
Security leaders are beginning to take notice, and the past few years have seen the industry start to shift away from the SIEM as the cornerstone of the SOC, and towards these newer cloud-based SOC platforms. Our CMO Lital Asher-Dotan and Senior Product Manager Erik Goldman recently hosted a live conversation over LinkedIn discussing this trend - what’s happening, why it’s happening, and where it’s going. You can find the full chat here, or read the key takeaways from the talk below.
Adapting to a changing world
When the first versions of SIEM came into use by SecOps teams, the cloud was an ambitious dream. At the time, an on-premises system to process and centralize unstructured logs was at the cutting-edge of security. In this earlier era, a SIEM could be used not only for security, but also to store, for example, product metrics to be used by the product team. As technology improved over time, other business functions moved to tools and practices that supported the higher data volumes these teams were now working with, but security lagged behind. Lately, security teams have begun to catch up to their counterparts, adopting SOC platforms with more modern data architectures equipped to handle higher log volumes.
An obvious benefit of these newer architectures is querying speed. While a team might have had 20 hours to wait for a query to run in their SIEM 15 years ago, this is no longer reasonable for a SOC that wants to run four such queries in an hour. Modern data architectures, such as security data lakes, have the power to vastly reduce these querying times, allowing analysts to change the way they investigate alerts. But the real game changer in a SOC platform is the separation between storage and compute.
In the past, logs were centralized and stored on a physical server, complete with a hard drive, processor, and memory. Storing more data entailed not only scaling out the hard drive, but also the rest of the server. The problem with this modality is that compute, or the “processor” side, doesn’t necessarily scale at the same rate as storage, or the “hard drive” side. In a perfect world, you would store all of your security data, because it’s nearly impossible to predict what data is going to be useful for an investigation. But most of these logs won’t actually be useful until an investigation takes place. Until then, storing these logs becomes very expensive, because you need to own or rent the processor as well, even if you never actually query this data.
SOC platforms built on a data warehouse like Snowflake or Databricks benefit from a pricing model where you pay only when you query the data, and not for the terabytes of the security data stored. This means that the platform can ingest unlimited data without taxing the security budget of the customer, and with that deliver better security outcomes.
The build or buy debate
A common discussion surrounding SIEMs is the “build vs. buy” debate. Storage in a SOC platform typically runs on a structure like an AWS S3 bucket. Even though this model makes data storage much cheaper than before, security leaders often see its relative simplicity as an opportunity to further cut costs by building it in-house. But anyone who’s attempted to DIY a project at this scale knows that what might appear simple quickly grows complex, requiring more and more resources to build and maintain it.
Entrepreneur and venture capitalist Paul Graham coined the term “general schlep” to describe a pain that’s generalized across an entire industry. In security, these are problems like ETL or centralized log storage, which almost every SOC team needs to deal with individually, and often involve a lot of frustration and resources to solve. A general schlep is an opportunity for a business to solve a common problem in an industry - on the data warehousing side, companies like Snowflake and Databricks have stepped in with their solutions. Security teams may have the budget and resources to build a solution in-house, but it’s not unreasonable to imagine that companies whose sole function is that specific problem might do a better job at it. Employing an external solution allows security teams to focus their resources on actually investigating threats, instead of maintaining complex data infrastructure.
Another general schlep that’s being solved in more and more SOC platforms is the problem of data normalization. For teams running SIEMs without an unlimited budget, data silos are a typical part of the job. As a result, the analysts working on these teams must endure the brain-melting monotony of constantly switching between a rotation of different tools to monitor all of their logs. And with every log source comes a set of log formats, each slightly different than the next and requiring deep understanding by already-stretched analysts.
The current generation of SOC platforms have taken this into account by doing the work of normalizing data into a unified schema. This means that when new tools are introduced that analysts haven’t necessarily seen before, they don’t need to spend time learning new log formats. Instead, analysts can immediately jump into investigating, triaging, and escalating the relevant threats.
A general rule for detections
General schleps appear yet again in the world of threat detection. It’s common for teams to have in-house staff writing all of their detections, even though most of them can apply to the majority of security teams. As a general rule, about 80% of threats are general to all security teams, and 20% are unique to individual organizations, based on their industry, location, or other factors. This is reflected in the increasing trend to adopt SOC platforms with built-in detections included, meaning that there’s a team working full time to cover those general 80% of detections. When a large event like SolarWinds or Log4Shell occurs, the SOC team gains access to IOC monitoring through the platform, instead of needing to reinvent the wheel. Engineers writing detections are then free to focus on the detections and investigations that are uniquely relevant to their business.
A platform to serve the SOC
The new generation of SOC platforms have a lot to offer, at every stage of the SOC workflow. Having been built in the cloud, these platforms are able to utilize their modern data architectures to more easily develop additional features and enhancements. This, along with the advantage of being able to ingest all security data at a fraction of the cost, has resulted in a trend towards increased automation embedded in SOC platforms.
For example, threat investigation is known by most analysts to be a tedious, manual task, involving sorting through a mountain of false positives. But today's SOC platforms have introduced some automation to significantly improve the investigation process. Improvements like automated cross log-source correlation, machine learning models for anomaly detection, and built-in data interrogation queries have emerged in the past few years to help analysts through the dullest and most laborious investigation tasks.
The human factor
Talk of automation can sometimes evoke the image of an empty room of computers, all humans gone after their jobs were automated away. But in reality, we’re far from having achieved the perfectly autonomous SOC. The benefits of automation aren’t about replacing your human workforce - they’re about making their jobs easier, more satisfying, and more productive. The less security teams need to execute manual tasks that can easily be automated, the more time they have for the hard security problems requiring the unique creativity of human beings.
While the biggest breakthroughs in security automation have yet to be made, the incredible innovations happening in the SOC platforms currently on the market give us all cause to be optimistic about the outlook of security operations. As the adoption of cloud-based SOC platforms becomes increasingly mainstream, more and more SOC members will find themselves doing the kinds of work they got into security for, instead of debugging an old SIEM in the middle of the night or looking through the same alert all day ad infinitum. The future of the SOC is not about killing jobs, but rather, about helping the people in those jobs flourish, and in turn more effectively protecting their organization.
Hunters allows teams to fully replace their SIEM with a lightweight, low-maintenance platform for the entire SOC workflow. To learn more, get a demo with us today.
