A SIEM that could actually be queried during an incident, on infrastructure they own
SIEM / security analytics cluster
Where they started
Log collection existed for compliance purposes only. Queries against a quarter of data did not return before the analyst gave up, and retention was set by disk capacity rather than by requirement.
The problem, precisely
Security analysis needs fast queries and long retention at the same time, which is a capacity and tiering problem before it is a tooling problem.
What we did
Sized retention against the actual compliance requirement and the actual query patterns, then built a tiered storage layout so recent data is fast and older data is merely available.
Delivered
- Indexer cluster sized for query performance rather than raw ingest
- Tiered storage so recent data stays fast and older data stays queryable
- Ingestion pipeline with backpressure handling so a log storm does not lose data
- Detection rules tuned against the environment, drastically cutting false positives
- Retention policy mapped to a compliance requirement instead of a disk budget
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 400 days retention, requirement-driven rather than disk-driven
- — query time over a 90-day window, measured before and after
- 2,400 endpoints reporting reliably
- 0 log loss during a simulated storm
Set retention from the compliance requirement downwards, then engineer storage tiers to meet it. Doing it the other way round produces a system nobody can query.