Introducing
"Historical Observations of Actionable Reputation Data" (HOARD) - a
new proof of concept that we've designed to help security defenders utilize
Threat Intelligence (Observable and Indicator data) in new ways. We understand
that there are a number of ways to address this challenge. The goal is not to
come up with the next "Product X" - but to bring awareness to another
use of threat intelligence (or reputation data).
Observable data is
frequently identified by computer security devices, intrusion detection
systems, and forensic investigators following an intrusion or other malicious
event. When observable data is paired with contextual information it becomes an
indicator. Indicators are usually given a reputation or risk score.
These Indicators are frequently classified with
the industry term "threat Intelligence" and disseminated by both
humans and machines to alert computer security teams about threats they may
have been previously unaware of. STIX is
a standard that is commonly used to communicate this type of data and inject it
into security device pipelines.
Many computer security technologies will import
this threat intelligence data and match it with same type observables. This has
been done through Security Incident and Event Monitoring (SIEM) solutions,
antivirus and network or system level intrusion detection systems.
Unfortunately, most of this searching is forward focused.
The issue with forward focused analysis of
indicators is the ephemeral nature. Once
identified, an adversary may change their attack profile and in doing so they
change the identified observables. This asserts that even the fastest sharing
platforms are likely to become less effective in the hours to days following
the initial discovery of a given observable.
HOARD aims to reduce the speed and storage
limitations needed for quickly matching observable data with historical
threats.
Once installed, the HOARD application will
monitor log events in real time by monitoring a queuing system (Currently
Redis) fed by RSYSLOG, Suricata or other technologies. Since context is not
required for the initial searches, the application extracts and stores only the
observable data identified by the analyst as being relevant. This immediately
reduces the data stored to a manageable size and provides raw data that can be
indexed in a probabilistic data structure known as sketches or Cuckoo Filters.
Once observable data has been added to a threat
intelligence exchange platform or a security team has been alerted to an
issue, a second application can be
utilized to rapidly search back in time by querying the sketches to determine
if the observable was probably seen in the past.
When a probabilistic match has been identified,
the organizations Security Incident and Event Monitor (SIEM) is queried using
the file date and timestamp information, this second query is used to validate
the hit and provide context.
Keeping in mind cuckoo filters are
probabilistic in nature, there will be false positives, but never false
negatives. This rapid searching capability should be used to narrow down the
potential timeframes at issue rather than relying on trigram or full context
searches through an already taxed SIEM product.
Furthermore, these cuckoo filters/sketches can
be provided to external organizations such as MSSPs, Incident Responders or Law
Enforcement without exposing any organization or sensitive data.
HOARD is open source, GPLv3. We are releasing
our operational POC in hopes that it will spark ideas and discussion among the
security community. Our goal is implementation in a wide variety of products to
continue to advance the future of threat intelligence and behavior or
reputation matching on observable data.
Please head on over to our GitHub repo (https://github.com/deependresearch/hoard) to take a look at our POC.
