Sunday, June 24, 2018

HOARD Concept Release


Introducing "Historical Observations of Actionable Reputation Data" (HOARD) - a new proof of concept that we've designed to help security defenders utilize Threat Intelligence (Observable and Indicator data) in new ways. We understand that there are a number of ways to address this challenge. The goal is not to come up with the next "Product X" - but to bring awareness to another use of threat intelligence (or reputation data).

Observable data is frequently identified by computer security devices, intrusion detection systems, and forensic investigators following an intrusion or other malicious event. When observable data is paired with contextual information it becomes an indicator. Indicators are usually given a reputation or risk score.

These Indicators are frequently classified with the industry term "threat Intelligence" and disseminated by both humans and machines to alert computer security teams about threats they may have been previously unaware of.  STIX is a standard that is commonly used to communicate this type of data and inject it into security device pipelines.
Many computer security technologies will import this threat intelligence data and match it with same type observables. This has been done through Security Incident and Event Monitoring (SIEM) solutions, antivirus and network or system level intrusion detection systems. Unfortunately, most of this searching is forward focused.
The issue with forward focused analysis of indicators is the ephemeral nature.  Once identified, an adversary may change their attack profile and in doing so they change the identified observables. This asserts that even the fastest sharing platforms are likely to become less effective in the hours to days following the initial discovery of a given observable.
HOARD aims to reduce the speed and storage limitations needed for quickly matching observable data with historical threats.
Once installed, the HOARD application will monitor log events in real time by monitoring a queuing system (Currently Redis) fed by RSYSLOG, Suricata or other technologies. Since context is not required for the initial searches, the application extracts and stores only the observable data identified by the analyst as being relevant. This immediately reduces the data stored to a manageable size and provides raw data that can be indexed in a probabilistic data structure known as sketches or Cuckoo Filters.
Once observable data has been added to a threat intelligence exchange platform or a security team has been alerted to an issue,  a second application can be utilized to rapidly search back in time by querying the sketches to determine if the observable was probably seen in the past.
When a probabilistic match has been identified, the organizations Security Incident and Event Monitor (SIEM) is queried using the file date and timestamp information, this second query is used to validate the hit and provide context.
Keeping in mind cuckoo filters are probabilistic in nature, there will be false positives, but never false negatives. This rapid searching capability should be used to narrow down the potential timeframes at issue rather than relying on trigram or full context searches through an already taxed SIEM product.
Furthermore, these cuckoo filters/sketches can be provided to external organizations such as MSSPs, Incident Responders or Law Enforcement without exposing any organization or sensitive data.
HOARD is open source, GPLv3. We are releasing our operational POC in hopes that it will spark ideas and discussion among the security community. Our goal is implementation in a wide variety of products to continue to advance the future of threat intelligence and behavior or reputation matching on observable data. 
Please head on over to our GitHub repo (https://github.com/deependresearch/hoard) to take a look at our POC.