Threat Intelligence Module
Overview
This module is designed for automated threat detection in IT infrastructure by correlating events from multiple sources with an Indicators of Compromise (IoC) database.
The module is built on the OpenSearch stack and uses Logstash for data collection and processing, correlation rules for threat detection, and OpenSearch Dashboards for incident visualization and analytics.
Key components:
- IoC database: centralized storage of indicators received from external providers (ThreatFox, CTT, etc.)
- Data sources: logs from network devices (Cisco ASA), endpoints (KSC), and other security systems
- Correlation engine: jobs that match event data with the IoC database in real time
- Incident system: dedicated index for storing enriched trigger details
Architecture and Workflow
The module workflow is split into three main stages:
- IoC collection: collection scripts extract data from provider IoC databases; Logstash processes and loads the data into dedicated indexes according to data models
- Correlation: real-time correlation rules perform lookups in IoC indexes for each relevant interaction event
- Incident creation: when a match is found, a new incident is created in Incident Manager. It contains complete details from both the source event and the matched IoC, enabling full-context analytics
Data Models
The module uses strictly typed and standardized data models to ensure efficient search and analysis. All event and IoC fields are organized into logical groups (for example, threat.indicator.*, file.*, network.*).
To ensure compatibility and consistency, data is aligned to ECS (Elastic Common Schema). Additional fields that start with a capital letter are also used. These fields are not part of ECS, but are required to enrich information for specific IoC types.
It is recommended to follow the data models described in the Domain, Hash, IP, and URL files. Links are provided in the supported IoC types section.
- Indicator of Compromise (IoC): describes a specific threat. Each IoC type is stored in a dedicated index and has a dedicated data model.
- Incident: records a match between an interaction event and an IoC. The incident data model combines fields from both entities.
Supported IoC Types
Currently, the module supports correlation for four primary indicator types.
| IoC Type | Key event field used for matching | Example fields in IoC storage |
|---|---|---|
| Domain | destination.address | threat.indicator.domain, threat.indicator.tld, threat.indicator.resolved.registrant |
| Hash | file.hash.md5, file.hash.sha1, file.hash.sha256 | threat.indicator.file.name, threat.indicator.file.sha256, Threat.indicator.malware.name |
| IP | destination.ip | threat.indicator.geo.country_name, Threat.indicator.category, Threat.indicator.blackists |
| URL | url.original | threat.indicator.url.domain, Threat.indicator.category, Threat.indicator.malware.name |
Each source should have its own alias (or aliases). Details about sources and required aliases are provided inside each IoC type data model.
Dashboards and Visualization
The module uses dashboards that provide the following information:
- Threat Intelligence: IoC Database Statistics
- total number of IoCs in the database
- IoC breakdown by type (IP, Domain, HASH, URL)
- IoC statistics by provider
- trend of new indicator ingestion
- Threat Intelligence: Incident Statistics
- list of recent incidents with severity
- top-10 attacked hosts and users
- incident breakdown by triggered rules and IoC types