Configuring Indexes

In the current module version, data storage has been redesigned: indicators are stored in indexes by IoC type and provider, and the expiration time is controlled for each indicator separately using the Threat.indicator.lifetime field. Active indexes are combined under the threat_intelligence_database_ioc alias.

The SCRIPT - Threat Intelligence - LifeTime IoC job manages retention. It runs every 10 minutes, moves expired indicators to the threat_intelligence_recycle_bin index, and removes them from the active database. Records are kept in the recycle bin for 30 days and then permanently deleted. When an indicator is restored, it is returned to the active database with a new expiration period of 90 days.

Because expiration is controlled for each IoC, this module does not use an ISM policy or index rollover for data storage. See IoC Lifecycle for information about the recycle bin.

---

Creating Index Templates

Open Main Menu - Settings - Dev Tools and create index templates for the IoC types and providers in use. Index names must match the index parameter in the Logstash configuration.

The following is an example template for AMTIP domains. Set the number of primary shards and replicas according to the cluster architecture.

Example template for AMTIP domains

PUT _index_template/threat_intelligence_template_domain_amtip
{
  "index_patterns": [
    "threat_intelligence_index-domain-amtip"
  ],
  "template": {
    "settings": {
      "index.number_of_shards": 1,
      "index.number_of_replicas": 1
    },
    "mappings": {
      "dynamic": "true",
      "dynamic_templates": [
        {
          "keyword_fields": {
            "path_unmatch": "^(event.original)$",
            "match_mapping_type": "string",
            "match_pattern": "regex",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "text_fields": {
            "path_match": "^(event.original)$",
            "match_mapping_type": "string",
            "match_pattern": "regex",
            "mapping": {
              "type": "text"
            }
          }
        },
        {
          "long_fields": {
            "path_match": "^(event.risk_score|threat.indicator.last_seen|threat.indicator.first_seen|threat.indicator.modified_at)$",
            "match_mapping_type": "long",
            "match_pattern": "regex",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "date_fields": {
            "path_match": "^(event.risk_score|threat.indicator.last_seen|threat.indicator.first_seen|threat.indicator.modified_at)$",
            "match_mapping_type": "date",
            "match_pattern": "regex",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "string_fields": {
            "path_match": "^(event.risk_score|threat.indicator.last_seen|threat.indicator.first_seen|threat.indicator.modified_at)$",
            "match_mapping_type": "string",
            "match_pattern": "regex",
            "mapping": {
              "type": "keyword"
            }
          }
        }
      ],
      "numeric_detection": false,
      "properties": {}
    },
    "aliases": {
      "threat_intelligence_index-domain": {},
      "threat_intelligence_database_ioc": {}
    }
  },
  "priority": 1
}

---

Creating Indexes

Create an index for each combination of IoC type and provider in use. The following is an example for AMTIP domains:

PUT threat_intelligence_index-domain-amtip
{
  "aliases": {
    "threat_intelligence_index-domain": {},
    "threat_intelligence_database_ioc": {}
  }
}

The threat_intelligence_index-domain alias combines domains from different providers for searches, while threat_intelligence_database_ioc combines all active IoCs. For other types, use the threat_intelligence_index-ip, threat_intelligence_index-hash, and threat_intelligence_index-url aliases. The recycle bin is stored in the separate threat_intelligence_recycle_bin index.

---

Creating Lookup Dictionaries

Lookup dictionaries use regular IoC indexes. To combine data from multiple providers, specify the alias for the corresponding indicator type.

Information

If the dictionaries were installed with the content module, proceed to the next section.

1. Open Main Menu - Lookup Manager - Create Dictionary
2. Select OpenSearch as the source type
3. Specify the configuration name and data source from the table

IoC type	Configuration name	Index or alias
Domain	threat_intelligence_lookup_domain	threat_intelligence_index-domain
Hash	threat_intelligence_lookup_hash	threat_intelligence_index-hash
IP	threat_intelligence_lookup_ip	threat_intelligence_index-ip
URL	threat_intelligence_lookup_url	threat_intelligence_index-url

4. Select the common fields and the additional fields for the corresponding IoC type listed below
5. Save the dictionary and repeat the setup for the other IoC types in use

Common Fields

Include these fields in every dictionary:

threat.indicator.name
event.provider
threat.indicator.type
Threat.indicator.category
threat.indicator.provider
threat.indicator.confidence
Threat.indicator.severity
event.risk_score
Threat.indicator.risk_score
threat.indicator.description
threat.indicator.last_seen
threat.indicator.first_seen
Threat.indicator.lifetime
tags
Threat.indicator.fp.alarm
Threat.indicator.fp.description
Threat.indicator.blackists

Additional Fields by IoC Type

IoC type	Additional fields
Domain	threat.indicator.domain, Threat.indicator.tld, Threat.indicator.resolved.registrant, Threat.indicator.resolved.registrar, Threat.indicator.malware.name, Threat.indicator.malware.printable
Hash	threat.indicator.file.name, threat.indicator.file.hash.sha256, threat.indicator.file.hash.sha1, threat.indicator.file.hash.md5, Threat.indicator.ttp
IP	threat.indicator.geo.country_name, threat.indicator.geo.city, threat.indicator.geo.latitude, threat.indicator.geo.longitude
URL	Threat.indicator.malware.name, Threat.indicator.malware.printable, threat.indicator.url.domain

---

Configuring Index Template Settings in Smart Monitor

Configure the index templates. In the left panel, go to Index Patterns (Main Menu - Settings - Module Settings - OPENSEARCH - Index Patterns) and create the following templates:

• threat_intelligence_index-domain-*
• threat_intelligence_index-hash-*
• threat_intelligence_index-ip-*
• threat_intelligence_index-url-*

Specify the @timestamp time field for each template.

---

Importing Correlation Rules

Attention

If the rules are already installed, proceed to the next step.

To add the rules, go to Jobs List (Main Menu - Job Scheduler - Jobs List) and import the jobs from the rules directory.

The rules detect interactions with the IoC asset database and create incidents. You can view incidents in Incident Manager (Main Menu - Incident Manager - Incident Manager).