Skip to main content
Version: 6.0

Setting up the index policy

From the index_settings directory, you need to add preliminary index settings for data collection. Through the navigation menu, go to System Settings - Index Management - Index Policies and click Create Policy. In the opened window, select the JSON Editor item. Fill in the settings according to the following description:

  1. Policy ID - threat_intelligence_policy
2. Policy definition
{
  "policy": {
    "description": "Rollover after 1d or 20GB, delete after 2d from rollover",
    "default_state": "init",
    "states": [
      {
        "name": "wait",
        "actions": [],
        "transitions": [
          {
            "state_name": "init",
            "conditions": {
              "min_doc_count": 1
            }
          }
        ]
      },
      {
        "name": "init",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_index_age": "1d",
              "min_primary_shard_size": "20gb",
              "copy_alias": false
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "2d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "sm_threat_intelligence_url_*vendor**",
          "sm_threat_intelligence_ip_*vendor**",
          "sm_threat_intelligence_hash_*vendor**",
          "sm_threat_intelligence_domain_*vendor**"
        ]
      }
    ]
  }
}
Information

The size of the rolled index and storage periods can be changed according to specific requirements. In this example, indexes are rolled daily and deleted after 2 days.

Creating patterns and index settings​

To create index patterns, you need to go to Developer Console (Navigation Menu - System Settings - Developer Console). Then execute requests according to the template below with the request body from the index_settings/template directory:

PUT _index_template/sm_threat_intelligence_template_*IoC type*_*Provider*

An example of an index template request for domain IoC is presented below:

Example of an index template request for domain IoC
PUT _index_template/sm_threat_intelligence_template_domain_*Provider*
{
  "index_patterns": [
    "sm_threat_intelligence_domain_*Provider*"
  ],
  "template": {
    "settings": {
      "opendistro.index_state_management.policy_id": "threat_intelligence_policy",
      "index.opendistro.index_state_management.rollover_alias": "sm_threat_intelligence_domain_*Provider*",
      "index.number_of_shards": 1,
      "index.number_of_replicas": 1
    },
    "mappings": {
      "dynamic": "true",
      "dynamic_templates": [
        {
          "keyword_fields": {
            "path_unmatch": "^(event.original)$",
            "match_mapping_type": "string",
            "match_pattern": "regex",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "text_fields": {
            "path_match": "^(event.original)$",
            "match_mapping_type": "string",
            "match_pattern": "regex",
            "mapping": {
              "type": "text"
            }
          }
        },
        {
          "long_fields": {
            "path_match": "^(event.risk_score|threat.indicator.last_seen|threat.indicator.first_seen|threat.indicator.modified_at)$",
            "match_mapping_type": "long",
            "match_pattern": "regex",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "date_fields": {
            "path_match": "^(event.risk_score|threat.indicator.last_seen|threat.indicator.first_seen|threat.indicator.modified_at)$",
            "match_mapping_type": "date",
            "match_pattern": "regex",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "string_fields": {
            "path_match": "^(event.risk_score|threat.indicator.last_seen|threat.indicator.first_seen|threat.indicator.modified_at)$",
            "match_mapping_type": "string",
            "match_pattern": "regex",
            "mapping": {
              "type": "keyword"
            }
          }
        }
      ],
      "numeric_detection": false,
      "properties": {}
    },
    "aliases": {
      "sm_threat_intelligence_indexes": {}
    }
  },
  "priority": 1
}

In this example, the number of primary shards and replicas is 1. These parameters are set depending on the number of nodes and cluster configurations.

Creating a partition​

To create the first rollable partition, you need to execute the following request in the developer console:

PUT sm_threat_intelligence_*IoC type*_*Provider*-000001
{
  "aliases": {
    "sm_threat_intelligence_*IoC type*_*Provider*": {
      "is_write_index": true
    }
  }
}
Information

Partitions must be created for each IoC type and provider.

Creating Lookup dictionaries​

Attention!

If dictionaries are installed, proceed to the next step.

After creating assets, it is required to create dictionaries based on Inventory data (Navigation Menu - Lookup Manager - Create Dictionary) and specify the following fields for each IoC type:

  • configuration name: sm_threat_intelligence_lookup_*IoC type*
  • index name: sm_threat_intelligence_inventory_*IoC type*
  • fields: list all fields from the Inventory asset

Creating dictionaries:

  • configuration name: sm_threat_intelligence_lookup_domain
  • index name: sm_threat_intelligence_inventory_domain
  • fields: base.IoC, base.Provider, base.IoC Type, advanced.Domain, advanced.Top Level Domain, advanced.Registrant, advanced.Registrar, advanced.IoC Categories, advanced.IoC Source, advanced.Provider Score, advanced.SIEM Score, advanced.Provider Risk Score, advanced.SIEM Risk Score, advanced.Additional Information, advanced.Last Updated, advanced.Creation Time, advanced.Relevance, advanced.Vulnerability List, advanced.FP Alert, advanced.FP Description, advanced.Black List, advanced.Malware Name, advanced.Malware Representative

    • Configuring index settings templates in Smart Monitor​

    You need to set index template settings. To do this, in the left panel go to Index Patterns (Navigation Menu - System Settings - Module Settings - OPENSEARCH - Index Patterns). Create the following template settings:

    • sm_threat_intelligence_domain_*
    • sm_threat_intelligence_hash_*
    • sm_threat_intelligence_ip_*
    • sm_threat_intelligence_url_*
    • sm_threat_intelligence_inventory_domain
    • sm_threat_intelligence_inventory_hash
    • sm_threat_intelligence_inventory_ip
    • sm_threat_intelligence_inventory_url

    Each template must specify the time field @timestamp.

    Importing correlation rules​

    Attention!

    If rules are installed, proceed to the next step.

    To add rules, go to Jobs List (Navigation Menu - Job Scheduler - Jobs List) and import jobs from the rules folder.

    The rules are aimed at detecting interaction with the IoC assets base and creating incidents. You can view incidents in Incident Manager (Navigation Menu - Incident Manager - Incident Manager).