Skip to main content
Version: 6.0

Setting up ThreatFox Provider Data Collection

Logstash Configuration​

To properly prepare Logstash for information collection, you need to create configurations in the /app/logstash/config/conf.d/ folder. The logstash folder contains the necessary files for creating a ready-made configuration.

In the input folder, there is a file describing the data collection method. The http_poller plugin is used to read data from the specified URL. The data collection frequency is set to 1 hour.

In the filter folder, there is a handler for received events.

In the output folder, the process of sending processed data to Smart Monitor is described. You need to replace the address with the addresses of hot data and warm data nodes separated by commas in the hosts key, for example:

hosts => ["https://<HOT_NODE_IP>:9200", "https://<WARM_NODE_IP>:9200"]

All 3 files must be combined into one in the following sequence: input, filter, output.

After creating the config, you need to specify a pipeline for its launch. To do this, in the /app/logstash/config/pipelines.yml directory, add lines from the logstash/pipelines/pipelines.yml file to the very end of the file.

Logrotate Configuration​

Logrotate automates the process of managing log files. Create a ti_rotate.conf file in the /etc/logrotate.d directory with the following content:

/app/TI/AMTIP/AM_report/*.json {
    daily
    rotate 1
    missingok
    nocompress
    nocreate
    nomail
    noolddir
    dateext
    dateformat -%Y.%m.%d
    extension .json
    sharedscripts
    postrotate
        # Postrotate for searching and deleting files
        find /app/TI/AMTIP/AM_report/ -name 'domain-*.json' -type f | sort -r | awk 'NR>2 {print}' | xargs rm -f
        find /app/TI/AMTIP/AM_report/ -name 'ip-*.json' -type f | sort -r | awk 'NR>2 {print}' | xargs rm -f
        find /app/TI/AMTIP/AM_report/ -name 'url-*.json' -type f | sort -r | awk 'NR>2 {print}' | xargs rm -f
        find /app/TI/AMTIP/AM_report/ -name 'hash-*.json' -type f | sort -r | awk 'NR>2 {print}' | xargs rm -f
    endscript
}