Setting up data collection from "AO Perspektivny Monitoring" provider

Installing the data collection script

On the VM where Logstash is installed, create a directory for storing scripts and related information:

mkdir /app/TI/AMTIP/amtip_script/
mkdir /app/TI/AMTIP/AM_report/

Copy the amtipfeed.py script file to the /app/TI/AMTIP/amtip_script/ directory. Create a .env file with the following content:

API_KEY = API Key
API_URL = Data collection endpoint.

Information

The API key can be obtained in your personal account. The key becomes available after purchasing a license.

---

Setting up Crontab

Using crontab settings, add the command to run the data collection script. The python interpreter included in the Smart Monitor installer is used to run the script. By default, it is located in the /app/logstash/utils/python/bin/python3 directory. The following flags are used in the script launch arguments:

• -c (--config) - path to the configuration file with API connection settings
• -o (--output) - path to the data files collected as a result of the script's work
• -sd (--start-date) - optional parameter specifying the start date of the export in YYYY-mm-ddT00:00:00Z format. The default value is yesterday
• -ed (--date-end) - optional parameter specifying the end date of the export in YYYY-mm-ddT00:00:00Z format. The default value is the current time

Before running the script, you need to install all dependencies for the script from the requirements.txt file.

Example command

/app/logstash/utils/python/bin/pip3 install -r /app/TI/AMTIP/amtip_script/requirements.txt

Example script launch command via crontab

0 3 * * * /app/logstash/utils/python/bin/python3 /app/TI/AMTIP/amtip_script/amtipfeed.py --config /app/TI/AMTIP/amtip_script/.env -o /app/TI/AMTIP/AM_report/

---

Setting up Logstash

To properly prepare Logstash for collecting information, you need to create configurations in the /app/logstash/config/conf.d/ folder. The necessary files for creating a ready configuration are located in the logstash folder.

In the input folder, there is a file describing the data collection method. The file plugin is used to read data from a file. Make sure that Logstash has sufficient rights to read files from the specified folder.

File example

input {
  file {
    path => "/app/TI/AMTIP/AM_report/*.json"
    exclude => "*-*-*.json"
    start_position => "beginning"
    sincedb_path => "/app/logstash/config/conf.d/last_run/.amtip"
    codec => "json"
  }
}

In the filter folder, there is a handler for received events.

In the output folder, the process of sending processed data to Smart Monitor is described. You need to replace the address under the hosts key with the addresses of hot data and warm data nodes, specifying them separated by commas.

Example

hosts => ["https://<HOT_NODE_IP>:9200", "https://<WARM_NODE_IP>:9200"]

All 3 files need to be combined into one in the following sequence: input, filter, output.

After creating the config, you need to specify the pipeline for its launch. To do this, add lines from the logstash/pipelines/pipelines.yml file to the very end of the /app/logstash/config/pipelines.yml file.

- pipeline.id: amtip
  path.config: "/app/logstash/config/conf.d/amtip.conf"