Skip to main content
Version: 6.1

Agent Data Collection in Smart Monitor

Agent data collection is a way to receive logs, metrics, and other events directly from endpoint nodes in Smart Monitor. For this purpose, the Smart Beat agent is installed on each node. Smart Beat is an aggregator of Elastic Beats applications that collect data from different sources. Agents, their configurations, and applications are managed centrally through Smart Beat Manager.

This approach allows different data types to be collected from one node at the same time, such as system metrics, application logs, and Windows security events. The set of applications and collection parameters can be assigned separately to each agent or to an entire agent group.

Agent Data Collection Architecture​

The agent collection process consists of the following steps:

  1. Smart Beat Manager assigns Elastic Beats application distributions and their configurations to an endpoint node.

  2. Smart Beat receives the assignment, installs the applications, and controls their startup.

  3. Each application collects its data type and sends events to Smart Monitor Data Collector or directly to Smart Monitor Data Storage.

Smart Beat Manager coordinates the agents but does not participate in transmitting collected events.

Agent data collection architecture

Smart Beat Manager​

Smart Beat Manager is a system component that coordinates Smart Beat agents. It provides centralized management of configurations and applications, as well as information about agent status, including activity, installed applications, and assigned groups.

For more information, see Smart Beat Management.

Smart Beat​

Smart Beat is installed on an endpoint node and receives assigned applications and configurations from Smart Beat Manager. The agent then installs, starts, and controls the corresponding Elastic Beats.

Several Beats can run on one node at the same time. For example, Filebeat can collect application logs, while Metricbeat can collect operating system metrics. The set of applications is determined by the collection tasks for a specific node.

Information

For information about installing and configuring the agent, see Smart Beat for Linux and Smart Beat for Windows.

Elastic Beats Overview​

Elastic Beats are lightweight specialized applications for collecting data from various sources. Each Beat type serves a specific purpose: collecting logs, system metrics, security events, availability check results, or network traffic.

Elastic Beats are installed and run inside Smart Beat. Because data types are separated, only the applications required for a specific collection scenario need to be assigned to an endpoint node.

Elastic Beats applications in Smart Beat

Filebeat​

Filebeat is designed to collect data from files. It monitors new entries and sends them to the data processing and storage system, so logs do not need to be uploaded manually.

The application supports text and structured logs, including JSON and CSV, and can combine multiline entries into a single event. Filebeat is commonly used to collect operating system and application logs.

Metricbeat​

Metricbeat collects system metrics and application performance indicators. It can provide information about CPU load, memory, disk, and network usage, as well as running processes and service status.

Collected data is used to monitor infrastructure status and analyze node performance. The metric set is determined by enabled modules and metric sets (metricsets).

Winlogbeat​

Winlogbeat is designed to collect events from Windows logs. It can transmit system errors, warnings, logon events, and other information required to monitor node operation and security.

The configuration can specify the standard Application, System, and Security channels, service and custom logs, as well as events forwarded from other nodes through ForwardedEvents.

Heartbeat​

Heartbeat regularly checks the availability of network nodes and services over ICMP, TCP, and HTTP. The results contain information about service status and response time, which helps detect unavailable or slow monitored resources.

Auditbeat​

Auditbeat collects information about user actions, system events, and file system changes. It can monitor process launches, file access, and changes to file attributes, as well as check the integrity of selected files and directories.

On Linux, Auditbeat can integrate with the Linux Audit Framework and receive kernel-level audit events. Filters can limit collection to the required event types, users, and file paths.

Packetbeat​

Packetbeat collects and analyzes network traffic at the level of supported application protocols, converting packets into events. The application groups related requests and responses into transactions and records response time, the amount of transferred data, and response codes.

This information can be used to analyze interactions between services, find delays, and detect network anomalies.

Elastic Beats Configuration​

Elastic Beats distributions and configurations are uploaded to Smart Beat Manager and assigned to individual agents or groups. Smart Beat receives the assignment, installs the corresponding Beat, and starts it with the specified configuration.

A configuration consists of parameters in the key: value format. Depending on the Beat type, it defines data sources and polling intervals, enabled modules, event filtering and processing, and the destination for collected data. For example, Filebeat configurations specify log file paths, while Metricbeat configurations specify modules and metric sets.

For more information, see Configuring Standard Data Collection Applications.