Version: 5.0

Installing Smart Beat Manager

warning

Starting from version 4.0.0, the interaction scheme between Smart Beat and Smart Beat Manager has changed, but Smart Beat Manager maintains backward compatibility with legacy agents.

Starting from version 4.2.0, the interaction API has been updated without backward compatibility. The Smart version must be at least 4.2.0.

Starting from version 4.3.0, the clients.yml snapshot file is deprecated. Data storage has been migrated to an embedded database. During the first launch of Smart Beat Manager, data will be migrated from the clients.yml file, after which the file will be renamed to clients_old.yml.

warning

Smart Beat can work with any version of ElasticBeat, but if you plan to send these examples to OpenSearch, then the ElasticBeat version should be 7.12.1 (or 7.10.2 is a more stable version). If you plan to use Logstash, then the ElasticBeat version does not matter. Check on the official website.

Symbols

${SBM_HOME} is the installation path of Smart Beat Manager, usually it is /app/smartBeatManager/

Installing Smart Beat Manager

Preparing to install Smart Beat Manager

Smart Beat Manager is available as part of the Smart installation archive in the directory ${SM_INSTALLER}/utils/smartBeatManager.

info

It is recommended to install under the root user. For a user with limited rights, you can make a preset.

Smart Beat Manager Service Installation

Smart Beat Manager supports the following service management commands:

install – installs the service
install -d <string> | --directory <string> - installs the service in a specified directory. Default: Smart Beat Manager executable's directory
install -u <string> | --user <string> - installs the service under specified user. Default: smartBeatManager
install -g <string> | --group <string> - installs the service under specified group. Default: smartBeatManager
install --ignore-selinux - installs the service without setting executable file permissions (if SELinux is present)
install --ignore-systemd - installs the service without creating smartBeatManager.service systemd file
remove – removes the service (in older versions: delete)
version – displays Smart Beat Manager version
help – shows help for any command

warning

For the install command:

if there was no user or group before, they will be created automatically. Including the default
if you specify an existing user and a group, but the user is not a member of this group, an error will occur and the installation will stop.

The installer is only compatible with the systemd service initialization and management subsystem.

The command to install the service in systemd may look like this:

mkdir -p /app/smartBeatManager/
cp ${SM_INSTALLER}/utils/smartBeatManager/SmartBeatManager-* /app/smartBeatManager/SmartBeatManager
cd /app/smartBeatManager/
./smartBeatManager install

When using a non-standard directory (e.g., $DIR), the installation command becomes:

cd /app/smartBeatManager/
./smartBeatManager install --directory $DIR

When using a non-standard user (e.g., $USER), the installation command becomes:

cd /app/smartBeatManager
./smartBeatManager install --user $USER

If SELinux is used, the rights to the executable file are set automatically when installing the service.

If you don't have enough rights to change the SELinux settings, you can set the --ignore-selinux flag when installing the service and run the command yourself:

sudo chcon -Rv -u system_u -r object_r -t bin_t /app/smartBeatManager/smartBeatManager-*

warning

The version of Smart Beat Manager may differ, and the file may be named differently. Also, pay attention to the owner of the group after changes in the working directory, since the service runs under the specified user during installation.

If you lack sufficient permissions to configure the systemd service, you can use the --ignore-systemd flag during service installation. After installation, manually create the service file:

sudo touch /etc/systemd/system/smartBeatManager.service

Modify the service configuration according to your requirements and save it to the created file:

[Unit]
Description=SmartBeatManager

[Service]
User=<SBM_USER>
Group=<SBM_GROUP>
WorkingDirectory=<SBM_HOME>
ExecStart=<FULL_PATH_TO_EXECUTABLE_FILE_SBM>
Restart=always

[Install]
WantedBy=multi-user.target

After creating the service, reload systemd configuration and enable autostart on server reboot:

systemctl daemon-reload
systemctl enable smartBeatManager

Upon completing the installation, all necessary directories and Smart Beat configuration files will be created, allowing service configuration before its first start.

Remaining directories and files will be created during the service's first launch.

Example of starting a systemd installed Smart Beat Manager service:

systemctl start smartBeatManager.service

To verify the service status, execute:

systemctl status smartBeatManager.service

Setting Up Smart Beat Manager

Smart Beat Manager Configuration File

Please note!

In the version of Smart Beat Manager 4.2.0, the name of the configuration file smartBeatManager.yaml was changed to config.yaml. Both files are supported, but config.yaml will be considered a priority.

Starting from version 4.3.0:

the updateSnapshot parameter is outdated and will not be taken into account when working

An example of the standard configuration of the configuration file config.yaml:

authorization:
  opensearch:
    ca_cert: ca-cert.pem
    host: https://127.0.0.1:9200
basic:
  appPath: ./apps
  binariesPath: ./binaries
  appsUnpackPath: ./config/apps
  binariesUnpackPath: ./bin
  portForSmartBeat: '7767'
  portForVisual: '7769'
  bindAddress: 127.0.0.1
  enableSnapshot: false
  enableDeletion: false
  timeoutDeletion: 24h
  disableGroupsCache: false
  badClientConnHours: 1h
logs:
  logPath: ./logs
  logMaxAge: 7d
  logRotate: 24h
ssl:
  enabled: true
  smartBeatTLSLevel: 0
  certPath: cert.pem
  keyPath: key.pem

The parameters are described below:

Parameter	Description	Default Value
`authorization.opensearch.ca_cert`	CA certificate in X509 format, used to connect to OpenSearch.	`ca-cert.pem`
`authorization.opensearch.host`	OpenSearch server node addresses.	`https://127.0.0.1:9200`
`basic.appPath`	Directory for searching for application configurations (Elastic Beats configurations).	`./apps`
`basic.binariesPath`	Elastic Beats archive search directory.	`./binaries`
`basic.appsUnpackPath`	Directory for unpacking application archives on Smart Beat agents.	`./config/apps`
`basic.binariesUnpackPath`	Directory for unpacking Elastic Beats archives on Smart Beat agents.	`./bin`
`basic.portForSmartBeat`	RestAPI port for connecting Smart Beat agents.	`7767`
`basic.portForVisual`	RestAPI port for connecting Smart Web.	`7769`
`basic.bindAddress`	At which address to accept connections.	`127.0.0.1`
`basic.enableSnapshot`	Enabling snapshots to save a list of connected agents. During operation, the load on the hard drive increases.	`false`
`basic.enableDeletion`	Whether to delete old agent entries in the snapshot.	`false`
`basic.timeoutDeletion`	The lifetime of the agent record in the snapshot.	`24h`
`basic.badClientConnHours`	The time after which it is assumed that there is no connection with the client.	`1h`
`basic.disableClientBatch`	Disabling batch data insertion by clients.	`false`
`basic.clientBatchConfig.maxClients`	The number of clients after which the recording takes place.	`50`
`basic.clientBatchConfig.timeLimit`	The time limit after which clients are recorded.	`10s`
`basic.clearCachePeriod`	Smart Beat Manager cached data clearing interval.	`24h`
`logs.logPath`	Logs directory.	`./logs`
`logs.logMaxAge`	Log storage time.	`7d`
`logs.logRotate`	The frequency of log rotation.	`24h`
`ssl.enabled`	Enables SSL.	`true`
`ssl.smartBeatTLSLevel`	Certificate verification level: 0 – client certificate is not requested and is not required, 1 – agent certificate will be requested, but it is not required and will not be verified, 2 – agent certificate is required, but any valid agent certificate is acceptable. It will not be verified by the CA certificate, 3 – the agent certificate will not be requested, but if it is present, it will be verified by the CA certificate, 4 – an agent certificate is required, which is verified for compliance with the certificate.	`0`
`ssl.certPath`	Certificate file in x509 Smart Beat Manager format.	`cert.pem`
`ssl.keyPath`	Private key file in x509 Smart Beat Manager format.	`key.pem`

note

The ssl.smartBeatTLSLevel parameter supports the following values:

0 – the client's certificate is not requested and is not required
1 – an agent certificate will be requested, but it is not required and will not be verified
2 – an agent certificate is required, but any valid agent certificate is acceptable. It will not be verified by the CA certificate
3 – the agent's certificate will not be requested, but if it is present, it will be verified against the CA certificate
4- an agent certificate is required, which is verified for compliance with the certificate

Install your Smart Beat Manager Certificates

warning

At the first launch, Smart Beat Manager generates its self-signed certificate and private key.

If you need to use ready-made certificates, then copy the CA certificate, the certificate and the Smart Beat Manager private key to the directory ${SBM_HOME}/etc/ssl/. You also need to configure the following settings:

authorization.opensearch.ca_cert - path to the CA certificate or certificate chain in x509 format
authorization.opensearch.host - OpenSearch server addresses must start with https
ssl.enabled - must be set to true
ssl.smartBeatTLSLevel - select the required certificate verification mode
ssl.CertPath - path to the Smart Beat Manager certificate file in x509 format
ssl.keyPath - path to the Smart Beat Manager private key file in x509 format

It is recommended to generate certificates with the following parameters:

keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
Subject Alternative Name - specify all possible IP and domain names of the server where the agent is installed

warning

Smart Beat Manager must have access to the specified files and directories

Configuring log collection configurations for Smart Beat agents

To send data collection configuration files to agents, you need to configure the file $SBM_HOME/etc/serverclasses.yml, which can contain several groups of settings for agents. The available group parameters are listed below:

Parameter	Description
`name`	The name of the group. Required parameter.
`filters`	A list of filters for connected agents, the values of the fields `ip`, `DNS name`, `hostname`, `guid` and `tags` of the agent can be used. You can use masks in the list. A logical `OR` is used between the conditions, i.e. it is sufficient to meet any of the listed conditions. Required parameter.
`apps`	List of configuration names (directory names) of ElasticBeats. Required parameter.
`binaries`	List of ElasticBeats archives that are required to perform the configurations listed in `apps`. You cannot specify multiple versions of the same ElasticBeat in the same group. Required parameter.
`systems`	A list of filters by the server platform on which the Smart Beat agent is running. The logical `OR` is used in the list, i.e. the agent falls into at least one of the filters. It is used as an additional filter with the `filters` field with the logical operation `And`.
`excludes`	List of agent filters to exclude from the group by `ip`, `DNS name`, `hostname`, `guid` and `tags`. You can use masks in the list. A logical `OR` is used between the conditions, i.e. it is sufficient to meet any of the listed conditions. It is used as an additional filter with the `filters` field with the logical operation `AND-NOT`.

warning

Starting from version 4.3.0 of Smart Beat Manager, for applications created through the Smart Beat management interface, a file with information about the application properties, including the type of ElasticBeat agent, will be stored.

For older applications created before version 4.3.0 that do not have a file with information about the application properties, backward compatibility has been preserved, that is, the name template will match <elasticbeat_name>_<apps_name>, where <elasticbeat_name> is the name of ElasticBeat in lowercase, <apps_name> is the name of the application.

warning

The DNS name is taken from the Smart Beat agent, for correct determination there must be a PTR record of the reverse DNS zone on the DNS server.

The following characters can be used as a mask in filters:

* - asterisk sign, replaces any character with a length of 0 (missing character) to infinity
? - replaces any 1 character

Example of setting up server classes.yml:

groups:
  - name: linux_test
    filters:
      - 192.168.1.?/24
      - server-test-linux.localhost
    apps:
      - filebeat_nginx
      - auditbeat_info_system
      - execbeat_script_python
    binaries:
      - filebeat-7.10.1-linux-x86_64.tar.gz
      - auditbeat-8.9.0-linux-x86_64.tar.gz
      - execbeat-3.3.0-linux-x86_64.tar.gz
  - name: windows_test
    filters:
      - 10.10.*.*
      - 78d17a171-a3c4-4e27-9c20-bc7005db068e
      - *our-site.org*
    apps:
      - filebeat_logs_system
      - execbeat_xspider
    binaries:
      - filebeat-8.9.2-windows-x86_64.zip
      - execbeat-3.3.0-windows-x86_64.zip

After configuring the file $SBM_HOME/etc/serverclasses.yml, you need to download the binaries archives from the official website (there should be archives with a pre-compiled ElasticBeats executable file) of oss versions and place them in the directory $SBM_HOME/binaries/.

info

It is recommended to use ElasticBeats versions of the same version that has Logstash installed. Smart version 4.2.0 uses Logstash version 8.9.0, so ElasticBeats is better to use versions 8.9.x.

Then you need to create the applications themselves (directories) as $SBM_HOME/etc/serverclasses.yml was specified in the $SBM_HOME/apps/ directory. Inside each application directory, all files and subdirectories will be sent to the agent. The name of the ElasticBeats configuration file must match the name used for the ElasticBeats application.

In the example below, there are applications execbeat_xspider, filebeat_nginx, filebeat_logs_system and others.

The execbeat_xspider application uses ElasticBeat - ExecBeat, the application name begins with execbeat\_, and the configuration file execbeat.ymlis located inside the directory. In this application, in addition to the configuration file forExecBeat, there are additional files, they will also be copied to Smart Beat agents.

The auditbeat_info_system application uses AuditBeat, so the name starts with auditbeat_, and the configuration file is called auditbeat.yml.

The contents of the ElasticBeats configuration files are filled in according to the official documentation of the corresponding ElasticBeats.

.
├── execbeat_xspider
│   ├── execbeat.yml
│   ├── xsipder_servers.ps1
│   ├── xsipder_skud.ps1
│   └── xsipder_switches.ps1
├── filebeat_nginx
│   └── filebeat.yml
├── filebeat_logs_system
│   └── filebeat.yml
├── auditbeat_info_system
│   └── auditbeat.yml
├── execbeat_script_python
│   └── execbeat.yml
└── metricbeat_logstash
    ├── metricbeat.yml
    └── modules.d
        ├── http.yml
        └── logstash-xpack.yml

The following is an example of the FileBeat configuration in the file bat.yml configuration file:

filebeat.inputs:
  - type: filestream
    id: logstash-logs
    enabled: true
    paths:
      - /app/logs/logstash/*.log

output.logstash:
  loadbalance: false
  hosts: ['server-logstash-01.com:5051', 'server-logstash-02.com:5051']
  ssl.certificate: '/app/smartBeat/cert/cert.pem'
  ssl.key: '/app/smartBeat/cert/key.pem'
  ssl.certificate_authorities: '/app/smartBeat/cert/ca-cert.pem'

The configuration above collects logs from the directory /app/logs/logstash/ with the file extension .log. The user running Smart Beat must have access to the files in this directory.

The collected data from the example above is sent to several Logstash servers via an encrypted SSL channel, the Logstash server is randomly selected and if the data is sent unsuccessfully, an attempt will be made to send it to another server.

To apply configurations, you can restart the systemd service or use the Smart Beat Manager API:

curl -k https://<SBM_HOST>:7769/reload -u sbm

Setting up `apps.yml`

warning

The apps.yml file is outdated and will be deleted in future versions.

The configuration file $SBM_HOME/etc/apps.yml is needed to configure the unpacking path on the configuration files agents. This file has a list of parameters:

appname is the name of the configuration file
path – the path of unpacking the file

Example of setting up apps.yml:

apps:
  - appname: scripts_exchange
    path: ./bin/scriptbeat-1.0.0-windows-x86_64/scripts
  - appname: scripts_activedirectory
    path: ./bin/scriptbeat-1.0.0-windows-x86_64/scripts

Setting up a connection to Smart Monitor Web

To configure the management of Smart Beat Manager and Smart Beat using Smart Monitor Web, you need to configure the parameters in the configuration file $OSD_HOME/config/opensearch_dashboards.ym.

Open the file and go to the Smart Beat Manager section and enter the correct settings. The configuration file already contains presets for these parameters. An example setup looks like this, where:

Parameter	Description
`smart_monitor.smartBeatManager.uri`	Connection string to the Smart Beat Manager API.
`smart_monitor.smartBeatManager.ssl.key`	Path to the OpenSearch Dashboards private key in x509 format.
`smart_monitor.smartBeatManager.ssl.certificate`	The path to the OpenSearch Dashboards certificate in x509 format.

An example of the settings is given below:

smart_monitor.smartbeatsmanagement.uri: 'https://127.0.0.1:7769'
smart_monitor.smartbeatsmanagement.login: sbm
smart_monitor.smartbeatsmanagement.ssl.key: /app/opensearch-dashboards/config/node-key.pem
smart_monitor.smartbeatsmanagement.ssl.certificate: /app/opensearch-dashboards/config/node-cert.pem

info

Smart Monitor already has a pre-installed user sbm for working via the Smart Beat Manager web interface. In the example above, the parameter smart_monitor.smartbeatsmanagement.password is not specified, as it is stored in the Keystore file. For authentication to work, you need to create a user in Smart Monitor in the menu Main Menu - System Settings - Security Settings - Internal Users.

Then add the password to the Keystore on the Smart Monitor Web server. From the console, run:

sudo -u opensearch $OSD_HOME/bin/opensearch-dashboards-keystore add smart_monitor.smartbeatsmanagement.password

Smart Monitor's Keystore already contains a saved password for the user sbm, so the command above can be omitted if you plan to use the default values.

Save the changes and restart the OpenSearch-Dashboards service:

sudo systemctl restart opensearch-dashboards.service

Symbols​

Installing Smart Beat Manager​

Preparing to install Smart Beat Manager​

Smart Beat Manager Service Installation​

Setting Up Smart Beat Manager​

Smart Beat Manager Configuration File​

Install your Smart Beat Manager Certificates​

Configuring log collection configurations for Smart Beat agents​

Setting up apps.yml​

Setting up a connection to Smart Monitor Web​

Symbols

Installing Smart Beat Manager

Preparing to install Smart Beat Manager

Smart Beat Manager Service Installation

Setting Up Smart Beat Manager

Smart Beat Manager Configuration File

Install your Smart Beat Manager Certificates

Configuring log collection configurations for Smart Beat agents

Setting up `apps.yml`

Setting up a connection to Smart Monitor Web