Updating Smart Monitor
This instruction describes the process of updating Smart Monitor from version 4.0.* to 4.1.*.
Information
Conditional designations:
SM_41- directory where theSmart Monitorversion 4.1 installation package is unpackedUSER- system user with administrator rights, usuallyadminOPENSEARCH_HOME- OpenSearch home directory, usually/app/opensearch/OPENSEARCH_DATA- directory where indexed data is stored, usually/app/data/OSD_HOME- OpenSearch Dashboards home directory, usually/app/opensearch-dashboards/PATH_SSL- location of certificate, privateadminkey, and alsoca-cert, usually coincides with/app/opensearch/config/
The primary step for updating is determining the currently installed version of Smart Monitor. This can be done by viewing module versions on the main page or executing a command in the command line:
curl https://127.0.0.1:9200/_cat/plugins -k -u $USER
After entering this command, you will need to enter the password for the $USER account. It is recommended to use the admin user.
In version 4.1, OpenSearch and OpenSearch Dashboards components and modules were updated, the OpenSearch version was raised to 2.13.0, the job-scheduler and incident severity document storage structure was changed, and the remote script execution module SME-RE was changed. Let's consider the procedure for updating each component. The 4.1 installer needs to be unpacked into a directory, for example, /app/distr/.
Before starting work, it is strictly recommended to make a backup of the main configuration files and Security settings.
Recommended actions
It is recommended to create a directory, for example, /app/backup, where the following will be saved:
-
configdirectory, usually$OPENSEARCH_HOME/configor$OSD_HOME/config -
systemdfiles, usually/etc/systemd/system/opensearch.serviceand/etc/systemd/system/opensearch-dashboards.service,/etc/systemd/system/sme-re.service -
file
/etc/sysctl.d/00-opensearch.conf -
copy of Security settings, this needs to be done once, for which you will need the certificate and private key of admin user
chmod +x $OPENSEARCH_HOME/plugins/opensearch-security/tools/securityadmin.sh
JAVA_HOME=$OPENSEARCH_HOME/jdk/ $OPENSEARCH_HOME/plugins/opensearch-security/tools/securityadmin.sh -backup my-backup-directory \
-icl \
-nhnv \
-cacert $OPENSEARCH_HOME/config/ca-cert.pem \
-cert $OPENSEARCH_HOME/config/admin-cert.pem \
-key $OPENSEARCH_HOME/config/admin-key.pem
Incident Severity and aggregations migration
Significant changes were made to the internal document storage structure of incidents and incident aggregations in version 4.1. These changes require data migration to ensure correct system operation after updating from version 4.0 to 4.1.
The migration process must be performed before carrying out the main update. These actions need to be performed once.
The installer includes a utility for migrating Incident Severity and aggregations. The utility is located in the directory $SM_41/utils/migrations_4.0-4.1/incident_severity_migrations/ and works on python. The main requirements of the utility:
- Python 3.8+
- plugin opensearch-py
The rest of the packages are included in the standard Python installation, a more detailed list of packages:
- certifi==2023.7.22
- charset-normalizer==3.3.2
- idna==3.4
- opensearch-py==2.3.2
- python-dateutil==2.8.2
- requests==2.31.0
- six==1.16.0
- urllib3==2.0.7
Python 3.8 with the required set of packages is included in the Smart Monitor 4.1 installer.
Configuration file
Before running the utility, configure the parameters in the file $SM_41/utils/migrations_4.0-4.1/incident_severity_migrations/default.ini. An example configuration file is shown below:
[server]
host = 127.0.0.1
port = 9200
[user]
name = admin
pass = password
In the server.host parameter, you need to specify the IP address of any OpenSearch node, it is recommended to specify a node with the data role and routing_mode: hot attribute. If you omit the user.pass parameter, the utility will request the password from the AD in interactive mode.
Utility startup parameters
The utility has the following startup parameters:
-c, --config- configuration file (Optional). Default -./default.ini-h, --help- display help
Utility operation modes
After entering the user password, the utility offers to choose 4 operation modes:
migration- performs migration of incidents to new severitymigration_aggs- performs migration of aggregationsbackup- creates a backup copy of all incidents in indexes with the prefixbackup_load_backup- copies all incidents from backup indexes to main ones
Running the utility
To perform migrations, you need to run the utility several times, executing the utility operation modes in the following sequence:
backupmigrationmigration_aggs
You can run the utility with the command:
$SM_41/utils/python/bin/python3 $SM_41/utils/migrations_4.0-4.1/incident_severity_migrations/main.py -c $SM_41/utils/migrations_4.0-4.1/incident_severity_migrations/default.ini
Updating OpenSearch
In version 4.1, OpenSearch and OpenSearch Dashboards plugins were updated, as well as inventory and SME-RE components. The 4.1 installer needs to be unpacked into a directory, for example, /app/distr/. Mark where you unpack the contents of the archive as $SM_41.
SM_41=/app/distr/sm_4.1
For clusters consisting of multiple nodes, it is recommended to disable allocation before updating through the developer console (Navigation Menu - System Settings - Developer Console) by executing the command:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "none"
}
}
When updating cluster nodes, do not use allocation disabling with the update script. After updating all cluster nodes, enable allocation:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "all"
}
}
Automatic mode
For the script to work, the following pre-installed packages are required:
curlzipunzip
If at the end you don't see the inscription that Smart Monitor has been updated, don't run the update script again, take a screenshot of where the script stopped and contact technical support.
The automatic update script allows automating actions during updating and is located at $SM_41/opensearch/update.sh. When calling the script, you can specify the configuration file $SM_41/opensearch/example_config_opensearch.yaml. The YAML file format is similar to the configuration file during installation.
The update script supports the following startup parameters:
-c, --config <path_to_config_file_yaml>- specify configuration file for updating-h, --help- displays help on available commands
Start updating with nodes without the master role, data nodes can connect to older versions of master nodes, but not vice versa.
To start the update, run the script:
$SM_41/opensearch/update.sh
After running the script, it will automatically find the paths of the main directories:
OpenSearch Home Directory- OpenSearch installation directory, usually/app/opensearchOpenSearch Conf Directory- OpenSearch configuration files directory, usually/app/opensearch/config/OpenSearch Data Directory- data directory, usually/app/data/OpenSearch Logs Directory- logs directory, usually/app/logs/
The update script does not perform any actions with the data and logs directories, configuration directory and systemd files will be saved to the temporary directory $SM_41/opensearch/staging/.
If you run the script again, the staging directory will be cleared.
================================================================================
SMART MONITOR UPDATE SCRIPT - OPENSEARCH
================================================================================
Current working directory: /app/distr/sm_4.1/opensearch
Current name of install's archive: opensearch-2.13.0-linux-x64.tar.gz
New version OpenSearch: 2.13.0
================================================================================
-- STEP 1. INSTALLATION DIRECTORIES
opensearch.service file found. Will get necessary paths from there
Final Opensearch home directory: /app/opensearch
Final Opensearch conf directory: /app/opensearch/config
Final Opensearch data directory: /app/data/opensearch
Final Opensearch logs directory: /app/logs/opensearch
Is this correct? [y/n]:
After entering the directories, you need to confirm the automatically found data by pressing y, or enter your own directories manually by pressing n.
At the second step, you need to answer the question about allocation. If you enter y, the script will disable allocation before updating and enable it at the end of the script's work.
-- STEP 2. CONFIGURE ALLOCATION
Do you want to disable allocation during update? [y/N]: n
You don't want to disable allocation: n
Is this correct? [y/n]:
At the third step, you will need to enter the password from admin user. The password will not be displayed when entering.
-- STEP 3. GET ADMIN PASSWORD
Enter password for user "admin":
Then preparatory actions will be performed before updating, before applying the update a question about continuing will be asked, until this moment no actions are performed in the system. Also, some information about the current node and the cluster as a whole will be displayed.
get current list of plugins
sm-core
sm-im
sm-inventory
sm-ism-action-clickhouse
sm-job-scheduler
sm-job-scheduler-actions-incident
sm-job-scheduler-actions-mitre
sm-knowledge-center
sm-mitre
sm-mssp
sm-rsm
sm-uba
sme
opensearch-security
Information about current node OpenSearch:
{
"name" : "smos-node-00",
"cluster_name" : "smos-cluster",
"cluster_uuid" : "yKPPDCHGSA6rHQT948jokQ",
"version" : {
"distribution" : "opensearch",
"number" : "2.11.1",
"build_type" : "tar",
"build_hash" : "6b1986e964d440be9137eba1413015c31c5a7752",
"build_date" : "2023-11-29T21:43:10.135035992Z",
"build_snapshot" : false,
"lucene_version" : "9.7.0",
"minimum_wire_compatibility_version" : "7.10.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
!!! AT THIS POINT WE START TO MAKE CHANGES IN OPERATING SYSTEM !!!
Do you want to continue? [y/N]:
If you press Enter - the update will be interrupted, to continue you need to press y.
Upon successful completion of the update, you should see the inscription SMART MONITOR SUCCESSFULLY UPDATED!, preliminary information about the cluster and current node will be displayed.
-- STEP 10. PRINT INFORMATION
current state of cluster
{
"cluster_name" : "smos-cluster",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"discovered_master" : true,
"discovered_cluster_manager" : true,
"active_primary_shards" : 50,
"active_shards" : 50,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 15,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 76.92307692307693
}
current state clusters nodes
172.16.0.27 14 99 8 1.47 0.63 0.26 dim data,ingest,master * smos-node-00
Information about current node OpenSearch:
{
"name" : "smos-node-00",
"cluster_name" : "smos-cluster",
"cluster_uuid" : "yKPPDCHGSA6rHQT948jokQ",
"version" : {
"distribution" : "opensearch",
"number" : "2.13.0",
"build_type" : "tar",
"build_hash" : "7ec678d1b7c87d6e779fdef94e33623e1f1e2647",
"build_date" : "2024-03-26T00:02:39.659767978Z",
"build_snapshot" : false,
"lucene_version" : "9.10.0",
"minimum_wire_compatibility_version" : "7.10.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
The following plugins cannot be installed:
-- sm-job-scheduler-actions-incident
-- sm-job-scheduler-actions-mitre
================================================================================
-- SMART MONITOR SUCCESSFULLY UPDATED!
================================================================================
If for some reason the update script could not update some plugins, then it will additionally display information about these plugins at the end, as in the example above (the text The following plugins cannot be installed).
It should be noted that the update script takes into account the current list of installed plugins on OpenSearch nodes, if you need to install some plugin additionally, then the action should be performed manually at the end of updating the node.
Updating OpenSearch Dashboards
For the script to work, the following pre-installed packages are required:
curlzipunzip
The automatic update script allows automating actions during updating and is located at $SM_41/opensearch-dashboards/update.sh. When calling the script, you can specify the configuration file $SM_41/opensearch-dashboards/example_config_dashboards.yaml. The YAML file format is similar to the configuration file during installation.
The update script supports the following startup parameters:
-c, --config <path_to_config_file_yaml>- specify configuration file for updating-h, --help- displays help on available commands
When working, the script will make a backup copy of the systemd service file, opensearch-dashboards.yml and the configuration directory to the temporary directory $SM_41/opensearch-dashboards/staging/.
The update script does not perform any actions with the data and logs directories, configuration directory and systemd files will be saved to the temporary directory $SM_41/opensearch-dashboards/staging/.
If you run the script again, the staging directory will be cleared.
To update, run the script:
$SM_41/opensearch-dashboards/update.sh
The script will automatically determine the main paths of the current server to the following directories:
OpenSearch Dashboards Home Directory- OpenSearch Dashboards installation directory, usually/app/opensearch-dashboardsOpenSearch Dashboards Conf Directory- OpenSearch Dashboards configuration files directory, usually/app/opensearch-dashboards/config/OpenSearch Dashboards Data Directory- data directory, usually/app/data/OpenSearch Dashboards Logs Directory- logs directory, usually/app/logs/
================================================================================
SMART MONITOR INSTALL SCRIPT - OPENSEARCH DASHBOARDS
================================================================================
Current working directory: /opt/sm_4.1/opensearch-dashboards
Current name of install's archive: opensearch-dashboards-2.13.0-linux-x64.tar.gz
Current version of OpenSearch-Dashboards: 2.13.0
================================================================================
-- STEP 1. INSTALLATION DIRECTORIES
opensearch-dashboards.service file found. Will get necessary paths from there
Final Opensearch Dashboards home directory: /app/opensearch-dashboards
Final Opensearch Dashboards conf directory: /app/opensearch-dashboards/config
Final Opensearch Dashboards data directory: /app/data/opensearch-dashboards
Final Opensearch Dashboards logs directory: /app/logs/opensearch-dashboards
Is this correct? [y/n]:
After entering the directories, you need to confirm the entered data by pressing y, or enter your own directories manually by pressing n.
Then preparatory actions will be performed before updating, before applying the update a question about continuing will be asked, until this moment no actions affecting system operability are performed. Also, some information about the current node and the cluster as a whole will be displayed.
Current list of plugins:
-- smartMonitor
-- smartMonitorColumnChart
-- smartMonitorCyberSecurity
-- smartMonitorDrawio
-- smartMonitorHeatmapChart
-- smartMonitorHtmlChart
-- smartMonitorIncidentManager
-- smartMonitorInventory
-- smartMonitorKnowledgeCenter
-- smartMonitorLineChart
-- smartMonitorLookupManager
-- smartMonitorMitreAttack
-- smartMonitorPDFExport
-- smartMonitorPieChart
-- smartMonitorSingleValue
-- smartMonitorTable
-- smartMonitorUserBehaviorAnalytics
Current version of OpenSearch-Dashboards: 2.13.0
!!! AT THIS POINT WE START TO MAKE CHANGES IN OPERATING SYSTEM !!!
Do you want to continue? [y/N]:
Upon successful completion of the update script, the corresponding text SMART MONITOR DASHBOARDS SUCCESSFULLY UPDATED will be displayed.
Job Scheduler migration
Significant changes were made to the internal document storage structure of job-scheduler in version 4.1. These changes require data migration to ensure correct system operation after updating from version 4.0 to 4.1.
The migration process must be performed after carrying out the main update of all cluster nodes. These actions need to be performed once.
The installer includes a utility for migrating job-scheduler. The utility is located in the directory $SM_41/utils/migrations_4.0-4.1/job_scheduler_migrations/ and works on python. The main requirements of the utility:
- Python 3.8+
- plugin opensearch-py
The rest of the packages are included in the standard Python installation, a more detailed list of packages:
- certifi==2023.7.22
- charset-normalizer==3.3.2
- idna==3.4
- opensearch-py==2.3.2
- python-dateutil==2.8.2
- requests==2.31.0
- six==1.16.0
- urllib3==2.0.7
Python 3.8 with the required set of packages is included in the Smart Monitor 4.1 installer.
Configuration file
Before running the utility, configure the parameters in the file $SM_41/utils/migrations_4.0-4.1/job_scheduler_migrations/default.ini. An example configuration file is shown below:
[server]
host = 127.0.0.1
port = 9200
[user]
name = admin
pass = password
In the server.host parameter, you need to specify the IP address of any OpenSearch node, it is recommended to specify a node with the data role and routing_mode: hot attribute. If you omit the user.pass parameter, the utility will request the password from the AD in interactive mode.
Utility startup parameters
The utility has the following startup parameters:
-c, --config- configuration file (Optional). Default -./default.ini-h, --help- display help
Preliminary actions
You need to make a backup copy of the .sm_settings index before running the utility, for this open the menu Navigation Menu - System Settings - Developer Console and execute the following request:
POST _reindex
{
"source": {
"index": ".sm_settings"
},
"dest": {
"index": "backup_.sm_settings"
}
}
Running the utility
You can run the utility with the command from the server terminal:
$SM_41/utils/python/bin/python3 $SM_41/utils/migrations_4.0-4.1/job_scheduler_migrations/main.py -c $SM_41/utils/migrations_4.0-4.1/job_scheduler_migrations/default.ini
Incident Manager settings migration
Migration is performed from the web interface in the developer console (menu Navigation Menu - System Settings - Developer Console) and consists of two requests. In the first request, old Incident Manager settings are deleted:
POST .sm_im_settings/_delete_by_query
{
"query": {
"term": {
"_meta.id.keyword": {
"value": "incident-manager-settings"
}
}
}
}
In the second request, new Incident Manager settings are copied:
POST _reindex
{
"source": {
"index": ".sm_settings",
"query": {
"term": {
"_meta.id.keyword": {
"value": "incident-manager-settings"
}
}
}
},
"dest": {
"index": ".sm_im_settings"
}, "script": {
"source": """ctx._source.get("editFields").get("severity").get("values").forEach(severity -> severity.put("color", params.get(severity.get("value"))))""",
"lang": "painless",
"params": {"3": "#DC4E41", "2": "#F8BE32", "1": "#3B8646", "4": "#6092C0"}
}
}
Setting up SME-RE connection
The way of connecting to SME-RE has changed in Smart Monitor 4.1. Several actions need to be performed to correctly set up the connection.
Configuration file
The configuration file ${OPENSEARCH_HOME}/utils/sme-re/application.properties should be located in the same directory as the SME-RE utility. An example of the file contents:
server.port=18080
spring.servlet.multipart.max-request-size=100000000
server.tomcat.max-http-form-post-size=104857600
sme.FileRotatePeriod=10000
sme.baseData=/app/data/sme-re/
sme.directoryPollerInterval=50
sme.scriptConfigPath=./scripts.yaml
sme.LogStringSize=300
sme.RapidDelete=True
# redis
sme.redis.url = "127.0.0.1"
sme.redis.port = 6379
sme.redis.pass = ""
# LOGs directory
logging.file.path=/app/logs/sme-re/
# ssl paramaters
server.ssl.enabled=true
server.ssl.certificate=/app/opensearch/config/node-cert.pem
server.ssl.certificate-private-key=/app/opensearch/config/node-key.pem
server.ssl.trust-certificate=/app/opensearch/config/ca-cert.pem
# can be need, none, want
server.ssl.client-auth=need
Make changes if necessary. If some parameter is not in the configuration file, the default value will be used. The parameter descriptions can be read in the table below.
| Parameter | Description | Default value |
|---|---|---|
server.port | Port on which SME-RE will accept incoming connections | 18080 |
spring.servlet.multipart.max-request-size | Maximum file size for uploading in bytes | 100000000 |
server.tomcat.max-http-form-post-size | Maximum size for multipart/form-data requests in bytes | 104857600 |
sme.FileRotatePeriod | Time for rotating temporary data files in seconds | 10000 |
sme.baseData | Directory for storing request data | ./data |
sme.directoryPollerInterval | Interval for checking the data storage directory in seconds | 50 |
sme.scriptConfigPath | Path to the scripts.yaml configuration file | ./scripts.yaml |
sme.LogStringSize | Log string size | 300 |
sme.RapidDelete | Whether to delete request data files | True |
sme.redis.url | Address for connecting to Redis | "127.0.0.1" |
sme.redis.port | Port for connecting to Redis | 6379 |
sme.redis.pass | Password for connecting to Redis | "" |
logging.file.path | Path for saving logs | logs |
server.ssl.enabled | Whether to use SSL/TLS when waiting for incoming connections | true |
server.ssl.certificate | Path to the public key (certificate) of the current SME-RE server in PKCS8 format | /app/opensearch/config/node-cert.pem |
server.ssl.certificate-private-key | Path to the private key of the current SME-RE server in PKCS8 format | /app/opensearch/config/node-key.pem |
server.ssl.trust-certificate | Path to the chain of certifying centers (CA) in PKCS8 format | /app/opensearch/config/ca-cert.pem |
server.ssl.client-auth | Certificate validation mode, can be need, none, want | need |
Adding a user
An internal account sme_re_user is used to connect to SME-RE. To create an account, open the menu Navigation Menu - System Settings - Security Settings - Internal Users. Click the Create Internal User button at the top right.
Fill in only the Username field and password. The password must meet password complexity requirements. The other fields can be left blank. After that, click the Create button.
After that, you need to add the password for the sme_re_user account to the keystore. This can be done in several ways.
You need to add only to OpenSearch nodes where there is a parameter node.attr.node_with_sme: true.
You can view the parameters on all cluster nodes with the command from the developer console (Navigation Menu - System Settings - Developer Console):
GET _cat/nodeattrs
Adding a password to the keystore using a request
In Smart Monitor 4.1, API for managing the keystore from the web interface has been added. Open the menu Navigation Menu - System Settings - Developer Console and execute the following request:
GET _core/keystore
The API allows changing keystore only for certain cluster nodes, for example, with a mask by node name. You can read more about the keystore management API in the corresponding article.
The contents of the keystore stores on all cluster nodes will be displayed. To add a key to all cluster nodes, edit and execute the request:
POST _core/keystore/sme.core.remote_script.password
{
"value" : "<PASSWORD_USER>"
}
Adding a password to the keystore manually
On OpenSearch nodes with SME-RE, you need to add the key sme.core.remote_script.password to the keystore. This can be done with the following command:
sudo -u opensearch $OPENSEARCH_HOME/bin/opensearch-keystore add sme.core.remote_script.password
When executing the script, it will ask you to enter the key value, enter the password of the sme_re_user user. After executing the command, restart the OpenSearch node.
Adding connection settings to the cluster
First, check that you don't have SME-RE connection parameters, open the menu Navigation Menu - System Settings - Developer Console and execute the following request:
GET _cluster/settings
In the right part, find the persistent.sme.core.remote_script object, for example:
{
"persistent": {
"sme": {
"core": {
"remote_script": {
"base_path": "/app/opensearch/utils/scripts/",
"enable_ssl": "true",
"port": "18080",
"base_interpreter_name": "bash",
"url": "https://127.0.0.1",
"interpreters": ["python3::/app/opensearch/utils/python/bin/python3", "bash::bash"]
}
}
}
},
"transient": {}
}
If you find it, then you already have SME-RE connection set up, the action below is not required. If the persistent.sme.core.remote_script object is missing, then you need to perform the action below.
For the OpenSearch cluster to be able to connect to SME-RE, you need to add parameters through the menu Navigation Menu - System Settings - Developer Console, execute the request after changing the necessary parameters:
PUT _cluster/settings
{
"persistent": {
"sme": {
"core" : {
"remote_script": {
"base_path": "/app/opensearch/utils/scripts/",
"enable_ssl": true,
"port": "18080",
"base_interpreter_name": "bash",
"url": "https://127.0.0.1",
"interpreters": [
"python3::/app/opensearch/utils/python/bin/python3",
"bash::bash"
]
}
}
}
}
}
Parameter descriptions are presented below:
| Parameter | Description |
|---|---|
sme.core.remote_script | Settings for executing external scripts |
sme.core.remote_script.base_path | Directory for searching external scripts |
sme.core.remote_script.enable_ssl | Whether to use SSL/TLS to connect to SME-RE |
sme.core.remote_script.port | Port for connecting to SME-RE |
sme.core.remote_script.base_interpreter_name | Default interpreter for external scripts |
sme.core.remote_script.url | URL for connecting to SME-RE, protocol can be omitted |
sme.core.remote_script.interpreters | Array of interpreters, each element is presented in the format short_intr_name::path_to_intr |
Creating a menu item
To view the new functionality, you need to create a navigation menu item, for this open the menu Navigation Menu - System Settings - Module Settings - Main - Menu Settings, find Incident Manager and expand it, then click the Add Section button.
Fill in the fields as follows:
| Field name | Content |
|---|---|
System name | incidents-group |
Title | Incident group settings |
Enable display | Yes, the flag must be enabled |
Type | Page |
After that, click the Save Changes button.