SML optimization recommendations
The section provides examples of search optimizations in query language Smart Monitor Language (further - SML). For more information about the list of commands and their purpose, see here.
Using full-text search
Full-text search simplifies the functionality for processing raw events, but its use increases the consumption of computing resources. If you know exactly which fields to search for, then it is better to specify them.
source sm_cs_auth_indexes
| search "*4624*"
The event code is stored in the event.code field and it is recommended to use the following search:
source sm_cs_auth_indexes
| search event.code=4624
Using wildcards
To search by value pattern in Smart Monitor it is possible to use wildcards. Searching using wildcards increases the number of events to search and requires more computing resources. The file.path field contains both the path to the file and its name
source sm_cs_auth_indexes
| search file.path="*/example.json"
To avoid similar situations, parsing data from the source according to the ECS standard will help.
source sm_cs_auth_indexes
| search file.name="example.json"
In such cases, it is recommended to use data parsing from the source according to the standard ECS. After applying ECS parsing, two fields will be available: file.path - path to the file, file.name - file name. The search will be performed on the specific value for the field, in this case file.name with the value example.json.
Both examples perform the same task, but in the second example the search is more optimized and uses minimal cluster resources.
Use filtering before data manipulation
It is recommended to reduce the sample size before using data processing commands when working with Smart Monitor. If you need to filter data that has already been processed, you can use the where command in SML.
The example searches for all events with event.code="4624" and then filters by the user.name field:
source sm_cs_auth_indexes
| search event.code=4624
| aggs count by user.name, host.name, source.ip
| where user.name=="maksimov.m"
Search with direct condition
When performing a search with a negation condition, more resources are consumed compared to specifying a direct condition. If you know in advance which data to search for, using a direct condition is the best solution.
source sm_cs_auth_indexes
| search user.name!="SMART-DC$" AND event.action!="logged-in"
source sm_cs_auth_indexes
| search user.name="maksimov.m" AND event.action="logged-out"
Storage-Level Commands
When constructing search queries, it is recommended to prioritize commands executed at the storage level, such as peval, search, aggs, and timeaggs, as they offer higher performance.
Statistics calculation commands
In Smart Monitor, there are commands for calculating statistics: aggs и stats.
When using the aggs command, the operation occurs at the SM Data Storage level and allows you to process a larger data set. For the stats command, it is important to take into account the qsize parameter, which filters the number of events processed. Changing the qsize parameter increases the load on RAM.
The commands for calculating statistics on a timeline work in a similar way: timeaggs и timechart.
Using сommands effectively
Properly chosen commands optimize search and reduce system load. Understanding the applicability of the commands allows you to get the maximum benefit from using Smart Monitor.
The search engine works in such a way that the input of the next command receives the results of the previous command. For example, if you need to perform transformations on data before aggregation (the aggs command), then the eval command will not work in this case and the corresponding error will be displayed. Example:
source sm_cs_auth_indexes
| eval user.domain=lower(user.domain)
| aggs count by user.domain
In this case, you must use the peval command, which uses the internal SM Data Storage mechanism:
source sm_cs_auth_indexes
| peval user.domain=lower(user.domain)
| aggs count by user.domain
To view the available fields in an event after processing, you can use the command | table *.
In Smart Monitor the user can enrich the search query data using a nested subquery. To do this, use the join command:
source sm_cs_auth_indexes
| search user.name!="*$" AND user.name!="unknown" AND event.code=4624
| peval source.address=coalesce(source.address, source.ip)
| aggs count, latest(@timestamp) as latest_time, earliest(@timestamp) as earliest_time, values(source.address) as source.address_values, values(winlog.computer_name) as winlog.computer_name_values by user.name
| eval duration=strptime(latest_time, "YYYY-MM-dd'T'HH:mm:ss.SSS'Z'") - strptime(earliest_time, "YYYY-MM-dd'T'HH:mm:ss.SSS'Z'")
| eval user.name=lower(user.name)
| join user.name
[ source sm_cs_auth_indexes
| search event.code=4634
| peval user.name=lower(user.name)
| aggs max(@timestamp) as time_password_last_change by user.name ]
Data can be combined from multiple sources, as the system imposes no restrictions.
Knowing these and other nuances will allow you to effectively utilize the capabilities of Smart Monitor for analyzing your data.