Searching Data Units and Search Spaces
A data unit assigns a name to an index, index pattern, or storage table. A search space combines multiple data units, for example, recent events from internal storage and an archive in ClickHouse. Queries use these names; connection parameters and physical mappings are stored in the configuration.
Connection, Data Unit, and Search Space
| Object | Defines | Example |
|---|---|---|
| Connection | Storage and access parameters | system_os - internal storage; ch_events - ClickHouse |
| Data unit | Connection, physical object, and optional time field | audit_events → auditd-* indices through system_os |
| Search space | A list of data units or patterns matching their names | security_events → audit_events, ch_audit_events |
Data units describe existing sources. Creating one does not create tables or indices, move events, or configure rotation or replication. Data retention and storage are controlled by the storage settings.
Search supports data units on the internal Smart Monitor Data Storage through system_os and on ClickHouse.
Searching a data unit connected to an external OpenSearch instance is not supported.
You do not need to create a search space to search one data unit. An internal storage data unit can point to one index, alias, or index pattern; a ClickHouse data unit can point to one table. Saving a data unit does not check whether its physical source exists: a query to a missing index may return no results, while a query to a missing ClickHouse table may return a storage error.
Enabling Search
Search by data unit and search space names is controlled by the dynamic cluster setting sme.core.data_units.search_enabled. It is disabled by default. An administrator can enable it in the Dev Tools console:
PUT _cluster/settings
{
"persistent": {
"sme.core.data_units.search_enabled": true
}
}
Creating and editing the three entities, checking connections, and collecting statistics work regardless of this setting. When disabled, search does not resolve data unit or search space names.
Before enabling search, check for conflicts among search space, data unit, and index names. When enabled, a short name is resolved first as a search space, then as a data unit, and finally as an index pattern. Creating an object with a conflicting name may change the source of an existing saved query.
Specifying a Source
Explicit forms select the source type:
source du:audit_events
source ss:security_events
A short name is resolved in this order: search space, data unit, index pattern:
source security_events
Names are case-sensitive. A pattern in source, such as source audit_*, remains an index pattern and is not expanded into data units. Patterns matching data unit names are specified in a search space's du_refs field.
If a name conflicts with an SML keyword, quote the explicit reference: source 'du:count'. Quote index names ss and du as well: source 'ss'.
An unknown name prefixed with du: or ss: causes an error. An unknown short name may fall through to index search and return no results. If an object exists but is inaccessible, the query does not fall through to an index with the same name.
You cannot list a data unit or search space with other sources separated by commas or apply exclude(...) to them. To combine data units, define a search space.
Source Names
Data unit and search space names contain 1 to 256 characters: Latin letters, digits, and _; single periods and hyphens separate non-empty name parts. The first part must contain at least one letter or _. Thus, 2026_logs, _archive, and archive.2026 are valid, while 123 and 1-archive are not. The names clk, had, elk, os, db, and api are reserved, regardless of case.
Search Space Membership
Membership consists of data unit names or patterns matching their names with *, without the du: prefix. For example, audit_* selects all matching data units available to the user. Membership is resolved for each query, so new matching data units are included automatically. A pattern is case-sensitive and does not search indices or tables directly. If a data unit matches multiple patterns or is listed explicitly, it participates only once.
The number of participating data units is limited by sme.core.search_spaces.max_branches: the default is 32, and the allowed range is 1 to 128. Exceeding the limit causes the search to fail.
Time Field and Time Window
The time field is selected in this order:
timefieldin thesourcecommand- Non-empty
time_fieldin the data unit dtFieldNamein the search request- The default cluster setting
The earliest and latest parameters in source override the request's dtStart and dtEnd boundaries. A search space has no time field of its own: each data unit uses its configured field unless timefield is explicitly set for the entire query.
If all data units store event time in event_time, set it for the entire search space:
source ss:security_events timefield=event_time earliest="now-1d" latest="now" qsize=1000
| table host, action
An explicit timefield must be compatible with every data unit in the search space. If the field is missing or has an incompatible type, results may be empty. Search issues a warning when it can detect the problem from field mappings.
Merging Results
When searching multiple data units, results are merged by time; newest events come first by default. qsize limits the overall result set, and the maxQsize cluster limit also applies. Commands after source, including stats, process the selected rows.
If different data units point to overlapping physical sources, an event may appear more than once in the results.
A search space that resolves to one accessible data unit runs as a search on that data unit: rows are not merged, and the aggregate function set is not restricted as it is for multiple data units.
Aggregations
The aggs and timeaggs sections calculate aggregates in the storage systems and then combine the results. For filtering and grouping across multiple data units, use fields with the same names and compatible types in all participating sources. The following functions are supported for a search space with multiple participating data units:
| Section | Functions |
|---|---|
aggs | count, sum, min, max, avg, values, dc |
timeaggs | count, sum, min, max, avg |
avg uses the sum and count of values from all participating data units. The median, perc, earliest, and latest functions are not supported in these sections across multiple data units. This restriction applies to aggregation functions; the earliest and latest time-window options in source are supported.
source ss:security_events earliest="now-1d" latest="now"
| aggs count by host
The values limit is 10,000 unique values. Truncating a merged list produces a warning. dc on a search space combines lists: if any data unit returns 10,000 or more unique values in a group, accuracy is not guaranteed and that group's value remains empty with a warning. Dates with different representations across storage systems may be treated as different values.
The number of groups and storage-specific limits are independent of the values limit. Exceeding the number of aggregation buckets may cause a request to an individual data unit to fail. With a partial response, the aggregate reflects only the data that participated.
Partial Results and Failures
An explicitly specified data unit that is unavailable or invalid may be excluded with a warning. A pattern includes only data units available to the user. If no usable data units remain or all requests to them fail, the search fails.
If some requests fail, results from the remaining data units may still be returned with warnings.
Permissions
Cluster permissions for connections, data units, and search spaces are listed in the roles reference.
Permissions for a search space and its data units are checked separately. Read permission for a search space does not grant access to all its data units. The object creator becomes its owner; the owner separately grants access to other users and roles.
For internal storage, users also need permissions on the physical indices.
ClickHouse searches use the connection account. Configure its access to tables in ClickHouse.
Source Statistics
Data unit statistics describe its physical source: size, row count, time range, and schema. Statistics are a stored snapshot collected every five minutes, regardless of whether search is enabled. Reading statistics does not start a new search. A manual refresh returns a snapshot younger than 60 seconds by default without collecting it again.
If collection fails, an individual metric may retain its previous value and collection time. After ten minutes, a snapshot is considered stale but is still returned. Check each metric's time and status: a missing value does not mean zero, and successful collection of the main metrics does not guarantee that all source details are available.
For internal storage, data size reflects the size of primary shards, while disk size includes replicas. For ClickHouse, both metrics come from table size and do not represent the size of uncompressed source events. The Replicated and Distributed engines provide a limited set of statistics. For a filtered alias, row count and time range account for the filter, while size refers to the indices behind the alias.
Cluster Settings
Settings are changed dynamically through /_cluster/settings. Specify the full setting name under persistent, as in the search enablement example. To restore the default, pass null instead of a value.
Search and Configurations
| Parameter | Default | Description |
|---|---|---|
sme.core.data_units.search_enabled | false | Enables search by data unit and search space names |
sme.core.search_spaces.max_branches | 32 | Maximum number of data units in a search space query; 1 to 128 |
sme.core.default_connection | system_os | Default connection name returned when reading storage settings |
sme.core.system_connections.index_name | .sm_sme_system_connections | Index from which SME reads connection configurations |
sme.core.data_units.index_name | .sm_sme_data_units | Index from which SME reads data unit configurations |
sme.core.search_spaces.index_name | .sm_sme_search_spaces | Index from which SME reads search space configurations |
State and Statistics Collection
| Parameter | Default | Description |
|---|---|---|
sme.core.data_unit_state.schema_max_fields | 1000 | Maximum number of fields in a schema snapshot; 1 to 10,000 |
sme.core.data_unit_state.schema_max_indices_per_field | 20 | Maximum number of index names for a schema field; 1 to 1,000 |
sme.core.data_unit_state.opensearch_request_timeout | 30s | Wait time for one internal storage request during statistics collection; 1s to 10m |
sme.core.data_unit_state.clickhouse_query_timeout | 30s | Timeout for one ClickHouse statistics query; 1s to 10m |
sme.core.data_unit_state.clickhouse_connect_timeout | 10s | Wait time for establishing a ClickHouse connection during statistics collection; 1s to 2m |
sme.core.data_unit_state.clickhouse_connection_budget | 120s | Total collection time for one ClickHouse connection, excluding connection setup and queue wait; 10s to 10m |
sme.core.data_unit_state.connection_check_timeout | 5s | Timeout for checking one connection during scheduled or manual status refresh; 1s to 1m |
sme.core.data_unit_state.refresh_min_age | 60s | Minimum age of a snapshot or connection status before manual collection is repeated; 0s to 1h; 0s disables the limit |
sme.core.data_unit_state.refresh_response_timeout | 45s | Wait time for a manual refresh response; 1s to 10m. Collection may continue in the background after timeout |
sme.core.data_unit_state.refresh_lock_duration | 300s | Lock duration preventing concurrent manual collection of the same object; 10s to 1h |
sme.core.data_unit_state.refresh_max_ids | 1000 | Maximum number of unique object names in one manual refresh; 1 to 1,000 |