Query Execution Statistics
Execution statistics show the duration of search stages, the number of documents processed, and storage access metrics. They help identify which commands and sources take the most time.
Contents
Enabling Statistics
When calling the Search API, the calculateExecutionCost parameter accepts ON and OFF. The default value is OFF. When set to ON, execution details are returned in service_data.
For request examples, see API SME.
Time and Documents
| Metric | Description |
|---|---|
took | Total request processing time in SME, in milliseconds |
execution_cost[].duration_sec | Stage duration, in seconds |
execution_cost[].input_count | Number of documents entering the stage |
execution_cost[].output_count | Number of documents leaving the stage |
execution_cost[].processing_bytes | Accounted size of data processed by the stage, in bytes |
Stages can contain nested stages. The duration of a nested stage may be included in the parent stage's duration.
processing_bytes is calculated when limits are enabled. When limits are disabled, the value is missing or equals null.
A missing value or null means that the metric was not calculated or is unavailable.
Storage Summary
The storage_summary block contains general information and a list of storage accesses grouped by storage.
| Field | Description |
|---|---|
qsize | Sum of the final effective sample sizes for storage accesses |
qsize_unbounded | Indicates whether at least one access has no document-count limit |
storage_duration_sec | Total duration of storage accesses, in seconds |
storages[].type | Storage type |
storages[].sources | Data sources |
storages[].selected_count | Number of selected documents |
storages[].available_count | Sum of effective qsize values in the summary row; -1 when an access is unlimited |
storages[].storage_duration_seconds | Duration of accesses in the summary row, in seconds |
The effective qsize accounts for applied limits. available_count does not show the total number of matching documents in storage. For example, if one million events match and the effective qsize=100, an access adds 100 to available_count.
The total qsize includes only final, finite effective access sizes. If all accesses are unlimited, it equals -1. If finite and unlimited accesses are combined, the sum of finite sizes is returned and qsize_unbounded is true.
Durations of parallel accesses are added together. For example, three accesses that each take 2 seconds result in 6 seconds of total storage work, although the request itself may finish in about 2 seconds.
For ClickHouse, the summary duration may be updated after server-side metrics are received.
ClickHouse Metrics
In background searches, calculateExecutionCost=ON enables stage statistics. A context for refreshing server-side ClickHouse metrics is not created for background or hybrid mode. Metrics for these results cannot be refreshed.
Information is collected from all available ClickHouse nodes. Metrics are returned in storage_resource_usage.
| Field | Description | Unit |
|---|---|---|
cpu_time_us | Processor time | Microseconds |
peak_memory_bytes | Estimated peak memory usage across collected executions | Bytes |
data_info.read_rows | Number of rows read | Rows |
data_info.read_bytes | Amount of data read | Bytes |
CPU represents processor time. With parallel execution, it may exceed the elapsed time.
ClickHouse writes its query log asynchronously, so metrics may become available after the search results. To retrieve them, use ClickHouse metrics refresh.
If records are not available yet, retry the refresh later. The message and error_code fields, when present, explain the collection status. Metrics are returned for the execution records found. Records for child executions may appear later, and unavailable nodes may be skipped.
Collection may finish after the record for the original SQL query is received but before records for child executions appear. After the context expires, later records cannot be guaranteed to be retrieved by refreshing it.
The refresh identifier is valid for a limited time. If it is no longer available, run the search again to obtain a new identifier.
Statistics and Limits
Statistics collection and limit application are enabled separately. Durations and document counts are calculated when statistics are enabled, regardless of whether limits are enabled. processing_bytes also requires limits to be enabled.
If a limit reduces the sample or time range, the statistics describe the search that was actually performed. Consider limit warnings when comparing queries.
For configuration details, see Search Limits.