source

Description

Retrieves data from sources.

Syntax

source <source-name> [source-options]

---

Required Arguments

Parameter	Syntax	Description
source-name	<string>	For the OpenSearch source, simply specify the index pattern name. To fetch data from Hadoop or Clickhouse sources, use the prefix hdh: or clk: respectively. Optionally, specify the sample size (qsize) after the prefix and source name with :.

Examples:

source sysmon_operational-*

source hdh:win_events

source clk:nix_events:1000

Multiple Sources Query

You can query multiple sources by listing them separated by commas. By default, the result is combined using the append command. To use internal storage mechanisms to combine results, see the append optional argument.

Example:

source cisco_asa-*, clk:nix_events:1000

---

Optional Arguments

• source-options - optional parameters that provide additional search settings

Parameter	Syntax	Default	Description
qsize	qsize=<int>	1000000, when requested via web interface 1000	Maximum number of records to fetch.
timefield	timefield=<field>	@timestamp	Name of the field containing the timestamp.
append	append=<bool>	true	Data merging mode when querying multiple sources. With append=true, the append command algorithm is used; with append=false, merging is performed using the storage engine's internal mechanisms. Append Notes.
earliest	earliest=<string>		Start timestamp for the search.
latest	latest=<string>		End timestamp for the search.
timeout	timeout=<int>	External query timeout from cluster settings	External query timeout in milliseconds.

For earliest and latest, the following time format is allowed: (+|-)<int>(s|m|h|d|w|month) | timestamp | unix-time:

• s/sec/secs/second/seconds - seconds
• m/min/mins/minute/minutes - minutes
• h/hr/hrs/hour/hours - hours
• d/day/days - days
• w/week/weeks - weeks
• mon/month/months - months

Local Search Parameters

Each source can use its own local search parameters that will override the global parameters. The source and its parameters should be enclosed in parentheses. See Example 4.

Append Notes

• the append = false option is only possible when querying OpenSearch
• the append parameter must be global
• when append = false, only global search parameters are considered

---

Default Time Parameters

If one or several time parameters are not specified for a source, default values for these parameters will be substituted during query compilation.

info

Default time parameters are stored in _cluster/settings in the sme.core.global_dt_params block.

Warning About Setting Default Time Parameters

In case default time parameters are substituted, a separate warning will be generated for each source, containing information about which values were substituted.

For example, executing a query to the index sysmon_operational-* without specifying time fields will result in a response containing a warning with the following message:

Default values were set for dtFieldName='@timestamp' from cluster settings, dtStart='now-24h' from cluster settings, dtEnd='now' from cluster settings for index 'sysmon_operational-*'

---

Time Parameter Priority

Local parameters for sources, such as timefield, earliest, and latest, have the highest priority. If they are not specified, global parameters from the search are used, such as dtFieldName, dtStart, and dtEnd. If a global time parameter is also not specified, the source's time parameter value will be taken from the cluster settings.

Time Field Priority for Search Anywhere Configs

Special rules apply for determining the time field in a Search Anywhere query. Local search parameters also have the highest priority. If timefield for a source is not explicitly specified, it is taken from the Default time field in the Search Anywhere configuration. If that field is empty, then, as in the general case, dtFieldName from the query or the value from the cluster settings is used.

---

Ignoring Time Parameters

If the dtStart or dtEnd parameter is passed in the query with a value equal to an empty string, that time boundary will be ignored.

In this example, all data up to the current time will be requested.

POST _sme
{
  "query": "source sysmon_operational-* | aggs count ",
  "dtStart": ""
}

tip

If the dtFieldName parameter is passed with an empty string value, time boundaries and time-based sorting will be disabled.

---

Query Examples

Example 1

source ldap_users-* timefield=event_time

---

Example 2

source hdh:nix_events, ldap_computers-* qsize=1000 earliest=-2d latest=1649344240

---

Example 3

In this example, data is combined using OpenSearch.

source sysmon_operational-*, zabbix-* qsize = 150 append = false

---

Example 4

In this example, when querying data from the server_warnings index, local parameters are used, so qsize and the time boundaries differ from the queries to winevent and zabbix. Since timefield is not specified in the local parameters, the global parameter value will be used for server_warnings.

source (server_warnings:10 earliest="now-10m" latest="now-5m"),
 winevent, zabbix timefield=@timestamp earliest="now-1d" latest="now" qsize = 100