dedup
Description
Keeps only unique records in the results based on specified fields.
Syntax
| dedup [<int>] <field-list> [sortby <sort-by-clause>]
Required Arguments
| Parameter | Syntax | Description |
|---|---|---|
<field-list> | <field> [, <field>] | List of fields by which deduplication should occur. |
Optional Arguments
| Parameter | Syntax | Default | Description |
|---|---|---|---|
maxnum | <int> | No limit | Maximum number of combinations for deduplication. |
[sortby <sort-by-clause>] | sortby <global-sort-options> (-|+)<sort-field> [(-|+)<sort-field> ...] | Specifying the sorting description. |
Sorting Options
| Parameter | Syntax | Default | Description |
|---|---|---|---|
<global-sort-options> | +|- | + | Sort + ascending, - descending. |
<sort-field> | <field> | auto(<field>) | str(<field>) | ip(<field>) | num(<field>) | Sorting type description. |
Sorting Types
| Parameter | Syntax | Description | Notes |
|---|---|---|---|
<field> | <field> | Name of the field for sorting. | |
<auto> | auto(<field>) | Automatically determines the sorting method. | |
ip | ip(<field>) | Sorts for IP addresses. | |
num | num(<field>) | Sorts the field as a number. | If the field does not contain a numeric value, the system will return an error. |
str | str(<field>) | Sorts the field as text. | If the field contains another data type (numbers, dates, boolean value), the field values will be converted to text. |
The nature of sorting depends on the data type (text—alphabetical order, numbers—ascending/descending, dates—earlier/later, etc.).
Examples
These examples demonstrate various ways to use the dedup command to remove duplicate records and obtain unique data.
Example 1
In this example, the dedup command leaves only one record for each unique value of the event field.
source radius_logs
| dedup event
Example 2
In this example, the parameter 3 limits the number of records that can remain for one combination of values.
source radius_logs
| dedup 3 event
Example 3
This example demonstrates the use of the sortby parameter to control the order of record selection during deduplication. Sorting is performed as follows:
- first by the
eventfield in descending order - then by the
@timestampfield in ascending order
source radius_logs
| dedup event sortby - event +@timestamp
Example 4
In this example, numeric sorting is used through the num() function. If the field does not contain a number, the command will return an error.
source radius_logs
| dedup event sortby num(event)