Cyber Security: Connecting and Configuring Data Collection

Contents

• Overview
  • Mapping Logs to Configurations and Indexes
• Configuration Sequence
  • Step 1. Verifying Module Installation
  • Step 2. Configuring Smart Monitor Data Collector
  • Step 3. Configuring Smart Beat Agents
  • Step 4. Verifying Data Ingestion
  • Step 5. Verifying Data Format
• Common Problems

---

Overview

The module's rules and dashboards read data from indexes with fixed names. The data collection configurations included with the module parse source events and normalize them into the required format. Use these configurations for the rules to work correctly.

Important

If source data already arrives in Smart Monitor in other indexes or a different format, the module's rules will not see it. To use different indexes or fields, you must change the rule queries.

Source-side configuration (audit policy, Sysmon, PowerShell logging, and Kubernetes auditing) is described in Configuring Infrastructure Sources.

Mapping Logs to Configurations and Indexes

Source	Log	Default Port	Index Pattern
Windows	Security	2001	windows_security-*
Windows	Microsoft-Windows-Sysmon/Operational	5001	windows_sysmon_operational-*
Windows	Microsoft-Windows-PowerShell/Operational	5002	windows_powershell_operational-*
Windows	System, Application, Microsoft-Windows-Windows Defender/Operational	5059	windows_events-*
Windows	DNS Server, Microsoft-Windows-DNSServer/Audit	5063	dns-events-*
Microsoft IIS	Microsoft-IIS-Logging/Logs, Microsoft-IIS-Configuration/Operational	5062	microsoft-iis-*
Linux	auditd	1313	auditd-*
nginx	access.log	51121	nginx_log_access-*
Kubernetes	kube-apiserver audit log	5060	kubernetes_audit-*

Active Directory uses the same logs and indexes as Windows. Domain controller events are stored in windows_security-*.

Windows collection configurations support collection from each host or through a Windows Event Collector (WEC, ForwardedEvents channel).

---

Configuration Sequence

Step 1. Verifying Module Installation

Install the module and run the installation utility before configuring data collection. The utility creates source index templates, write aliases, and index patterns. See Installation.

Important

Enable collection only after running the installation utility. If events arrive before the write alias exists, Smart Monitor Data Storage creates a regular index with the alias name (for example, auditd), and the alias cannot be created.

Data Retention

By default, an index rolls over once a day or when it reaches 10 GB, and is deleted 45 days after rollover. Behavioral baseline rules analyze 90 days and three months of history. They can use the full history only if the index retention period is at least as long as those windows.

Step 2. Configuring Smart Monitor Data Collector

2.1. Placing Configurations

Place the *.conf pipelines for each source in /app/logstash/config/conf.d.

Set installation-specific values in the output section of each configuration.

Configuration Value	What to Set
https://HOST:PORT	address of Smart Monitor Data Storage
/app/logstash/config/ca-cert.pem	CA certificate path, if different

The pipelines read the logstash user's password from the ES_PWD key in logstash.keystore.

sudo -u logstash /app/logstash/bin/logstash-keystore add ES_PWD

2.2. Connecting Pipelines

Add pipeline IDs and paths to /app/logstash/config/pipelines.yml.

### Cyber Security

- pipeline.id: windows_security
  path.config: "/app/logstash/config/conf.d/windows_security.conf"
- pipeline.id: windows_sysmon_operational
  path.config: "/app/logstash/config/conf.d/windows_sysmon_operational.conf"
# ... other module configurations

2.3. Connection Ports

Default ports are listed in the mapping table. If a port is already in use, change port in the pipeline's input section and the address in the agent configuration's output section.

Step 3. Configuring Smart Beat Agents

Deploy agent configurations through Smart Beat Manager. See Agent Data Collection.

3.1. Windows

• deploy winlogbeat_security, winlogbeat_sysmon_operational, winlogbeat_powershell_operational, and winlogbeat_windows_events to all Windows hosts
• deploy winlogbeat_dns_server only to Windows DNS servers
• deploy winlogbeat_iis only to IIS servers
• collect the domain controllers' Security log using the same winlogbeat_security configuration
• replace SM_DATA_COLLECTOR in the output section with the Smart Monitor Data Collector address

3.2. Linux

• auditbeat_auditd contains kernel audit rules (audit_rules)
• replace SM_DATA_COLLECTOR with the Smart Monitor Data Collector address

3.3. nginx

• filebeat_nginx reads /var/log/nginx/access.log by default
• if the log is elsewhere, set its path in paths

3.4. Kubernetes

• deploy filebeat_kubernetes_audit to control plane hosts
• by default, it reads /var/log/kubernetes/audit/audit.log, /var/log/kube-apiserver/audit.log, and /var/lib/rancher/k3s/server/logs/audit.log
• if the audit log is elsewhere, set its path in paths

Step 4. Verifying Data Ingestion

Open Main Menu - Search and run a query for each connected source over the last 15 minutes. The following example uses the windows_security index.

source windows_security-*
| stats count

The event count must be greater than zero for each connected source.

Step 5. Verifying Data Format

Rules depend on normalized fields. Verify that the key fields are populated in source events.

Index	Key Fields
windows_security-*	event.code, host.name, user.name, process.command_line (event 4688), source.ip (events 4624 and 4625)
windows_sysmon_operational-*	event.code, process.executable, process.command_line, process.parent.executable, destination.ip, dns.question.name
windows_powershell_operational-*	event.code, powershell.file.script_block_text (event 4104)
windows_events-*	event.code, winlog.channel, service.name
auditd-*	process.executable, process.args, user.name
nginx_log_access-*, microsoft-iis-*	source.ip, http.request.method, url.path, http.response.status_code, user_agent.original
kubernetes_audit-*	kubernetes.audit.verb, kubernetes.audit.objectRef.resource, kubernetes.audit.user.username

Run the following checks in Search over the last 15 minutes. An empty result or a zero value in a column means the field is not populated.

Command lines in Windows process creation events:

source windows_security-*
| search event.code="4688"
| stats count, dc(process.command_line) as command_lines by host.name

Sysmon event types by host:

source windows_sysmon_operational-*
| stats count by host.name, event.code

PowerShell script block text:

source windows_powershell_operational-*
| search event.code="4104"
| stats count, dc(powershell.file.script_block_text) as script_blocks by host.name

System, Application, and Windows Defender log channels:

source windows_events-*
| stats count by winlog.channel

Process starts on Linux hosts:

source auditd-*
| stats count, dc(process.executable) as executables by host.name

Web server requests:

source nginx_log_access-*, microsoft-iis-*
| stats count, dc(source.ip) as clients by http.response.status_code

Actions in the Kubernetes API:

source kubernetes_audit-*
| stats count by kubernetes.audit.verb, kubernetes.audit.objectRef.resource

Also check value formats.

• host names (host.name) and account names (user.name) must use the same format as the lookup data; see Lookups
• event times (@timestamp) must match the source time, accounting for the time zone

---

Common Problems

Symptom	Possible Cause	What to Check
no data arrives	port blocked or already in use	firewall, ss -ltnp output, agent log
data arrives in an index without a suffix (windows_security instead of windows_security-000001)	missing write alias	Installation
process.command_line is empty in event 4688	command line logging is disabled in process creation auditing	Configuring Infrastructure Sources
no event 4104	PowerShell script block logging is disabled	Configuring Infrastructure Sources