Linux Server - Integration Guide

This guide will show you how to integrate a Linux server into the Secure60 platform.

Overview

This guide will explain how to monitor Linux instances through configuration of Syslog. As always if you have any additional questions, don’t hesitate to contact Secure60 Support.

This page is the complete Linux baseline

Syslog transport, the auditd setup, and the full Secure60 audit rule baseline are all here. Follow this page and a Linux host is done.

Once it is running, Linux Audit and File Integrity Monitoring (auditd) covers what comes next: adding your own file integrity watches, tuning the volume down on busy hosts, and turning the raw records into structured fields.

Install Syslog

  1. (Prerequisite) Install the Secure60 Collector
  2. (Optional) Install rsyslog on Linux Most Linux instances come preconfigured with syslog installed. Adding TLS Support normally requires an additional package. If needed manually install:
#Check if already exists:
systemctl status rsyslog

#Ubuntu
sudo apt install rsyslog
#RHEL / Rocky
sudo dnf install rsyslog

Then if needed add TLS Support

#RHEL / Rocky:
sudo dnf install rsyslog-gnutls
  1. Create a new file at nano /etc/rsyslog.d/10-tls-forward.conf with the following content:

# without ssl, default syslog port using UDP
*.* @<S60_COLLECTOR_IP_ADDRESS>:514 

# TLS settings for self-signed certs
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

# Forward all logs via TLS
*.* @@S60_COLLECTOR_IP_ADDRESS:6514

Ensure you replace <S60_COLLECTOR_IP_ADDRESS> with the actual IP address of your Secure60 collector. Also enable TLS validation with signed certificates as needed.

Capture systemd journal messages

We recommend also capturing systemd journal messages so that auditd/audisp output and other journal-based logs are forwarded to Secure60. This requires loading the imjournal module in rsyslog. If you already have a TLS forward block above, you can add just the module(load="imjournal"...) block to the same file and ensure *.* is forwarded; otherwise use the following complete example.

Update /etc/rsyslog.d/10-tls-forward.conf to contain the below (or a separate file in /etc/rsyslog.d/). Replace the example IP and port with your Secure60 collector address and port.

##################################################
# Secure60 journal input (needed for auditd/audisp)
##################################################
module(
  load="imjournal"
)

##################################################
# TLS settings for self-signed certs
##################################################
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

##################################################
# Forward all logs via TLS to Secure60 collector
##################################################
*.* @@S60_COLLECTOR_IP_ADDRESS:6514

Keeping the PID on long program names

rsyslog’s default forwarding template truncates the program tag at 32 characters. A tag such as run-parts(/etc/cron.hourly)[2651]: leaves the host as run-parts(/etc/cron.hourly)[2651 — the closing ]: is lost, so the PID cannot be separated from the program name.

This happens on the sending host, so it applies whether you forward straight to a collector or through a relay. Secure60 normalises app_name correctly either way, so this only matters if you want source_procid populated for programs with long tags.

To keep it, replace the *.* @@S60_COLLECTOR_IP_ADDRESS:6514 line with either of the following.

RFC3164 with the full, untruncated tag:

template(name="s60_fulltag" type="string"
 string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%\n")

*.* action(type="omfwd" target="S60_COLLECTOR_IP_ADDRESS" port="6514" protocol="tcp"
           StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="anon"
           template="s60_fulltag")

Or RFC5424, which carries the PID in its own header field:

*.* action(type="omfwd" target="S60_COLLECTOR_IP_ADDRESS" port="6514" protocol="tcp"
           StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="anon"
           template="RSYSLOG_SyslogProtocol23Format")

The three StreamDriver* settings must be repeated on the action. The $ActionSendStreamDriver* directives above do not apply to an action() block, and without them the host connects in plaintext, the collector rejects it, and the host forwards nothing while systemctl status rsyslog still reports active.

Forwarding through a central relay?

If these hosts send to a central syslog relay rather than straight to a collector, ip_src_address will show the relay for every host behind it. Syslog Relays - Keeping the Original Client covers the one change on the relay that fixes it.

To restart the Syslog service, execute the following commands:

systemctl restart rsyslog

To check the status of Syslog, use the command:

systemctl status rsyslog
journalctl -u rsyslog

Enabling Linux audit logging

Secure60 uses auditd, the kernel’s native audit subsystem, for host-level detection telemetry on Linux. The baseline below records what ran, who exists, who can escalate, what will run again after a reboot, and the mechanisms an intruder uses to hide — using auditctl rules, the format CIS benchmarks and audit standards are written in.

No third-party agent is involved. Audit records travel the syslog path this page has already configured.

Most distributions ship the auditd daemon, but the syslog plugin is a separate package on every distribution — it is not part of the base audit package. Install both, then enable the plugin:

# Ubuntu / Debian
sudo apt install -y auditd audispd-plugins

# RHEL / Rocky / Alma
sudo dnf install -y audit audispd-plugins

sudo nano /etc/audit/plugins.d/syslog.conf
# Change active = no to active = yes
sudo systemctl restart auditd

Without audispd-plugins there is no /etc/audit/plugins.d/syslog.conf to edit and no audisp-syslog binary to run — on a stock Rocky 9 host the base audit package ships only af_unix.conf.

Change only active. Leave path and type as your distribution ships them: auditd 3.1 and later use path = /sbin/audisp-syslog with type = always, while 3.0 and earlier use path = builtin_syslog with type = builtin. Setting the older values on auditd 3.1 stops forwarding silently — rules still load and the local audit log still fills, but nothing reaches Secure60 and no error is logged.

The Secure60 audit rule baseline

Nothing arrives until rules are loaded — a freshly installed auditd has no syscall rules and no watches, so it produces almost no records.

Create /etc/audit/rules.d/50-secure60.rules:

## Secure60 - baseline audit rules
## Place at: /etc/audit/rules.d/50-secure60.rules
##
## Rule order matters. For a given syscall the kernel walks the list and
## stops at the FIRST match, so any rule that filters execve more tightly
## than "-S execve" must sit above the broad rule or it never fires.

############################
# 0) Volume control
#    PROCTITLE repeats, hex-encoded, the command line the EXECVE record
#    already carries. Roughly one record in seven, and pure duplication.
#    Do NOT add PATH here - it is the only record carrying the filename
#    for the file watches below.
############################
-a always,exclude -F msgtype=PROCTITLE

############################
# 1) Privilege escalation
#    Placed above the broad execve rule on purpose - see the note above.
############################
-a always,exit -F arch=b64 -S execve -C uid!=euid -F euid=0 -k s60_privesc
-a always,exit -F arch=b32 -S execve -C uid!=euid -F euid=0 -k s60_privesc

############################
# 2) Process execution
#    The largest source of events on most hosts. Everything else in this
#    file is close to free; this is the rule that costs volume.
############################
-a always,exit -F arch=b64 -S execve -k s60_exec
-a always,exit -F arch=b32 -S execve -k s60_exec

############################
# 3) Identity and group management
#    Captures useradd/usermod/groupadd/passwd changes via file writes.
############################
-w /etc/passwd           -p wa -k s60_identity
-w /etc/shadow           -p wa -k s60_identity
-w /etc/group            -p wa -k s60_identity
-w /etc/gshadow          -p wa -k s60_identity
-w /etc/security/opasswd -p wa -k s60_identity
-w /etc/login.defs       -p wa -k s60_identity

############################
# 4) Privilege configuration changes
############################
-w /etc/sudoers    -p wa -k s60_privcfg
-w /etc/sudoers.d/ -p wa -k s60_privcfg

############################
# 5) SSH and authentication control changes
############################
-w /etc/ssh/sshd_config -p wa -k s60_authcfg
-w /etc/pam.d/          -p wa -k s60_authcfg

############################
# 6) Persistence mechanisms
############################
-w /etc/crontab       -p wa -k s60_persist
-w /etc/cron.d/       -p wa -k s60_persist
-w /etc/cron.hourly/  -p wa -k s60_persist
-w /etc/cron.daily/   -p wa -k s60_persist
-w /etc/cron.weekly/  -p wa -k s60_persist
-w /etc/cron.monthly/ -p wa -k s60_persist
-w /var/spool/cron/   -p wa -k s60_persist

-w /etc/systemd/system/     -p wa -k s60_persist
-w /usr/lib/systemd/system/ -p wa -k s60_persist
-w /lib/systemd/system/     -p wa -k s60_persist

-w /etc/profile    -p wa -k s60_persist
-w /etc/profile.d/ -p wa -k s60_persist
-w /etc/bashrc      -p wa -k s60_persist   # RHEL family
-w /etc/bash.bashrc -p wa -k s60_persist   # Debian / Ubuntu

############################
# 7) Kernel module loading
#    Rootkits and kernel-level persistence.
############################
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module -k s60_modules
-a always,exit -F arch=b32 -S init_module,finit_module,delete_module -k s60_modules
-w /etc/modprobe.d/     -p wa -k s60_modules
-w /etc/modules-load.d/ -p wa -k s60_modules

############################
# 8) Dynamic loader configuration
#    LD_PRELOAD-style library hijacking. /etc/ld.so.preload usually does
#    not exist, and its creation is exactly what you want to catch - the
#    watch loads fine because its parent directory exists.
############################
-w /etc/ld.so.conf    -p wa -k s60_loader
-w /etc/ld.so.conf.d/ -p wa -k s60_loader
-w /etc/ld.so.preload -p wa -k s60_loader

############################
# 9) System time changes (anti-forensics)
############################
-a always,exit -F arch=b64 -S adjtimex,settimeofday,clock_settime -k s60_time
-a always,exit -F arch=b32 -S adjtimex,settimeofday,clock_settime -k s60_time
-w /etc/localtime -p wa -k s60_time

############################
# 10) Network and name-resolution configuration
############################
-a always,exit -F arch=b64 -S sethostname,setdomainname -k s60_netcfg
-a always,exit -F arch=b32 -S sethostname,setdomainname -k s60_netcfg
-w /etc/hosts         -p wa -k s60_netcfg
-w /etc/hosts.allow   -p wa -k s60_netcfg
-w /etc/hosts.deny    -p wa -k s60_netcfg
-w /etc/resolv.conf   -p wa -k s60_netcfg
-w /etc/nsswitch.conf -p wa -k s60_netcfg
-w /etc/sysctl.conf   -p wa -k s60_netcfg
-w /etc/sysctl.d/     -p wa -k s60_netcfg

############################
# 11) Mandatory access control
#     Keep the line that matches your distribution.
############################
-w /etc/selinux/    -p wa -k s60_mac   # RHEL family
-w /etc/apparmor/   -p wa -k s60_mac   # Debian / Ubuntu
-w /etc/apparmor.d/ -p wa -k s60_mac   # Debian / Ubuntu

############################
# 12) Audit configuration changes (tamper detection)
############################
-w /etc/audit/         -p wa -k s60_auditcfg
-w /etc/audit/rules.d/ -p wa -k s60_auditcfg

############################
# 13) Login, session and lockout records
############################
-w /var/log/lastlog  -p wa -k s60_logins
-w /var/run/faillock -p wa -k s60_logins
-w /var/log/wtmp -p wa -k s60_session
-w /var/log/btmp -p wa -k s60_session
-w /var/run/utmp -p wa -k s60_session

############################
# 14) Process inspection
#     Debuggers and process injection. The execve rules in (2) already
#     cover sudo, su and pkexec by name, so the per-binary "privileged
#     command" lists in older benchmarks are redundant here.
############################
-a always,exit -F arch=b64 -S ptrace -k s60_ptrace

############################
# 15) Removable media and unexpected mounts
############################
-a always,exit -F arch=b64 -S mount -F auid>=1000 -F auid!=unset -k s60_mount

Load and validate:

sudo augenrules --load
sudo auditctl -l

augenrules merges every file in rules.d into one ruleset in filename order, so an error is reported against line numbers in the merged output — check the source files if it complains.

One bad line stops every rule below it

augenrules --load halts at the first line it cannot load. Rules above that point are live; everything below is silently absent. There is no second error, and auditctl -l returns a shorter list than was written.

A watch resolves its path when the rule loads, and it fails if the parent directory is missing. The file itself need not exist — /etc/ld.so.preload watches fine on a host that has never had one, because /etc exists — but /etc/apparmor.d/ on a RHEL host takes out every rule after it.

This is why the baseline marks its distribution-specific lines. /etc/bashrc exists on RHEL but not Ubuntu, /etc/bash.bashrc the other way round, and /etc/selinux/ and /etc/apparmor/ never both. Delete the lines that do not apply to your distribution before loading.

Then verify the count, every time:

sudo augenrules --load          # any "There was an error in line N" is a hard failure
sudo auditctl -l | grep -c .    # compare against what you wrote
sudo auditctl -l | grep s60_mount   # the last rule in the file - if it is here, all of it loaded

Checking the last rule is the cheapest complete test: because loading stops at the first failure, the presence of the final rule proves every rule before it loaded too.

Confirm records are reaching Secure60

Do this after loading the rules — a host with no audit rules produces no SYSCALL records, so the check fails on a perfectly healthy system.

/usr/bin/id > /dev/null

# 1. Are the rules producing records at all?
sudo ausearch -k s60_exec -ts recent | head

# 2. Is the syslog plugin running?
pgrep -af audisp-syslog

# 3. Did the records reach the syslog daemon?
sudo grep -c "type=SYSCALL" /var/log/messages   # RHEL family
sudo grep -c "type=SYSCALL" /var/log/syslog     # Debian / Ubuntu

Check 3 is the one that matters — it is the last point on the host before the records go to the collector.

Symptom Cause
Check 1 empty Rules did not load. Run sudo auditctl -l and compare against your rules file
Check 1 fine, check 2 empty audispd-plugins is not installed, or active = no in syslog.conf
Checks 1 and 2 fine, check 3 zero path or type is wrong for your audit version — see above

Then confirm in the portal. Open Search and query:

app_name = 'audisp-syslog'

Re-run all of this after any patching that upgrades the audit packages.

journalctl is not a reliable check here

Two things make it return a confident-looking zero on a healthy host.

Without sudo it shows almost nothing. A non-root journalctl is scoped to the invoking user’s journal, so audit records written by root are absent from it.

On some hosts the journal is not in the path at all. audisp-syslog writes to /dev/log. Where that socket belongs to systemd — the RHEL family default — the records pass through the journal. Where rsyslog owns /dev/log directly through imuxsock, or the host does not run systemd, a healthy host has zero audit records in its journal.

Check the file rsyslog writes, not the journal.

What this costs, and what to do if it is too much

Only one rule in the baseline generates meaningful volume: process execution. Measured on a steady-state Rocky 9 host, every other group in the file — kernel modules, loader configuration, time, network, MAC policy, audit-config tamper, login records, ptrace, mounts — produced zero records in sixty seconds. Detection breadth on a Linux host is close to free; execution auditing is what costs money.

The execve pair audits every process launch, on both the 64-bit and 32-bit interfaces, with no filtering. On a busy server it is commonly the dominant share of that host’s ingest — larger than the application logs beside it.

Enable the baseline on one representative host, measure a full day, and multiply that figure across the fleet rather than using a general estimate. If the number is too high, narrow the execve rules rather than removing them — Linux Audit and File Integrity Monitoring (auditd) has the narrowing recipes with worked examples, along with the auditd disk-full settings and the rate limits that silently drop records on high-volume hosts.

Do not exclude PATH

-F msgtype=PATH is widely recommended alongside PROCTITLE as a volume measure, and on a host doing execution auditing only, it is fine.

On a host running the file watches in this baseline it removes the filename from every event they produce. The PATH record is the only record that carries the name of the file that changed. With it excluded, a write to /etc/shadow still produces a SYSCALL record with the user, the process and the syscall, and no record names the file. CWD carries the working directory needed to resolve relative paths, and excluding it has the same effect on those.

If PATH volume is the problem, narrow the execve rules instead. Those are what generate the PATH records you do not want; the file watches generate very few.

Getting structured fields out of the records

Raw auditd records arrive as a single blob of key=value text in message_text. You do not need to write a parser for them — Secure60 ships one.

Go to Integrations → Secure60 Collector → Parser Templates, deploy the Linux Platform template into your organisation, and attach it to the collector profile assigned to your collector group. It covers every auditd record type, plus sshd, sudo, su, cron, PAM, systemd and iptables from the same hosts. See Parser Templates for the workflow.

Once it is running, the rule keys above become a searchable field, so each group in the baseline is one query:

Query What it shows
product = 'auditd' Every audit record from the estate
event_audit_key = 's60_exec' Process execution
event_audit_key = 's60_privesc' Execution that crossed into root
event_audit_key = 's60_identity' Changes to /etc/passwd, /etc/shadow, group files
event_audit_key = 's60_privcfg' Changes to sudo configuration
event_audit_key = 's60_persist' New cron jobs, systemd units, profile scripts
event_audit_key = 's60_modules' Kernel module loads and unloads
event_audit_key = 's60_loader' Library preload and loader configuration
event_audit_key = 's60_auditcfg' Attempts to change auditing itself

Before the template is deployed, app_name = 'audisp-syslog' selects the same events and the keys are only present inside message_text.

The parser also combines the records of one audit event — SYSCALL, EXECVE, CWD and every PATH — into a single event, so the user, the process, the command line and the file all land together. For file operations the file is event_target_filename, its directory is file_directory, and a rename or link keeps the previous name in event_source_filename; event_directory is the working directory from the CWD record and is what relative names are resolved against. The auditd file integrity guide has the full field table.

Back to top