Syslog Relays - Keeping the Original Client

Most sites do not point every host at the collector. They point every host at one central syslog relay and forward from there. Syslog has no field for “who connected to me”, so by default the collector records the relay as the source of every event behind it.

This page shows what a relay costs you, and the one configuration change that fixes it.

Overview

The short version

Change the relay’s outbound forwarding to RFC5424 with an origin element. That is one file on one host, and it restores the client IP for every host behind it. The hosts behind the relay need no change.

What a relay does and does not damage

Measured with rsyslog 8.2510 relaying to a Secure60 Collector over TLS:

Field Sent direct to the collector Sent through a default relay
host_name the origin host the origin host - survives
app_name the program the program - survives
message_text intact intact
ip_src_address the origin host the relay

Host identity is not the problem. A relay passes the original syslog HOSTNAME header through unchanged, and Secure60 reads host_name from that header rather than from the network connection.

The address is the problem, and it is wrong rather than missing, which is worse:

The client address is genuinely absent from the message. The relay is the only party that knows it, so the relay is where the fix goes.

Choosing a syslog format

Format Carries the client IP? Notes
RFC3164 (BSD) No - the format has no field for it rsyslog’s default. Timestamp has no year, no timezone and no sub-second precision
RFC5424 Yes, in an origin structured-data element Recommended. RFC3339 timestamps with timezone and milliseconds; program name and PID in their own header fields
EWMM (syslog-ng only) Not by default Carries the origin hostname inside a JSON body, but no source IP unless you add one. Secure60 does not map its field names today

RFC5424’s structured-data section is the only standard place a relay can record the address it received a message from. That is why the fix requires it.

Configuring an rsyslog relay

On the relay, replace the forwarding configuration in /etc/rsyslog.d/ with the following. Replace S60_COLLECTOR_IP_ADDRESS with your collector address.

##################################################
# Required for the buffer below. Without a workDirectory rsyslog cannot
# create queue files, logs one error at startup, and then runs memory-only.
##################################################
global(workDirectory="/var/lib/rsyslog")

##################################################
# Receive from the hosts behind this relay
##################################################
module(load="imtcp")
input(type="imtcp" port="514")

module(load="imudp")
input(type="imudp" port="514")

##################################################
# TLS settings for self-signed certs
##################################################
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

##################################################
# Preserve structured data the sender already set ("-" means none)
##################################################
set $.s60sd = "";
if ($structured-data != "-") then { set $.s60sd = $structured-data; }

##################################################
# Forward in RFC5424, naming the real client
##################################################
template(name="s60_relay_fwd" type="string"
 string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% [origin ip=\"%FROMHOST-IP%\"]%$.s60sd% %MSG%\n")

*.* action(type="omfwd" target="S60_COLLECTOR_IP_ADDRESS" port="6514" protocol="tcp"
           StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="anon"
           template="s60_relay_fwd"
           queue.type="LinkedList"
           queue.filename="s60fwd"
           queue.maxdiskspace="1g"
           queue.saveOnShutdown="on"
           action.resumeRetryCount="-1")

Then restart and confirm it is shipping:

rsyslogd -N1 -f /etc/rsyslog.conf
systemctl restart rsyslog
ss -tnp | grep 6514

The connection must reach ESTABLISHED. Requires rsyslog 7 or later, which covers RHEL/Rocky 7 and newer.

Remove the old forwarding line

Delete any existing *.* @@collector:6514 line. Leaving it in place alongside the new action ships every event twice, once in each format.

The TLS settings must be repeated on the action

The $ActionSendStreamDriver* directives at the top do not apply to an action() block. Without the three StreamDriver* settings on the action itself, the relay connects in plaintext, the collector rejects it, and the relay ships nothing - while systemctl status rsyslog still reports active. The tell is ptcp network driver: CheckConnection detected broken connection in the relay’s log.

Defining the template is not enough

A template() on its own does nothing. It only takes effect when an action() references it by name.

Configuring a syslog-ng relay

destination d_s60 {
  syslog("S60_COLLECTOR_IP_ADDRESS" port(6514) transport("tls")
         template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} - [origin ip=\"${SOURCEIP}\"] ${MSG}\n"));
};

Use this rather than the syslog-ng() destination driver, which emits EWMM. EWMM moves the original hostname into a JSON body and replaces the syslog header with the relay’s own details, which Secure60 does not currently unpack.

What Secure60 records

Field Meaning
ip_src_address The host that produced the event, taken from the origin element
syslog_origin_address The same value, kept separately so you can see the event was relayed
syslog_ip_src_address The relay that delivered it
host_name The syslog HOSTNAME header - the original host

An address found inside the message itself always wins. An SSH record reading Accepted password for jsmith from 198.51.100.9 still reports 198.51.100.9, because that is the address that matters for that event.

Requires a collector running the stable image. On older collectors the address still arrives — as a raw origin.ip field — but ip_src_address continues to show the relay. See Updating the Collector.

Buffering through a collector outage

The queue settings in the example above buffer to disk if the collector becomes unreachable. Without them rsyslog discards - and on a relay that means every host behind it, for the whole outage.

Always set queue.maxdiskspace

Setting queue.filename switches rsyslog into disk-assisted mode, where maxdiskspace defaults to unlimited. Its goal there is to never lose a message, so paired with action.resumeRetryCount="-1" and an unreachable collector it will grow until the filesystem is full. Size it to what you can afford to give logs.

If you would rather drop than buffer, remove the five queue.* and action.* lines. Nothing else changes.

Program names and PIDs are a sender-side problem

Separately from the client IP, rsyslog’s default forwarding template truncates the program tag at 32 characters. A tag such as run-parts(/etc/cron.hourly)[2651]: goes out as run-parts(/etc/cron.hourly)[2651 - the closing ]: is lost and the PID can no longer be separated.

This happens on the sending host at the first hop, before any relay is involved, so the relay template above cannot undo it. Secure60 normalises app_name correctly regardless, so this only matters if you want source_procid populated. The fix belongs on the sending hosts and is covered in the Linux Server Integration Guide.

A note on structured data

Secure60 parses well-formed RFC5424 structured data and adds it to the event. Malformed structured data is a different matter - a device that puts its entire payload where structured data belongs can have its message truncated, or dropped altogether.

The template on this page only ever emits a single well-formed [origin ip="..."] element, so it is safe. If you are configuring a device that emits its payload in the structured-data position - CheckPoint Log Exporter in its “Generic” or “Syslog” format is the common case - change the device to JSON or CEF output instead.

Confirming it worked

Search for events from a host behind the relay and check that ip_src_address is that host rather than the relay:

Query What it shows
host_name = 'yourhost' Events from one host behind the relay
syslog_origin_address != '' Every event that arrived via a relay that names its client
ip_src_address = 'RELAY_IP' Should be empty once the relay is configured
Back to top