Reliable Forwarding of syslog Messages with Rsyslog

Abstract

In this paper, I describe how to forward syslog messages (quite) reliable to a central rsyslog server. This depends on rsyslog being installed on the client system and it is recommended to have it installed on the server system. Please note that industry-standard plain TCP syslog protocol is not fully reliable (thus the “quite reliable”). If you need a truly reliable solution, you need to look into RELP (natively supported by rsyslog).*

The Intention

Whenever two systems talk over a network, something can go wrong. For example, the communications link may go down, or a client or server may abort. Even in regular cases, the server may be offline for a short period of time because of routine maintenance.

A logging system should be capable of avoiding message loss in situations where the server is not reachable. To do so, unsent data needs to be buffered at the client while the server is offline. Then, once the server is up again, this data is to be sent.

This can easily be accomplished by rsyslog. In rsyslog, every action runs on its own queue and each queue can be set to buffer data if the action is not ready. Of course, you must be able to detect that “the action is not ready”, which means the remote server is offline. This can be detected with plain TCP syslog and RELP, but not with UDP. So you need to use either of the two. In this howto, we use plain TCP syslog.

Please note that we are using rsyslog-specific features. They are required on the client, but not on the server. So the client system must run rsyslog (at least version 3.12.0), while on the server another syslogd may be running, as long as it supports plain tcp syslog.

The rsyslog queueing subsystem tries to buffer to memory. So even if the remote server goes offline, no disk file is generated. File on disk are created only if there is need to, for example if rsyslog runs out of (configured) memory queue space or needs to shutdown (and thus persist yet unsent messages). Using main memory and going to the disk when needed is a huge performance benefit. You do not need to care about it, because, all of it is handled automatically and transparently by rsyslog.

How To Setup

First, you need to create a working directory for rsyslog. This is where it stores its queue files (should need arise). You may use any location on your local system.

Next, you need to instruct rsyslog to use a disk queue and then configure your action. There is nothing else to do. With the following simple config file, you forward anything you receive to a remote server and have buffering applied automatically when it goes down. This must be done on the client machine.

$ModLoad imuxsock # local message reception
$WorkDirectory /rsyslog/work # default location for work (spool) files
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server:port

The port given above is optional. It may not be specified, in which case you only provide the server name. The “$ActionQueueFileName” is used to create queue files, should need arise. This value must be unique inside rsyslog.conf. No two rules must use the same queue file. Also, for obvious reasons, it must only contain those characters that can be used inside a valid file name. Rsyslog possibly adds some characters in front and/or at the end of that name when it creates files. So that name should not be at the file size name length limit (which should not be a problem these days).

Please note that actual spool files are only created if the remote server is down and there is no more space in the in-memory queue. By default, a short failure of the remote server will never result in the creation of a disk file as a couple of hundred messages can be held in memory by default. [These parameters can be fine-tuned. However, then you need to either fully understand how the queue works (read elaborate doc) or use professional services to have it done based on your specs ;) - what that means is that fine-tuning queue parameters is far from being trivial…]

If you would like to test if your buffering scenario works, you need to stop, wait a while and restart your central server. Do not watch for files being created, as this usually does not happen and never happens immediately.

$ModLoad imuxsock # local message reception
$WorkDirectory /rsyslog/work # default location for work (spool) files

# start forwarding rule 1
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd1 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server1:port
# end forwarding rule 1

# start forwarding rule 2
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd2 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server2
# end forwarding rule 2

Some Final Words on Reliability …

Using plain TCP syslog provides a lot of reliability over UDP syslog. However, plain TCP syslog is not a fully reliable transport. In order to get full reliability, you need to use the RELP protocol.

Follow the next link to learn more about the problems you may encounter with plain tcp syslog.

Revision History

• 2008-06-27 * Rainer Gerhards * Initial Version created

Copyright 2008-2023 Rainer Gerhards (Großrinderfeld), and Others.