Templates

Constant statement

This provides a way to specify constant text. The text is used literally. It is primarily intended for text-based output, so that some constant text can be included. For example, if a complex template is built for file output, one usually needs to finish it by a newline, which can be introduced by a constant statement. Here is an actual sample of that use case from the rsyslog testbench:

template(name="outfmt" type="list") {
    property(name="$!usr!msgnum")
    constant(value="\n")
}

The following escape sequences are recognized inside the constant text:

• \\ - single backslash

• \n - LF

• \ooo - (three octal digits) - represents a character with this octal numerical value (e.g. \101 equals “A”). Note that three octal digits must be given (in contrast to some languages, where between one and three are valid). While we support octal notation, we recommend to use hex notation as this is better known.

• \xhh - (where h is a hex digit) - represents a character with this hexadecimal numerical value (e.g. \x41 equals “A”). Note that two hexadecimal digits must be given (in contrast to some languages where either one or two are valid).

• … some others … list needs to be extended

Note: if an unsupported character follows a backslash, this is treated as an error. Behaviour is unpredictable in this case.

To aid usage of the same template both for text-based outputs and structured ones, constant text without an “outname” parameter will be ignored when creating the name/value tree for structured outputs. So if you want to supply some constant text e.g. to mongodb, you must include an outname, as can be seen here:

template(name="outfmt" type="list") {
    property(name="$!usr!msgnum")
    constant(value="\n" outname="IWantThisInMyDB")
}

To generate a constant json field, the format parameter can be used, as in this example

template(name="outfmt" type="list" option.jsonf="on") {
          property(outname="message" name="msg" format="jsonf")
          constant(outname="@version" value="1" format="jsonf")
}

The constant statement in this case will generate “@version”:”1”. Note that to do this, both the value and the format parameters must be given.

The “constant” statement supports the following parameters:

• value - the constant value to use

• outname - the output field name (for structured outputs)

• format - can be either empty or jsonf

Property statement

This statement is used to include property values. It can access all properties. Also, options permit to specify picking only part of a property or modifying it. It supports the following parameters:

• name - the name of the property to access

• outname - the output field name (for structured outputs)

• dateformat - the date format to use (only for date-related properties). Here you can find a list of all property options. TODO: right now, the property replacer documentation contains property format options for string templates, only. The formats for non-string templates differ. For example, date format options in string templates start with “date-” whereas those in property statements do not (e.g. “date-year” vs. just “year”). The technical reason behind this is that inside string templates, the option must include what it applies to whereas with the explicit format that is part of the parameter name.

  To create a customized format you can use multiple property options together. The following example would result in YYYY-MM-DD:

property(name="timereported" dateformat="year")
constant(value="-")
property(name="timereported" dateformat="month")
constant(value="-")
property(name="timereported" dateformat="day")

• date.inUTC - date shall be shown in UTC (please note that this requires a bit more performance due to the necessary conversions) Available since 8.18.0.

• caseconversion - permits to convert case of the text. Supported values are “lower” and “upper”

• controlcharacters - specifies how to handle control characters. Supported values are “escape”, which escapes them, “space”, which replaces them by a single space, and “drop”, which simply removes them from the string.

• securepath - used for creating pathnames suitable for use in dynafile templates. Supported values are “drop” and “replace”.

• format - specify format on a field basis. Supported values are:

  • “csv“ for use when csv-data is generated

  • “json“ which formats proper json content (but without a field header)

  • “jsonf“ which formats as a complete json field

  • “jsonr“ which avoids double escaping the value but makes it safe for a json field

  • “jsonfr“ which is the combination of “jsonf” and “jsonr”.

• position.from - obtain substring starting from this position (1 is the first position)

• position.to - obtain substring up to this position. As a special extension available since 8.2302.0. the position.to value may be negative (-n). In this case, all characters from position.from to the end of the actual property string except the last n characters are extracted. This makes it easy to strip the beginning and end of a string with changing legth. Let us assume you have a string “[abc]” and want to remove the braces. In this case, use position.from=”2” position.to=”-1” in the property() object. This will result in the string “abc” after template processing. This is especially useful if you want to drop the braces around the STRUCTURED-DATA message property.

• position.relativeToEnd - the from and to position is relative to the end of the string instead of the usual start of string. (available since rsyslog v7.3.10)

• fixedwidth - changes behaviour of position.to so that it pads the source string with spaces up to the value of position.to if the source string is shorter. “on” or “off” (default) (available since rsyslog v8.13.0)

• compressspace - compresses multiple spaces (US-ASCII SP character) inside the string to a single one. This compression happens at a very late stage in processing. Most importantly, it happens after substring extraction, so the position.from and position.to positions are NOT affected by this option. (available since v8.18.0).

• field.number - obtain this field match

• field.delimiter - decimal value of delimiter character for field extraction

• regex.expression - expression to use

• regex.type - either ERE or BRE

• regex.nomatchmode - what to do if we have no match

• regex.match - match to use

• regex.submatch - submatch to use

• droplastlf - drop a trailing LF, if it is present

• mandatory - signifies a field as mandatory. If set to “on”, this field will always be present in data passed to structured outputs, even if it is empty. If “off” (the default) empty fields will not be passed to structured outputs. This is especially useful for outputs that support dynamic schemas (like ommongodb).

• spifno1stsp - expert options for RFC3164 template processing

• datatype - for “jsonf” format ONLY; permits to set a datatype Log messages as string data types natively. Thus every property inside rsyslog is string based. However, in some end systems you need different data types like numbers of boolean. This setting, in jsonf mode, permits to configure a desired data type. Supported data types are:

  • number - value is treated as a JSON number and not enclosed in quotes.

    If the property is empty, the value 0 is generated.

  • string - value is a string and enclosed in quotes

  • auto - value is treated as number if numeric and as string otherwise.

    The current implementation treats only integers as numeric to avoid confusion.

  • bool - the value is treated as boolean. If it is empty or 0, it will generate “false”, else “true”.

  If not specified, ‘string’ datatype is assumed. This is a feature of rsyslog 8.1905.0 or later.

• onEmpty - for “jsonf” format ONLY; specifies how empty values shall be handled. Possible values are:

  • keep - emit the empty element

  • skip - completely ignore the element, do not emit anything

  • null - emit a JSON ‘null’ value

  If not specified, ‘keep’ is assumed. This is a feature of rsyslog 8.1905.0 or later.

Use case

A typical use case is to first create a custom subtree and then include it into the template, like in this small example:

set $!usr!tpl2!msg = $msg;
set $!usr!tpl2!dataflow = field($msg, 58, 2);
template(name="tpl2" type="subtree" subtree="$!usr!tpl2")

Here, we assume that $msg contains various fields, and the data from a field is to be extracted and stored - together with the message - as field content.

Options

The <options> part is optional. It carries options influencing the template as a whole and is a part of the template parameters. See details below. Be sure NOT to mistake template options with property options - the latter ones are processed by the property replacer and apply to a SINGLE property, only (and not the whole template). Template options are case-insensitive. Currently defined are:

option.sql - format the string suitable for a SQL statement in MariaDB/ MySQL format. This will replace single quotes (”’”) and the backslash character by their backslash-escaped counterpart (”\’” and “\\”) inside each field. Please note that in MySQL configuration, the NO_BACKSLASH_ESCAPES mode must be turned off for this format to work (this is the default).

option.stdsql - format the string suitable for a SQL statement that is to be sent to a standards-compliant sql server. This will replace single quotes (”’”) by two single quotes (”’’”) inside each field. You must use stdsql together with MySQL if in MySQL configuration the NO_BACKSLASH_ESCAPES is turned on.

option.json - format the string suitable for a json statement. This will replace single quotes (”’”) by two single quotes (”’’”) inside each field.

option.jsonf - format the string as JSON object. This means a leading and trailing curly brace “{” will be added as well as a comma between all non-terminal properties and constants.

option.casesensitive - treat property name references as case sensitive. The default is “off”, where all property name references are first converted to lowercase during template definition. With this option turned “on”, property names are looked up as defined in the template. Use this option if you have JSON ($!*), local (!.*), or global ($!\\*) properties which contain uppercase letters. The normal Rsyslog properties are case-insensitive, so this option is not needed for properly referencing those properties.

Use of the options option.sql, option.stdsql, and option.json are mutually exclusive. Using more than one at the same time can cause unpredictable behaviour.

Either the sql or stdsql option must be specified when a template is used for writing to a database, otherwise injection might occur. Please note that due to the unfortunate fact that several vendors have violated the sql standard and introduced their own escape methods, it is impossible to have a single option doing all the work.  So you yourself must make sure you are using the right format. If you choose the wrong one, you are still vulnerable to sql injection. Please note that the database writer checks that the sql option is present in the template. If it is not present, the write database action is disabled. This is to guard you against accidentally forgetting it and then becoming vulnerable to SQL injection. The sql option can also be useful with files - especially if you want to import them into a database on another machine for performance reasons. However, do NOT use it if you do not have a real need for it - among others, it takes some toll on the processing time. Not much, but on a really busy system you might notice it.

The default template for the write to database action has the sql option set. As we currently support only MariaDB/MySQL and the sql option matches the default MariaDB/MySQL configuration, this is a good choice. However, if you have turned on NO_BACKSLASH_ESCAPES in your MySQL config, you need to supply a template with the stdsql option. Otherwise you will become vulnerable to SQL injection.

template (name="TraditionalFormat" type="string"
string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n")