[syslog-ng] How to manage the Syslog-NG messages coming from different sites? Each sites has their Syslog-NG clients?

[Balazs Scheidler]

On Wed, 2011-08-03 at 20:36 +0200, Fekete Róbert wrote:
> Hi, 
> 
> A template is actually a way the message is formatted. Usually it includes only macros, but it can include any text. For example:
> template t_mytemplate { 
>              template("site1#$MSG\n");}; 
> and then modify this template for site2, and so on.
> 
> Or if you are using syslog-ng OSE 3.2 or later, you can use a variable in the template, and define the sitename earlier in the config file: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/config-global-variables.html
> 
> Now that I think about it, it might be possible to avoid templates and modify the value of the $MSG macro using a rewrite rule while keeping its value. 
> 
> rewrite r_rewrite_set{set("sitecode#$MSG", value("MSG"));};, 
> But I am not sure that macros are permitted in the first parameter - I have to check it with Bazsi or someone more well-versed in the source code, but I would be surprised if it was not possible. 
> 
> Then on the server side, you can define a parser that segments the message into two parts at the # character (or any other delimiter of your liking), something like:
> csv_parser(columns("SITECODE", "MYMESSAGE") delimiters("#") flags(greedy));
> 
> Then you can refer to the sitecode using $SITECODE, and to the rest of the message using $MYMESSAGE
> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_parsers.html
> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/reference_parsers_csv.html
> 
> Of course, you are not necessarily limited to adding only the sitecode to the messages, you can add other things as well (like the role of the host, or the department it belongs to, etc.), but for this using the SDATA part of the RFC5424 message format is more suitable.
> 
> Check out Chapters 11 and 12 in the admin guide, I am sure you'll get some more ideas about how to solve this problem.

Instead of adding the site information to the $MSG part, I'd add this to
the hostname, it's much more natural and easier to handle even with
defaults.

Or as Robert suggested in his earlier e-mail, use RFC5424 format and
structured data.

Also, a message within syslog-ng has a number of things that enhance the
original "syslog" idea. Each message within syslog-ng has the following
properties:

  * facility/severity: just like good old syslog
  * timestamps: two complete sets, one for the reception time stamp, the
other for the time as was received from the peer ($R_DATE and S_DATE
respectively)
  * name-value pairs: are named properties that can store textual
information. You can define any kind of name-value pairs as you see fit,
using parsers (csv-parser and db-parser), rewrite rules (set & subst).

You can use  information stored in the message using templates, a syntax
used at a lot of different places inside syslog-ng. Templates can be
used to customize the output format in a file, but templates can be also
used to name files or SQL tables.

Templates can contain both the "builtin" properties of a message, but
with the same syntax it can also extract user-defined stuff.

For instance:

# this defines a name-value pair named FOO
rewrite r_foo { set("foobar" value("FOO")); };

# later this can be referenced in a destination file template
destination d_file { file("/var/log/${FOO}"); };

# and this one glues these together
log { source(...); rewrite(r_foo); destination(d_file); };

At the destination side, it doesn't matter how the name-value pair named
$FOO gets its value. If it is set, it'll be used.

-- 
Bazsi