[syslog-ng] How to manage the Syslog-NG messages coming from different sites? Each sites has their Syslog-NG clients?

[Fekete Róbert]

Hi, 

A template is actually a way the message is formatted. Usually it includes only macros, but it can include any text. For example:
template t_mytemplate { 
             template("site1#$MSG\n");}; 
and then modify this template for site2, and so on.

Or if you are using syslog-ng OSE 3.2 or later, you can use a variable in the template, and define the sitename earlier in the config file: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/config-global-variables.html

Now that I think about it, it might be possible to avoid templates and modify the value of the $MSG macro using a rewrite rule while keeping its value. 

rewrite r_rewrite_set{set("sitecode#$MSG", value("MSG"));};, 
But I am not sure that macros are permitted in the first parameter - I have to check it with Bazsi or someone more well-versed in the source code, but I would be surprised if it was not possible. 

Then on the server side, you can define a parser that segments the message into two parts at the # character (or any other delimiter of your liking), something like:
csv_parser(columns("SITECODE", "MYMESSAGE") delimiters("#") flags(greedy));

Then you can refer to the sitecode using $SITECODE, and to the rest of the message using $MYMESSAGE
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/configuring_parsers.html
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guide-admin-en.html/reference_parsers_csv.html

Of course, you are not necessarily limited to adding only the sitecode to the messages, you can add other things as well (like the role of the host, or the department it belongs to, etc.), but for this using the SDATA part of the RFC5424 message format is more suitable.

Check out Chapters 11 and 12 in the admin guide, I am sure you'll get some more ideas about how to solve this problem.

Robert

On Wednesday, August 3, 2011 18:44 CEST, Marcos Tang <marcostang2002 at yahoo.com> wrote: 
 
> Hi Robert,
> 
> Thanks for your suggestion and I am now trying to follow your 3rd suggestion. I will let you know if I can make it or not.
> 
> However, I would also like to know more the details of your 1st suggestion. My understanding of template is Syslog-NG has some built-in MACRO such as HOST, FACILITY, PRIORITY, TAG, PROGRAM, MSG and etc. Other than those built-in MACRO, can I create some for myself, such as SITE_CODE? It would be appreciated if you can let me know more about it or show me the pointer how to learn more about the template.
>  
> Regards,
> Marcos
> 
> 
> ________________________________
> From: Fekete Robert <frobert at balabit.hu>
> To: Marcos Tang <marcostang2002 at yahoo.com>; Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
> Sent: Wednesday, August 3, 2011 9:05 PM
> Subject: Re: [syslog-ng] How to manage the Syslog-NG messages coming from different sites? Each sites has their Syslog-NG clients?
> 
> Hi,
> 
> There are several ways to achieve this:
> - Use a custom template in the destination definition of your clients (or the 
> site's main syslog server that forwards the messages to your central logserver) 
> that includes the sitename in the log message, then extract this info from the 
> message (for example, with a csv parser) on the server side
> 
> - If you use the RFC5424 syslog message format, you can add the sitename into 
> the SDATA part of the message, and access it with a macro on the central server
> 
> - You can configure your central server to listen on different IP addresses or 
> different ports, one for each site, and create separate sources in your server's 
> config for every site. syslog-ng automatically adds a tag (the id of the source) 
> to every message, so you can use this tag to filter the messages to different 
> databases or tables (or just include it in an extra column).
> 
> HTH,
> 
> Robert
> 
> On 08/03/2011 12:20 PM, Marcos Tang wrote:
> 
> > Hi,
> >
> > I have a problem to manage the Syslog-NG messages sending from different
> > Syslog-NG clients at different remote sites.
> >
> > For example, I have 10 sites (site1, site2, site3 to site 10) running Syslog-NG
> > clients. Each site has one Syslog-NG server and all the Syslog messages will be
> > forwarded to the centralized Syslog-NG server and insert into the MySQL database
> > at the headquarter finally.
> >
> > Now, I can see all the Syslog messages at the headquarter MySQL database; but it
> > is hard for me to manage them.
> >
> > For example, if I know the hostname of a particular host, I can query the MySQL
> > database to search for the Syslog related to that host. However, if I want to
> > know the Syslog messages coming from a particular site such as "site1", how can
> > I do that? There is no "site1" information inside the Syslog messages.
> >
> > I am thinking if I can modify the Syslog-NG configuration file such that I can
> > add some information such as "site1" and make it as part of the Syslog messages,
> > I can query the MySQL database for "site1" pattern. But I am not sure if I can
> > do that or not.
> >
> > If you have any suggestions/ideas, please let me know that.
> > Regards,
> > Marcos
> >
> >
> >
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >