No, I think he needs a way to distinguish be site1 and site2, where site1 have servera/serverc/serverf, and site2 have serverb/serverd/servere/serveri I would consider the chain_hostnames() option on the relay/site-syslog server On Wed, Aug 3, 2011 at 2:51 PM, Clayton Dukes <cdukes@gmail.com> wrote:

Hi Marcos, Are you saying that the recieving server is not keeping the original hostname? If so, just use keep_hostname(yes); in your global options.

Here are my settings: options {       long_hostnames(off);       # doesn't actually help on Solaris, log(3) truncates at 1024 chars       log_msg_size(8192);       # buffer just a little for performance       flush_lines(1);       # memory is cheap, buffer messages unable to write (like to loghost)       log_fifo_size(16384);       # The time to wait before a dead connection is reestablished (seconds)       time_reopen(10);       #Use DNS so that our good names are used, not hostnames       use_dns(yes);       dns_cache(yes);       #Use the whole DNS name       use_fqdn(yes);       keep_hostname(yes);       chain_hostnames(no);       #Read permission for everyone       perm(0644); };

______________________________________________________________

Clayton Dukes ______________________________________________________________

On Wed, Aug 3, 2011 at 6:20 AM, Marcos Tang <marcostang2002@yahoo.com> wrote:

...

Hi, I have a problem to manage the Syslog-NG messages sending from different Syslog-NG clients at different remote sites. For example, I have 10 sites (site1, site2, site3 to site 10) running Syslog-NG clients. Each site has one Syslog-NG server and all the Syslog messages will be forwarded to the centralized Syslog-NG server and insert into the MySQL database at the headquarter finally. Now, I can see all the Syslog messages at the headquarter MySQL database; but it is hard for me to manage them. For example, if I know the hostname of a particular host, I can query the MySQL database to search for the Syslog related to that host. However, if I want to know the Syslog messages coming from a particular site such as "site1", how can I do that? There is no "site1" information inside the Syslog messages. I am thinking if I can modify the Syslog-NG configuration file such that I can add some information such as "site1" and make it as part of the Syslog messages, I can query the MySQL database for "site1" pattern. But I am not sure if I can do that or not. If you have any suggestions/ideas, please let me know that.

Regards, Marcos

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Hi Robert, Thanks for your suggestion and I am now trying to follow your 3rd suggestion. I will let you know if I can make it or not. However, I would also like to know more the details of your 1st suggestion. My understanding of template is Syslog-NG has some built-in MACRO such as HOST, FACILITY, PRIORITY, TAG, PROGRAM, MSG and etc. Other than those built-in MACRO, can I create some for myself, such as SITE_CODE? It would be appreciated if you can let me know more about it or show me the pointer how to learn more about the template.   Regards, Marcos ________________________________ From: Fekete Robert <frobert@balabit.hu> To: Marcos Tang <marcostang2002@yahoo.com>; Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Sent: Wednesday, August 3, 2011 9:05 PM Subject: Re: [syslog-ng] How to manage the Syslog-NG messages coming from different sites? Each sites has their Syslog-NG clients? Hi, There are several ways to achieve this: - Use a custom template in the destination definition of your clients (or the site's main syslog server that forwards the messages to your central logserver) that includes the sitename in the log message, then extract this info from the message (for example, with a csv parser) on the server side - If you use the RFC5424 syslog message format, you can add the sitename into the SDATA part of the message, and access it with a macro on the central server - You can configure your central server to listen on different IP addresses or different ports, one for each site, and create separate sources in your server's config for every site. syslog-ng automatically adds a tag (the id of the source) to every message, so you can use this tag to filter the messages to different databases or tables (or just include it in an extra column). HTH, Robert On 08/03/2011 12:20 PM, Marcos Tang wrote:

Hi,

I have a problem to manage the Syslog-NG messages sending from different Syslog-NG clients at different remote sites.

For example, I have 10 sites (site1, site2, site3 to site 10) running Syslog-NG clients. Each site has one Syslog-NG server and all the Syslog messages will be forwarded to the centralized Syslog-NG server and insert into the MySQL database at the headquarter finally.

Now, I can see all the Syslog messages at the headquarter MySQL database; but it is hard for me to manage them.

For example, if I know the hostname of a particular host, I can query the MySQL database to search for the Syslog related to that host. However, if I want to know the Syslog messages coming from a particular site such as "site1", how can I do that? There is no "site1" information inside the Syslog messages.

I am thinking if I can modify the Syslog-NG configuration file such that I can add some information such as "site1" and make it as part of the Syslog messages, I can query the MySQL database for "site1" pattern. But I am not sure if I can do that or not.

If you have any suggestions/ideas, please let me know that. Regards, Marcos

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

On Wed, 2011-08-03 at 20:36 +0200, Fekete Róbert wrote:

Hi,

A template is actually a way the message is formatted. Usually it includes only macros, but it can include any text. For example: template t_mytemplate { template("site1#$MSG\n");}; and then modify this template for site2, and so on.

Or if you are using syslog-ng OSE 3.2 or later, you can use a variable in the template, and define the sitename earlier in the config file: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid...

Now that I think about it, it might be possible to avoid templates and modify the value of the $MSG macro using a rewrite rule while keeping its value.

rewrite r_rewrite_set{set("sitecode#$MSG", value("MSG"));};, But I am not sure that macros are permitted in the first parameter - I have to check it with Bazsi or someone more well-versed in the source code, but I would be surprised if it was not possible.

Then on the server side, you can define a parser that segments the message into two parts at the # character (or any other delimiter of your liking), something like: csv_parser(columns("SITECODE", "MYMESSAGE") delimiters("#") flags(greedy));

Then you can refer to the sitecode using $SITECODE, and to the rest of the message using $MYMESSAGE http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid... http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid...

Of course, you are not necessarily limited to adding only the sitecode to the messages, you can add other things as well (like the role of the host, or the department it belongs to, etc.), but for this using the SDATA part of the RFC5424 message format is more suitable.

Check out Chapters 11 and 12 in the admin guide, I am sure you'll get some more ideas about how to solve this problem.

Instead of adding the site information to the $MSG part, I'd add this to the hostname, it's much more natural and easier to handle even with defaults. Or as Robert suggested in his earlier e-mail, use RFC5424 format and structured data. Also, a message within syslog-ng has a number of things that enhance the original "syslog" idea. Each message within syslog-ng has the following properties: * facility/severity: just like good old syslog * timestamps: two complete sets, one for the reception time stamp, the other for the time as was received from the peer ($R_DATE and S_DATE respectively) * name-value pairs: are named properties that can store textual information. You can define any kind of name-value pairs as you see fit, using parsers (csv-parser and db-parser), rewrite rules (set & subst). You can use information stored in the message using templates, a syntax used at a lot of different places inside syslog-ng. Templates can be used to customize the output format in a file, but templates can be also used to name files or SQL tables. Templates can contain both the "builtin" properties of a message, but with the same syntax it can also extract user-defined stuff. For instance: # this defines a name-value pair named FOO rewrite r_foo { set("foobar" value("FOO")); }; # later this can be referenced in a destination file template destination d_file { file("/var/log/${FOO}"); }; # and this one glues these together log { source(...); rewrite(r_foo); destination(d_file); }; At the destination side, it doesn't matter how the name-value pair named $FOO gets its value. If it is set, it'll be used. -- Bazsi