snmptrapd and HOST macro mapping
Dear Syslog-ng users, I'm trying to understand (using syslog-ng community source code and some testing) if it is possible to map a given hostname located into MSGONLY macro to the FULLHOST or HOST macro. I'm trying to achieve this in order to fetch the hostname (SNMP trap source) provided by my snmptrapd collector (which is located on the same server as syslog-ng). The snmptrapd collector is logging its trap using syslog API as follow (snmptrapd source code): int log_handler_syslog( netsnmp_log_handler* logh, int pri, const char *str){ /* * XXX * We've got three items of information to work with: * Is the syslog currently open? * What ident string to use? * What facility to log to? * * We've got two "magic" locations (imagic & magic) plus the token */ if (!(logh->imagic)) { const char *ident = logh->token; int facility = (int)(intptr_t)logh->magic; if (!ident) ident = netsnmp_ds_get_string(NETSNMP_DS_LIBRARY_ID, NETSNMP_DS_LIB_APPTYPE); openlog(ident, LOG_CONS | LOG_PID, facility); logh->imagic = 1; } syslog( pri, "%s", str ); return 1; } As provided by syslog.h, there is no way to set the hostname used into syslog message (it is automatically set by the API). So my logs, provided by snmptrapd, look like the following (where HOST macro is related to the server hosting syslog-ng and snmptrapd): 12/11/2008 16:57:14 SYSLOG-COLLECTOR LEVEL=info snmptrapd[29592]: wlc02.mydomain.com Cold Start In order to be able to get wlc02.mydomain.com as $HOST or $FULLHOST macro, I set: chain_hostnames(yes); keep_hostname(yes); and I formated my snmptrapd logs MSGONLY content (using snmptrad config statements) as RFC 3164 need it: 12/11/2008 16:58:00 SYSLOG-COLLECTOR LEVEL=info snmptrapd[29592]: Nov 12 16:57:59 wlc02.mydomain.com Cold Start I thought that the above log would be seen as a relayed message by syslog-ng allowing me to fetch wlc02.mydomain.com into $HOST or $FULLHOST. Unfortunately this config provide me the following string into $FULLHOST macro: s_source@SYSLOG-COLLECTOR Does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ? I saw another hint about this here: http://osdir.com/ml/syslog-ng/2004-10/msg00062.html It says that: "If you still have your snmptrapd hostname, then you have to modify snmptrapd to use the original hostname when it sends the message about the trap" But as given above (syslog API usage into snmptrapd) how can I set the hostname of the syslog message ? Any hint is welcome ! Thanks in advance, Joël
On Wed, 2008-11-12 at 17:51 +0100, joël Winteregg wrote:
Dear Syslog-ng users,
I'm trying to understand (using syslog-ng community source code and some testing) if it is possible to map a given hostname located into MSGONLY macro to the FULLHOST or HOST macro. I'm trying to achieve this in order to fetch the hostname (SNMP trap source) provided by my snmptrapd collector (which is located on the same server as syslog-ng).
The snmptrapd collector is logging its trap using syslog API as follow (snmptrapd source code):
int log_handler_syslog( netsnmp_log_handler* logh, int pri, const char *str){ /* * XXX * We've got three items of information to work with: * Is the syslog currently open? * What ident string to use? * What facility to log to? * * We've got two "magic" locations (imagic & magic) plus the token */ if (!(logh->imagic)) { const char *ident = logh->token; int facility = (int)(intptr_t)logh->magic; if (!ident) ident = netsnmp_ds_get_string(NETSNMP_DS_LIBRARY_ID, NETSNMP_DS_LIB_APPTYPE); openlog(ident, LOG_CONS | LOG_PID, facility); logh->imagic = 1; } syslog( pri, "%s", str ); return 1; }
As provided by syslog.h, there is no way to set the hostname used into syslog message (it is automatically set by the API). So my logs, provided by snmptrapd, look like the following (where HOST macro is related to the server hosting syslog-ng and snmptrapd):
12/11/2008 16:57:14 SYSLOG-COLLECTOR LEVEL=info snmptrapd[29592]: wlc02.mydomain.com Cold Start
In order to be able to get wlc02.mydomain.com as $HOST or $FULLHOST macro, I set: chain_hostnames(yes); keep_hostname(yes);
and I formated my snmptrapd logs MSGONLY content (using snmptrad config statements) as RFC 3164 need it:
12/11/2008 16:58:00 SYSLOG-COLLECTOR LEVEL=info snmptrapd[29592]: Nov 12 16:57:59 wlc02.mydomain.com Cold Start
I thought that the above log would be seen as a relayed message by syslog-ng allowing me to fetch wlc02.mydomain.com into $HOST or $FULLHOST. Unfortunately this config provide me the following string into $FULLHOST macro: s_source@SYSLOG-COLLECTOR
Does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ?
I saw another hint about this here: http://osdir.com/ml/syslog-ng/2004-10/msg00062.html It says that: "If you still have your snmptrapd hostname, then you have to modify snmptrapd to use the original hostname when it sends the message about the trap"
But as given above (syslog API usage into snmptrapd) how can I set the hostname of the syslog message ?
Any hint is welcome !
Well, one way is to patch snmptrapd to avoid using the syslog() API and go to syslog-ng directly _OR_ you can try the latest 3.0 devel snapshot, which has support for rewrite rules. You could do something like this with 3.0: filter f_snmptrapd { program("snmptrapd"); }; # this parser assumes this snmptrapd format # <host> "<message payload>" parser p_snmptrapd { csv-parser(columns("SNMPTRAP.HOST", "SNMPTRAP.MSG") flags(escape-backslash, strip-whitespace)); }; rewrite r_snmptrapd { set("${SNMPTRAP.HOST}" value("HOST")); set("${SNMPTRAP.MSG}" value("MESSAGE")); } log { source(...); filter(f_snmptrapd); parser(p_snmptrapd); rewrite(r_snmptrapd); destination(...); }; As it seems it would be useful to add a new flag to csv-parser(): last-column-greedy, in which case any non-processed input would be added to the last column. If you are willing to try if this works in your setup, I'm willing to invest the time to implement that csv parser option. If that would be done, it would be easier to process even the original input. -- Bazsi
Hello Balazs, Thank you very much for you quick reply ! The csv-xxx functionality looks really great. You're doing a really good job ! Thanks ! Indeed, today, I was looking for such a functionality into the doc ;-) I think I will try your proposal as soon as possible and I will let you know how it works in my use case. Juste to know, does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ? I'm asking this, because I'm wondering if I forward my SNMP trap to syslogd and then to syslog-ng through udp (@SYSLOG-COLLECTOR defined in syslog.conf), syslog-ng will maybe see the SNMP trap as a compliant RFC 3164 forwarded message ? Thanks again and best regards, Joël On Wed, 2008-11-12 at 22:48 +0100, Balazs Scheidler wrote:
On Wed, 2008-11-12 at 17:51 +0100, joël Winteregg wrote:
Dear Syslog-ng users,
I'm trying to understand (using syslog-ng community source code and some testing) if it is possible to map a given hostname located into MSGONLY macro to the FULLHOST or HOST macro. I'm trying to achieve this in order to fetch the hostname (SNMP trap source) provided by my snmptrapd collector (which is located on the same server as syslog-ng).
The snmptrapd collector is logging its trap using syslog API as follow (snmptrapd source code):
int log_handler_syslog( netsnmp_log_handler* logh, int pri, const char *str){ /* * XXX * We've got three items of information to work with: * Is the syslog currently open? * What ident string to use? * What facility to log to? * * We've got two "magic" locations (imagic & magic) plus the token */ if (!(logh->imagic)) { const char *ident = logh->token; int facility = (int)(intptr_t)logh->magic; if (!ident) ident = netsnmp_ds_get_string(NETSNMP_DS_LIBRARY_ID, NETSNMP_DS_LIB_APPTYPE); openlog(ident, LOG_CONS | LOG_PID, facility); logh->imagic = 1; } syslog( pri, "%s", str ); return 1; }
As provided by syslog.h, there is no way to set the hostname used into syslog message (it is automatically set by the API). So my logs, provided by snmptrapd, look like the following (where HOST macro is related to the server hosting syslog-ng and snmptrapd):
12/11/2008 16:57:14 SYSLOG-COLLECTOR LEVEL=info snmptrapd[29592]: wlc02.mydomain.com Cold Start
In order to be able to get wlc02.mydomain.com as $HOST or $FULLHOST macro, I set: chain_hostnames(yes); keep_hostname(yes);
and I formated my snmptrapd logs MSGONLY content (using snmptrad config statements) as RFC 3164 need it:
12/11/2008 16:58:00 SYSLOG-COLLECTOR LEVEL=info snmptrapd[29592]: Nov 12 16:57:59 wlc02.mydomain.com Cold Start
I thought that the above log would be seen as a relayed message by syslog-ng allowing me to fetch wlc02.mydomain.com into $HOST or $FULLHOST. Unfortunately this config provide me the following string into $FULLHOST macro: s_source@SYSLOG-COLLECTOR
Does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ?
I saw another hint about this here: http://osdir.com/ml/syslog-ng/2004-10/msg00062.html It says that: "If you still have your snmptrapd hostname, then you have to modify snmptrapd to use the original hostname when it sends the message about the trap"
But as given above (syslog API usage into snmptrapd) how can I set the hostname of the syslog message ?
Any hint is welcome !
Well, one way is to patch snmptrapd to avoid using the syslog() API and go to syslog-ng directly _OR_ you can try the latest 3.0 devel snapshot, which has support for rewrite rules.
You could do something like this with 3.0:
filter f_snmptrapd { program("snmptrapd"); };
# this parser assumes this snmptrapd format # <host> "<message payload>" parser p_snmptrapd { csv-parser(columns("SNMPTRAP.HOST", "SNMPTRAP.MSG") flags(escape-backslash, strip-whitespace)); };
rewrite r_snmptrapd { set("${SNMPTRAP.HOST}" value("HOST")); set("${SNMPTRAP.MSG}" value("MESSAGE")); }
log { source(...); filter(f_snmptrapd); parser(p_snmptrapd); rewrite(r_snmptrapd); destination(...); };
As it seems it would be useful to add a new flag to csv-parser(): last-column-greedy, in which case any non-processed input would be added to the last column.
If you are willing to try if this works in your setup, I'm willing to invest the time to implement that csv parser option. If that would be done, it would be easier to process even the original input.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2008-11-12 at 23:32 +0100, joël Winteregg wrote:
Hello Balazs,
Thank you very much for you quick reply ! The csv-xxx functionality looks really great. You're doing a really good job ! Thanks ! Indeed, today, I was looking for such a functionality into the doc ;-)
I think I will try your proposal as soon as possible and I will let you know how it works in my use case.
Juste to know, does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ?
no, keep_hostname is always applied. in 3.0, it is even possible to specify hostname related options on a per-source basis.
I'm asking this, because I'm wondering if I forward my SNMP trap to syslogd and then to syslog-ng through udp (@SYSLOG-COLLECTOR defined in syslog.conf), syslog-ng will maybe see the SNMP trap as a compliant RFC 3164 forwarded message ?
That wouldn't work. the problem is inherent in the syslog API, it does not let you change the hostname. The only way to work around that is to have snmptrapd to send its output to syslog-ng directly (and format the message according to the syslog protocol). There are multiple options: * pipe: make snmptrapd output go to a pipe, and reference this from syslog-ng; writing a pipe is about the same as writing a file, so this would probably work * program source: in 3.0, I introduced program source, which is basically a syslog-ng managed program, whose output is parsed as a syslog message, line by line. We use the latter in our syslog appliance. -- Bazsi
Hi, Thanks again for your support.
Juste to know, does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ?
no, keep_hostname is always applied. in 3.0, it is even possible to specify hostname related options on a per-source basis.
Okay, interesting ! You can hardcode (into config file) a given HOST macro value associated to a source config ?
I'm asking this, because I'm wondering if I forward my SNMP trap to syslogd and then to syslog-ng through udp (@SYSLOG-COLLECTOR defined in syslog.conf), syslog-ng will maybe see the SNMP trap as a compliant RFC 3164 forwarded message ?
That wouldn't work. the problem is inherent in the syslog API, it does not let you change the hostname.
Okay, but here, what I wanted to achieve was the following. Log this SNMP message using snmptrapd syslog functionality: "Nov 12 16:57:59 wlc02.mydomain.com Cold Start" The given snmptrapd output message formatting (header): "Nov 12 16:57:59 wlc02.mydomain.com" set before every snmptrapd message is here to provide a RFC 3164 compliant message => this should allow syslog-ng to think that "Nov 12 16:57:59 wlc02.mydomain.com Cold Start" is a forwarded syslog message ? If so, this would allow me to fetch "wlc02.mydomain.com" as HOST macro using keep_hostname(on), no ?
The only way to work around that is to have snmptrapd to send its output to syslog-ng directly (and format the message according to the syslog protocol). There are multiple options:
* pipe: make snmptrapd output go to a pipe, and reference this from syslog-ng; writing a pipe is about the same as writing a file, so this would probably work
Ahhh, yeah ! That's much easier than my relayed message style ! If, as I did before, I format snmptrad message as follow (to a named pipe), it should work: "Nov 12 16:57:59 wlc02.mydomain.com Cold Start"
* program source: in 3.0, I introduced program source, which is basically a syslog-ng managed program, whose output is parsed as a syslog message, line by line.
Okay, really interesting too ;-) It reads logs from stdout and stderr of the given program ? Will try this (named pipe stuff) before the csv-parser option. As I'm also interested into csv-parser option I will invest time to try it too. Will let you know about last-column-greedy. Thanks again, Regards. Joël
On Thu, 2008-11-13 at 21:18 +0100, joël Winteregg wrote:
Hi,
Thanks again for your support.
Juste to know, does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ?
no, keep_hostname is always applied. in 3.0, it is even possible to specify hostname related options on a per-source basis.
Okay, interesting ! You can hardcode (into config file) a given HOST macro value associated to a source config ?
Yes, there are two ways to do this: - host-override(): this is a new option, that let's you specify a fixed hostname for each source, this effectively overrides the hostname parsing routines - rewrite rule that changes the HOST value after parsing The first looks like this: pipe("/tmp/snmptrapd.pipe" host-override("overridden-host")); everything coming from this pipe will use "overridden-host" as hostname. The second one looks like this: rewrite r_host { set("overridden-host" value("HOST")); }; The rewrite rule can even use macros, like this: rewrite r_host { set("${HOST}-append" value("HOST")); }; This will append the string '-append' to the hostname.
I'm asking this, because I'm wondering if I forward my SNMP trap to syslogd and then to syslog-ng through udp (@SYSLOG-COLLECTOR defined in syslog.conf), syslog-ng will maybe see the SNMP trap as a compliant RFC 3164 forwarded message ?
That wouldn't work. the problem is inherent in the syslog API, it does not let you change the hostname.
Okay, but here, what I wanted to achieve was the following. Log this SNMP message using snmptrapd syslog functionality: "Nov 12 16:57:59 wlc02.mydomain.com Cold Start"
The given snmptrapd output message formatting (header): "Nov 12 16:57:59 wlc02.mydomain.com" set before every snmptrapd message is here to provide a RFC 3164 compliant message => this should allow syslog-ng to think that "Nov 12 16:57:59 wlc02.mydomain.com Cold Start" is a forwarded syslog message ? If so, this would allow me to fetch "wlc02.mydomain.com" as HOST macro using keep_hostname(on), no ?
you misunderstand the relayed message format. the header is not duplicated in case a message is relayed, the format is still the same. e.g. original message looks like this: <7>Nov 12 16:57:59 wlc02.mydomain.com Cold Start Then a relayed message looks the same, except in some cases the relay host mangles the syslog header, and changes the hostname for instance.
The only way to work around that is to have snmptrapd to send its output to syslog-ng directly (and format the message according to the syslog protocol). There are multiple options:
* pipe: make snmptrapd output go to a pipe, and reference this from syslog-ng; writing a pipe is about the same as writing a file, so this would probably work
Ahhh, yeah ! That's much easier than my relayed message style ! If, as I did before, I format snmptrad message as follow (to a named pipe), it should work: "Nov 12 16:57:59 wlc02.mydomain.com Cold Start"
yes. you might add a priority field though.
* program source: in 3.0, I introduced program source, which is basically a syslog-ng managed program, whose output is parsed as a syslog message, line by line.
Okay, really interesting too ;-) It reads logs from stdout and stderr of the given program ?
it only fetches the standard output.
Will try this (named pipe stuff) before the csv-parser option. As I'm also interested into csv-parser option I will invest time to try it too. Will let you know about last-column-greedy.
the named pipe should work with any syslog-ng version, csv-parser is added in 3.0. -- Bazsi
Hello,
Juste to know, does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ?
no, keep_hostname is always applied. in 3.0, it is even possible to specify hostname related options on a per-source basis.
Okay, interesting ! You can hardcode (into config file) a given HOST macro value associated to a source config ?
Yes, there are two ways to do this: - host-override(): this is a new option, that let's you specify a fixed hostname for each source, this effectively overrides the hostname parsing routines - rewrite rule that changes the HOST value after parsing
The first looks like this:
pipe("/tmp/snmptrapd.pipe" host-override("overridden-host"));
everything coming from this pipe will use "overridden-host" as hostname.
The second one looks like this:
rewrite r_host { set("overridden-host" value("HOST")); };
The rewrite rule can even use macros, like this:
rewrite r_host { set("${HOST}-append" value("HOST")); };
This will append the string '-append' to the hostname.
Great ! Thanks for the input !
I'm asking this, because I'm wondering if I forward my SNMP trap to syslogd and then to syslog-ng through udp (@SYSLOG-COLLECTOR defined in syslog.conf), syslog-ng will maybe see the SNMP trap as a compliant RFC 3164 forwarded message ?
That wouldn't work. the problem is inherent in the syslog API, it does not let you change the hostname.
Okay, but here, what I wanted to achieve was the following. Log this SNMP message using snmptrapd syslog functionality: "Nov 12 16:57:59 wlc02.mydomain.com Cold Start"
The given snmptrapd output message formatting (header): "Nov 12 16:57:59 wlc02.mydomain.com" set before every snmptrapd message is here to provide a RFC 3164 compliant message => this should allow syslog-ng to think that "Nov 12 16:57:59 wlc02.mydomain.com Cold Start" is a forwarded syslog message ? If so, this would allow me to fetch "wlc02.mydomain.com" as HOST macro using keep_hostname(on), no ?
you misunderstand the relayed message format. the header is not duplicated in case a message is relayed, the format is still the same.
Ah, okay ! Yes you're right I totally misunderstood the relayed message format ;-)
The only way to work around that is to have snmptrapd to send its output to syslog-ng directly (and format the message according to the syslog protocol). There are multiple options:
* pipe: make snmptrapd output go to a pipe, and reference this from syslog-ng; writing a pipe is about the same as writing a file, so this would probably work
Ahhh, yeah ! That's much easier than my relayed message style ! If, as I did before, I format snmptrad message as follow (to a named pipe), it should work: "Nov 12 16:57:59 wlc02.mydomain.com Cold Start"
yes. you might add a priority field though.
Ok ! Will try this !
* program source: in 3.0, I introduced program source, which is basically a syslog-ng managed program, whose output is parsed as a syslog message, line by line.
Okay, really interesting too ;-) It reads logs from stdout and stderr of the given program ?
it only fetches the standard output.
Will try this (named pipe stuff) before the csv-parser option. As I'm also interested into csv-parser option I will invest time to try it too. Will let you know about last-column-greedy.
the named pipe should work with any syslog-ng version, csv-parser is added in 3.0.
Thanks again for your support ! Joël
participants (2)
-
Balazs Scheidler
-
joël Winteregg