Insert hostname instead of overwrite ?
Guys I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e. Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated) I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ? -- stucky
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
Guys
I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e.
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this
Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting
I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ?
what about keep_hostname(yes) ? -- Bazsi
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this : Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated) So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong. On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
Guys
I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e.
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this
Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting
I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ?
what about keep_hostname(yes) ?
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
Hmm. Well, if you can't put an intermediary syslog server with use_dns enabled between your Dell app and the load balancer, I think you're outta luck. If the log message doesn't contain a hostname, and the sending IP is that of the load balancer, then syslog really has no way to know where the message came from. You could write Dell and ask them to conform to RFC syslog standards but I don't think that's going to happen any time soon :). The only other thing I can think of is that if you only have one dell openManage box, you could filter for something specific to those logs then apply a static hostname using a template. But that method sucks and doesn't work as soon as you have two openManage boxes forwarding syslogs. On Thu, Sep 2, 2010 at 3:39 PM, stucky <stucky101@gmail.com> wrote:
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this :
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong.
On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
Guys
I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e.
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this
Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting
I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ?
what about keep_hostname(yes) ?
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
Guys We're not on the same page here. I have already addressed the missing hostname by forcing syslog-ng to use dns to lookup its own hostname and then insert it. All I was asking is if I can make syslog truly "insert" the hostname. Currently it simply overwrites whatever is in this field (This this case the word "Server") and replaces it with the correct hostname. I was simply saying that this field which was just overwritten might have contained important loginfo - that's all. It doesn't in this case but what if it did. So to make this clear syslog can do this Replace "Server Administrator" with "{hostname} Administrator" I was wondering if it could instead do this : Replace "Server Administrator" with "{hostname} Server Administrator" in order not to truncate the log content. On a side node instead of using dns wouldn't it be great if syslog could do a "gethostbyname" instead to figure out its own hostname ? Should be much more efficient for local log source like this. On Thu, Sep 2, 2010 at 5:28 PM, Lance Laursen <lance@demonware.net> wrote:
Hmm. Well, if you can't put an intermediary syslog server with use_dns enabled between your Dell app and the load balancer, I think you're outta luck. If the log message doesn't contain a hostname, and the sending IP is that of the load balancer, then syslog really has no way to know where the message came from. You could write Dell and ask them to conform to RFC syslog standards but I don't think that's going to happen any time soon :). The only other thing I can think of is that if you only have one dell openManage box, you could filter for something specific to those logs then apply a static hostname using a template. But that method sucks and doesn't work as soon as you have two openManage boxes forwarding syslogs.
On Thu, Sep 2, 2010 at 3:39 PM, stucky <stucky101@gmail.com> wrote:
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this :
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong.
On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi@balabit.hu>wrote:
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
Guys
I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e.
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this
Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting
I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ?
what about keep_hostname(yes) ?
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
I think you would be able to do this. You can set the no-parse flag on the tcp source the bad messages come in on, and then use a filter on the $MSG macro to grab things out. Like a pcre filter that does '(?<PROGRAM>some.regex). I'm not certain if filters can set macros such as PROGRAM though, but worth a shot. Sent: Thursday, September 02, 2010 7:40:38 PM From: stucky <stucky101@gmail.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Insert hostname instead of overwrite ?
Guys
We're not on the same page here. I have already addressed the missing hostname by forcing syslog-ng to use dns to lookup its own hostname and then insert it. All I was asking is if I can make syslog truly "insert" the hostname. Currently it simply overwrites whatever is in this field (This this case the word "Server") and replaces it with the correct hostname. I was simply saying that this field which was just overwritten might have contained important loginfo - that's all. It doesn't in this case but what if it did. So to make this clear syslog can do this
Replace "Server Administrator" with "{hostname} Administrator"
I was wondering if it could instead do this :
Replace "Server Administrator" with "{hostname} Server Administrator" in order not to truncate the log content.
On a side node instead of using dns wouldn't it be great if syslog could do a "gethostbyname" instead to figure out its own hostname ? Should be much more efficient for local log source like this.
On Thu, Sep 2, 2010 at 5:28 PM, Lance Laursen <lance@demonware.net <mailto:lance@demonware.net>> wrote:
Hmm. Well, if you can't put an intermediary syslog server with use_dns enabled between your Dell app and the load balancer, I think you're outta luck. If the log message doesn't contain a hostname, and the sending IP is that of the load balancer, then syslog really has no way to know where the message came from. You could write Dell and ask them to conform to RFC syslog standards but I don't think that's going to happen any time soon :). The only other thing I can think of is that if you only have one dell openManage box, you could filter for something specific to those logs then apply a static hostname using a template. But that method sucks and doesn't work as soon as you have two openManage boxes forwarding syslogs.
On Thu, Sep 2, 2010 at 3:39 PM, stucky <stucky101@gmail.com <mailto:stucky101@gmail.com>> wrote:
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this :
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong.
On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi@balabit.hu <mailto:bazsi@balabit.hu>> wrote:
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote: > Guys > > I'm trying to log to a loadbalanced VIP. It seems to work ok except > that the loadbalancer uses SNAT so I loose my source IP. > This means I cannot use dns or even the source ip to get the source > host as all logs appear to come from the same source (the > loadbalancer). > This means I have no choice but to rely on the hostname field which > works about 98% of the time but some stuff like Dell OpenManage skips > the hostname field. > So I'd get logs like this on host "cage" f.e. > > Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 > The Patrol Read has started.: Controller 0 (PERC 5/i Integrated) > > I fixed that by telling the syslog-ng client to force itself to figure > out a proper hostname and now the log looks like this > > Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: > 1000 Server Administrator starting > > I thought syslog-ng inserts the hostname but by the looks of it it > simply replaces whatever is in the expected field with the hostname it > has just figured out. > As you can see it overwrote the entry "Server". > No biggie in the above case but what if this field contained valuable > information ? I'd loose that. > Any way to squeeze in the hostname so to speak ?
what about keep_hostname(yes) ?
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky ------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Guys Let me clarify. This is not on the syslog server but the clients. I need to make 100% sure clients have the correct hostname set _before_ they even send the message to the log server. So this is for the /dev/log, proc/kmsg source. I need the correct hostname field set by the time the server gets it since I log to a SNAT VIP so the server thinks everything comes from the loadbalancer hence dns is out. Having said that I could try your approach on any src not only a tcp source I assume. Not sure if the regex is worth the pain. It might introduce more issues. So the answer is out of the box syslog only "overwrites" the hostfield. It doesn't squeeze it in right ? On Thu, Sep 2, 2010 at 7:09 PM, <syslogng@feystorm.net> wrote:
I think you would be able to do this. You can set the no-parse flag on the tcp source the bad messages come in on, and then use a filter on the $MSG macro to grab things out. Like a pcre filter that does '(?<PROGRAM>some.regex). I'm not certain if filters can set macros such as PROGRAM though, but worth a shot.
Sent: Thursday, September 02, 2010 7:40:38 PM From: stucky <stucky101@gmail.com> <stucky101@gmail.com>
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Insert hostname instead of overwrite ?
Guys
We're not on the same page here. I have already addressed the missing hostname by forcing syslog-ng to use dns to lookup its own hostname and then insert it. All I was asking is if I can make syslog truly "insert" the hostname. Currently it simply overwrites whatever is in this field (This this case the word "Server") and replaces it with the correct hostname. I was simply saying that this field which was just overwritten might have contained important loginfo - that's all. It doesn't in this case but what if it did. So to make this clear syslog can do this
Replace "Server Administrator" with "{hostname} Administrator"
I was wondering if it could instead do this :
Replace "Server Administrator" with "{hostname} Server Administrator" in order not to truncate the log content.
On a side node instead of using dns wouldn't it be great if syslog could do a "gethostbyname" instead to figure out its own hostname ? Should be much more efficient for local log source like this.
On Thu, Sep 2, 2010 at 5:28 PM, Lance Laursen <lance@demonware.net> wrote:
Hmm. Well, if you can't put an intermediary syslog server with use_dns enabled between your Dell app and the load balancer, I think you're outta luck. If the log message doesn't contain a hostname, and the sending IP is that of the load balancer, then syslog really has no way to know where the message came from. You could write Dell and ask them to conform to RFC syslog standards but I don't think that's going to happen any time soon :). The only other thing I can think of is that if you only have one dell openManage box, you could filter for something specific to those logs then apply a static hostname using a template. But that method sucks and doesn't work as soon as you have two openManage boxes forwarding syslogs.
On Thu, Sep 2, 2010 at 3:39 PM, stucky <stucky101@gmail.com> wrote:
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this :
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong.
On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi@balabit.hu>wrote:
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
Guys
I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e.
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this
Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting
I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ?
what about keep_hostname(yes) ?
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
Hey, Sorry I didn't read your original email hard enough. So basically, out of the box syslog-ng is attempting to parse your non-rfc-standard log entries, and as a result the "Server" column after the time and date is being assigned the $HOST macro, whilst the rest of the message is getting thrown into $MSG. Out of the box syslog-ng tries its best to parse messages appropriately, but in this case it's close enough to RFC standard that it's just not getting it right. Dell stuff is a bit too windowy. Setting flags(no-parse) will solve your problem, except that if you assign it to the /dev/log source, all of your other logs are going to be affected. This does suck. Can you get dell openmanage to send to a network location? If so, you could make a network source (127.0.0.1:12345) and set the flags(no-parse) on that. After you set no-parse, you'll have to either use csv-parser or patterndb to pull out the fields (ie: the date, then the rest of the message), and write out/relay the message using a template. Isolating this source will negate any issues with other logs. I would recommend googling how to parse apache logs (or another common app with non-rfc logs) with syslog-ng for examples on templating and the no-parse option. On Thu, Sep 2, 2010 at 7:52 PM, stucky <stucky101@gmail.com> wrote:
Guys
Let me clarify. This is not on the syslog server but the clients. I need to make 100% sure clients have the correct hostname set _before_ they even send the message to the log server. So this is for the /dev/log, proc/kmsg source. I need the correct hostname field set by the time the server gets it since I log to a SNAT VIP so the server thinks everything comes from the loadbalancer hence dns is out. Having said that I could try your approach on any src not only a tcp source I assume. Not sure if the regex is worth the pain. It might introduce more issues.
So the answer is out of the box syslog only "overwrites" the hostfield. It doesn't squeeze it in right ?
On Thu, Sep 2, 2010 at 7:09 PM, <syslogng@feystorm.net> wrote:
I think you would be able to do this. You can set the no-parse flag on the tcp source the bad messages come in on, and then use a filter on the $MSG macro to grab things out. Like a pcre filter that does '(?<PROGRAM>some.regex). I'm not certain if filters can set macros such as PROGRAM though, but worth a shot.
Sent: Thursday, September 02, 2010 7:40:38 PM From: stucky <stucky101@gmail.com> <stucky101@gmail.com>
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Insert hostname instead of overwrite ?
Guys
We're not on the same page here. I have already addressed the missing hostname by forcing syslog-ng to use dns to lookup its own hostname and then insert it. All I was asking is if I can make syslog truly "insert" the hostname. Currently it simply overwrites whatever is in this field (This this case the word "Server") and replaces it with the correct hostname. I was simply saying that this field which was just overwritten might have contained important loginfo - that's all. It doesn't in this case but what if it did. So to make this clear syslog can do this
Replace "Server Administrator" with "{hostname} Administrator"
I was wondering if it could instead do this :
Replace "Server Administrator" with "{hostname} Server Administrator" in order not to truncate the log content.
On a side node instead of using dns wouldn't it be great if syslog could do a "gethostbyname" instead to figure out its own hostname ? Should be much more efficient for local log source like this.
On Thu, Sep 2, 2010 at 5:28 PM, Lance Laursen <lance@demonware.net>wrote:
Hmm. Well, if you can't put an intermediary syslog server with use_dns enabled between your Dell app and the load balancer, I think you're outta luck. If the log message doesn't contain a hostname, and the sending IP is that of the load balancer, then syslog really has no way to know where the message came from. You could write Dell and ask them to conform to RFC syslog standards but I don't think that's going to happen any time soon :). The only other thing I can think of is that if you only have one dell openManage box, you could filter for something specific to those logs then apply a static hostname using a template. But that method sucks and doesn't work as soon as you have two openManage boxes forwarding syslogs.
On Thu, Sep 2, 2010 at 3:39 PM, stucky <stucky101@gmail.com> wrote:
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this :
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong.
On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi@balabit.hu>wrote:
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
Guys
I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e.
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this
Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting
I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ?
what about keep_hostname(yes) ?
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
Lance Thanks for your insight. This hackjob wouldn't really be worth the time since its only a few logs and only from the server manager. Most DELL logs are rfc compliant just not a few lines now and then. I hope I don't encounter any more serious non-standard programs but if I do I have a better idea now what I could do. I guess the problem is isolating the specific log source. My solution works pretty well otherwise. Btw. any idea on whether syslog-ng can do "gethostbyname" call ? DNS works but seems awkward considering that I only wanna look up my own hostname and shouldn't need a dns lookup for that. I don't use /etc/hosts at all but I did turn on the dns cache option. I assume this means syslog-ng does a lookup once at startup per each client and then caches the result ? On Thu, Sep 2, 2010 at 11:26 PM, Lance Laursen <lance@demonware.net> wrote:
Hey,
Sorry I didn't read your original email hard enough. So basically, out of the box syslog-ng is attempting to parse your non-rfc-standard log entries, and as a result the "Server" column after the time and date is being assigned the $HOST macro, whilst the rest of the message is getting thrown into $MSG. Out of the box syslog-ng tries its best to parse messages appropriately, but in this case it's close enough to RFC standard that it's just not getting it right. Dell stuff is a bit too windowy. Setting flags(no-parse) will solve your problem, except that if you assign it to the /dev/log source, all of your other logs are going to be affected. This does suck. Can you get dell openmanage to send to a network location? If so, you could make a network source (127.0.0.1:12345) and set the flags(no-parse) on that. After you set no-parse, you'll have to either use csv-parser or patterndb to pull out the fields (ie: the date, then the rest of the message), and write out/relay the message using a template. Isolating this source will negate any issues with other logs. I would recommend googling how to parse apache logs (or another common app with non-rfc logs) with syslog-ng for examples on templating and the no-parse option.
On Thu, Sep 2, 2010 at 7:52 PM, stucky <stucky101@gmail.com> wrote:
Guys
Let me clarify. This is not on the syslog server but the clients. I need to make 100% sure clients have the correct hostname set _before_ they even send the message to the log server. So this is for the /dev/log, proc/kmsg source. I need the correct hostname field set by the time the server gets it since I log to a SNAT VIP so the server thinks everything comes from the loadbalancer hence dns is out. Having said that I could try your approach on any src not only a tcp source I assume. Not sure if the regex is worth the pain. It might introduce more issues.
So the answer is out of the box syslog only "overwrites" the hostfield. It doesn't squeeze it in right ?
On Thu, Sep 2, 2010 at 7:09 PM, <syslogng@feystorm.net> wrote:
I think you would be able to do this. You can set the no-parse flag on the tcp source the bad messages come in on, and then use a filter on the $MSG macro to grab things out. Like a pcre filter that does '(?<PROGRAM>some.regex). I'm not certain if filters can set macros such as PROGRAM though, but worth a shot.
Sent: Thursday, September 02, 2010 7:40:38 PM From: stucky <stucky101@gmail.com> <stucky101@gmail.com>
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Insert hostname instead of overwrite ?
Guys
We're not on the same page here. I have already addressed the missing hostname by forcing syslog-ng to use dns to lookup its own hostname and then insert it. All I was asking is if I can make syslog truly "insert" the hostname. Currently it simply overwrites whatever is in this field (This this case the word "Server") and replaces it with the correct hostname. I was simply saying that this field which was just overwritten might have contained important loginfo - that's all. It doesn't in this case but what if it did. So to make this clear syslog can do this
Replace "Server Administrator" with "{hostname} Administrator"
I was wondering if it could instead do this :
Replace "Server Administrator" with "{hostname} Server Administrator" in order not to truncate the log content.
On a side node instead of using dns wouldn't it be great if syslog could do a "gethostbyname" instead to figure out its own hostname ? Should be much more efficient for local log source like this.
On Thu, Sep 2, 2010 at 5:28 PM, Lance Laursen <lance@demonware.net>wrote:
Hmm. Well, if you can't put an intermediary syslog server with use_dns enabled between your Dell app and the load balancer, I think you're outta luck. If the log message doesn't contain a hostname, and the sending IP is that of the load balancer, then syslog really has no way to know where the message came from. You could write Dell and ask them to conform to RFC syslog standards but I don't think that's going to happen any time soon :). The only other thing I can think of is that if you only have one dell openManage box, you could filter for something specific to those logs then apply a static hostname using a template. But that method sucks and doesn't work as soon as you have two openManage boxes forwarding syslogs.
On Thu, Sep 2, 2010 at 3:39 PM, stucky <stucky101@gmail.com> wrote:
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this :
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong.
On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi@balabit.hu>wrote:
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote: > Guys > > I'm trying to log to a loadbalanced VIP. It seems to work ok except > that the loadbalancer uses SNAT so I loose my source IP. > This means I cannot use dns or even the source ip to get the source > host as all logs appear to come from the same source (the > loadbalancer). > This means I have no choice but to rely on the hostname field which > works about 98% of the time but some stuff like Dell OpenManage skips > the hostname field. > So I'd get logs like this on host "cage" f.e. > > Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 > The Patrol Read has started.: Controller 0 (PERC 5/i Integrated) > > I fixed that by telling the syslog-ng client to force itself to figure > out a proper hostname and now the log looks like this > > Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: > 1000 Server Administrator starting > > I thought syslog-ng inserts the hostname but by the looks of it it > simply replaces whatever is in the expected field with the hostname it > has just figured out. > As you can see it overwrote the entry "Server". > No biggie in the above case but what if this field contained valuable > information ? I'd loose that. > Any way to squeeze in the hostname so to speak ?
what about keep_hostname(yes) ?
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
On Fri, 2010-09-03 at 01:06 -0700, stucky wrote:
Lance
Thanks for your insight. This hackjob wouldn't really be worth the time since its only a few logs and only from the server manager. Most DELL logs are rfc compliant just not a few lines now and then. I hope I don't encounter any more serious non-standard programs but if I do I have a better idea now what I could do. I guess the problem is isolating the specific log source.
My solution works pretty well otherwise. Btw. any idea on whether syslog-ng can do "gethostbyname" call ? DNS works but seems awkward considering that I only wanna look up my own hostname and shouldn't need a dns lookup for that.
gethostbyname() automatically does a DNS lookup through NSS (if the name is not in /etc/hosts), and yes syslog-ng uses that to get the hostname information from the IP address. Its own hostname is found out by using gethostname(), so no DNS lookup in that case.
I don't use /etc/hosts at all but I did turn on the dns cache option. I assume this means syslog-ng does a lookup once at startup per each client and then caches the result ?
Once when it encounters it, yes. The dns cache size is limited but can be tuned, as the time a given entry is assumed to be valid in the cache (just in order to follow DNS name changes).
On Thu, Sep 2, 2010 at 11:26 PM, Lance Laursen <lance@demonware.net> wrote: Hey,
Sorry I didn't read your original email hard enough. So basically, out of the box syslog-ng is attempting to parse your non-rfc-standard log entries, and as a result the "Server" column after the time and date is being assigned the $HOST macro, whilst the rest of the message is getting thrown into $MSG. Out of the box syslog-ng tries its best to parse messages appropriately, but in this case it's close enough to RFC standard that it's just not getting it right. Dell stuff is a bit too windowy. Setting flags(no-parse) will solve your problem, except that if you assign it to the /dev/log source, all of your other logs are going to be affected. This does suck. Can you get dell openmanage to send to a network location? If so, you could make a network source (127.0.0.1:12345) and set the flags(no-parse) on that. After you set no-parse, you'll have to either use csv-parser or patterndb to pull out the fields (ie: the date, then the rest of the message), and write out/relay the message using a template. Isolating this source will negate any issues with other logs. I would recommend googling how to parse apache logs (or another common app with non-rfc logs) with syslog-ng for examples on templating and the no-parse option.
On Thu, Sep 2, 2010 at 7:52 PM, stucky <stucky101@gmail.com> wrote: Guys
Let me clarify. This is not on the syslog server but the clients. I need to make 100% sure clients have the correct hostname set _before_ they even send the message to the log server. So this is for the /dev/log, proc/kmsg source. I need the correct hostname field set by the time the server gets it since I log to a SNAT VIP so the server thinks everything comes from the loadbalancer hence dns is out. Having said that I could try your approach on any src not only a tcp source I assume. Not sure if the regex is worth the pain. It might introduce more issues.
So the answer is out of the box syslog only "overwrites" the hostfield. It doesn't squeeze it in right ?
On Thu, Sep 2, 2010 at 7:09 PM, <syslogng@feystorm.net> wrote: I think you would be able to do this. You can set the no-parse flag on the tcp source the bad messages come in on, and then use a filter on the $MSG macro to grab things out. Like a pcre filter that does '(?<PROGRAM>some.regex). I'm not certain if filters can set macros such as PROGRAM though, but worth a shot.
Sent: Thursday, September 02, 2010 7:40:38 PM From: stucky <stucky101@gmail.com>
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>
Subject: Re: [syslog-ng] Insert hostname instead of overwrite ? > > Guys > > We're not on the same page here. I have > already addressed the missing hostname by > forcing syslog-ng to use dns to lookup its > own hostname and then insert it. > All I was asking is if I can make syslog > truly "insert" the hostname. Currently it > simply overwrites whatever is in this field > (This this case the word "Server") > and replaces it with the correct hostname. > I was simply saying that this field which > was just overwritten might have contained > important loginfo - that's all. It doesn't > in this case but what if it did. > So to make this clear syslog can do this > > Replace "Server Administrator" with > "{hostname} Administrator" > > I was wondering if it could instead do > this : > > Replace "Server Administrator" with > "{hostname} Server Administrator" in order > not to truncate the log content. > > On a side node instead of using dns wouldn't > it be great if syslog could do a > "gethostbyname" instead to figure out its > own hostname ? Should be much more efficient > for local log source like this. > > On Thu, Sep 2, 2010 at 5:28 PM, Lance > Laursen <lance@demonware.net> wrote: > Hmm. Well, if you can't put an > intermediary syslog server with > use_dns enabled between your Dell > app and the load balancer, I think > you're outta luck. If the log > message doesn't contain a hostname, > and the sending IP is that of the > load balancer, then syslog really > has no way to know where the message > came from. You could write Dell and > ask them to conform to RFC syslog > standards but I don't think that's > going to happen any time soon :). > The only other thing I can think of > is that if you only have one dell > openManage box, you could filter for > something specific to those logs > then apply a static hostname using a > template. But that method sucks and > doesn't work as soon as you have two > openManage boxes forwarding syslogs. > > > On Thu, Sep 2, 2010 at 3:39 PM, > stucky <stucky101@gmail.com> wrote: > That's exactly the problem. > I cannot keep a hostname > that was never written in > the first place. > The DELL server > administrator doesn't send > it. As per my email below it > sends this : > > > Aug 16 21:47:22 Server > Administrator: Storage > Service EventID: 2242> The > Patrol Read has started.: > Controller 0 (PERC 5/i > Integrated) > > > So If I do a "keep_hostname" > syslog-ng assumes that the > server is called "Server" > which is of course wrong. > > > On Thu, Sep 2, 2010 at 8:21 > AM, Balazs Scheidler > <bazsi@balabit.hu> wrote: > On Tue, 2010-08-17 > at 18:26 -0700, > stucky wrote: > > Guys > > > > I'm trying to log > to a loadbalanced > VIP. It seems to > work ok except > > that the > loadbalancer uses > SNAT so I loose my > source IP. > > This means I > cannot use dns or > even the source ip > to get the source > > host as all logs > appear to come from > the same source (the > > loadbalancer). > > This means I have > no choice but to > rely on the hostname > field which > > works about 98% of > the time but some > stuff like Dell > OpenManage skips > > the hostname > field. > > So I'd get logs > like this on host > "cage" f.e. > > > > Aug 16 21:47:22 > Server > Administrator: > Storage Service > EventID: 2242 > > The Patrol Read > has started.: > Controller 0 (PERC > 5/i Integrated) > > > > I fixed that by > telling the > syslog-ng client to > force itself to > figure > > out a proper > hostname and now the > log looks like this > > > > Aug 17 13:51:10 > cage > Administrator[]: > Instrumentation > Service EventID: > > 1000 Server > Administrator > starting > > > > I thought > syslog-ng inserts > the hostname but by > the looks of it it > > simply replaces > whatever is in the > expected field with > the hostname it > > has just figured > out. > > As you can see it > overwrote the entry > "Server". > > No biggie in the > above case but what > if this field > contained valuable > > information ? I'd > loose that. > > Any way to squeeze > in the hostname so > to speak ? > > > what about > keep_hostname(yes) ? > > > -- > Bazsi > > ______________________________________________________________________________ > Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: > http://www.campin.net/syslog-ng/faq.html > > > > > > > -- > stucky > > ______________________________________________________________________________ > Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: > http://www.campin.net/syslog-ng/faq.html > > > > > > -- > > Lance Laursen > Demonware Systems Engineer > > > ______________________________________________________________________________ > Member info: > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: > http://www.campin.net/syslog-ng/faq.html > > > > > > -- > stucky > > > ____________________________________________ > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Lance Laursen Demonware Systems Engineer
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
Hi, On Fri, Sep 3, 2010 at 12:39 AM, stucky <stucky101@gmail.com> wrote:
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this :
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong.
For the legacy syslog format bad_hostname("^Server$") should do the trick. Regards, Sandor
So it would consider "Server" to be a bad hostname but what will it do with it ? I'm not sure I understand this option 100%. If I set "keep_hostname" to "yes" but then tell it that "Server" is a bad hostname what will it use as a hostname in this case ? On Fri, Sep 3, 2010 at 12:32 AM, Sandor Geller < Sandor.Geller@morganstanley.com> wrote:
Hi,
On Fri, Sep 3, 2010 at 12:39 AM, stucky <stucky101@gmail.com> wrote:
That's exactly the problem. I cannot keep a hostname that was never written in the first place. The DELL server administrator doesn't send it. As per my email below it sends this :
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
So If I do a "keep_hostname" syslog-ng assumes that the server is called "Server" which is of course wrong.
For the legacy syslog format bad_hostname("^Server$") should do the trick.
Regards,
Sandor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- stucky
On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
Guys
I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e.
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)
I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this
Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting
I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ?
Well, apart from using bad_hostname() patterns to instruct syslog-ng to assume that string is part of the message I only have one idea: in syslog-ng 3.2 you could use the "no-hostname" flag. Can you configure these clients to send their logs to a different port/ip? If you can, then the following would work with pre-3.2: source s_dell { udp(port(2000) bad_hostname(".")); }; This instructs syslog-ng that the first word after the timestamp is never the hostname. It's simpler with 3.2 though: source s_dell { udp(port(2000) flags(no-hostname)); }; -- Bazsi
participants (5)
-
Balazs Scheidler
-
Lance Laursen
-
Sandor Geller
-
stucky
-
syslogng＠feystorm.net