syslog-ng 3.0.2 issues: Extraneous colon (:) in forwarded log; two instances of syslog-ng
Hi. I'm been running syslog-ng 2.0.9 for eons, and wanted to get 3.0.2 in place, but have run into two issues. First, the prerequisite info: This is on Solaris 10 x86, and I'm running the eventlog-0.2.9 component. It has all compiled cleanly. I'm using the same script to initialize syslog-ng as I was using before. In the conf file, added in some value() parameters on my filters, in accordance with the 3.0 "warnings", and added in the @version directive, but other than that, it's essentially the same (reasonably simple) conf file I was using with my 2.0.9 version of the code. For reference, I've appended my conf file at the end of the email. "Generally" syslog-ng, seems to start-up fine, and does feed the output files that I've indicated in the conf file. If I didn't "look too closely", I would have thought it was all fine, but... My problems: As part of my test environment, I'm forwarding data from another (production 2.0.9) server, to my test server, so that I can see how things are working. Here are some (truncated) messages, showing both the original message (as it looks in the output file on the 2.0.9 server), and the second message being what gets written to the file by the 3.0.2 component: Original: May 29 09:15:05 10.132.240.25 id=firewall sn=0006B10B2E88 <truncated> Received and written by 3.0.2: May 29 09:15:05 10.132.240.25 id=firewall: sn=0006B10B2E88 <truncated> So the "bug" (I assume it to be a bug) is that "colon" that is now sitting just behind the word "firewall". When my test server is running a 2.0.9 daemon, these two lines match, but the 3.0.2 daemon has inserted that colon. I can only assume that this is tied to a bug in the filtering activity, because if you look at my conf file, you will see these lines: filter SonicWallNoise { match("id=firewall" value("MSG")) and filter(SonicWallMsgs); }; Bottom line, I am doing a filter that happens to "match on" that exact "firewall" string, and it seems like more than just coincidence that this is the exact location where that unwanted colon now appears. It seems like that filtering process has somehow injected that character into the output. (Maybe I'm just missing some "new functionality or feature" in the 3.0 docs, that would intentionally cause this?) Anyway... I'm obviously looking for any input as to why this is different, and/or how to fix it (or even just work-around the problem until it is fixed). And secondly, as my subject line implies, when I run my "/etc/init.d/syslog-ng start" scripting (the same one I've always used), I find, right away, that I now have two instances of syslog-ng running. I saw a bug report from back in 2006, mentioning a similar behavior that was related to (I think) syslog-ng restarting itself, but that was supposedly fixed. I'm ass-u-me-ing that something is now causing the daemon to restart itself, but again, when I run my 2.0.9 code on place of the 3.0.2 code, I get just the one copy running. Again, I'm looking for any guidance as to something that might be driving that behavior (and as I said, maybe it's just something stupid on my part, because I'm missing something "new" in the 3.0 behavior). Thanks, in advance, for any help and/or guidance as to what these issues might be. I'm glad to provide further information if it will help. Marvin Nipper syslog-ng.conf: @version:3.0 options { dir_perm(0755); perm(0600); chain_hostnames(no); keep_hostname(yes); log_fifo_size(1000); dns_cache_size(5000); dns_cache_expire(86400); dns_cache_expire_failed(86400); }; source any_udp { udp(); }; source any_tcp { tcp(port(601)); }; destination SEC {pipe("/tmp/sec"); }; destination routers_log { file("/var/adm/log/routers.log" create_dirs(yes)); }; destination ravlin_log { file("/var/adm/log/ravlin.log" create_dirs(yes)); }; destination windows_log { file("/var/adm/log/windows.log" create_dirs(yes)); }; destination workstation_log { file("/var/adm/log/workstation.log" create_dirs(yes)); }; destination catch-all_log { file("/var/adm/log/catch-all.log" create_dirs(yes)); }; destination test { file("/var/adm/log/test.log" create_dirs(yes)); }; destination dev_null {}; filter f_4 { facility(syslog) and level(info..emerg); }; log { source(any_udp); filter(f_4); destination(windows_log); flags(final); }; filter f_1 { facility(local5) and level(debug..emerg); }; log { source(any_udp); filter(f_1); destination(routers_log); flags(final); }; filter SonicWallNoise { match("id=firewall" value("MSG")) and filter(SonicWallMsgs); }; filter SonicWallMsgs { match("m=97" value("MSG")) or match("m=98" value("MSG")) or match("m=537" value("MSG")); }; log { source(any_udp); filter(SonicWallNoise); destination(dev_null); flags(final); }; filter f_3 { facility(local0) and level(debug..emerg); }; log { source(any_udp); filter(f_3); destination(ravlin_log); flags(final); }; log { source(any_tcp); destination(workstation_log); flags(final); }; log { source(any_udp); destination(catch-all_log); flags(final); }; This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
On Fri, 2009-05-29 at 13:07 -0500, Marvin Nipper wrote:
Hi. I’m been running syslog-ng 2.0.9 for eons, and wanted to get 3.0.2 in place, but have run into two issues.
First, the prerequisite info: This is on Solaris 10 x86, and I’m running the eventlog-0.2.9 component. It has all compiled cleanly. I’m using the same script to initialize syslog-ng as I was using before. In the conf file, added in some value() parameters on my filters, in accordance with the 3.0 “warnings”, and added in the @version directive, but other than that, it’s essentially the same (reasonably simple) conf file I was using with my 2.0.9 version of the code.
For reference, I’ve appended my conf file at the end of the email.
“Generally” syslog-ng, seems to start-up fine, and does feed the output files that I’ve indicated in the conf file. If I didn’t “look too closely”, I would have thought it was all fine, but…
My problems: As part of my test environment, I’m forwarding data from another (production 2.0.9) server, to my test server, so that I can see how things are working. Here are some (truncated) messages, showing both the original message (as it looks in the output file on the 2.0.9 server), and the second message being what gets written to the file by the 3.0.2 component: Original: May 29 09:15:05 10.132.240.25 id=firewall sn=0006B10B2E88 <truncated>
Received and written by 3.0.2: May 29 09:15:05 10.132.240.25 id=firewall: sn=0006B10B2E88 <truncated>
So the “bug” (I assume it to be a bug) is that “colon” that is now sitting just behind the word “firewall”. When my test server is running a 2.0.9 daemon, these two lines match, but the 3.0.2 daemon has inserted that colon. I can only assume that this is tied to a bug in the filtering activity, because if you look at my conf file, you will see these lines: filter SonicWallNoise { match("id=firewall" value("MSG")) and filter(SonicWallMsgs); }; Bottom line, I am doing a filter that happens to “match on” that exact “firewall” string, and it seems like more than just coincidence that this is the exact location where that unwanted colon now appears. It seems like that filtering process has somehow injected that character into the output. (Maybe I’m just missing some “new functionality or feature” in the 3.0 docs, that would intentionally cause this?)
Anyway… I’m obviously looking for any input as to why this is different, and/or how to fix it (or even just work-around the problem until it is fixed).
The internal representation of messages was optimized, every incoming data is processed and parsed into the internal representation and then the message is reconstructed. Your sample message has no "program-name" field, thus the incoming syslog-ng thinks that the first word is the program name and when it is reconstructing the message it uses the normal program[pid]: msg format, thus adds a ':'. You can work this around by using the 'store-legacy-msghdr' flag on your source, this causes some performance degradation, but otherwise should solve your problem. The root cause of the problem is the ambigous format of the syslog message. I'm usually quite reluctant to change the parsing code (as it causes problems for others), but I might add a workaround to this issue, as others have ran into this as well. The workaround would be to assume that there's no program name if there's no colon. in that case $MSGHDR would expand to nothing.
And secondly, as my subject line implies, when I run my “/etc/init.d/syslog-ng start” scripting (the same one I’ve always used), I find, right away, that I now have two instances of syslog-ng running. I saw a bug report from back in 2006, mentioning a similar behavior that was related to (I think) syslog-ng restarting itself, but that was supposedly fixed. I’m ass-u-me-ing that something is now causing the daemon to restart itself, but again, when I run my 2.0.9 code on place of the 3.0.2 code, I get just the one copy running.
This is not two instances of syslog-ng, syslog-ng now has a 'supervisor' process that monitors the child, and if it crashes, it automatically restarts it. So this is not a problem, it is intended behaviour. You can change it with the --process-mode command line option.
Again, I’m looking for any guidance as to something that might be driving that behavior (and as I said, maybe it’s just something stupid on my part, because I’m missing something “new” in the 3.0 behavior).
Thanks, in advance, for any help and/or guidance as to what these issues might be. I’m glad to provide further information if it will help.
Hope this helps. -- Bazsi
THANKS for your replies. I just wanted to confirm that I did discover the store-legacy-msghdr yesterday afternoon, and that did provide a method for retaining the same message formats as I was getting in 2.0.x. Also, now knowing that there are now two operational processes in 3.x obviously resolves my concern about seeing two instances. THANKS again for all of your efforts, and your help. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Wednesday, June 03, 2009 3:06 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng 3.0.2 issues: Extraneous colon (:) in forwarded log; two instances of syslog-ng On Fri, 2009-05-29 at 13:07 -0500, Marvin Nipper wrote:
Hi. I’m been running syslog-ng 2.0.9 for eons, and wanted to get 3.0.2 in place, but have run into two issues.
First, the prerequisite info: This is on Solaris 10 x86, and I’m running the eventlog-0.2.9 component. It has all compiled cleanly. I’m using the same script to initialize syslog-ng as I was using before. In the conf file, added in some value() parameters on my filters, in accordance with the 3.0 “warnings”, and added in the @version directive, but other than that, it’s essentially the same (reasonably simple) conf file I was using with my 2.0.9 version of the code.
For reference, I’ve appended my conf file at the end of the email.
“Generally” syslog-ng, seems to start-up fine, and does feed the output files that I’ve indicated in the conf file. If I didn’t “look too closely”, I would have thought it was all fine, but…
My problems: As part of my test environment, I’m forwarding data from another (production 2.0.9) server, to my test server, so that I can see how things are working. Here are some (truncated) messages, showing both the original message (as it looks in the output file on the 2.0.9 server), and the second message being what gets written to the file by the 3.0.2 component: Original: May 29 09:15:05 10.132.240.25 id=firewall sn=0006B10B2E88 <truncated>
Received and written by 3.0.2: May 29 09:15:05 10.132.240.25 id=firewall: sn=0006B10B2E88 <truncated>
So the “bug” (I assume it to be a bug) is that “colon” that is now sitting just behind the word “firewall”. When my test server is running a 2.0.9 daemon, these two lines match, but the 3.0.2 daemon has inserted that colon. I can only assume that this is tied to a bug in the filtering activity, because if you look at my conf file, you will see these lines: filter SonicWallNoise { match("id=firewall" value("MSG")) and filter(SonicWallMsgs); }; Bottom line, I am doing a filter that happens to “match on” that exact “firewall” string, and it seems like more than just coincidence that this is the exact location where that unwanted colon now appears. It seems like that filtering process has somehow injected that character into the output. (Maybe I’m just missing some “new functionality or feature” in the 3.0 docs, that would intentionally cause this?)
Anyway… I’m obviously looking for any input as to why this is different, and/or how to fix it (or even just work-around the problem until it is fixed).
The internal representation of messages was optimized, every incoming data is processed and parsed into the internal representation and then the message is reconstructed. Your sample message has no "program-name" field, thus the incoming syslog-ng thinks that the first word is the program name and when it is reconstructing the message it uses the normal program[pid]: msg format, thus adds a ':'. You can work this around by using the 'store-legacy-msghdr' flag on your source, this causes some performance degradation, but otherwise should solve your problem. The root cause of the problem is the ambigous format of the syslog message. I'm usually quite reluctant to change the parsing code (as it causes problems for others), but I might add a workaround to this issue, as others have ran into this as well. The workaround would be to assume that there's no program name if there's no colon. in that case $MSGHDR would expand to nothing.
And secondly, as my subject line implies, when I run my “/etc/init.d/syslog-ng start” scripting (the same one I’ve always used), I find, right away, that I now have two instances of syslog-ng running. I saw a bug report from back in 2006, mentioning a similar behavior that was related to (I think) syslog-ng restarting itself, but that was supposedly fixed. I’m ass-u-me-ing that something is now causing the daemon to restart itself, but again, when I run my 2.0.9 code on place of the 3.0.2 code, I get just the one copy running.
This is not two instances of syslog-ng, syslog-ng now has a 'supervisor' process that monitors the child, and if it crashes, it automatically restarts it. So this is not a problem, it is intended behaviour. You can change it with the --process-mode command line option.
Again, I’m looking for any guidance as to something that might be driving that behavior (and as I said, maybe it’s just something stupid on my part, because I’m missing something “new” in the 3.0 behavior).
Thanks, in advance, for any help and/or guidance as to what these issues might be. I’m glad to provide further information if it will help.
Hope this helps. -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
participants (2)
-
Balazs Scheidler
-
Marvin Nipper