- syslog-ng - lists.balabit.hu

Re: [syslog-ng] syslog-ng deadlock if /dev/console locks?
by Patrick H. 26 Jan '11

26 Jan '11

Ah, there might be other differences that you have then. All our machines are almost identical, same OS, only 2 hardware models, same syslog versions, etc, so there wasnt much variance to cause problems. Definitely seems like an HP issue though, and common enough I would have thought something they would have caught and corrected by now :-/ -Patrick Sent: Wed Jan 26 2011 12:55:47 GMT-0500 (Eastern Standard Time) From: Sowell, Brett <brett.sowell(a)amd.com> To: Patrick H. <syslogng(a)feystorm.net> "Krizak, Paul" <Paul.Krizak(a)amd.com>, Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu>, "Petrini, Bryce" <Bryce.Petrini(a)amd.com>, "Hart, Corey" <Corey.Hart(a)amd.com> Subject: Re: [syslog-ng] syslog-ng deadlock if /dev/console locks? > Hi Patrick, > > I had come across your post during research and added the suggested > echo 'h' to our break/recovery test scripts. > > It seemed that, in our case, echoing 'h' to sysrq-trigger would result > in a stable recovery of /dev/console 2/10 times, sometimes temporary > recovery, but more resulted in no effect. > > -Brett > > On 01/26/11 10:17, Patrick H. wrote: >> Nope, just 'echo h', that was it. >> >> -Patrick >> >> Sent: Wed Jan 26 2011 11:16:31 GMT-0500 (Eastern Standard Time) >> From: Paul Krizak <paul.krizak(a)amd.com> >> To: Syslog-ng users' and developers' mailing list >> <syslog-ng(a)lists.balabit.hu> "Patrick H." <syslogng(a)feystorm.net>, >> "Sowell, Brett" <Brett.Sowell(a)amd.com>, "Petrini, Bryce" >> <Bryce.Petrini(a)amd.com>, "Hart, Corey" <Corey.Hart(a)amd.com> >> Subject: Re: [syslog-ng] syslog-ng deadlock if /dev/console locks? >>> Fascinating. So just triggering the kernel to print something to >>> the console (h is "help") caused /dev/console to properly realign >>> and syslog-ng woke back up? You didn't have to restart syslog-ng or >>> reboot the box or anything? >>> >>> >>> Paul Krizak 7171 Southwest Pkwy MS B200.3A >>> MTS Systems Engineer Austin, TX 78735 >>> Advanced Micro Devices Desk: (512) 602-8775 >>> Linux/Unix Systems Engineering Cell: (512) 791-0686 >>> Global IT Infrastructure Fax: (512) 602-0468 >>> >>> On 01/26/11 10:11, Patrick H. wrote: >>>> We ran into this issue when upgrading iLO on all our boxes. When >>>> the iLO >>>> was upgraded, /dev/console went completely unresponsive, and things >>>> started to hang. The solution turned out to be 'echo h > >>>> /proc/sysrq-trigger'. Apparently when the kernel went to write out to >>>> the serial port, it ran into problems and would reinitialize it. After >>>> that everything started working fine. >>>> >>>> -Patrick >>>> >>>> Sent: Wed Jan 26 2011 11:03:37 GMT-0500 (Eastern Standard Time) >>>> From: Sandor Geller <Sandor.Geller(a)morganstanley.com> >>>> To: Syslog-ng users' and developers' mailing list >>>> <syslog-ng(a)lists.balabit.hu> "Sowell, Brett" <Brett.Sowell(a)amd.com>, >>>> "Petrini, Bryce" <Bryce.Petrini(a)amd.com>, "Hart, Corey" >>>> <Corey.Hart(a)amd.com> >>>> Subject: Re: [syslog-ng] syslog-ng deadlock if /dev/console locks? >>>>> Hello, >>>>> >>>>> On Wed, Jan 26, 2011 at 4:12 PM, Paul Krizak<paul.krizak(a)amd.com> >>>>> wrote: >>>>> >>>>>> Hi, we're using syslog-ng 3.1.2 and have run into what appears to >>>>>> be a >>>>>> bug, but I'd like to get the community's opinion before we dig >>>>>> further >>>>>> into it. >>>>>> >>>>>> We have a bunch of HP servers with iLO2 and iLO3 devices, configured >>>>>> with their virtual serial ports on COM1 (ttyS0). We subsequently >>>>>> have >>>>>> the OS (RHEL4, RHEL5) configured to use COM1 as its console (e.g. >>>>>> /dev/console). This is a very standard configuration that allows >>>>>> us to >>>>>> get remote access to the machines without having to purchase the iLO >>>>>> Advanced KVM feature. It also lets us use the Magic SysRq keys >>>>>> to probe >>>>>> dead systems and stuff, so in general it's not something we're >>>>>> keen to >>>>>> change. >>>>>> >>>>>> What we have found, however, is that there are some cases where >>>>>> the iLO >>>>>> will freeze and requires a reboot. When the iLO reboots, >>>>>> however, the >>>>>> kernel's connection to /dev/console (through the virtual serial >>>>>> port) >>>>>> hangs and blocks. Any traffic to /dev/console just sits in the >>>>>> kernel's >>>>>> buffer and is never delivered. Once the buffer is full, the kernel >>>>>> simply blocks on any write to /dev/console. >>>>>> >>>>>> Now this is a Bad Thing in general, and we're working with HP to >>>>>> try and >>>>>> remedy this bug. However, what concerns me is that syslog-ng, when >>>>>> faced with this behavior, also blocks, even for log messages not >>>>>> bound >>>>>> for /dev/console. >>>>>> >>>>> >>>>> syslog-ng uses a single thread (with the exception of database >>>>> destinations) running the event loop so when a read() or a write() >>>>> blocks then it affects the whole log processing >>>>> >>>>> >>>>>> What we have observed is that a system with syslog-ng will keep >>>>>> delivering the occasional console message to /dev/console (ex. >>>>>> *.emerg >>>>>> messages) and meanwhile the file-based log paths keep working. >>>>>> But once >>>>>> /dev/console blocks, the next time a console message is >>>>>> delivered, *all* >>>>>> of syslog-ng blocks waiting for that message to be delivered, and >>>>>> all of >>>>>> the file-based paths block as well. The result is that pretty much >>>>>> everything on the system stops working. For example, you can't >>>>>> log in, >>>>>> even as root, because the login process blocks on the syslog command >>>>>> that writes to /var/log/secure. Anything that uses syslog >>>>>> suddenly blocks. >>>>>> >>>>>> Is this expected behavior? I would think that syslog-ng would be >>>>>> able >>>>>> to continue accepting and delivering messages, even if one of the >>>>>> log >>>>>> paths is stalled on a blocked write. >>>>>> >>>>> >>>>> syslog-ng uses non-blocking I/O for all sources / destinations but >>>>> despite of this the kernel could still block it therefore syslog-ng >>>>> protects reads/writes in logtransport.c with alarm() so it should >>>>> recover when timeout is set and a read/write blocked. For me it looks >>>>> like the timeout is not set in all cases, only file and program >>>>> sources initialise transport->timeout to 10 secs so I'd say this >>>>> isn't >>>>> expected behaviour - it is a bug. >>>>> >>>>> Regards, >>>>> >>>>> Sandor >>>>> ______________________________________________________________________________ >>>>> >>>>> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>> Documentation:http://www.balabit.com/support/documentation/?product=syslog-… >>>>> >>>>> FAQ:http://www.campin.net/syslog-ng/faq.html >>>>> >>>>> >>>

1 0

syslog-ng - Apache Logging
by Steven Shepherd 26 Jan '11

26 Jan '11

I have Apache logging working via a named pipe, however the log looks like: Jan 25 14:44:13 $HOSTNAME $VIRTUAL_HOST: 000.000.000.000 - - [25/Jan/2011:14:44:13 -0600] "GET /foo.html HTTP/1.0" 200 24040 "-" "Wget/1.10.2 (Red Hat modified)" I want to strip the syslog timestamp and $HOSTNAME. However, if I use "template("$MSGONLY\n")", it strips the timestamp, hostname *and* the $virtual_host (obtained from including "%V" in LogFormat and using 'vcommon' on apache server). Any ideas on how to remove the syslog data but leave the vhost data in place? Cheers!

2 2

Problem with getting syslog-ng statistics
by Aleksei Plotnikov 25 Jan '11

25 Jan '11

Hello everyone, I have a problem with getting statistics from syslog-ng 3.0.5 (under Debian 5.0.4) using it's socket file. In our configuration logs received by syslog-ng are being forwarded to python scripts (which in turn send logs to database). Since sometimes these scripts cannot handle the load, we want to monitor syslog-ng statistics for dropped logs. Currently, I run a script from cron which get statistics from syslog-ng socket file like this "echo STATS | socat - /var/lib/syslog-ng/syslog-ng.ctl" and then parse output. Most of time it works great, but during peak times when number of logs get higher ( ~2000 logs/sec instead of ~1500 logs/sec), this cron script cannot get any output from statistics. Also this message appears in /var/log/syslog: syslog-ng[21677]: Error writing control channel; error='Broken pipe (32)' Is it possible to tune syslog-ng statistics somehow, or maybe there is a better way to get statistics? Thanks, Aleksei ---------------------------- Võimalik, et oled TASUTA 1600€ soojamaareisist vaid CV sisestamise kaugusel! Vaata lähemalt SIIT. http://www.cv.ee

2 2

AUTO: Shanif Lakhani is out of the office (returning 02/14/2011)
by Shanif Lakhani 25 Jan '11

25 Jan '11

I am out of the office until 02/14/2011. Please contact Robert Holoweckyj and/or Lori-Ann Walker for any urgent matters. Thanks Note: This is an automated response to your message "syslog-ng Digest, Vol 69, Issue 44" sent on 1/24/11 20:55:19. This is the only notification you will receive while this person is away.

1 0

[review request]: MongoDB destination
by Gergely Nagy 24 Jan '11

24 Jan '11

I prepared three branches, in preparation for a pull request, but there's still some minor work left to do, so this is more like a "review the structure and plan before the pull request, please" request. All three branches are integration branches, and as such, I will rebase them from time to time. All three branches are folded into one commit over their base, signed-off, and should have proper descriptions. * integration/libmongo-client This one adds a git submodule, which pulls in the libmongo-client library from github to modules/afmongodb/libmongo-client. Also adds infrastructure to make autogen.sh, configure and make enter this directory and Do The Right Thing. At the moment, linking with the supplied libmongo-client is the only option, because the library is not yet ready for system-wide install. It is easy to extend this framework to use the system library if it exists - that is one of the missing features which should go into this branch, but isn't there yet. * integration/afmongodb/base This is the basic, bare-bones driver, without support for dynamic values: it depends on the previous one, and doesn't change anything outside of modules/afmongodb/, except lib/stats.[ch], where it adds SCS_MONGO for stat counter support. The current state of the driver is reasonably stable, there are three known problems that I am aware of: + Segfaults whenever the writer thread emits an msg_debug() or similar itself. This is actually a bug in ivykis, for which Bazsi had a fix for, but I can't find the fix in his git tree yet (only the log message, but without the patch). + Locking issues: it's fairly easy to push the driver into a deadlock, where it will just sit and wait, not doing a thing. I've only seen this happen with flow-control enabled, though. This will have to be solved before I'm confident enough to ask for a pull. + The libmongo-client library is built as a shared library, and is installed as such. It should be static-only, and not installed, only linked to afmongodb. This will be implemented mostly in libmongo-client, and an update of the integration/libmongo-client branch will pull that in; then this branch will be updated accordingly. * integration/afmongodb/dyn-vars Based on integration/afmongodb/base, this adds dynamic_values() support. Minor modifications are done to nvtable too, but it's a very small patch, and very straightforward too. However, as discussed earlier, the goal in the long run is to develop a generic framework for this kind of thing, so this here is more like a temporary solution so that people can experiment with the key feature the mongodb provides over the others. Until the generic framework is ready, this, I believe, is an acceptable compromise. -- |8]

2 2

Error starting syslog-ng
by Fabien Bagard 24 Jan '11

24 Jan '11

Hi list ! I have a problem to start syslog-ng. I got a syntax error on line in bold : ###### # sources # all known message sources source s_all { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" log_prefix("kernel: ")); }; source s_unix { tcp( port(514) * tls(* key_file("/etc/syslog-ng/certs/logger.parrot.biz.key") cert_file("/etc/syslog-ng/certs/logger.parrot.biz.crt") ca_dir("/etc/syslog-ng/certs/CA/") peer_verify(optional-untrusted) ) ); }; But I can't figure out what ! Can someone point this error to me ? Thanks -- Fabien Bagard IT Department tel + 33 (0)1 48 03 60 40 -------------------------------------------------------------------------------- Parrot SA 174, Quai de Jemmapes | 75010 Paris - France tel + 33 (0)1 48 03 60 60 | fax + 33 (0)1 48 03 70 08 http://www.parrot.com -------------------------------------------------------------------------------- This e-mail message and any attached document(s) are for the sole use of the intended recipient(s)and may contain confidential and legally privileged information. Any unauthorized review, copy, use and/or disclosure is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original.

4 9

Syslog-ng error
by Shawn Cannon 24 Jan '11

24 Jan '11

I am trying to setup a second syslog-ng server that will be part of a load balanced cluster. When the 2nd server tries to connect to my mssql database, it tries to run the CREATE TABLE commands which fails because the table already exists since the first syslog-ng server already created them. How do I tell the 2nd server not to try the CREATE TABLE stuff so it will start working? Shawn Cannon IT Professional http://www.shawncannon.com Contact Me <http://www.linkedin.com/pub/shawn-cannon/5/84/81a> LinkedIn<http://www.linkedin.com/pub/shawn-cannon/5/84/81a> <http://www.facebook.com/rolltidega72> Facebook<http://www.facebook.com/rolltidega72> <http://twitter.com/rolltidega> Twitter <http://twitter.com/rolltidega> My blog: My thoughts on VMworld… a week later…<http://www.shawncannon.com/?p=19>

5 10

syslog-ng 3.1.3-2 - working examples of remote logging
by Steven Shepherd 21 Jan '11

21 Jan '11

OK, so syslog has come a long way and many configuration changes in the years :) Can someone provide working examples of (3.1.X) client/server syslog-ng.conf set up for remote logging purposes such as /var/log/messages, /var/log/kern.log, /var/log/auth.log...etc? Would want on local machine as well as remote syslog server. Will also want to remotely log apache access/error log via syslog-ng as well. Any help/tips/pointers is greatly appreciated! Cheers!

3 2

Open source version: syslog-ng-3.1.3-1.rhel5.i386.rpm
by julian 20 Jan '11

20 Jan '11

what install options are assumed for syslog-ng-3.1.3-1.rhel5.i386.rpm: server, relay, client? It is acting as relay.

3 2

oracle db + syslog-ng integration
by vijay 19 Jan '11

19 Jan '11

hi, Iam Using syslog-ng 3.0.4.when i used below configuration destination d_sql { sql(type(oracle) username("myusername") password("mypassword") database("LOGS") table("msgs_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar2") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; iam facing error while starting service syntax error in /usr/local/etc/syslog-ng.conf at line 322. syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng the line 322 is---------- sql(type(oracle) username("myusername") password("mypassword") vijay.p

2 3