- syslog-ng - lists.balabit.hu

program() destination in 1.9.6: respawning not automatic?
by Paul Krizak 28 Oct '05

28 Oct '05

I built the latest stable copy of 1.9 (1.9.6) a few days ago and I'm seeing some behavior that doesn't seem to jive with the documentation. The docs say that the program() destination in 2.0 (which is presumably what 1.9.X is preparing for) should respawn the program if it dies. However, this doesn't seem to be the case; you must signal syslog-ng with a HUP before it'll respawn the program. Is this expected behavior? And if it is, is there some way to make syslog-ng respawn the program immediately instead of waiting for a HUP signal? Thanks! -- Paul Krizak 5900 E. Ben White Blvd. MS 625 Advanced Micro Devices Austin, Tx 78741 Linux/Unix Systems Engineering Phone: (512) 602-8775 Microprocessor Solutions Sector

2 1

syslog-ng.conf : option group(<gid>); does not work ( stable version )
by Yan M. 28 Oct '05

28 Oct '05

Hi, in syslog-ng.conf man page it is said that the option group(gid); is used to set group on created logfiles. Im trying to use that and it does not work it is setting the group, not to the group I specify in that group() directive, but to the group syslog-ng runs as. Also, the man page specifies you have to enter the gid in the group() option but the syslog-ng daemon wont start when the gid is specified in its numeral form. It should be group(groupname); instead of group(gid); in the man page. Does the chgrp works only when syslog-ng is running as root ? If so, it is not a documented feature .. --------------------------------- Yahoo! FareChase - Search multiple travel sites in one click.

3 4

filter help
by Greg Rodenhiser 27 Oct '05

27 Oct '05

I have several remote servers that are logging via regular syslog to my central syslog-ng server. This is working fine. Now, I have several distinct messages from different servers that I want to filter to a single log file (without having anything else in this file). I have tried setting it up but I'm having problems... 1.) I have made six filter lines in syslog-ng.conf that contain the host/message combination I want to put in my special log. I have named these six filters the same name (f_interest). I then have a log line that contains the name of the filer (the filter name is used for six filter declarations), and the destination for the file I want to log to. My problem/question is, How does syslog-ng handle multiple filters that have the same name? Each of these 6 filters contain a host and word match I want to have interpreted. My problem is nothing is getting logged to my log. Does syslog-ng somehow combine these 6 filters because they have the same name? I want each one to be interpreted separately from the others with the same name. I only named them the same to prevent having to make 6 different log directives to the same file. 2.) How do I do a multiple not match. For instance I want to log messages from serverA that do NOT contain the text, message1, or message2, or message3, but log everything else from serverA. Right now I do... filter f_interest { not match("(message1|message2|message3)") and host(serverA); }; Does that look right? Thanks in advance!!

1 0

high performance server
by JR Mayberry 27 Oct '05

27 Oct '05

I've googled this topic and read the mailing list and haven't had much luck improving my situation. I have a very high volume syslog-ng server. I currently have logs that are being received across the network but not being written to disk. This could be as much as 25% of the logs being dropped. The STAT to syslog always says 0 drops. It's a Netro 1405 w/ two CPUs and 1 gig of memory connected to a SAN. syslog-ng is always at 49% CPU and never goes beyond that. What settings can I use to stop dropping logs? This is an output of iostate with an interval of 1. This will give you an idea of performance. As you can see there are no disk waits. Thanks. device r/s w/s Mr/s Mw/s wait actv svc_t %w %b md89 2.0 86.0 0.0 0.4 0.0 0.2 1.9 0 5 md89 0.0 3.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 2.5 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.2 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 1.2 0 0 md89 0.0 61.0 0.0 0.3 0.0 0.1 1.3 0 3 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.5 0 0 md89 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.7 0 0 md89 0.0 83.0 0.0 0.4 0.0 0.2 2.2 0 3 md89 2.0 4.0 0.0 0.0 0.0 0.0 3.0 0 2 md89 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 1.0 86.0 0.0 0.4 0.0 0.1 1.2 0 4 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 5.0 0.0 0.0 0.0 0.0 0.9 0 0 md89 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0 0 md89 0.0 2.0 0.0 0.0 0.0 0.0 0.8 0 0 md89 2.0 751.9 0.0 5.6 0.0 6.5 8.6 0 25

3 5

1.6.5 performance
by Scott C 24 Oct '05

24 Oct '05

I am currently experiencing an odd performance issue with syslog-ng 1.6.5. I use several load-balanced relays which receive syslog traffic via both udp and tcp. They forward all traffic to the back-end collection hosts exclusively via tcp. Overall, this environment collects around 60GB of log data per day. All of the relays have identical Solaris 8 builds, as well as identical syslog-ng configurations. Here is the pertinent snippet from a configuration file: options { stats(300); sync(0); time_reopen(10); log_fifo_size(3000000); gc_busy_threshold(200000); check_hostname(no); keep_hostname(yes); }; As you can see, my fifo queue is enormous. Despite that fact, syslog-ng on any given relay will begin dropping messages at an alarming rate (600-700 per second) when combined inbound and outbound tcp traffic increases beyond a certain threshold. Said threshold seems to be a bit low to me, though. We're talking somewhere in the neighborhood of 700 tcp segments per second. The corresponding inbound UDP dgrams are usually between 200 and 400 per second. It should be noted that the IP stack is handling the load just fine. Also, CPU utilization rarely exceeds 60 or 70 percent. The collection hosts are fairly beefy systems that are barely breathing even during peak load. I've watched both the IP stack and syslog-ng on these systems, and they're keeping up just fine. I know the FIFO queue size probably seems unreasonable here, but the lower I make it, the worse the drops. Of course, the higher I make it, the more memory it eats up. And in some cases, I've seen syslog-ng eventually eat it all up and the scanner kicks in. But what's really most peculiar in this scenario is the fact that the numbers simply don't add up. Why does syslog-ng appear (on the surface) to be dropping a very large percentage of the messages that it receives? I realize that it's not, but the numbers tell a different story. And how could it possibly drop so many messages when the FIFO queue is configured to buffer three million lines? How preposterous! Am I way off base here? --------------------------------- Yahoo! FareChase - Search multiple travel sites in one click.

4 13

Fwd: [syslog-ng] spoof_source not working
by Chance Ellis 22 Oct '05

22 Oct '05

Ok, I have tried everything you have given men and I am still coming up with nothing... Here is the output of my ldd: ldd /usr/local/sbin/syslog-ng libpthread.so.1 => /usr/lib/libpthread.so.1 libnsl.so.1 => /usr/lib/libnsl.so.1 libsocket.so.1 => /usr/lib/libsocket.so.1 libdoor.so.1 => /usr/lib/libdoor.so.1 libresolv.so.2 => /usr/lib/libresolv.so.2 libxnet.so.1 => /usr/lib/libxnet.so.1 libc.so.1 => /usr/lib/libc.so.1 libdl.so.1 => /usr/lib/libdl.so.1 libmp.so.2 => /usr/lib/libmp.so.2 libthread.so.1 => /usr/lib/libthread.so.1 /usr/platform/SUNW,Sun-Fire-V210/lib/libc_psr.so.1 Notice libxnet. Should I be using it? Can it be causing the problem? How do I tell syslog-ng at compile time to use libnet.a or libnet.so? On 10/14/05, Nate Campi <nate(a)campin.net> wrote: > > Ellis, > > I didn't mention in my email to you that you'll want the libnet.so file > in /tmp/foo - but first make sure you're dynamically liked using the ldd > command: > > -0-[root@duo:masterfiles]# ldd /sbin/syslog-ng > libnsl.so.1 => /lib/libnsl.so.1 (0x40028000) > libresolv.so.2 => /lib/libresolv.so.2 (0x4003c000) > libc.so.6 => /lib/libc.so.6 (0x4004e000) > /lib/ld-linux.so.2 (0x40000000) > > You should see libnet.so in there somewhere if you're dynamically > linked. If you're statically linked then you have to narrow it down with > truss as Bazsi has said. > > Good luck, > Nate > > On Fri, Oct 14, 2005 at 05:35:13PM +0200, Balazs Scheidler wrote: > > On Thu, 2005-10-13 at 15:57 -0400, Chance Ellis wrote: > > > Ok, > > > > > > I copied the the Solaris 8 libnet-config file to the /tmp/foo folder > > > on Solaris 9. I then ran: > > > > > > LD_LIBRARY_PATH=/tmp/foo:$LD_LIBRARY_PATH > > > truss /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -F & > > > > > > > > > I get the same result... Whenever I apply the spoof_source(yes) to the > > > config I do not get any messages forwarded to the destination. If I > > > remove the spoof_source(yes) messages flow but with the source IP > > > address from the syslog-ng server... > > > > > > The truss output is quite huge! Is there any piece of the truss output > > > that would help me to troubleshoot this? Is libnet-config the only > > > thing I need or do I need something in addition to libnet-config? > > > > libnet-config is not used at all while running syslog-ng, it is a > > build-time script that is invoked to query header location and linking > > information about the libnet library. > > > > libnet is usually linked statically, so there's no point in setting > > LD_LIBRARY_PATH either, you can check which one syslog-ng is using by > > using "ldd" and/or checking if you have a libnet.a file (static), or > > libnet.so file (dynamic), or both. If you have both, the linking > > parameters determine which one is used by syslog-ng, in this case ldd > > should tell you the truth. > > > > To analyze the truss path, you should look for the pattern of the > > message that should be sent out with a spoofed source address, and then > > you should see system calls like recvmsg() when the message is received > > and either send() or write() when it is sent. libnet uses raw sockets to > > send messages, syslog-ng is probably opening AF_INET, SOCK_RAW sockets > > in that case. > > > > Although Solaris uses socket emulation and thus the actual system calls > > you see in the truss output might not be the same as in Linux, this is > > pretty straightforward. By the way, it might be easier to analyze the > > truss output if you are running a syslog-ng instance which does not > > actually deliver a lot of messages, but only a single one, this way the > > truss output will not be so large. > > -- > Nate > > "Often, when I am reading a good book, I stop and thank my teacher. That > is, I used to, until she got an unlisted number." - Unknown 15-year-old > > >

3 4

Queueing of messages if remote loghost is down
by Eric Brown 22 Oct '05

22 Oct '05

I'd like to centralize logs from 2-data centers to my corporate offices via tcp destinations at my data centers and a tcp source at my corporate office. What if my data centers suffer network connectivity problems or I loose power at my corporate office? How long will syslog-ng keep queueing messages before I start loosing data? Will its attempts to retry connecting to my corporate office adversely affect overall system performance of the machine it is running on? Will it recover on its own after a few hours of lost connectivity? Thanks, Eric

1 0

Fwd: Re: [syslog-ng] I need to add more facilities
by Yan M. 22 Oct '05

22 Oct '05

On Fri, 2005-10-21 at 01:03 -0500, Kevin wrote:> On 10/20/05, Yan M. <yannnick_m(a)yahoo.com> wrote:> > searching through the documentation, FAQ, README> > and INSTALL I didnt find anywhere mentionned that I can> > create new facilities.> >> > I read that syslog-ng has the same facilities than syslog in Solaris.> > Technically, syslog-ng has the same facilities as all> standards-compliant syslog daemons, because the facility range is> defined in RFC 3164> > There are only 24 legal values for the facility, and eight legal> values for severity.> > > What I want to do is to have more facilities, with significative names> > like httpd0, httpd1, httpd2, httpd3.....and so on to httpd50.> >> > This is to split message received from my apache VirtualHosts AccessLog and> > ErrorLog> > How would you cause Apache to generate "syslog" messages with> non-standards-conformant facility values, corresponding to your> virtual hosts?> > > > I dont want to filter using regexp, only with facility> > One ugly hack to accomplish what you ask would be to use the standard> facilities and severities to indicate which virtual host sourced the> traffic. So "kern.debug" would be httpd0, kern.info would be httpd1,> and local7.emergency would be httpd191> > It might be easier to just use regex.syslog-ng 2.x allows specifying facility values by number, even stilldifficult to generate non-compliant facility messages in the firstplace.The reason why the facility based filtering failed in the first placewas that no appropriate processes were created to allocate new numbers.-- Bazsi_______________________________________________syslog-ng maillist - syslog-ng@lists.balabit.huhttps://lists.balabit.hu/mailman/listinfo/syslog-ngFrequently asked questions at http://www.campin.net/syslog-ng/faq.html Rethinking about it, I found another way to do it. It is under Solaris8 so I can use pipes, apache ErrorLog and AccessLog can write to pipes. Apache runs in a chroot jail, and ErrorLog's/AccessLog's of all VirtualHosts write to their own unique pipe in the jail. Syslog-ng runs not chrooted and can read the pipes so I defines a source for each pipes in the config I can split logs from apache that way. Thanks for your replies Yan --------------------------------- Yahoo! FareChase - Search multiple travel sites in one click.

1 0

new list: syslog-ng-commit
by Balazs Scheidler 21 Oct '05

21 Oct '05

Hi, For those interested I've created a new list named syslog-ng-commit where information about the commits to my internal syslog-ng 2.0 tree will be published. To subcribe go to https://lists.balabit.hu/mailman/listinfo/syslog-ng-commit/ The number of commits is usually low, you don't need to expect more than 1-5 commits weekly. I hope this list can help you to track the progress towards syslog-ng 2.x. Each message lists commit meta information, changelog entry and a diffstat on the commit. For now the patch will not be available, I'm going to publish my internal GNU arch tree if requested. -- Bazsi

1 0

I need to add more facilities
by Yan M. 21 Oct '05

21 Oct '05

Hi list, searching through the documentation, FAQ, README and INSTALL I didnt find anywhere mentionned that I can create new facilities. I read that syslog-ng has the same facilities than syslog in Solaris. What I want to do is to have more facilities, with significative names like httpd0, httpd1, httpd2, httpd3.....and so on to httpd50. This is to split message received from my apache VirtualHosts AccessLog and ErrorLog I dont want to filter using regexp, only with facility Thank you Yan --------------------------------- Yahoo! FareChase - Search multiple travel sites in one click.

3 2