- syslog-ng - lists.balabit.hu

29 Aug '08

Hi! I'm trying to filter some logs that have no program field. kind of "logger -t '' " logs. I've try to use program(""); or program(NULL); as filter but none of them works. How can I filter this empty program field logs ? Thanks! Gaël.

4 7

Re: [syslog-ng] destination program is killed due to broken pipe
by prmarino1＠gmail.com 28 Aug '08

28 Aug '08

well the script on the surface sounds like a good simple idea, the simple way is not always the best or most relyable way. There are several better options open to you that should work better and shouldn't be too hard to impliment. The first is to write to a log file then use something like a "tail -f" in your script. Another option open to you is if you know how to write unix network program's in any languages (I use perl for this kind of thing my self) it wouldn't be difficult to write a simple application read from a socket, or even udp syslog on an alternitive port on the loopback address. Syslog is a relitivly simple protocal to write a listener for and there are several perl modules that can do most of the work for you. ----------------------- Sent from my Treo(r) smartphone -----Original Message----- From: "midnight codder" <midnight.codder(a)gmail.com> Date: Tuesday, Aug 19, 2008 5:08 pm Subject: [syslog-ng] destination program is killed due to broken pipe To: syslog-ng(a)lists.balabit.huReply-To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> I am trying to redirect all log messages to a destination program (written in java). This did not do much so I redirected all messages to a single catch-all log file AND also to a bash shell script that runs an endless loop, reads a line and spit it out to a text file. The shell script is killed by syslog-ng after the 5th message. The catch-all file, however, keeps getting all of the messages (that's how I know who is killing who). Here is the config file for syslog-ng: options { ts_format(iso); frac_digits(3); }; template bds_template{ template('<pri=$PRI> <date=$ISODATE> <host=$HOST> <msg=$MSG>\n'); template_escape(no); }; source src{ internal(); unix_stream('/dev/log'); unix-dgram('/dev/log'); file('/proc/kmsg' log_prefix('kernel:')); udp(); tcp(); }; destination test_adapter{ program('bash /usr/local/bds/logger/syslogAdapterTest.sh' template(bds_template)); }; log { source(src); destination(test_adapter); }; destination catchAll{ file('/var/log/syslog-ng-catch-all.log', template(bds_template)); }; log { source(src); destination(catchAll); }; Here is the bash script: #!/bin/bash file='/root/syslogAdapterTest' counter=1 limit=3 while true do read line echo '$line' > $file.$counter ((counter++)) done and finally the content of the catch-all file, in blue are the log messages that the shell script still captures and spits to text files, in red are the messages are the messages the shell does not get: <priE> <date 08-08-19T15:13:56.191-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Server license found, starting in server mode; customer='University of Illinois', serial='49544f30-573e-11dd-9892-0007e9195011', limit='25'> <priF> <date 08-08-19T15:13:56.191-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Starting destination program; cmdline='bash /usr/local/bds/logger/syslogAdapterTest.sh'> <priE> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: syslog-ng starting up; version='2.1.11a'> <priF> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Initializing destination file writer; template='/var/log/syslog-ng-catch-all.log', filename='/var/log/syslog-ng-catch-all.log'> <priE> <date 08-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: License validity expires in one week, logging service will terminate at that point; error='License outside the validity period (option=syslog-ng, not-before 08/07/21, not-after 08/08/21)'> <priC> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <priF> <date 08-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Sending child a TERM signal; child_pid='27972'> Any idea?

3 6

Can syslog-ng support TCP and UDP simultaneously?
by Guy Fleegman 28 Aug '08

28 Aug '08

Hello: Currently we have all of our devices sending syslog data to a 2.05 syslog-ng server. However... there is one instance where we need to use TCP as a delivery mechanism Can syslog-ng support having both TCP and UDP sources simultaneously? If so ..where might I find a configuration example? Please advise and thanks, -Guy

3 2

Lots of dropped=tcp(AF_INET(
by Ivan Lezhnjov Jr. 27 Aug '08

27 Aug '08

Hello I have two hosts on my LAN running syslog-ng. Host A acts as central loghost and host B sends its log messages via tcp/ip network connection to host A to store. The issue is that I've got a lot of messages like this one (and almost no normal ones) generated by host B: Aug 19 19:25:23 xerxes syslog-ng[20806]: Log statistics; dropped='tcp(AF_INET(217.117.75.2:1999))=0', processed='center(queued)=112', processed='center(receive d)=23', processed='destination(newsnotice)=0', processed='destination(console)=0', processed='destination(debug)=23', processed='destination(mailinfo)=0', proc essed='destination(mail)=0', processed='destination(user)=0', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(u ucp)=0', processed='destination(messages)=19', processed='destination(mailwarn)=0', processed='destination(ppp)=1', processed='destination(loghost)=23', proces sed='destination(lpr)=0', processed='destination(cron)=0', processed='destination(syslog)=23', processed='destination(authlog)=0', processed='destination(maile rr)=0', processed='destination(kern)=0', processed='destination(daemon)=0', processed='destination(xconsole)=0', processed='destination(console_all)=23', proce ssed='source(net)=0', processed='s While it is obvious that dropped tcp speaks of network issues it isn't really clear to me (being a novice syslog-ng user) what exactly goes wrong here. So, could you please explain what's wrong with host B and how I can fix that? -- Ivan Lezhnjov Jr. Europe, Ukraine, Simferopol Running Source Mage GNU/Linux, kernel version 2.6.24 build #5 +----------------------------------------------------------------------+ Key ID 0x5811D90C Key Fingerprint 2A52 5C8C 38BE C04F D8DE A169 19E2 E49A 5811 D90C Use GPG Exercise Your Right To Privacy

5 9

Re: [syslog-ng] syslog-ng Digest, Vol 40, Issue 16
by vadi 27 Aug '08

27 Aug '08

Hi Joe, Thanks for your replay. I am not able to understand the regular expression statement with are there in the internet. For example: \s(\S+)\s(\S+)\[\d+\]\:\s\[ID \d+ (\S+)\.(\S+)\]\s \S+\s+\d+\s+(\d+)\:\d+\:\d+\s I want to know what exactly text of this pattern? Is there any tool will convert RegExp to text?. So that it will helps for me to understand better. Regards, Vadiraj On Mon, Aug 25, 2008 at 3:30 PM, <syslog-ng-request(a)lists.balabit.hu> wrote: > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. Re: tool to convert regular expression to text (Fegan, Joe) > 2. Re: syslog-ng 2 nics (Balazs Scheidler) > 3. Re: [patch] Add follow_freq_ms option (Balazs Scheidler) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 24 Aug 2008 23:34:59 +0000 > From: "Fegan, Joe" <Joe.Fegan(a)hp.com> > Subject: Re: [syslog-ng] tool to convert regular expression to text > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: > < > 0E6222894DE49B40A3A9FD929C630BCB379EB5A2EF(a)GVW1121EXC.americas.hpqcorp.net > > > > Content-Type: text/plain; charset="us-ascii" > > Hi Vadiraj, > > Your question doesn't really make sense. Can you please elaborate. That > might help. > > Joe. > > ________________________________ > From: syslog-ng-bounces(a)lists.balabit.hu [mailto: > syslog-ng-bounces(a)lists.balabit.hu] On Behalf Of vadi > Sent: 22 August 2008 22:15 > To: syslog-ng(a)lists.balabit.hu > Subject: [syslog-ng] tool to convert regular expression to text > > Dear All, > > Do we have any tool to convert regular expression to text?. Please let me > know about it. > > Thanking u, > > Regards, > Vadiraj > >

2 1

Re: [syslog-ng] FreeBSD dist syslog-ng.conf-sample dumps mail.info to maillog & messages
by Len Conrad 26 Aug '08

26 Aug '08

> > > > postfix is sending only warnings to messages > >Well, this rings a bell. When you run a chrooted postfix then it >could fall back using /dev/log (or its FreeBSD equivalent) in the >chroot when syslog-ng is restarted. A commonly used workaround on >linux is to create the /var/spool/postfix/dev directory by hand >and add /var/spool/postfix/dev/log to the sources of syslog-ng. Interesting point, thanks. We aren't running chroot, but noted for the future. To the "log" directives feeding stuff into destination file "messages", I added a NOT filter for "postfix.*warning" and for "postgrey". >I don't see any problems with your config, although I did only >a quick check. fixed my error: filter f_conn_from_unk { match(": connection from unknown"); }; to filter f_conn_from_unk { match(": connect from unknown"); }; thanks, Len

1 0

FreeBSD dist syslog-ng.conf-sample dumps mail.info to maillog & messages
by Len Conrad 26 Aug '08

26 Aug '08

I can't see how this is happening. I can stop the mail.info to maillog, but I can't stop it to messages since I can't find any destination (messages) being fed by mail.info I found this and commented it out, but no change: #log { source(src); filter(f_mail); filter(f_crit); destination(messages); }; I'm looking for : log { source(src); filter(f_mail); filter(f_info); destination(messages); }; or anything similar but can't find it. It's weird that this is the vanilla syslog-ng.conf-sample with the FreeBSD 6 package. thanks, Len

2 2

Re: [syslog-ng] FreeBSD dist syslog-ng.conf-sample dumps mail.info tomaillog & messages
by infosec＠gmail.com 26 Aug '08

26 Aug '08

Len, Go ahead and send the whole file to the list, or if it's really large post it somewhere and send us the link. Is it possible that you're still running another syslog daemon and it isn't syslog-ng sending the messages there? -----Original Message----- From: Len Conrad <LConrad(a)Go2France.com> Subj: [syslog-ng] FreeBSD dist syslog-ng.conf-sample dumps mail.info tomaillog & messages Date: Mon Aug 25, 2008 9:03 pm Size: 852 bytes To: syslog-ng(a)lists.balabit.hu I can't see how this is happening. I can stop the mail.info to maillog, but I can't stop it to messages since I can't find any destination (messages) being fed by mail.info I found this and commented it out, but no change: #log { source(src); filter(f_mail); filter(f_crit); destination(messages); }; I'm looking for : log { source(src); filter(f_mail); filter(f_info); destination(messages); }; or anything similar but can't find it. It's weird that this is the vanilla syslog-ng.conf-sample with the FreeBSD 6 package. thanks, Len ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

1 0

tool to convert regular expression to text
by vadi 25 Aug '08

25 Aug '08

Dear All, Do we have any tool to convert regular expression to text?. Please let me know about it. Thanking u, Regards, Vadiraj

2 1

lost messages with follow_freq()?
by Joe Shaw 21 Aug '08

21 Aug '08

Hi, I'm trying to use the follow_freq() option to tail a growing log file, but not all of the messages are making it from the source end to the destination end. However, according to the statistics, no messages are dropped. I am using syslog-ng 2.0.9. The file on the source side looks like this: -rw-r--r-- 1 jshaw ita 251M Aug 19 15:07 dump-file and is growing rather rapidly. (It is basically being created by replaying another stored log file.) At this same point on the destination side, this is the corresponding file: -rw-r--r-- 1 root root 30M Aug 19 15:07 syslog-messages And doing a diff on those files does show large missing chunks from this file. The source's statistics say that nothing has been dropped: Aug 19 15:07:45 source-host syslog-ng[18574]: Log statistics; dropped='tcp(AF_INET(10.1.73.18:2000))=0', processed='center(queued)=303967', processed='center(received)=304017', processed='destination(d_file)=6', processed='destination(d_remote)=303961', processed='source(s_sys)=56', processed='source(s_internal)=6', processed='source(s_file)=303955' And on the destination side: Aug 19 15:08:04 dest-host syslog-ng[21023]: Log statistics; processed='center(queued)=318524', processed='center(received)=318521', processed='destination(d_file)=318521', processed='destination(d_stats)=3', processed='source(s_tcp)=318518', processed='source(s_internal)=3' The source syslog-ng.conf file looks like this: options { sync(0); time_reopen(10); log_fifo_size(1000); long_hostnames(off); use_dns(yes); dns_cache(yes); use_fqdn(no); keep_hostname(yes); use_time_recvd(no); log_msg_size(65536); stats_freq(180); }; source s_internal { internal(); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); }; source s_file { file("/ita/dump-file" follow_freq(1) flags(no-parse)); }; # Local destination for statistics destination d_file { file("/ita/syslog-messages" perm(0644)); }; # Remote destination destination d_remote { tcp("dest-host" port(2000)); }; # Send stats locally log { source(s_internal); destination(d_file); }; # Send everything remotely log { source(s_internal); source(s_file); destination(d_remote); }; (Yes, the s_sys source is unused.) On the destination side: options { sync(0); time_reopen(10); log_fifo_size(1000); long_hostnames(off); use_dns(yes); dns_cache(yes); use_fqdn(no); keep_hostname(yes); use_time_recvd(no); log_msg_size(65536); stats_freq(180); }; # Remote source source s_tcp { tcp(port(2000) log-fetch-limit(128) max-connections(1000)); }; # syslog-ng statistics source s_internal { internal(); }; destination d_file { file("/ita/syslog-messages" perm(0644) log_fifo_size(100)); }; destination d_stats { file("/ita/syslog-stats" perm(0644)); }; # Save stats separately log { source(s_internal); destination(d_stats); }; # Take all remote data and save it locally log { source(s_tcp); source(s_internal); destination(d_file); }; Any ideas what might be going on, or how to analyze this further? Thanks, Joe

2 6