- syslog-ng - lists.balabit.hu

rewriteable fields
by chris packham 03 Dec '09

03 Dec '09

Hi List, Just looking for a bit of clarification on exactly what fields I can rewrite in a message. Basically I've got an annoying debug message coming out of a program and while I could filter it off to a separate file, what I really want to do is drop its level to debug (from warning) and let my existing filters take care of it. The docs say "Setting a field can operate on any value available via macros, e.g., HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers" so modifying LEVEL is not listed but its not explicitly excluded either. I tried "rewrite r_norm {set("1", value("LEVEL")); };" but that didn't work. However "rewrite r_norm {set("FOO", value("MESSAGE")); };" and "rewrite r_norm {set("BAR", value("PROGRAM")); };" work as expected. Are there limitations as to which parts of a message I can rewrite? Are there some fuller examples hanging around? Thanks, Chris

3 2

syslog-ng OSE 3.1beta1 released
by Balazs Scheidler 03 Dec '09

03 Dec '09

Dear syslog-ng users, I'm proud to announce that syslog-ng OSE 3.1 has been released and uploaded to our webserver. This version is new in two ways: 1) of course it has new features, see below for the most interesting bits 2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link: https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement. For those who hurry, here's a link for the source code: https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1be… And here are the binaries for Linux/FreeBSD systems: https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… Select the Downloads tab, and in the Version selector select 3.1beta1. What is new in syslog-ng OSE 3.1 -------------------------------- * Support for patterndb v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT. Patterndb (more exactly the db-parser()) is a high performance message classifier and information extraction tool, that makes it easy to get away from the unstructured nature of syslog. Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at the 3rd iteration, hopefully slowly reaching its final form. Patterndb in general and the v1 format database is described in the syslog-ng manual at: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html The XML schemas that describe the different patterndb versions are available in the syslog-ng source tree: http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD The changes in the patterndb format as they evolved were described in Marton Illes's blog at http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html But see the other related posts as well. Old patterndb databases can be converted to the new format by putting them in a directory and using the pdbtool utility using the command: $ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml -D /opt/syslog-ng/etc/patterns.d Assuming the installation prefix of syslog-ng is /opt/syslog-ng Some v2 format patterns are distributed by BalaBit itself for its SSB product, download location: https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/ You can convert these db files using pdbtool as described above. Work is ongoing to publish a more comprehensive patterndb, but more on that in a separate post. * Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message. See the manpage and Marci's post: http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html * Support for message tags: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering. http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html * Added support for rewriting structured data. Earlier structured data fields in the new RFC5424 style syslog protocol were only read-only values that could be referenced in a template, but they couldn't be changed, and neither was it possible to add new fields in an already existing syslog message. Now all these became possible by using the same syntax that didn't work earlier, e.g. rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); }; * Macros and name-value pairs got a little tighter integration, in filters where syslog-ng 3.0 was limited to only use name-value pairs, with 3.1 you can also use macros. The following now works: match("<regexp>" value("R_DATE")); syslog-ng is now warning you in case you are using '$' prefix in the value syntax. * Enhanced dynamic name-value performance by a factor of three. The summary says it all, the performance dynamic name-value pairs that the various parsers produce got faster, thus the performance penalty of structuring the incoming messages got smaller. * Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These are patterndb parsers to make it easier to describe log messages. * Added non-standard and non-portable facility codes (range 10-15), decouple syslog-ng facility name information from the system used to compile syslog-ng on. Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems. Any feedback, success/failure reports is more than appreciated. -- Bazsi

2 1

hi all.Why $HOST variable valued "250" , "250-localhost^M" ?
by [焱龙企鹅]韩友洪 01 Dec '09

01 Dec '09

Hi.all I configured syslog-ng (3)(CentOS 5) to collect logs sent by 1xx syslog servers.(freebsd6).I got two strange dirs: "250" and "250-localhost^M" my syslog-ng.conf : ============================================== source s_udp { udp(ip(172.16.18.10) port(514) ); }; destination d_udp_data{ file ("/data3/syslogng/logcollect/$YEAR-$MONTH-$DAY/$HOST/$FACILITY/$PROGRAM.log" create_dirs(yes) template("\n$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $SOURCEIP $PROGRAM $MSG") ); }; ================================================= my syslog.conf !webface.login *.info /var/log/mail/webface/webface-login.log *.* @172.16.18.10 ======================================= root@localhost 2009-12-01]# pwd /data3/syslogng/logcollect/2009-12-01 [root@localhost 2009-12-01]# ls ............... 10.55.2.35 10.55.2.46 10.55.2.57 10.55.2.68 10.55.2.79 10.55.2.90 250 250-localhost^M [root@localhost 2009-12-01]# ls -R 250 250: user 250/user: 2.1.0.log [root@localhost 2009-12-01]# more 250/user/2.1.0.log 2009-12-01 13:46:12 10.55.2.11 2.1.0 Ok^M 2009-12-01 13:46:27 10.55.2.11 2.1.0 Ok^M ........ [root@localhost 2009-12-01]# ls -R 250-localhost^M/ 250-localhost^M/: user 250-localhost^M/user: 250-SIZE^M.log [root@localhost 2009-12-01]# more 250-localhost^M/user/250-SIZE^M.log 2009-12-01 14:07:44 10.55.2.128 250-SIZE^M 250-8BITMIME^M 250 ENHANCEDSTATUSCODES^M -- 祝：身体安康，万事如意！ ________________________________________________________________ Mr. Hunter - 韩友洪焱龙企鹅 youhong(a)staff.sina.com.cn 新浪 - 产品事业部 -邮箱 MSN：hf_linux(a)msn.com 电话：5392 手机：15001328768 地址：北京市海淀区北四环西路58号理想国际大厦18层 ________________________________________________________________ http://www.sina.com.cn You're the One 新浪.北京一切由你开始

2 1

AUTO: Phil Newlon/Corporate/Wendys is out of the office. (returning 12/02/2009)
by Phil.Newlon＠wendysarbys.com 01 Dec '09

01 Dec '09

I am out of the office until 12/02/2009. I have received the email message that you sent and will respond to it when I return. If you feel your message needs immediate attention, you may call me on my cell at 614-569-1328. Note: This is an automated response to your message "[syslog-ng] hi all.Why $HOST variable valued "250" , "250-localhost^M" ?" sent on 12/1/2009 2:50:50 AM. This is the only notification you will receive while this person is away. Notice: This e-mail message and its attachments are the property of Wendy's/Arby's Group Inc. or one of its subsidiaries and may contain confidential or legally privileged information intended solely for the use of the addressee(s). If you are not an intended recipient, then any use, copying or distribution of this message or its attachments is strictly prohibited. If you received this message in error, please notify the sender and delete this message entirely from your system.

1 0

Question about flags(ignore-case) option
by PAUL WILLIAMSON 30 Nov '09

30 Nov '09

I have the following filter set up: filter f_wireless_devices { host("system") flags(ignore-case); }; and I'm getting a syntax error. If I remove the flags(ignore-case), filter f_wireless_devices { host("system"); }; It is fine. On page 58 of the 3.0 sys admin guide, I have the syntax correct. Is there any other way to do this? Maybe put an ignore_case(yes) in the options sections at the top of the configuration file? Thanks, Paul ************************************ This email may contain privileged and/or confidential information that is intended solely for the use of the addressee. If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act. You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information. There are risks associated with the use of electronic transmission. The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission. ************************************

7 21

Troubles with the pattern database
by Guillaume Rousse 30 Nov '09

30 Nov '09

Hello list. I'm the mandriva maintainer for syslog-ng. I'm trying to play with the pattern database, with syslog-ng 3.0.4. I rebuild the package with the attached patch, so as to use /usr/share/syslog-ng a database location (a bit more FHS-compliant than /var), and extracted the files downloaded from http://www.balabit.com/downloads/files/patterndb there. However, loading them fails with this message: Error parsing pattern database file; filename='/usr/share/syslog-ng/patterndb.xml', error='Unexpected <rule> element' Error reloading pattern database, no pattern recognition will be done; It looks like some DB format issue. According to http://www.balabit.com/dl/html/syslog-ng-v3.0-guide-admin-en.html/ch08s06.h… the supported format is v1 until syslog-ng 3.0.2, and the NEWS file doesn't list any change here, while the patterndb file is already using v2. Am I correct ? And in this case, is there any way to easily convert the base to the old format ? -- BOFH excuse #252: Our ISP is having {switching,routing,SMDS,frame relay} problems diff -Naur -x '*~' syslog-ng-3.0.4/configure.in syslog-ng-3.0.4-fix-pattern-database-location/configure.in --- syslog-ng-3.0.4/configure.in 2009-04-30 18:32:19.000000000 +0200 +++ syslog-ng-3.0.4-fix-pattern-database-location/configure.in 2009-10-12 15:20:11.000000000 +0200 @@ -722,6 +722,7 @@ AC_DEFINE_UNQUOTED(PATH_SYSCONFDIR, "`patheval $sysconfdir`", [sysconfdir]) AC_DEFINE_UNQUOTED(PATH_LOCALSTATEDIR, "`patheval $localstatedir`", [local state directory]) AC_DEFINE_UNQUOTED(PATH_PIDFILEDIR, "`patheval $pidfiledir`", [local state directory]) +AC_DEFINE_UNQUOTED(PATH_DATADIR, "`patheval $datadir`", [datadir]) if test -n "$timezonedir"; then AC_DEFINE_UNQUOTED(PATH_TIMEZONEDIR, "`patheval $timezonedir`", [timezone base directory]) fi diff -Naur -x '*~' syslog-ng-3.0.4/src/syslog-ng.h syslog-ng-3.0.4-fix-pattern-database-location/src/syslog-ng.h --- syslog-ng-3.0.4/src/syslog-ng.h 2009-04-22 13:06:57.000000000 +0200 +++ syslog-ng-3.0.4-fix-pattern-database-location/src/syslog-ng.h 2009-10-12 15:14:10.000000000 +0200 @@ -52,7 +52,7 @@ #define PATH_PERSIST_CONFIG PATH_LOCALSTATEDIR "/syslog-ng.persist" #define PATH_QDISK PATH_LOCALSTATEDIR -#define PATH_PATTERNDB_FILE PATH_LOCALSTATEDIR "/patterndb.xml" +#define PATH_PATTERNDB_FILE PATH_DATADIR "/syslog-ng/patterndb.xml" #define LOG_PRIORITY_LISTEN 0 #define LOG_PRIORITY_READER 0

3 3

Beginner's questions for deploying syslog-ng
by Hailu Meng 29 Nov '09

29 Nov '09

Hi All, I'm planning create a centralized logging server for our network. We have 100 cisco routers and switches. Not too big organization. Suppose every cisco must have log to be sent to centralized logging server. I'm considering to deploy syslog-ng in CentOS 5. As I understand, I also need a web front end for user to search the log easily. Someone recommend splunk to do this front end job. I checked it out and found the free version of splunk has a 500MB/day for the indexing size. I'm not sure how this limit will affect on our network. Or is there any other good web front end for use with syslog-ng? Any help is appreciated. Lou

1 0

filter logger tags from syslog
by Jose Sanchez 28 Nov '09

28 Nov '09

Hello, I've been using classic syslog for centralizing apache access logs from one server to a remote syslog server, the thing is syslog adds some nasty tags before the lines in the access logs and I cant get them off, ie: "Nov 25 21:25:37 server1 logger:" I would like to know if syslog-ng has the option to filter this kind of stuff, I just want to have the logs sent to the syslog server exactly like I was saving them in a local access.log file. Thanks in advance.

3 5

problems with syslog-ng-3.0.4 and -C option
by Chris Fabri 26 Nov '09

26 Nov '09

Greetings- I believe I've discovered an issue in syslog-ng 3.0.4. I've compiled this version on RH 3 (which might be the problem), and when you try for example: syslog-ng -C /var/log -u syslog-ng you get: Error opening configuration file; filename='/usr/local/etc/syslog-ng.conf', error='No such file or directory (2)' even specifying "-f /usr/local/etc/syslog-ng.conf" causes the same problem. Placing syslog-ng.conf in the directory specified by -C works, so perhaps the -C option has some issue with using the actual path instead of the chroot'ed path. I didn't see anything in the last several months of emails from the list, and I'm surprised this wouldn't have bitten more people, so it could be an RH3-ism, but wanted to make sure I reported it, in case it was more of a problem. chris

2 1

Some issues with syntax checking
by Guillaume Rousse 21 Nov '09

21 Nov '09

Hello list. Unresolved object errors seems not to be catched by 'syslog-ng --syntax-only', whereas they are fatal errors. For instance, with this configuration file: @version:3.0 log { source(s_foo); destination(d_foo); }; [root@oberkampf guillaume]# syslog-ng --syntax-only [root@oberkampf guillaume]# syslog-ng Error in configuration, unresolved source reference; source='s_foo' Moreover, the error message is sometimes a bit misleading. with this configuration file: @version:3.0 source s_foo { unix-stream ("/dev/log" max-connections(100)); }; log { source(s_foo); destination(d_foo); }; [root@oberkampf guillaume]# syslog-ng --syntax-only [root@oberkampf guillaume]# syslog-ng Error in configuration, unresolved destination reference; filter='d_foo' What is missing is a destination, not a filter. -- BOFH excuse #327: The POP server is out of Coke

2 1

log action or discard ?
by Jan Schaumann 18 Nov '09

18 Nov '09

Hello, I have a syslog server that receives syslog messages and will forward a selection of them to a central host via TLS. It will log all other messages to a local file. The current configuration looks something like this: destination forward { syslog(<options here>); }; destination all { file("/var/log/all"); }; filter forward { match("mumble"); }; log { all-my-sources filter(forward); destination(forward); flags(final); }; log { all-my-sources destination(all); }; Now if my remote server to which I'm forwarding messages to goes down or stops accepting connections for any reason, _all_ my logging will come to a grinding halt. That is, nothing gets appended to /var/log/all. What I want is for my forwarding to be a "try this or discard" kind of scenario. That is, if I can't forward the messages, just drop them, but continue to log all other messages unphased. Is that possible? If so, how? Thanks in advance, -Jan

1 0

Re: [syslog-ng] syslog-ng on solaris locks up after a while
by Balazs Scheidler 18 Nov '09

18 Nov '09

Thanks. It indeeed seems to be the same as Author: Balazs Scheidler <bazsi(a)balabit.hu> 2009-08-30 11:41:24 Committer: Balazs Scheidler <bazsi(a)balabit.hu> 2009-08-30 11:41:24 Parent: 1ad4da07d5305ba0140ac385d661ab6de25fc5f3 ([patterndb] estring parser length calculation must include ending quote) Child: c2e8aa58763a89cab58d05fb7a2b2a18021413b4 ([logmsg] added support for ASA timestamps) Branches: master, remotes/balabit/master, remotes/origin/master Follows: v3.0.4 Precedes: [afinter] don't block on the internal_msg_queue even in the threaded case (fixes: pub#48) A hang was reported in bugzilla ticket #48 which seems to have been caused by MARK messages interfering with local messages: * if the MARK is due in the same poll iteration as a local message * the MARK timeout is checked and the internal source is marked as having input available * then the local message comes in pushing the mark timeout further ahead in time * then the internal() dispatch callback checks the mark timeout again, but at this time it is already in the future -> * the dispatch callback falls back to fetching the internal message from internal_msg_queue, assuming it was that which caused the dispatch callback to be scheduled * this blocks indefinitely. The solution is very simple: use g_async_queue_try_pop() instead of g_async_queue_pop(), the dispatch code already takes care about a NULL message value. Thanks for the helpful reporters to hunt down the issue. Reported-By: Arkadiusz Miśkiewicz, Elan Ruusamäe A recent snapshot (and the git tree) should contain the fix. On Thu, 2009-11-12 at 11:10 -0500, Manassypov, Igor wrote: > Here is the information: > > (gdb) info thread > 2 process 153634 0xfed4b000 in _door_return () from /lib/libc.so.1 > * 1 process 88098 0xfed46df0 in __lwp_park () from /lib/libc.so.1 > (gdb) thread 1 > [Switching to thread 1 (process 88098 )]#0 0xfed46df0 in __lwp_park () from /lib/libc.so.1 > (gdb) bt > #0 0xfed46df0 in __lwp_park () from /lib/libc.so.1 > #1 0xfed40c44 in cond_sleep_queue () from /lib/libc.so.1 > #2 0xfed40e08 in cond_wait_queue () from /lib/libc.so.1 > #3 0xfed41350 in cond_wait () from /lib/libc.so.1 > #4 0xfed4138c in pthread_cond_wait () from /lib/libc.so.1 > #5 0xff119d80 in g_async_queue_pop_intern_unlocked (queue=0x757e0, try=0, end_time=0x75618) at gasyncqueue.c:359 > #6 0xff119e80 in g_async_queue_pop (queue=0x757e0) at gasyncqueue.c:398 > #7 0x0003e984 in afinter_source_dispatch (source=0x8d260, callback=0x3e9dc <afinter_source_dispatch_msg>, user_data=0x8d1e0) > at afinter.c:112 > #8 0xff143564 in g_main_context_dispatch (context=0x8d158) at gmain.c:2144 > #9 0xff1459a4 in g_main_context_iterate (context=0x8d158, block=1, dispatch=1, self=0x76030) at gmain.c:2778 > #10 0xff146050 in g_main_context_iteration (context=0x8d158, may_block=1) at gmain.c:2841 > #11 0x0001bc20 in main_loop_run (cfg=0xffbffbc8) at main.c:149 > #12 0x0001c260 in main (argc=1, argv=0xffbffd44) at main.c:394 > (gdb) thread 2 > [Switching to thread 2 (process 153634 )]#0 0xfed4b000 in _door_return () from /lib/libc.so.1 > (gdb) bt > #0 0xfed4b000 in _door_return () from /lib/libc.so.1 > #1 0xff370bdc in door_return () from /lib/libdoor.so.1 > #2 0xff370c38 in door_create_func () from /lib/libdoor.so.1 > #3 0xfed46d54 in _lwp_start () from /lib/libc.so.1 > #4 0xfed46d54 in _lwp_start () from /lib/libc.so.1 > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > (gdb) > > > > > ----------------------------------------------------------------------------------- > Igor R. Manassypov, M.Eng., P.Eng., CCIE 23032, CCVP > Network Architect, CI Investments > 416.795.3147 > > -----Original Message----- > From: Balazs Scheidler [mailto:bazsi@balabit.hu] > Sent: Thursday, November 12, 2009 11:02 AM > To: imanassypov(a)rogers.com > Cc: Syslog-ng users' and developers' mailing list; Pallagi Zoltán; Network > Subject: Re: [syslog-ng] syslog-ng on solaris locks up after a while > > Hi, > > not really, are there multiple threads in the same core file? > > e.g. what is the response for "info threads"? > > It would be nice to have the backtrace for all threads, like this: > > (gdb) thread 1 > (gdb) bt > (gdb) thread 2 > (gdb) bt > > and so on, for each threadid that "info thread" lists. > > > On Fri, 2009-11-06 at 11:41 -0800, Igor Manassypov wrote: > > Would this one make more sense? > > > > > > > > bash-3.00# ps -eaf | grep syslog > > root 22562 22561 0 Nov 04 ? > > 0:30 /usr/local/sbin/syslog-ng > > root 22561 1 0 Nov 04 ? > > 0:00 /usr/local/sbin/syslog-ng > > > > bash-3.00# truss -f -p 22562 > > 22562/2: door_return(0x00000000, 0, 0x00000000, 0) > > (sleeping...) > > 22562/1: lwp_park(0x00000000, 0) (sleeping....) > > 22562/1: Received signal #11, SIGSEGV, in lwp_park() > > [default] > > 22562/1: siginfo: SIGSEGV pid=12717 uid=0 > > 22562/1: lwp_park(0x00000000, 0) Err#4 > > EINTR > > > > Core was generated by `/usr/local/sbin/syslog-ng'. > > Program terminated with signal 11, Segmentation fault. > > [New process 88098 ] > > [New process 153634 ] > > #0 0xfed46df0 in __lwp_park () from /lib/libc.so.1 #0 0xfed46df0 in > > __lwp_park () from /lib/libc.so.1 > > > > bash-3.00# gdb syslog-ng core > > > > Core was generated by `/usr/local/sbin/syslog-ng'. > > Program terminated with signal 11, Segmentation fault. > > [New process 88098 ] > > [New process 153634 ] > > #0 0xfed46df0 in __lwp_park () from /lib/libc.so.1 > > (gdb) > > > > --- On Tue, 11/3/09, Balazs Scheidler <bazsi(a)balabit.hu> wrote: > > > > From: Balazs Scheidler <bazsi@balabit..hu> > > Subject: Re: [syslog-ng] syslog-ng on solaris locks up after a > > while > > To: imanassypov(a)rogers.com, "Syslog-ng users' and developers' > > mailing list" <syslog-ng(a)lists.balabit.hu> > > Cc: "Pallagi Zoltán" <pzolee(a)balabit.hu>, network(a)ci.com > > Date: Tuesday, November 3, 2009, 2:11 PM > > > > Hi, > > > > The problem is that you killed the supervisor process, which > > restarts > > syslog-ng in case it crashes.. However the hang is not in this > > part, but > > in its child. > > > > So by looking at the ps output, I'd say that in this situation > > you > > should have trussed 13621 and not its parent. > > > > On Tue, 2009-11-03 at 08:54 -0800, Igor Manassypov wrote: > > > Hi Zoltan, > > > > > > > > > Here are the traces: > > > > > > bash-3.00# ps -eaf | grep syslog > > > root 12694 12616 0 11:37:07 pts/1 0:00 grep > > syslog > > > root 13012 1 0 Oct 21 ? 0:00 syslog-ng > > -v > > > root 13013 13012 0 Oct 21 ? 0:41 syslog-ng > > -v > > > root 13620 1 0 Oct 08 ? > > > 0:00 /usr/local/sbin/syslog-ng > > > root 13621 13620 0 Oct 08 ? > > > 6:16 /usr/local/sbin/syslog-ng > > > bash-3.00# truss -f -p "13620" > > > 13620: waitid(P_PID, 13621, 0xFFBFF468, WEXITED|WTRAPPED) > > > (sleeping...) > > > > > > 13620: Received signal #11, SIGSEGV, in waitid() > > [default] > > > 13620: siginfo: SIGSEGV pid=12717 uid=0 > > > 13620: waitid(P_PID, 13621, 0xFFBFF468, WEXITED|WTRAPPED) > > Err#4 EINTR > > > > > > Core was generated by `/usr/local/sbin/syslog-ng'. > > > Program terminated with signal 11, Segmentation fault. > > > [New process 79156 ] > > > #0 0xfed4ad80 in _waitid () from /lib/libc.so.1 > > > (gdb) bt full > > > #0 0xfed4ad80 in _waitid () from /lib/libc.so.1 > > > No symbol table info available. > > > #1 0xfecee038 in _waitpid () from /lib/libc.so.1 > > > No symbol table info available. > > > #2 0xfed3a70c in waitpid () from /lib/libc.so.1 > > > No symbol table info available. > > > #3 0x0003017c in g_process_start () at gprocess.c:1042 > > > rc = 0 > > > deadlock = 0 > > > pid = 13621 > > > __PRETTY_FUNCTION__ = "g_process_start" > > > #4 0x0001c214 in main (argc=1, argv=0xffbffd14) at > > main.c:371 > > > cfg = (GlobalConfig *) 0x10034 > > > rc = 310272 > > > ctx = (GOptionContext *) 0x76030 > > > error = (GError *) 0x0 > > > > > > Please let me know if I can provide you with more > > information, > > > > > > Thanks! > > > > > > --- On Tue, 11/3/09, Pallagi Zoltán <pzolee(a)balabit.hu> > > wrote: > > > > > > From: Pallagi Zoltán <pzolee(a)balabit.hu> > > > Subject: Re: [syslog-ng] syslog-ng on solaris locks > > up after a > > > while > > > To: imanassypov(a)rogers.com, "Syslog-ng users' and > > developers' > > > mailing list" <syslog-ng(a)lists.balabit.hu> > > > Received: Tuesday, November 3, 2009, 11:10 AM > > > > > > Hi Igor, > > > > > > Can you show me truss output or backtrace of the > > stuck > > > syslog-ng?: > > > truss: > > > > > > truss -f -p "syslog-ng pid" > > > > > > backtrace: > > > > > > kill -11 "syslog-ng pid" (syslog-ng will drop a core > > file) > > > gdb syslog-ng core > > > bt full > > > > > > Igor Manassypov írta: > > > > Hello, > > > > > > > > > > > > I am having an issue with a solaris installation > > of the > > > > syslog-ng. It is configured such that all the logs > > are > > > > stored different per-ip folders. This is my > > centralized > > > > logging device, so it is fairly heavily loaded > > with > > > > receiving logs from a few dozen hosts. The > > syslog-ng process > > > > locks up every two to three weeks, with no > > messages logging > > > > to any of the files. The only way of getting it > > back is kill > > > > -9 the process and restart it. > > > > > > > > Is there any known issue of same sorts and is > > there any > > > > other way around it other than recycling the > > daemon every > > > > night? > > > > > > > > > > > > here is the version info: > > > > > > > > bash-3.00# syslog-ng --version > > > > syslog-ng 3.0.4 > > > > Revision: ssh > > > > > > +git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 > > > > Compile-Date: Aug 11 2009 10:44:17 > > > > Enable-Threads: on > > > > Enable-Debug: off > > > > Enable-GProf: off > > > > Enable-Memtrace: off > > > > Enable-Sun-STREAMS: on > > > > Enable-Sun-Door: on > > > > Enable-IPv6: off > > > > Enable-Spoof-Source: on > > > > Enable-TCP-Wrapper: off > > > > Enable-SSL: on > > > > Enable-SQL: on > > > > Enable-Linux-Caps: off > > > > Enable-Pcre: on > > > > > > > > bash-3.00# uname -a > > > > SunOS prelude 5.10 Generic_137137-09 sun4v sparc > > SUNW,T5240 > > > > Thanks! > > > > > > > > -igor > > > > > > > > Igor Manassypov., M.Eng, P.Eng, CCIE 23032, CCVP > > Network > > > > Architect > > > > > > > > > > ____________________________________________________________ > > > > > > > > > > ______________________________________________________________________________ > > > > Member info: > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > Member info: > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > -- > > Bazsi > > > > > > > -- > Bazsi > > ################################################################################### > This communication is confidential and may be > privileged. If you received it in error, please destroy > without copying and advise the sender. > > By submitting personal information to CI Investments, > you agree to the collection, use and disclosure of > such personal information for the purposes described > in our Privacy Policy available at www.ci.com. > > > Cette communication est confidentielle et pourrait être > privilégiée. Si vous la recevez par erreur, veuillez > l'éliminer sans en faire une copie et aviser l'expéditeur. > > Lorsque vous soumettez des renseignements > personnels à Placements CI, vous nous permettez de > conserver, utiliser et divulguer ces renseignements > personnels aux fins décrites dans nos Principes > directeurs en matière de protection des renseignements personnels > qui sont disponibles au www.ci.com. > > ################################################################ > -- Bazsi

1 0

udp6 source declaration causes syslog+tls to fail
by Jan Schaumann 17 Nov '09

17 Nov '09

Hello, I'm seeing what appears to be an odd interaction between the declaration of an udp6 source and trying to forward messages via tls to another host. I have the following configuration: # UDP will use the regular port 514 as per /etc/services. source s_udp { udp(log_fetch_limit(400000000)); # UDP6 definition requires an explicit port statement. # See upstream bug # https://bugzilla.balabit.com/show_bug.cgi?id=58 # udp6( # port(514) # log_fetch_limit(400000000) # ); }; destination d_amslog { syslog("<hostname>" port(6514) suppress(30) transport("tls") tls( ca_dir("<dir>") cert_file("<cert>") key_file("<key>") trusted_keys("<fp>") ) ); }; log { source(s_udp); destination(d_amslog); flags(final); }; Now this works just fine with the "udp6" part commented out as above. However, if I uncomment the udp6 part, I get the following error: ERROR:gsockaddr.h:86:g_sockaddr_inet_get_sa: assertion failed: (g_sockaddr_inet_check(s)) failed! Note that if I remove the destination from the log directive (ie comment out the line "destination(d_amslog);", I still get the same error. If I comment out the entire syslog directive and replace it with a file destination, there is no problem (even though the udp6 source definition is there). It seems as if -- and I'm guessing here -- the udp6 declaration causes syslog-ng to try to treat the subsequent "syslog" directive to be an IPv6 type connection, which fails since the host does not (currently) have IPv6 configured. Ie, I believe syslog-ng assumes that just by the existence of an IPv6 source all connections will be IPv6 (which, if correct, seems like a bug to me). The version of syslog-ng here is: syslog-ng 3.0.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#08c9bf9d98e4e021756adc12c42605840140ba8b Compile-Date: Sep 12 2009 02:43:20 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Any suggestion on how to fix this would be appreciated. Thanks in advance, -Jan

1 1

Dear syslog-ng@lists.balabit.hu SALE 80% OFF on Pfizer
by Pfizer Inc 17 Nov '09

17 Nov '09

1 0

Logging only certain logs to a remote syslog server
by Paras Fadte 16 Nov '09

16 Nov '09

Hi, Is it possible to log only a particular logs to a remote syslog server ? For example logging only mysql/postgres logs to a remote host . syslog-ng version used is syslog-ng 1.6.8 Thank you. -Paras

4 6

Re: [syslog-ng] syslog-ng Digest, Vol 55, Issue 18
by Adam Chou 13 Nov '09

13 Nov '09

> > > On Friday 13 of November 2009, Adam Chou wrote: > > > > Just in case you're wondering why I'm trying to build from source, the > > current packages are built for amd64 and I'm running x86_64 on an > > Intel. > > https://lists.balabit.hu/pipermail/syslog-ng/2009-November/013551.html > > HTH > > thanks, that was informative. That link seems to be only the client version, however. Is there a server binary that is also compiled to work with yum's x86_64?

2 1

Can't build syslog-ng RPM
by Adam Chou 13 Nov '09

13 Nov '09

I'm trying to build the syslog-ng RPM and I'm having an issue with the libnet-devel package. I've tried searching for it and I can't find a devel package from Fedora, CentOS, or RHEL. I found the libnet-devel page but the project appears to have forked now so I'm no longer sure if its still good to use. Just in case you're wondering why I'm trying to build from source, the current packages are built for amd64 and I'm running x86_64 on an Intel. Any help with how I can get libnet-devel installed would be much appreciated. Thanks.

3 2

stats_freq ignored in config version 3.0?
by Peter Volkov 12 Nov '09

12 Nov '09

Hi, I've just upgraded syslog-ng up to 3.0.4 and configuration files up to version 3.0. Now I have the following config: ======================================================== @version:3.0 options { stats_freq(4320); }; source src { unix-stream("/dev/log" max-connections(256)); internal(); }; destination messages { file("/var/log/messages"); }; log { source(src); destination(messages); }; ======================================================== The problem I experience is that -- MARK -- is printed every 20 minutes no matter what I put into /etc/syslog-ng/syslog-ng.conf stats_freq option argument: Nov 12 17:56:32 stage4 syslog-ng[25815]: syslog-ng starting up; version='3.0.4' Nov 12 18:16:32 stage4 -- MARK -- Nov 12 18:36:32 stage4 -- MARK -- Does anybody else experience this? What could cause this? I've managed to reproduce this problem on two linux systems with kernel 2.6.27 and 2.6.31. How to debug this kind of problem? Thanks in advance for any hint. -- Peter.

2 2

Re: [syslog-ng] flags(final)
by Michael J. Bauer 12 Nov '09

12 Nov '09

flags(final) is now working with my rewritten match() statements. I wonder if it was a combination of the older version I was initially running and the odd structure of the log entry I was trying to use, but I really am disinclined to try to chase it down. However, in rewriting things, I did discover something else. In cutting and pasting, I had accidentally done this in my config: destination d_dhcpd { file("/var/log/dhcpd" flush_lines(10) flush_timeout(1000)); }; destination d_dhcpd { file("/var/log/named" flush_lines(10) flush_timeout(1000)); }; filter f_dhcpd { match("dhcpd" value(PROGRAM)); }; filter f_named { match("named" value(PROGRAM)); }; log { source(s_sys); filter(f_dhcpd); destination(d_dhcpd); flags(final); }; log { source(s_sys); filter(f_named); destination(d_named); flags(final); }; log { source(s_sys); destination(d_default); flags(fallback); }; Note the log statement containing the non-existent destination(d_named). When I ran "/etc/init.d/syslog-ng reload", it didn't crash, nor did it throw any error where I could see it. What it did do was start logging multiple copies of each line going to destination(d_default), apparently one more line each time I did a reload. I didn't notice what was going on until it was at 8 copies of each line, and it took me until 17 copies of each line to finally give up and run "/etc/init.d/syslog-ng stop; /etc/init.d/syslog-ng start". Once I did that, I got an error and corrected the problem. This is probably an obscure bug, but it may be worth looking at. It was certainly entertaining after the fact. MJB Michael J. Bauer wrote: > I think I am misunderstanding what flags(final) is supposed to do. I'm > running syslog-ng 2.1.4 on RHEL 5.4 (Tikanga). > > I have a fairly simple syslog-ng configuration, which I've attached > below. I'm trying to pick off individual groups of log entries and put > them in their own individual files. I want to ensure that each gets > logged exactly once, so I'm using flags(final). I also have a catch-all > at the end in case I've missed something, but the ultimate goal is to > have that file present, but empty. > > However, with this configuration, the log entries that appear in > d_network_address_translation (/var/log/network-address-translation) > also appear in d_default (/var/log/default) despite the presence of > flags(final) on an earlier log() line. Should it work this way? If so, > what can I do to get the desired behavior? > > Thanks, > MJB > > options { > sync (0); > time_reopen (10); > log_fifo_size (1000); > long_hostnames (off); > use_fqdn (no); > create_dirs (no); > keep_hostname (yes); > }; > > source s_sys { > file ("/proc/kmsg" log_prefix("kernel: ")); > unix-stream ("/dev/log"); > internal(); > udp(ip(0.0.0.0) port(514)); > }; > > destination d_network_address_translation { > file("/var/log/network-address-translation"); }; > destination d_default { file("/var/log/default"); }; > > filter f_network_address_translation { host("router-service-interface") and > priority(info) and > facility(local2) and > match("FWNAT"); }; > > log { source(s_sys); > filter(f_network_address_translation); > destination(d_network_address_translation); > flags(final); }; > log { source(s_sys); > destination(d_default); }; > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > >

2 2

stupid question: syslogging plaintext non-syslog-format messages unchanged (netconsole)
by Nix 12 Nov '09

12 Nov '09

So I'm trying to get the Linux kernel netconsole working, to diagnose some nasty persistent panics... and after some struggle I've got most of it working, but I can't make syslog-ng (3.0 from git as of tonight) cooperate. The problem seems to be that netconsole is handing syslog-ng completely bare text lines over UDP, and syslog-ng is persistently trying to linebreak them at apparently random space characters, even with the no-parse flag turned on. Some specifics. I'm receiving the netconsole messages with the following (trimmed down from the real 500-line-long config, all global options preserved in this trim, I'm pretty sure this is everything relevant but I haven't tested this on its own yet 'cos it would break syslogging for a fairly large network to actually replace the syslog-ng config with this): ,---- | options | { | keep_hostname(no); | create_dirs(1); | perm(0644); | dir_perm(0755); | log_fifo_size(1000); | use_dns(yes); | dns_cache(yes); | use_fqdn(no); | log_iw_size(5192); | flush_lines(100); | flush_timeout(1000); | stats_freq(3600); | chain_hostnames(yes); | }; | | source netconsole | { | udp(ip(0.0.0.0) port(24183) log_fetch_limit(100) flags(no-parse no-multi-line)); | }; | | # A bare template, for netconsole messages. | template bare | { | template("$MSG\n"); | template_escape(no); | }; | | destination netconsole | { | file("/var/log/$HOST-netconsole.log" | template(bare)); | }; | | log | { | source(netconsole); | destination(netconsole); | }; `---- but this is not enough. Most of the time the same message is received over an ordinary syslog source, too, so I can compare the two. If the syslogged message reads: ,---- | Oct 20 21:17:53 mutilate warning: : [ 6783.237435] pktcdvd: Fixed packets, 32 blocks, Mode-2 disc | Oct 20 21:17:53 mutilate warning: : [ 6783.238541] pktcdvd: write caching control failed | Oct 20 21:17:53 mutilate warning: : [ 6783.238547] pktcdvd: 55 10 00 00 00 00 00 00 14 00 00 00 - sense 05.00.00 (Illegal request) `---- the netconsole message that's logged using the configuration above reads ,---- | [ 6783.237435] pktcdvd: Fixed packets, | 32 blocks, | Mode-2 disc | [ 6783.238541] pktcdvd: write caching control failed | [ 6783.238547] pktcdvd: | 55 | 10 | 00 | 00 | 00 | 00 | 00 | 00 | 14 | 00 | 00 | 00 | - | sense 05.00.00 | (Illegal request) `---- This is obviously not sane. Removing the no-multi-line flag doesn't change anything at all (I only added it in desperation, because I *want* linebreaks in the original message to be reproduced in the output: I just don't want *more* linebreaks than are present in the original output), so that's not it either. I'm logging kernel messages using ,---- | source local | { | unix-stream("/dev/log" flags(no-multi-line)); | internal(); | file("/proc/kmsg" program_override("kernel: ") flags(kernel no-multi-line)); | }; `---- and things hanging off that, without incident: no random linebreaks or anything. So, what am I missing? It's probably incredibly obvious, but I've blown several hours hacking at configs and digging through the syslog-ng source and have seen nothing to indicate what's going on...

2 3

syslog-ng on solaris locks up after a while
by Igor Manassypov 12 Nov '09

12 Nov '09

Hello, I am having an issue with a solaris installation of the syslog-ng. It is configured such that all the logs are stored different per-ip folders. This is my centralized logging device, so it is fairly heavily loaded with receiving logs from a few dozen hosts. The syslog-ng process locks up every two to three weeks, with no messages logging to any of the files. The only way of getting it back is kill -9 the process and restart it. Is there any known issue of same sorts and is there any other way around it other than recycling the daemon every night? here is the version info: bash-3.00# syslog-ng --version syslog-ng 3.0.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 Compile-Date: Aug 11 2009 10:44:17 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: off Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on bash-3.00# uname -a SunOS prelude 5.10 Generic_137137-09 sun4v sparc SUNW,T5240 Thanks! -igor Igor Manassypov., M.Eng, P.Eng, CCIE 23032, CCVP Network Architect

3 11

Prefiltering
by Kārlis Repsons 12 Nov '09

12 Nov '09

People, is there currently some good way to filter source and then use it *filtered* for numerous "log" invocations in configuration file? If no, it takes writing the same thing many times...

4 4

syslog-ng 3.0 Administrator Guide, 9th Edition
by Robert Fekete 12 Nov '09

12 Nov '09

Dear syslog-ng users, A new version of the syslog-ng 3.0 Administrator Guide is available on our website. You can read about the changes in detail at the BalaBit Documentation Blog at http://robert.blogs.balabit.com/2009/11/syslog-ng-administrator-guide-9th.h… Kind Regards, Robert Fekete

2 1

rm /var/log/{dmesg,lastlog,wtmp,faillog,..?}?!
by Kārlis Repsons 11 Nov '09

11 Nov '09

If one has set up a dedicated syslog-ng configuration and wants to remove files like those, what would happen? Can anyone list the software entities responsible for writing to those? Has anyone configured a system, which doesn't needs them?

3 3

Using pipes for prefiltering?
by Kārlis Repsons 10 Nov '09

10 Nov '09

Hello, Recently I've been trying to squeeze out the most of syslog-ng 3.0 and one of problems pestering me is this: there are some messages, which I would like never to appear in any log file, so I created filters for them, directed real sources through them to pipe destination and created prefiltered sources pointing to those same pipes. Well and what happens is - some logs reach the final destinations, but some not. For example, nothing at all now gets logged form kernel through pipe, even if filters get thrown out! Its on Linux... Also I had to disable syslogng logs, because otherwise CPU was loaded 100% sometimes (one core). After all, how should I do prefiltering? Strange thing is, that even with pipes first of these works, while the other doesn't: log { source(s_compname_usp_f); source(s_compname_kern_f); destination(d_compname_vt12); }; log { source(s_compname_kern_f); filter(f__kern); destination(d_compname_kern); }; filter f__kern { facility(0); }; You can see pipes fragment below. Hope someone has suggestion... Best, Kārlis # prefiltering # destination d_compname_usp_f { pipe("/var/log/filtered-source"); }; destination d_compname_kern_f { pipe("/var/log/filtered-kernel-source"); }; # abusers/spammers dropping here # filter f_compname_antispam_usp { ( not ( program(upsd) and message("some shit there") ) ) and ( not program("syslog") ) and ( not ( program("cron") and ( message(".*rm \-f /var/spool/cron/lastrun/cron\..*") or message(".*test \-x /usr/sbin/run-crons \&\& /usr/sbin/run\-crons.*") ) ) ) and ( not ( program("acpid") and ( message(".*client connected from.*") or message(".*1 client rule loaded") or message(".*client .* has disconnected") ) ) ) ; }; filter f_compname_antispam_kern { ( not message(".*ata1: exception .*") ) and ( not message(".*ata1: .*connection status changed") ) and ( not message(".*Hangcheck\: hangcheck value past margin\!") ) ; }; log { source(s_compname_usp); filter(f_compname_antispam_usp); destination(d_compname_usp_f); }; log { source(s_compname_kern); destination(d_compname_kern_f); }; source s_compname_usp_f { pipe("/var/log/filtered-source"); }; source s_compname_kern_f { pipe("/var/log/filtered-kernel-source"); };

1 1