- syslog-ng - lists.balabit.hu

[patch] syslog-ng 3.0.4 init script doesn't remove stale PID file on startup
by Gordon McAllister 04 Dec '09

04 Dec '09

Hello, I'm running the syslog-ng 3.0.4 RPM on CentOS 5.4 and came across a problem where syslog-ng would not start properly after an unclean shutdown, since the old PID file was still in place. Attached is a patch to the init script supplied with the RPM that attempts to correct this. Regards, ---Gordon

3 3

rewriteable fields
by chris packham 03 Dec '09

03 Dec '09

Hi List, Just looking for a bit of clarification on exactly what fields I can rewrite in a message. Basically I've got an annoying debug message coming out of a program and while I could filter it off to a separate file, what I really want to do is drop its level to debug (from warning) and let my existing filters take care of it. The docs say "Setting a field can operate on any value available via macros, e.g., HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers" so modifying LEVEL is not listed but its not explicitly excluded either. I tried "rewrite r_norm {set("1", value("LEVEL")); };" but that didn't work. However "rewrite r_norm {set("FOO", value("MESSAGE")); };" and "rewrite r_norm {set("BAR", value("PROGRAM")); };" work as expected. Are there limitations as to which parts of a message I can rewrite? Are there some fuller examples hanging around? Thanks, Chris

3 2

syslog-ng OSE 3.1beta1 released
by Balazs Scheidler 03 Dec '09

03 Dec '09

Dear syslog-ng users, I'm proud to announce that syslog-ng OSE 3.1 has been released and uploaded to our webserver. This version is new in two ways: 1) of course it has new features, see below for the most interesting bits 2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link: https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement. For those who hurry, here's a link for the source code: https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1be… And here are the binaries for Linux/FreeBSD systems: https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… Select the Downloads tab, and in the Version selector select 3.1beta1. What is new in syslog-ng OSE 3.1 -------------------------------- * Support for patterndb v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT. Patterndb (more exactly the db-parser()) is a high performance message classifier and information extraction tool, that makes it easy to get away from the unstructured nature of syslog. Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at the 3rd iteration, hopefully slowly reaching its final form. Patterndb in general and the v1 format database is described in the syslog-ng manual at: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html The XML schemas that describe the different patterndb versions are available in the syslog-ng source tree: http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD The changes in the patterndb format as they evolved were described in Marton Illes's blog at http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html But see the other related posts as well. Old patterndb databases can be converted to the new format by putting them in a directory and using the pdbtool utility using the command: $ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml -D /opt/syslog-ng/etc/patterns.d Assuming the installation prefix of syslog-ng is /opt/syslog-ng Some v2 format patterns are distributed by BalaBit itself for its SSB product, download location: https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/ You can convert these db files using pdbtool as described above. Work is ongoing to publish a more comprehensive patterndb, but more on that in a separate post. * Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message. See the manpage and Marci's post: http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html * Support for message tags: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering. http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html * Added support for rewriting structured data. Earlier structured data fields in the new RFC5424 style syslog protocol were only read-only values that could be referenced in a template, but they couldn't be changed, and neither was it possible to add new fields in an already existing syslog message. Now all these became possible by using the same syntax that didn't work earlier, e.g. rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); }; * Macros and name-value pairs got a little tighter integration, in filters where syslog-ng 3.0 was limited to only use name-value pairs, with 3.1 you can also use macros. The following now works: match("<regexp>" value("R_DATE")); syslog-ng is now warning you in case you are using '$' prefix in the value syntax. * Enhanced dynamic name-value performance by a factor of three. The summary says it all, the performance dynamic name-value pairs that the various parsers produce got faster, thus the performance penalty of structuring the incoming messages got smaller. * Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These are patterndb parsers to make it easier to describe log messages. * Added non-standard and non-portable facility codes (range 10-15), decouple syslog-ng facility name information from the system used to compile syslog-ng on. Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems. Any feedback, success/failure reports is more than appreciated. -- Bazsi

2 1

hi all.Why $HOST variable valued "250" , "250-localhost^M" ?
by [焱龙企鹅]韩友洪 01 Dec '09

01 Dec '09

Hi.all I configured syslog-ng (3)(CentOS 5) to collect logs sent by 1xx syslog servers.(freebsd6).I got two strange dirs: "250" and "250-localhost^M" my syslog-ng.conf : ============================================== source s_udp { udp(ip(172.16.18.10) port(514) ); }; destination d_udp_data{ file ("/data3/syslogng/logcollect/$YEAR-$MONTH-$DAY/$HOST/$FACILITY/$PROGRAM.log" create_dirs(yes) template("\n$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $SOURCEIP $PROGRAM $MSG") ); }; ================================================= my syslog.conf !webface.login *.info /var/log/mail/webface/webface-login.log *.* @172.16.18.10 ======================================= root@localhost 2009-12-01]# pwd /data3/syslogng/logcollect/2009-12-01 [root@localhost 2009-12-01]# ls ............... 10.55.2.35 10.55.2.46 10.55.2.57 10.55.2.68 10.55.2.79 10.55.2.90 250 250-localhost^M [root@localhost 2009-12-01]# ls -R 250 250: user 250/user: 2.1.0.log [root@localhost 2009-12-01]# more 250/user/2.1.0.log 2009-12-01 13:46:12 10.55.2.11 2.1.0 Ok^M 2009-12-01 13:46:27 10.55.2.11 2.1.0 Ok^M ........ [root@localhost 2009-12-01]# ls -R 250-localhost^M/ 250-localhost^M/: user 250-localhost^M/user: 250-SIZE^M.log [root@localhost 2009-12-01]# more 250-localhost^M/user/250-SIZE^M.log 2009-12-01 14:07:44 10.55.2.128 250-SIZE^M 250-8BITMIME^M 250 ENHANCEDSTATUSCODES^M -- 祝：身体安康，万事如意！ ________________________________________________________________ Mr. Hunter - 韩友洪焱龙企鹅 youhong(a)staff.sina.com.cn 新浪 - 产品事业部 -邮箱 MSN：hf_linux(a)msn.com 电话：5392 手机：15001328768 地址：北京市海淀区北四环西路58号理想国际大厦18层 ________________________________________________________________ http://www.sina.com.cn You're the One 新浪.北京一切由你开始

2 1

AUTO: Phil Newlon/Corporate/Wendys is out of the office. (returning 12/02/2009)
by Phil.Newlon＠wendysarbys.com 01 Dec '09

01 Dec '09

I am out of the office until 12/02/2009. I have received the email message that you sent and will respond to it when I return. If you feel your message needs immediate attention, you may call me on my cell at 614-569-1328. Note: This is an automated response to your message "[syslog-ng] hi all.Why $HOST variable valued "250" , "250-localhost^M" ?" sent on 12/1/2009 2:50:50 AM. This is the only notification you will receive while this person is away. Notice: This e-mail message and its attachments are the property of Wendy's/Arby's Group Inc. or one of its subsidiaries and may contain confidential or legally privileged information intended solely for the use of the addressee(s). If you are not an intended recipient, then any use, copying or distribution of this message or its attachments is strictly prohibited. If you received this message in error, please notify the sender and delete this message entirely from your system.

1 0

Question about flags(ignore-case) option
by PAUL WILLIAMSON 30 Nov '09

30 Nov '09

I have the following filter set up: filter f_wireless_devices { host("system") flags(ignore-case); }; and I'm getting a syntax error. If I remove the flags(ignore-case), filter f_wireless_devices { host("system"); }; It is fine. On page 58 of the 3.0 sys admin guide, I have the syntax correct. Is there any other way to do this? Maybe put an ignore_case(yes) in the options sections at the top of the configuration file? Thanks, Paul ************************************ This email may contain privileged and/or confidential information that is intended solely for the use of the addressee. If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act. You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information. There are risks associated with the use of electronic transmission. The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission. ************************************

7 21

Troubles with the pattern database
by Guillaume Rousse 30 Nov '09

30 Nov '09

Hello list. I'm the mandriva maintainer for syslog-ng. I'm trying to play with the pattern database, with syslog-ng 3.0.4. I rebuild the package with the attached patch, so as to use /usr/share/syslog-ng a database location (a bit more FHS-compliant than /var), and extracted the files downloaded from http://www.balabit.com/downloads/files/patterndb there. However, loading them fails with this message: Error parsing pattern database file; filename='/usr/share/syslog-ng/patterndb.xml', error='Unexpected <rule> element' Error reloading pattern database, no pattern recognition will be done; It looks like some DB format issue. According to http://www.balabit.com/dl/html/syslog-ng-v3.0-guide-admin-en.html/ch08s06.h… the supported format is v1 until syslog-ng 3.0.2, and the NEWS file doesn't list any change here, while the patterndb file is already using v2. Am I correct ? And in this case, is there any way to easily convert the base to the old format ? -- BOFH excuse #252: Our ISP is having {switching,routing,SMDS,frame relay} problems diff -Naur -x '*~' syslog-ng-3.0.4/configure.in syslog-ng-3.0.4-fix-pattern-database-location/configure.in --- syslog-ng-3.0.4/configure.in 2009-04-30 18:32:19.000000000 +0200 +++ syslog-ng-3.0.4-fix-pattern-database-location/configure.in 2009-10-12 15:20:11.000000000 +0200 @@ -722,6 +722,7 @@ AC_DEFINE_UNQUOTED(PATH_SYSCONFDIR, "`patheval $sysconfdir`", [sysconfdir]) AC_DEFINE_UNQUOTED(PATH_LOCALSTATEDIR, "`patheval $localstatedir`", [local state directory]) AC_DEFINE_UNQUOTED(PATH_PIDFILEDIR, "`patheval $pidfiledir`", [local state directory]) +AC_DEFINE_UNQUOTED(PATH_DATADIR, "`patheval $datadir`", [datadir]) if test -n "$timezonedir"; then AC_DEFINE_UNQUOTED(PATH_TIMEZONEDIR, "`patheval $timezonedir`", [timezone base directory]) fi diff -Naur -x '*~' syslog-ng-3.0.4/src/syslog-ng.h syslog-ng-3.0.4-fix-pattern-database-location/src/syslog-ng.h --- syslog-ng-3.0.4/src/syslog-ng.h 2009-04-22 13:06:57.000000000 +0200 +++ syslog-ng-3.0.4-fix-pattern-database-location/src/syslog-ng.h 2009-10-12 15:14:10.000000000 +0200 @@ -52,7 +52,7 @@ #define PATH_PERSIST_CONFIG PATH_LOCALSTATEDIR "/syslog-ng.persist" #define PATH_QDISK PATH_LOCALSTATEDIR -#define PATH_PATTERNDB_FILE PATH_LOCALSTATEDIR "/patterndb.xml" +#define PATH_PATTERNDB_FILE PATH_DATADIR "/syslog-ng/patterndb.xml" #define LOG_PRIORITY_LISTEN 0 #define LOG_PRIORITY_READER 0

3 3

Beginner's questions for deploying syslog-ng
by Hailu Meng 29 Nov '09

29 Nov '09

Hi All, I'm planning create a centralized logging server for our network. We have 100 cisco routers and switches. Not too big organization. Suppose every cisco must have log to be sent to centralized logging server. I'm considering to deploy syslog-ng in CentOS 5. As I understand, I also need a web front end for user to search the log easily. Someone recommend splunk to do this front end job. I checked it out and found the free version of splunk has a 500MB/day for the indexing size. I'm not sure how this limit will affect on our network. Or is there any other good web front end for use with syslog-ng? Any help is appreciated. Lou

1 0

filter logger tags from syslog
by Jose Sanchez 28 Nov '09

28 Nov '09

Hello, I've been using classic syslog for centralizing apache access logs from one server to a remote syslog server, the thing is syslog adds some nasty tags before the lines in the access logs and I cant get them off, ie: "Nov 25 21:25:37 server1 logger:" I would like to know if syslog-ng has the option to filter this kind of stuff, I just want to have the logs sent to the syslog server exactly like I was saving them in a local access.log file. Thanks in advance.

3 5

problems with syslog-ng-3.0.4 and -C option
by Chris Fabri 26 Nov '09

26 Nov '09

Greetings- I believe I've discovered an issue in syslog-ng 3.0.4. I've compiled this version on RH 3 (which might be the problem), and when you try for example: syslog-ng -C /var/log -u syslog-ng you get: Error opening configuration file; filename='/usr/local/etc/syslog-ng.conf', error='No such file or directory (2)' even specifying "-f /usr/local/etc/syslog-ng.conf" causes the same problem. Placing syslog-ng.conf in the directory specified by -C works, so perhaps the -C option has some issue with using the actual path instead of the chroot'ed path. I didn't see anything in the last several months of emails from the list, and I'm surprised this wouldn't have bitten more people, so it could be an RH3-ism, but wanted to make sure I reported it, in case it was more of a problem. chris

2 1