April 2009 - syslog-ng - lists.balabit.hu

Parser-DB Memory Leak
by Martin Holste 27 Apr '09

27 Apr '09

I'm running tests with the db-parser() parser module and finding that the memory grows exponentially until the program finally dies with: ***MEMORY-ERROR***: syslog-ng[2769]: GSlice: failed to allocate 248 bytes (alignment: 256): Cannot allocate memory Here is my configuration: @version: 3.0 options { }; source s_tcp { tcp(ip(0.0.0.0) port(445)); }; parser p_db { db-parser(); }; filter f_security_class { match("security" value(".classifier.class")); }; filter f_class_unknown { match("unknown" value(".classifier.class")); }; filter f_not_class_unknown { not match("unknown" value(".classifier.class")); }; filter f_FIREWALL_ACCESS_DENY { match("2" value(".classifier.rule_id")); }; template t_FIREWALL_ACCESS_DENY { template("$R_UNIXTIME\t$SOURCEIP\t$FACILITY\t$PROGRAM\t${.classifier.class}\t${.classifier.rule_id}\t${FIREWALL.proto}\t${FIREWALL.o_int}\t${FIREWALL.srcip}\t${FIREWALL.srcport}\t${FIREWALL.i_int}\t${FIREWALL.dstip}\t${FIREWALL.dstport}\t${FIREWALL.access_group}\t${FWSM.PRIORITY}\t${FWSM.MSG_CODE}\t${MSGONLY}\n"); }; destination d_FIREWALL_ACCESS_DENY { fifo("/tmp/db_parsed_fifo", template(t_FIREWALL_ACCESS_DENY)); }; filter f_FIREWALL_CONNECTION_END { match("3" value(".classifier.rule_id")); }; template t_FIREWALL_CONNECTION_END { template("$R_UNIXTIME\t$SOURCEIP\t$FACILITY\t$PROGRAM\t${.classifier.class}\t${.classifier.rule_id}\t${FIREWALL.proto}\t${FIREWALL.o_int}\t${FIREWALL.srcip}\t${FIREWALL.srcport}\t${FIREWALL.i_int}\t${FIREWALL.dstip}\t${FIREWALL.dstport}\t${FIREWALL.conn_duration}\t${FIREWALL.conn_bytes}\t$MSGONLY\n"); }; destination d_FIREWALL_CONNECTION_END { fifo("/tmp/db_parsed_fifo", template(t_FIREWALL_CONNECTION_END)); }; template t_DEFAULT { template("$R_UNIXTIME\t$SOURCEIP\t\$FACILITY\t$PROGRAM\t1\t1\t$MSGONLY\n"); }; destination d_default_fifo { fifo("/tmp/db_parsed_fifo", template(t_DEFAULT)); }; log { source(s_tcp); log { parser(p_db); filter(f_not_class_unknown); log { filter(f_FIREWALL_ACCESS_DENY); destination(d_FIREWALL_ACCESS_DENY); }; log { filter(f_FIREWALL_CONNECTION_END); destination(d_FIREWALL_CONNECTION_END); }; flags(final); }; log { destination(d_default_fifo); }; }; var/patterndb.xml: <patterndb version='1' pub_date='2009-04-16'> <program name='FWSM'> <pattern>%FWSM</pattern> <rule id='2' class='2'> <pattern>Deny@QSTRING:FIREWALL.proto: @src@QSTRING:FIREWALL.o_int: :@@IPv4:FIREWALL.srcip:@/@NUMBER:FIREWALL.srcport:@ dst@QSTRING:FIREWALL.i_int: :@@IPv4:FIREWALL.dstip:@/@NUMBER:FIREWALL.dstport:@ by access-group @QSTRING:FIREWALL.access_group:"@</pattern> </rule> <rule id='3' class='3'> <pattern>Teardown@QSTRING:FIREWALL.proto: @connection @NUMBER::@ for@QSTRING:FIREWALL.o_int: :@@IPv4:FIREWALL.srcip:@ /@NUMBER:FIREWALL.srcport@ to@QSTRING:FIREWALL.i_int: :@@IPv4:FIREWALL.dstip:@/@NUMBER:FIREWALL.dstport@ duration@QSTRING:FIREWALL.conn_duration: @bytes @NUMBER:FIREWALL.conn_bytes:@</pattern> </rule> </program> </patterndb> syslog-ng -V syslog-ng 3.0.1+binpkg4 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#555574a984eaef9410a2869db0af1be0d52b269b Compile-Date: Apr 16 2009 14:42:21 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Should I recompile with memtrace on? Thanks, Martin

4 10

syslog-ng 3.0.1 dies on reload with program() destination
by Jan Rekorajski 24 Apr '09

24 Apr '09

Sorry for possible double post, I sent the first from wrong email. Hi, Syslog-ng 3.0.1 dies on reload (kill -HUP pid) when config file contains log { destination { program() } } directive set. 100% repeatable. Here are the relevant config lines: destination d_sshguard { program("/usr/sbin/sshguard -a 6"); }; # A desperate experiment #destination d_sshguard { program("read"); }; log { source(s_sys); filter(f_authpriv); destination(d_sshguard); }; >From what I managed to find with strace/ltrace, syslog-ng kills the program, does not clear some data structures and then falls face down with sigsegv/term/ill trying to restart/reattach(?) to the program. The exact same configuration works perfectly with version 2.0.10. -- Jan Rekorajski | ALL SUSPECTS ARE GUILTY. PERIOD! baggins<at>mimuw.edu.pl | OTHERWISE THEY WOULDN'T BE SUSPECTS, WOULD THEY? BOFH, MANIAC | -- TROOPS by Kevin Rubio

5 8

sunfreeware syslog-ng v 3.0.1
by Ngim, Hyrum CTR ADCF/Arrowpoint 23 Apr '09

23 Apr '09

I get this error when I introduce the owner when defining a destination: destination firewall { file("/data/firewall/fwlog" owner(sysadmin) group(sysadmin) perm(0644)); }; When I run a syntax check I get this: # syslog-ng -s Error resolving user; user='sysadmin' # pkginfo -l SMCsyslng PKGINST: SMCsyslng NAME: syslogng CATEGORY: application ARCH: sparc VERSION: 3.0.1 BASEDIR: /usr/local VENDOR: BalaBit IT Ltd PSTAMP: Steve Christensen INSTDATE: Apr 15 2009 11:56 EMAIL: steve(a)smc.vnet.net STATUS: completely installed FILES: 91 installed pathnames 4 shared pathnames 25 directories 7 executables 1610 blocks used (approx) I also checked the misc.c file in the sunfreeware source and it didn't seem like it had the same errors that were identified in posts like this one: https://lists.balabit.hu/pipermail/syslog-ng/2009-April/012761.html

2 2

sunfreeware syslog-ng v 3.0.1
by Ngim, Hyrum CTR ADCF/Arrowpoint 23 Apr '09

23 Apr '09

I have installed and am configuring the sunfreeware package for syslog-ng. I ran into this issue that I saw on some other emails: Error resolving user; user='sysadmin' Is there a way around this to get it working? TIA

2 1

IPv6 addresses logged as 127.0.0.1
by Michael Granzow 22 Apr '09

22 Apr '09

When using the template parameter $SOURCEIP, connections originating from IPv6 addresses are logged as "127.0.0.1". The following patch fixes that issue. Listening on :: on dual-stack hosts will lead to IPv4 connections being logged in IPv6-mapped format, e.g. ::ffff:10.11.12.13. This is not what people usually expect, but it's not wrong. If desired, the place to fix that would be g_sockaddr_inet6_format().

2 2

Error compiling syslog-ng-3.0.1 on HP-UX 11.23 (Itanium)
by Monz Roswitha 22 Apr '09

22 Apr '09

Hi, I am trying to build syslog-ng-3.0.1 on HP-UX Itanium (64-bit) box. I am running ./configure CFLAGS="-U_XOPEN_SOURCE -U_XOPEN_SOURCE_EXTENDED -D_HPUX_SOURCE" and it is giving following error messages while 'make': if gcc -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/include/glib-2.0 -I/usr/local/lib/hpux32/glib-2.0/include -I/usr/local/include/eventlog -I/usr/local/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -U_XOPEN_SOURCE -U_XOPEN_SOURCE_EXTENDED -D_HPUX_SOURCE -Wall -U_XOPEN_SOURCE -U_XOPEN_SOURCE_EXTENDED -D_HPUX_SOURCE -MT compat.o -MD -MP -MF ".deps/compat.Tpo" -c -o compat.o compat.c; \ then mv -f ".deps/compat.Tpo" ".deps/compat.Po"; else rm -f ".deps/compat.Tpo"; exit 1; fi compat.c:35: error: redefinition of '__pread' /usr/local/lib/gcc/ia64-hp-hpux11.23/3.4.3/include/sys/unistd.h:584: error: previous definition of '__pread' was here compat.c:57: error: redefinition of '__pwrite' /usr/local/lib/gcc/ia64-hp-hpux11.23/3.4.3/include/sys/unistd.h:586: error: previous definition of '__pwrite' was here *** Error exit code 1 Stop. *** Error exit code 1 Stop. *** Error exit code 1 Stop. How can I solve this problem? Is a binary for HP-UX 11.23 Itanium (IA64) planed? Thanks for help. Kind regards, Roswitha Mit freundlichen Grüßen Roswitha Monz, OCP -- Systemadministratorin Unix & Oracle Server-Netzwerk-Betrieb _____________________________________________________________ Gemeinnützige Salzburger Landeskliniken Betriebsgesellschaft mbH Servicebereich Informatik und Medizintechnik Müllner Hauptstraße 48, A-5020 Salzburg Tel.: +43-(0)662-4482-4938 Mob.: +43(0)664-604141-4938 Fax +43(0)662 4482-4908 mailto:r.monz@salk.at Internet: http://www.salk.at

2 1

Failure to source one file prevents entire syslog-ng logger from running
by Hari Sekhon 22 Apr '09

22 Apr '09

Hi, While I was away on holiday, another sysadmin changed the path of a 3rd party application logfile that I was I sourcing in syslog-ng (it cannot log directly to syslog). This caused the whole syslog-ng logger to basically fail to work/(re)start, giving an the error: Starting syslog-ng: Persistent configuration file is in invalid format, ignoring; Error opening file for reading; filename='/path/to/file.log', error='No such file or directory (2)' Error initializing source driver; source='s_file_blah' And auto-restarting the logger of course didn't work either since every time a restart is initiated the thing simply bailed out with this error. Shouldn't this error simply throw up a warning and continue logging the rest of what's happening on the system, rather than leaving a hole in the audit trail because it cannot lock in one source??? (Even better: start regardless, do what it can and then periodically check the source again, and every time it cannot get the source open, log a warning in syslog so that it can be caught by the monitoring infrastructure) Any suggestions for a workaround/option to prevent anything like this ever happening again in future would be welcome. -h -- Hari Sekhon Always open to interesting opportunities http://www.linkedin.com/in/harisekhon

2 3

db-parser QSTRING delmiter limitations
by Martin Holste 20 Apr '09

20 Apr '09

Hi, I'm new to the list and syslog-ng in general. I'm building a centralized log collector and am very interested in the power of the db-parser() parsing module. It really has amazing potential, and I'm eager to implement it. I've been playing with it quite a bit with a proof-of-concept to parse firewall logs from Cisco FWSM blades. The $MSGONLY part looks like this for a firewall deny: Deny udp src OUTSIDE:10.0.0.0/1234 dst INSIDE:192.168.0.0/5678 by access-group "OUTSIDE" [0xb74026ad, 0x0] My working parser entry is thus: <patterndb version='1' pub_date='2009-04-17'> <program name='FWSM'> <pattern>%FWSM</pattern> <rule id='1' class='security'> <pattern>Deny@QSTRING:FIREWALL.DENY_PROTO: @src</pattern> </rule> </program> </patterndb> This works great and returns udp and tcp in the ${FIREWALL.DENY_PROTO} macro for logging, along with the ${.classifier.class} and ${.classifier.rule_id} macros. However, when I try to parse out the interface, IP, and port numbers from "OUTSIDE:10.0.0.0/1234" part, the delimiters fail to capture correctly and the whole pattern misses. Here's what I'm trying to do: <patterndb version='1' pub_date='2009-04-17'> <program name='FWSM'> <pattern>%FWSM</pattern> <rule id='1' class='security'> <pattern>Deny@QSTRING:FIREWALL.DENY_PROTO: @src@QSTRING:FIREWALL.DENY_O_INT: @:@IPv4$:FIREWALL.DENY_SRCIP:@/@NUMBER:FIREWALL.DENY_SRCPORT: @dst</pattern> </rule> </program> </patterndb> After much debugging, it appears that there is a problem using QSTRING to match non-space-delimited parsing boundaries. That is, you cannot parse arbitrarily, you have to match on space boundaries. Is this true, or am I doing something wrong? I even tried to parse the 'n' out of the word 'Deny' with a pattern like <pattern>De@QSTRING:test: @y</pattern> and that fails. >From the debug, it appears that unless there is a space present, the radix key is off by one: Looking up node in the radix tree; i='0', nodelen='0', keylen='138', root_key='', key='Deny udp src<snip></snip>' Looking up node in the radix tree; i='2', nodelen='2', keylen='138', root_key='De', key='Deny udp src<snip></snip>' It looks like the key for the second entry should be key='ny udp src<snip></snip>' since the original 'De' match already hit. I put a lot of printf debugging statements in the code to see if I could figure out what was going wrong, but I havent' been able to conclude what the problem is yet, assuming arbitrary pattern delimiting was the intended goal. Is anyone able to successfully get db-parser() to parse on arbitrary characters? Also, the source code refers to STRING and ESTRING, how are those different from QSTRING? It looked like ESTRING was probably just an offset-based version of QSTRING. Thanks, Martin

2 3

message source hostname not displayed relay's hostname displaied instedd
by gatfi sami 17 Apr '09

17 Apr '09

hi i have syslog-ng 2.O.9-27 on sles11 32bite (SUSE Entreprise server 11) on 3 virtual machine i use the following syslog organisation a *syslog clien*t (read error log and sends them to a relay) the *relay* frorwords the message to a syslog-server The *syslog server* writes the messages in a pipe binde to a mysql database untill now everithing is working perfectly but in the syslog server i olways founf the relai host name ip adress i tried HOST_FROM in the template writing to the pipe it is always the same the relay's hostname destination d_mysql { pipe("/opt/topnet/mnt/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( '$HOST_FROM', '$FACILITY', '$PRIORITY', '$LEVEL','$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; imagine if i have aot of cilents with each one a diiferent error log i will see nothing but the relai's hostname need hepl here thks

2 1

GThread error when starting syslog-ng
by Scott Foster 17 Apr '09

17 Apr '09

I have installed syslog-ng on a CentOS v5.2 server. I have also installed the required dependency rpms. I am not using any database server for this (just plain old logging). When I attempt to start the syslog-ng service I get the below error: --------------------------------------------------------- [root@it188v ~]# service syslog-ng start Starting syslog-ng: GThread-ERROR **: file gthread-posix.c: line 140 (): error 'Operation not permitted' during 'pthread_getschedparam (pthread_self(), &policy, &sched)' aborting... /bin/bash: line 1: 8888 Aborted /sbin/syslog-ng [FAILED] [root@it188v ~]# --------------------------------------------------------- I've Googled anything about gthread and can't find anything useful. Any yes I am a newbie to Linux. Would someone clue me in on this issue? Froste ****************************************************************************This e-mail and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom it is addressed. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error, and that any use, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender at Merrick Bank and delete it from your computer. Although Merrick Bank attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses.

2 1