October 2010 - syslog-ng - lists.balabit.hu

Diagnosing patterndb load errors
by Matthew Hall 15 Oct '10

15 Oct '10

I am getting the following sequence of load errors for my patterndbs. Duplicates eliminated, timestamps zeroed out, host info anonymized. Has anybody else gotten these and learned how to fix them? I'm having a hard time figuring out what 'type', 'name', and 'value' mean, to track them down in the XML files. Thanks, Matthew Hall. Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Name-value pairs cannot have a zero-length name; value='' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='Neighbor Down: Dead timer expired', value='19' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='26' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='28' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='34' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='36' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='39' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=' packet', value='40' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='41' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='s', value='44' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key=', domain = <not specified>', value='8' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='12' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='user', value='13' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='20' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='21' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='22' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='23' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='24' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='26' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='27' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='28' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='29' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='30' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='31' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='32' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='33' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='34' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='5', name='reason', value='35' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='4', name='None', value='38' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='vpn-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='load-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='firewall-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate parser node in radix tree; type='0', name='group', value='32' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Duplicate key in radix tree; key='. Please check authentication server radius/ldap/tacacs logs.', value='50' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Log pattern database reloaded; file='wireless-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23005]: Termination requested via signal, terminating; Jan 01 00:00:00 s_internal@host syslog-ng[23005]: syslog-ng shutting down; version='3.1.1' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='wireless-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='vpn-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='switch-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='load-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: Log pattern database reloaded; file='firewall-db.xml', version='3', pub_date='2010-08-25' Jan 01 00:00:00 s_internal@host syslog-ng[23379]: syslog-ng starting up; version='3.1.1'

3 3

Bazsi's blog: syslog-ng correllation updated
by Balazs Scheidler 15 Oct '10

15 Oct '10

syslog-ng correllation updated I’m trying to push syslog-ng 3.2beta1 out on the door, but as I was writing the NEWS entry I had to realize that the latest state of the patterndb correllation functions are undocumented so far. So here goes a blog post which tries to summarize how it works, so that I can include it in the NEWS entry :) My previous post on the topic used a syntax that included explicit “store” and “join” elements, but I’ve decided to drop those, as they stood in the way for some more juicy functionality. What remained is that the correllation is still focused around a “correllation state”, or as I’ve called internally a “context”. A context consists of a series of log messages related to each other in some way. As new messages come in, they may be associated with a context (e.g. added to the context). Also, when an incoming message is identified it can trigger some actions to be performed. And these actions can use all the information that was stored previously in the context. Let’s see how this work out with concrete examples: each rule in the patterndb has a “context-id” attribute telling db-parser() which context the given message should be associated with. This example covers an SSH login message: <rule id=”…” context-id=”$HOST:$PROGRAM:$PID” context-timeout=”86400″ context-scope=”global”> <patterns> <pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern> </patterns> … </rule> Since multiple rules can reference the same context, multiple different kind of messages may be added into the same context as a result. E.g. the logout event looks like this: <rule context-id=”$HOST:$PROGRAM:$PID” context-timeout=”0″ context-scope=”global”> <patterns> <pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:usracct.username:@</pattern> </patterns> … </rule> As you can see a “session” is identified using the triplet ($HOST, $PROGRAM, $PID) and these two rules correllate the login/logout events into the same context, which means that you can create a derived event that contains information from both of them. Please note that it is fairly common that messages only need to be correllated if they originate from the same host: e.g. this SSH login message needs only to be correllated to its logout counterpart if they both originate from the same host. In the previous example this was achieved using explicit macros in the context-id attribute, however since this is quite often the case, this was worked into a function of its own right: each rule can have a context-scope attribute: <rule id=”…” context-scope=”process” context-id=”ssh-login” context-timeout=”5″> </rule> The context-scope tells syslog-ng which messages need to be considered when looking for correllations: * process: only consider messages that have matching $HOST, $PROGRAM and $PID values * program: only consider messages that have matching $HOST and $PROGRAM values * host: only consider messages that have matching $HOST values * global: any kind of message is fine The default is to use “process”, which means that if it is true that the same process is emitting all the messages that you want to correllate, then you don’t need to use a variable part in your context-id attribute. But it is also important to know that it is way faster to specify the scope this way than it’d be to add all relevant macros to your context-id attribute. So far so good, we have all the functions that we used to have with the previous versions of the functionality. But I mentioned something about “actions” to be performed. Until now a patterndb rule basically only identified the incoming message, possibly associated tags and name-value pairs, but didn’t perform anything else. This is being changed: one or more actions can be associated with a patterndb rule in order to make it possible to react to more complex situations. Here’s an example action: <action rate=”1/86400″> <message> <values> <value name=”MESSAGE”>a patterndb rule matched</value> </values> </message> </action> Right now the only real response to a message is to generate another message, but this allows us to do a couple of powerful transformations, especially with the following options that you can specify for an action tag: * condition: specifies a syslog-ng filter expression that needs to be matched in order to really perform the action. It is evaluated on the current message that matched the rule. * rate: <num>/<period> specifies how much messages are to be generated (num), in the specified time period (period). Excess messages are dropped. For example: “1/60″ allows 1 message / minute. Rates apply to the given scope for the given rule/action. E.g. context-scope=”host”, rate=”1/60″ means that one message gets generated for _each host_ per minute. * trigger: specifies when to execute the action, there are two possible triggers right now: * match: execute immediately once the rule matches * timeout: execute when the correllation timer expires I’d like to highlight two things: * it is possible to react to the expiration of a correllation timer (e.g. trigger=”timeout”) * it is possible to generate a message only in case a given condition is met (e.g. “$PID” == “”) Right now new messages are posted to the internal() driver. This is not the way I wanted it to be, but doing my original plan would require an enormous refactorization of the code, and it is too late for that too happen. My original idea was to let the db-parser() emit multiple messages, but since the current state of affairs in syslog-ng assumes that only sources generate messages, that needs a lot of work. But hey, we need something to do for syslog-ng 3.3, right?

5 10

syslog-ng and ntsyslog
by Fiorenzi Alessandro 15 Oct '10

15 Oct '10

Hi, We have syslog-ng 3.05 as log server, and datagram syslog agent on windows system (originary ntsyslog) form e windows 2003 server with syslogagent configure I have this event on eventviewer Event Type: Success Audit Event Source: Security Event Category: Logon/Logoff Event ID: 538 Date: 10/12/2010 Time: 12:26:43 PM User: DOMAINXXX\A.Fiorenzi Computer: XXXXXX Description: User Logoff: User Name: A.Fiorenzi Domain: DOMAINXXX Logon ID: (0x0,0xF78F137) Logon Type: 10 and on syslog-ng server i get this: Oct 12 12:26:43 XXXXXX security[success]: 538 DOMAINXXX\a.fiorenzi User Logoff User Name: A.Fiorenz Domain: DOMAINXX Logo n ID: (0x0,0xF78F137 Logon Type: 1 where the descrition field has UserName, Domain, logon ID an Logon Type cutted. I have record the network traffic via tcpdump and I have seen data arrive correctly. So have set in syslog-ng.conf options the statement log_msg_size(8192); The problem is still open and I do not know how to solve, anyone can help me? Alessandro Fiorenzi Prima di stampare, pensa all'ambiente ** Think about the environment before printing ________________________________ Il presente messaggio, inclusi gli eventuali allegati, ha natura aziendale e potrebbe contenere informazioni confidenziali e/o riservate. Chiunque lo ricevesse per errore, ? pregato di avvisare tempestivamente il mittente e di cancellarlo. E' strettamente vietata qualsiasi forma di utilizzo, riproduzione o diffusione non autorizzata del contenuto di questo messaggio o di parte di esso. Pur essendo state assunte le dovute precauzioni per ridurre al minimo il rischio di trasmissione di virus, si suggerisce di effettuare gli opportuni controlli sui documenti allegati al presente messaggio. Non si assume alcuna responsabilit? per eventuali danni o perdite derivanti dalla presenza di virus. *** This email (including any attachment) is a corporate message and may contain confidential and/or privileged and/or proprietary information. If you have received this email in error, please notify the sender immediately, do not use or share it and destroy this email. Any unauthorised use, copying or disclosure of the material in this email or of parts hereof (including reliance thereon) is strictly forbidden. We have taken precautions to minimize the risk of transmitting software viruses but nevertheless advise you to carry out your own virus checks on any attachment of this message. We accept no liability for loss or damage caused by software viruses. For the conduct of investment business in the UK, the Company is authorized by Bank of Italy and regulated by the Financial Services Authority.

4 3

Open source versions of syslog-ng with sql() destination
by Liam Kirsher 15 Oct '10

15 Oct '10

Hi -- Just upgraded to a new server (CentOS 5) and the yum installed version of syslog-ng (2.1.4) is gagging on the previous version of syslog-ng.conf, which was working fine for syslog-ng (1.6.12). I commented out a couple of lines like @version and file() but find I can't get past sql(). The documentation says that for versions 2.x, sql() is only available for Premium version. Just want to verify that I will have to roll back to the old 1.6.12 version to have the sql() feature, which we are using quite a bit. Is that version still available? Thanks, Liam -- Liam Kirsher PGP: http://liam.numenet.com/pgp/

4 4

[Bug 52] CAP_DAC_OVERRIDE is missing when capabilities are enabled, unable to write to files created with the owner() option.
by bugzilla＠bugzilla.balabit.com 15 Oct '10

15 Oct '10

https://bugzilla.balabit.com/show_bug.cgi?id=52 --- Comment #3 from Balazs Scheidler <bazsi(a)balabit.hu> 2010-10-15 22:09:19 --- sounds great. I'll try to get this integrated. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

Re: [syslog-ng] error in syslog-ng 3.2alpha2: missing m4 macros directory
by Guillaume Rousse 15 Oct '10

15 Oct '10

Le 12/10/2010 11:37, Guillaume Rousse a écrit : > Le 06/10/2010 15:40, Balazs Scheidler a écrit : >> On Sat, 2010-10-02 at 23:01 +0200, Guillaume Rousse wrote: >>> Hello. >>> >>> configure.in has this macro: >>> AC_CONFIG_MACRO_DIR([m4]) >>> >>> However, the m4 directory doesn't exist. As a consequence, running >>> 'autoreconf' fails immediatly. I guess it's just missing, as previous >>> release had it. >> >> Right. I've fixed this in the git tree already. Could you try using a >> git snapshot? > I just tried, the tree still doesn't have any m4 directory... Forget this, I just understood it is dynamically created by autogen.sh script. [..] > Anyway, I just realized I didn't need to regenerate autotools toolchain, > as my patch doesn't impact any relevant file. Here it is, its goal is > just to ensure than syslog-ng doesn't create files directly under > $localstatedir directory, which is actually /var, but only under its own > specific directory, $localstatedir/lib/syslog-ng. It avoid the need to > override configure defaults through --localstatedir option, as does the > spec file shipped in the tree. Actually, the patch itself only deals with localestatedir, whereas the same issue appears with other shared directories (datadir and sysconfdir). By default, with a standard prefix, all those locations matches shared directories (/etc, /usr/share, /var), forcing packagers to override all of them to isolate syslog-ng specific content. Having all of them correct by default would be nicer for FHS-addicts. However, it also allow people using a distinct prefix (/opt/syslog-ng for instance) to avoid uselessly long path for different component (/opt/syslog-ng/share instead of /opt/syslog-ng/share/syslog-ng). I guess this is the goal here. -- BOFH excuse #368: Failure to adjust for daylight savings time.

2 1

Re: [syslog-ng] error in syslog-ng 3.2alpha2: missing m4 macros directory
by Guillaume Rousse 15 Oct '10

15 Oct '10

Le 06/10/2010 15:40, Balazs Scheidler a écrit : > On Sat, 2010-10-02 at 23:01 +0200, Guillaume Rousse wrote: >> Hello. >> >> configure.in has this macro: >> AC_CONFIG_MACRO_DIR([m4]) >> >> However, the m4 directory doesn't exist. As a consequence, running >> 'autoreconf' fails immediatly. I guess it's just missing, as previous >> release had it. > > Right. I've fixed this in the git tree already. Could you try using a > git snapshot? I just tried, the tree still doesn't have any m4 directory... > No need to use a git client, the gitweb interface is able to generate a > tar.gz on the fly: > > http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=snapshot;h=master > > In order to make syslog-ng compile from the git snapshot, rather than a > tar.gz, you'll a couple more tools, like (IIRC): > > * bison 2.4.1 > * automake/autoconf/libtool > > Once you have those you should run ./autogen.sh from the source tree, > then you'll have everything starting with a configure script to compile > it. Excepted the hack on PKG_CONFIG minimal version, autoreconf should work as well. Anyway, I just realized I didn't need to regenerate autotools toolchain, as my patch doesn't impact any relevant file. Here it is, its goal is just to ensure than syslog-ng doesn't create files directly under $localstatedir directory, which is actually /var, but only under its own specific directory, $localstatedir/lib/syslog-ng. It avoid the need to override configure defaults through --localstatedir option, as does the spec file shipped in the tree. -- BOFH excuse #334: 50% of the manual is in .pdf readme files

2 1

Windows event logs vs syslog format
by Jerry Riedel 15 Oct '10

15 Oct '10

I have the latest syslog-ng on an Opensuse 11.2 acting as a syslog server and it is working well except for one thing - the Windows event logs that are being sent with the Datagram Syslog Agent contain a space that causes issues. Initially, all of these were going into /var/log/messages until I added the keep_hostname(yes) argument. After doing that, it now puts the Windows logs into the appropriate folder under /var/log/hosts/ but it still puts a copy into the /var/log/messages file. I would like to have that log only contain log messages from Opensuse. Is there a configuration setting I am missing, or is this caused by the fact that the syslog agent does not correct the eventlog message so that it adheres to the standard syslog message format? If the latter, does anyone know of an open source/free agent that does this? An example of one of the problematic messages is: Oct 6 11:02:08 cli-fs-1 security[success] 576 NT AUTHORITY\SYSTEM Special privileges assigned to new logon: User Name:CLI-FS-1$ Domain:(obscured) Logon ID:(0x0,0x11331C8) Thanks, Jerry Riedel

4 12

request for smtp/imap/pop3 server log samples
by Peter Czanik 14 Oct '10

14 Oct '10

Hello, As you can see from my last few e-mails, I started to work on patterns for mail related (smtp/imap/pop3) servers. For now I'm only concentrating on login/login failure/logout events. There are many related servers, so it is a lot of work to install/configure/collect logs from them, even from the most important ones. So I'd like to ask for some help here, with an offer, which is beneficial also for you. Here is my offer: - you send me log samples for the following situations from your mail related servers: successful login, logout, invalid password, invalid username) - I create patterns, discuss here when I run into troubles - I push the results into the git tree, so patterns will be available for you and the syslog-ng community - you have patterns you can use immediately with your software (vs. patterns for software I find interesting :-) ) - you see how your log samples turn into patterns If you send the log samples to the list, please make sure, that confidential information is replaced. If you send them directly to me, I can also do it for you, but obviously it's better when sensitive information never leaves your network. Please use the method I showed at http://czanik.blogs.balabit.com/2010/10/pattern-writing-tips-and-tricks/ to collect your logs. If you don't want to read those few paragraphs, here is the most important part, a short syslog-ng.conf snippet: filter f_myprogi { program('pure-ftpd'); }; destination d_myprogi { file('/var/log/myprogi'); }; log { source(src); filter(f_myprogi); destination(d_myprogi); }; As other way messages are scattered among many log files, and difficult to see which messages cover which event. Thanks for your help! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

uw-imapd patterns
by Peter Czanik 13 Oct '10

13 Oct '10

Hello, Attached you can find UW-imapd samples and patterns. Please test it and let me know, if you have any login/logout related log message left as "Unknown". Known problem: login failures generate the same name value pairs twice. These patterns were created on Ubuntu 10.04 LTS. I'd be very interested to know, if using them on other distributions or UNIX variants makes a difference. -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0