- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
by Worsham, Michael 29 Sep '10

29 Sep '10

I must be doing something wrong or the syntax is really dependant on the program name now. Orginally under the v2.1 build, the intent of the filter was to strip any incoming message going to the syslog-ng server from the data being sent from the syslog-ng client. There was no one specific program attached to the incoming data. It could come from the client's syslog-ng side via logger command, apache data, kernel, etc. Server-side syslog-ng.conf filter section: filter M_audit { not match("Auditing" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_repeat { not match("Repeating" value("last message repeated") flags(ignore-case)); }; filter M_stats { not match("Stats" value("Log statistics") flags(ignore-case)); }; On the client side, I would just do 'logger daemon' to send over the message to the syslog-ng server. On the server side, running syslog-ng -d reveals this output: Incoming log entry; line='<13>Sep 20 17:16:31 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='Audit daemon rotating log files' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_audit' Filter rule evaluation begins; filter_rule='M_repeat' No such value known; value='last message repeated' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_repeat' Filter rule evaluation begins; filter_rule='M_stats' No such value known; value='Log statistics' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_stats' Server-side syslog output being seen in the data file: Sep 20 17:16:31 drupal root: daemon ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

5 18

pub_date
by Peter Czanik 27 Sep '10

27 Sep '10

Hello, All of the pdb files in git have pub_date set to 2010-07-13. Should I also use this date, or it's just a coincidence and I can use the actual date? Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

[Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent
by bugzilla＠bugzilla.balabit.com 23 Sep '10

23 Sep '10

https://bugzilla.balabit.com/show_bug.cgi?id=93 Summary: filter() functionality between 2.1 to 3.0 not consistent Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: mworsham(a)scires.com Type of the Report: bug Estimated Hours: 0.0 In the older syslong-ng v2.1, this line works perfectly: filter M_audit { not match("Audit daemon rotating log files"); }; Under 3.0.8, none of the following are working (if added one line at a time) and the daemon restarted: filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY) flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files")); }; What I am looking to do is if any incoming data has the following keywords (i.e. "Audit daemon rotating log files") being detected, it should be filtered (i.e. dropped), and then not show up in the actual message log file. For example, if I go over on the syslog-ng client and do 'logger daemon': - The older v2.1 syslog-ng server detects the embedded keyword and drops the message. <-- This is correct - On the v3.0.8, the message passes through and found in the data file. <-- This is incorrect Attempted to use the following URLs to correct the formatting, but it still not working: - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui… - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui… -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

5 10

Converting filtering from 2.1 to 3.0?
by Worsham, Michael 20 Sep '10

20 Sep '10

We are running syslog-ng v2.1 in a multiple client-server configuration. Due to new encryption requirements, we planning to upgrade to v3.0.8 soon and use the TLS capabilities. Upon one of our test servers, I discovered that the following filtering works just fine under syslog-ng v2.1 and blocks the messages from reaching the remote syslog-ng server: filter M_audit { not match("Audit daemon rotating log files"); }; filter M_snmp { not match("snmpd"); }; filter M_repeat { not match("last message repeated"); }; filter M_stats { not match("Log statistics"); }; log { source(s_localhost); filter(M_audit); filter(M_snmp); filter(M_repeat); filter(M_stats); destination(d_general); # Remote Syslog server }; However under syslog-ng v3.0.8, these messages are getting passed over to the remote syslog server. What do I need to do to strip these messages from getting passed through? I tried using "not message" but that doesn't seem to work. -- Michael ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

2 1

[patterndb] classification
by Balazs Scheidler 20 Sep '10

20 Sep '10

Hi, As you probably know one goal for patterndb is to implement message classification. E.g. in addition to extracting information from log messages, it also associates a "class", later available in the "${.classifier.class}" value. Right now, syslog-ng doesn't really care what this string is. But the XML schema validating patterndb file lists the following four classes (taken from the logcheck project) violation - security violation security - other security events system - system information unknown - no rule matches One one hand, the tagging functionality (e.g. the ability to also associate tags with each message) is superior to classes. On the other hand, all tags are equivalent, thus if a message has 5 tags, then currently syslog-ng only provides functions to _filter_ based on tags, but not use it as a macro. So for example it is possible to do: file d_class_files { file("/var/log/messages.${.classifier.class}.log"); }; But it is difficult to do with tags (except for using filters and different destinations), as there's no such functionality. Another problem is that tags/classes are completely independent, in order to filter on the class of the message, one would have to use a match() filter like this: filter f_class { match("violation" value(".classifier.class")); }; My conclusion is that classes are better when used in templates, tags are better when filtering. The two should be merged somehow. So I'm thinking on how to move forward. Here are the alternatives I'm considering: 1) the class of the message is always a tag in some generated format (e.g. if a message has class XXX, then a tag named ".class.XXX" would be automatically associated with the message. This is somewhat cumbersome. 2) the class of the message is created as a tag as well, with the same name as the class. e.g. we'd have a tag named "violation", but that'd preclude the use of the "violation" name as a tag. 3) drop the class stuff and implement a macro trick that makes it possible to use tags in macro context One way to do this: file d_class_files { file("/var/log/messages.$(expand-tag-name violation security system unknown).log"); }; The "expand-tag-name" macro function would try to look for the tags listed as parameters, and if the message matches it'd expand to the tagname. This is not intuitive and if someone wants to use such an expansion in a lot of templates, it is also irritating and difficult to get right. On an independent matter, the set of classes may need some thought. As I said the original list is borrowed from logcheck, but I think it probably needs to be expanded. Last time I got patterns for DNS queries, and although I could shove them into "system", right now I feel that the point of classification is to categorize events by "importance", in a similar spirit to syslog severity, but one that works even if the application developer uses a bogus severity when sending syslog messages. So one email, two questions, feedback appreciated. Thanks. -- Bazsi

4 17

sql driver does'nt work on RHEL5 syslog-ng 3.0.7
by LEBRETON Philippe 20 Sep '10

20 Sep '10

Hello i want to use mysl driver. My configuration file is : destination d_mysql_drv { sql(type(mysql) host("l1-154901l333sng") port("4058") username("root") password("admin") database("Syslog") table("logs") columns("host", "facility", "priority", "pid", "tag", "datetime", "program", "msg") values( '$HOST', '$FACILITY_NUM', '$LEVEL_NUM', '$PID', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' )); }; but in system log i have this error : Sep 8 15:54:01 154901l333sng syslog-ng[15030]: Error establishing SQL connection; type='mysql', host='l1-154901l333sng', port='4058', username='root', database='Syslog', error='2003: Can\'t connect to MySQL server on \'l1-154901l333sng\' (111)' on shell prompt, i use : mysql -u admin -p -h l1-154901l333sng -P 4058 Syslog an i can connect to database. Where is the problem? a bug ? a bad configuration? Help me. Philippe ***************************************************** "Le contenu de ce courriel et ses eventuelles pièces jointes sont confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas violer le secret des correspondances, vous ne devez pas le transmettre à d'autres personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire. Attention : L'Organisme de l'émetteur du message ne pourra être tenu responsable de l'altération du présent courriel. Il appartient au destinataire de vérifier que les messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles ne reflètent pas la position de l'Organisme sauf s'il en est disposé autrement dans le présent courriel." ******************************************************

5 9

Log statistics
by Stefan Jakobs 20 Sep '10

20 Sep '10

Hello list, I'm using syslog-ng 3.1.1 on a single user system. I'm getting these statistics in my logfile: syslog-ng[1653]: Log statistics; processed='destination(newsnotice)=0', processed='destination(firewall)=1', processed='destination(newscrit)=0', processed='src.internal(src#0)=2', stamp='src.internal(src#0)=1284066773', processed='center(queued)=0', processed='destination(cron)=15', processed='destination(acpid)=0', processed='destination(d_freshclam)=13', processed='destination(messages)=175', processed='destination(localmessages)=24', processed='global(payload_reallocs)=0', processed='global(msg_clones)=0', processed='destination(d_clamd)=36', processed='destination(console)=13', processed='destination(mail)=52', processed='destination(netmgm)=0', processed='destination(xconsole)=13', processed='destination(warn)=16', processed='destination(newserr)=0', processed='source(chroots)=10', processed='global(sdata_updates)=0', processed='destination(mailerr)=0', processed='center(received)=0', processed='destination(devnull)=0', processed='destination(mailwarn)=1', processed='source(src)=242', processed='destination(mailinfo)=2' I'm wondering why processed='center(received) and processed='center(queued) are zero. The documentation says: "The center(received) entry shows the total number of messages received from every configured sources." Any ideas? Is it still true that center(received)=sum(source(*))+sum(src.internal(*))? In this example: center(received)=src.internal(src#0)+source(src). And is this also right: center(queued)=sum(destination(*))? And what is the meaning of stamp='src.internal(src#0)=1284066773', processed='global(payload_reallocs)=0', processed='global(msg_clones)=0' and processed='global(sdata_updates)=0' These are a lot of questions, I know. Would be nice if someone can help out. Thanks and best regards Stefan

2 1

Re: [syslog-ng] syslog-ng under HP-UX processing unwanted "padded" data
by Scott Rochford 20 Sep '10

20 Sep '10

(resending as my first attempt didn't make it to the list, apologies if there are duplicates) > > If you still think it is, you might want to use tusc to check if > > syslog-ng reads /dev/log in 2048 byte chunks (which is should if > > pad_size() is enabled and which it doesn't if it is not). > > Most of the time it does, but sometimes it doesn't: > > 7197: read(4, "< 3 8 > S e p 7 1 0 : 5 2 ".., 8192) = 2048 > 7197: read(4, 0 <---- snip -----> > 7197: read(4, "\0\0\0\0\0< 3 8 > S e p 7 ".., 8192) = 6149 > 7197: read(4, "< 2 2 > S e p 7 1 0 : 5 5 ".., 8187) = 8187 > 7197: read(4, "u s . n e ", 8173) = 5 > 7197: read(4, 0 <---- snip -----> I have since tried the syslog-ng 1.6.12 package from the HP-UX Porting and Archive Centre again. This version is a little flaky (core dumps on SIGHUP or .conf syntax errors) however when configured correctly it does appear to read the /dev/log device with the correct padding. Therefore it would appear to be more recent enhancements that have changed this behaviour - although I assume at least some people are running syslog-ng v3 successfully under HP-UX. I tried to use gdb to determine whether the padding value in the LogProto object had the expected value of 2048, but... I'm not sure that's the right variable to look at, or even whether I'm doing it correctly, being a gdb newbie. Is there any more information I can provide or tests I can perform to help identify the cause, as I would prefer to use a more recent and hopefully stable version than 1.6 if possible? Regards, Scott

2 1

where does this mean in log status
by fedora fedora 20 Sep '10

20 Sep '10

I have a syslog-ng server running and i noticed in the stats info there are some big number of log gets dropped, see below. Log statistics; dropped='pipe(/dev/xconsole)=542760' Can someone tell me what exactly is /dev/xconsole? what it drops so many logs? Thanks FD

2 1

Re: [syslog-ng] Matching against an embedded @
by Peter Mills 17 Sep '10

17 Sep '10

I have just found the answer to my own question by trial and error: @@ Peter ________________________________ From: Peter Mills Sent: 17 September 2010 09:56 To: syslog-ng(a)lists.balabit.hu. Cc: Sam Moore Subject: Matching against an embedded @ Since @ is a special character, how do you match against strings containing an @? Do you escape them e.g. \@ in order to parse them correctly? I am trying to skip over/match the string [wx-event(a)juniper.net and have had no success. Thanks Peter root@nagios-collector:/opt/syslog-ng/var <mailto:root@nagios-collector:/opt/syslog-ng/var> # /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/db/patterns/v1/test2.xml -M "- - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime="1284641119" metric="Primary Reg Server Unreachable" sev="major" type="sys"] REG: Self registration failed. IP=192.168.61.242" Pattern matching part: - - WXC-@IPv4:.wxc.host_ip=192.168.61.202 <mailto:WXC-@IPv4:.wxc.host_ip=192.168.61.202> @ @STRING:.wxc.process=register@ @NUMBER:.wxc.pid=1102@ @ANYSTRING:rest=[wx-event@juniper.net eventtime=1284641119 metric=Primary@ Matching part: - - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime=1284641119 metric=Primary Values: MESSAGE=- - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime=1284641119 metric=Primary .classifier.class=system .classifier.rule_id=1 .wxc.host_ip=192.168.61.202 .wxc.process=register .wxc.pid=1102 rest=[wx-event(a)juniper.net eventtime=1284641119 metric=Primary Imtech Telecom Global is a limited company registered in England and Wales. Registered number: GB04407184. Registered office: Viables 3, Jays Close, Basingstoke, Hampshire, RG22 4BS. ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________

1 0

Matching against an embedded @
by Peter Mills 17 Sep '10

17 Sep '10

Since @ is a special character, how do you match against strings containing an @? Do you escape them e.g. \@ in order to parse them correctly? I am trying to skip over/match the string [wx-event(a)juniper.net and have had no success. Thanks Peter root@nagios-collector:/opt/syslog-ng/var <mailto:root@nagios-collector:/opt/syslog-ng/var> # /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/db/patterns/v1/test2.xml -M "- - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime="1284641119" metric="Primary Reg Server Unreachable" sev="major" type="sys"] REG: Self registration failed. IP=192.168.61.242" Pattern matching part: - - WXC-@IPv4:.wxc.host_ip=192.168.61.202 <mailto:WXC-@IPv4:.wxc.host_ip=192.168.61.202> @ @STRING:.wxc.process=register@ @NUMBER:.wxc.pid=1102@ @ANYSTRING:rest=[wx-event@juniper.net eventtime=1284641119 metric=Primary@ Matching part: - - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime=1284641119 metric=Primary Values: MESSAGE=- - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime=1284641119 metric=Primary .classifier.class=system .classifier.rule_id=1 .wxc.host_ip=192.168.61.202 .wxc.process=register .wxc.pid=1102 rest=[wx-event(a)juniper.net eventtime=1284641119 metric=Primary Imtech Telecom Global is a limited company registered in England and Wales. Registered number: GB04407184. Registered office: Viables 3, Jays Close, Basingstoke, Hampshire, RG22 4BS. ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________

1 0

Pattern Matching Issues
by Peter Mills 16 Sep '10

16 Sep '10

I am new to syslog-ng and just trying to get a basic version 1 XML pattern match file working properly. The output with an associated template is redirected to a file. It currently claims there is no such value. Does this mean that the pattern match has failed? Using syslog-ng (3.0.8) parser p_patterndb { db_parser(file("/opt/syslog-ng/var/db/patterns/v1/test.xml")); }; # Check pattern matching is working log { source(s_ext); parser(p_patterndb); <<<<< destination(df_udp_pattern_output); }; #Check pattern matching destination df_udp_pattern_output { file("/var/log/pattern_output" template("$PROGRAM,${rest},,$MSG\n") template_escape(no)); }; Output of: /opt/syslog-ng/sbin/syslog-ng -e -F -d -v > /tmp/syslog-ng.out 2>&1 Incoming log entry; line='<131>1 - - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime="1284626839" metric="Primary Reg Server Unreachable" sev="major" type="sys"] REG: Self registration failed. IP=192.168.61.242.' Filter rule evaluation begins; filter_rule='f_messages' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_messages' Filter rule evaluation begins; filter_rule='f_udp_nagios_check' Filter node evaluation result; filter_result='not-match' Filter rule evaluation result; filter_result='not-match', filter_rule='f_udp_nagios_check' Filter rule evaluation begins; filter_rule='wxc_all' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_all' Filter rule evaluation begins; filter_rule='wxc_tunnel_info' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_tunnel_info' Filter rule evaluation begins; filter_rule='wxc_tunnel_warn' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_tunnel_warn' Filter rule evaluation begins; filter_rule='wxc_tunnel_crit' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_tunnel_crit' Filter rule evaluation begins; filter_rule='wxc_license_crit' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='wxc_license_crit' No such value known; value='rest' <<<< root@nagios-collector:/opt/syslog-ng/var/db/patterns/v1 <mailto:root@nagios-collector:/opt/syslog-ng/var/db/patterns/v1> # cat test1.xml <?xml version="1.0" encoding="utf-8"?> <patterndb version="1" pub_date="2009-04-17"> <program name="1"> <pattern>wxc</pattern> <rule id="1" class="system"> <pattern>@ANYSTRING:rest@</pattern> </rule> </program> </patterndb> root@nagios-collector:/opt/syslog-ng/var/db/patterns/v1 <mailto:root@nagios-collector:/opt/syslog-ng/var/db/patterns/v1> # Thanks Peter Imtech Telecom Global is a limited company registered in England and Wales. Registered number: GB04407184. Registered office: Viables 3, Jays Close, Basingstoke, Hampshire, RG22 4BS. ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________

3 5

syslog-ng under HP-UX processing unwanted "padded" data
by Scott Rochford 16 Sep '10

16 Sep '10

Hi, I'm trying to test syslog-ng under HP-UX 11iv2 (both PA-RISC and IA64) to use as an alternative to the supplied syslogd. I have tried using the precompiled 3.0.5 binaries supplied with HP's DSAU (Distributed Systems Administration Utilities) pack, as well as compiling 3.0.5 and also version 3.1.2 from source. In all 3 cases I have the same problem; it seems to be attempting to process the padded data remaining in the 2048-byte chunk that is read from /dev/log, as you can see from the test below. In other words it does not stop processing the 2048-byte chunk of data when it reaches the first null byte. This means that the first log entry after startup is processed correctly, but then all sorts of random things start to happen, including blank (other than date stamp and host name) messages to every connected terminal. I used gcc-3.4.3, glib-2.24.2, popt 1.16, and pkgconfig 0.25 from the HP-UX Porting and Archive Centre to compile it. Have I overlooked something obvious here? I can't see any other similar reports on this mailing list or in the wild. I'd appreciate any suggestions! Regards, Scott. P.S. It's a little confusing that the HP-UX notes in the INSTALL file and the contrib/syslog-ng.conf.HP-UX files disagree with each other about the source syntax; will these be reconciled in a future release? # nohup /usr/local/sbin/syslog-ng --debug --foreground --stderr > syslog-ng.out 2>&1 & # logger -p user.notice "a user.notice test" # grep Incoming syslog-ng.out | head Incoming log entry; line='<13>Sep 6 05:16:52 syslog: a user.notice test' Incoming log entry; line='L\x84yD' Incoming log entry; line='\x01' Incoming log entry; line='\x01' Incoming log entry; line='\x01' Incoming log entry; line='\x09' Incoming log entry; line='\x01\x90T' Incoming log entry; line='\xd1\x9d' Incoming log entry; line='h\x0f\x1c,h\x0f\x1c\x98@' Incoming log entry; line='[Hh\x0dG\xc4h\x0dG\xc4\xc0\x1d?K\xc0\x1d?\x93' # /usr/local/sbin/syslog-ng -V syslog-ng 3.1.2 Installer-Version: 3.1.2 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.1#master#8bf13c304b6ab5fc1a372b49d55c78370efe14ca Compile-Date: Sep 6 2010 01:21:23 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: off # cat /usr/local/etc/syslog-ng.conf @version: 3.0 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on HP-UX. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(8) for more information. # # 20000925 gb(a)sysfive.com options { time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; # default contrib'd version # source s_sys { pipe("/dev/log"); internal(); }; # recommended by INSTALL source s_sys { pipe("/dev/log" pad_size(2048)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/adm/syslog/syslog.log"); }; destination d_mail { file("/var/adm/syslog/mail.log"); }; destination d_mlrt { usertty("root"); }; destination d_mlal { usertty("*"); }; filter f_filter1 { facility(mail) and level(debug); }; filter f_filter2 { (facility(mail) and level(debug)) or level(info); }; filter f_filter3 { level(alert); }; filter f_filter4 { level(emerg); }; log { source(s_sys); filter(f_filter1); destination(d_mail); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_cons); destination(d_mlrt); }; log { source(s_sys); filter(f_filter4); destination(d_mlal); }; #

2 5

rpm GPG keys for syslog-ng RPMs
by Tim Edwards 15 Sep '10

15 Sep '10

Hi, I've downloaded the syslog-ng RPMs from your site but can't find the GPG keys for RPM. I need these to put the RPMs in our local yum repository. Thanks Tim Edwards

2 1

Mass text log files processing over network.
by Tomasz Wrona 13 Sep '10

13 Sep '10

Hello, Could somebody point me if is it possible to configure syslog-ng to automatic, mass, text log files processing over network? Which version eventually support following case? My case: Webservers run virtual servers, which logs their own php error logs [text files]. I want to send this logs to central log server keeping orginal distribution based on virtual servers in the simplest way possible. I would like to have something like this virtual setup: #*** Client config ***# source s_php { # Wildcard match [only for commercial edition?] and TAG it somehow: file("/var/log/php/*-error.log" follow_freq(1) flags(no-parse) TAG("$FILENAME: ")); # eventually manual config for each file if above not possible: file("/var/log/php/service1-error.log" follow_freq(1) flags(no-parse) log_prefix("service1: ")); file("/var/log/php/service2-error.log" follow_freq(1) flags(no-parse) log_prefix("service2: ")); }; destination d_collector_php { udp("collector" port(5501)); }; log { source(s_php); destination(d_collector_php); }; #*** Collector Server config ***# source s_network_php { udp(ip(0.0.0.0) port(5501)); }; # Can I split log stream using MACRO based on custom information send in log [or syslog header]? destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$TAG.log"); }; # or maybe: destination d_php { file("/var/log/hosts/webservers/php/$HOST/$YEAR$MONTH$DAY/$log_prefix.log"); }; log { source(s_network_php); destination(d_php); }; Regards Tom

4 3

3.2 alpha 2 build error
by Matthew Hall 13 Sep '10

13 Sep '10

When building using debian/rules binary on 3.2 alpha 2 my build fails here. Any advice on this one? Thanks, Matthew. /bin/bash ../../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../src -I.. -DENABLE_SQL=1 -pthread -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I/usr/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -MT libafsql_la-afsql.lo -MD -MP -MF .deps/libafsql_la-afsql.Tpo -c -o libafsql_la-afsql.lo `test -f 'afsql.c' || echo './'`afsql.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../src -I.. -DENABLE_SQL=1 -pthread -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I/usr/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -MT libafsql_la-afsql.lo -MD -MP -MF .deps/libafsql_la-afsql.Tpo -c afsql.c -fPIC -DPIC -o .libs/libafsql_la-afsql.o afsql.c:120: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘dbi_instance’ afsql.c: In function ‘afsql_dd_insert_db’: afsql.c:583: warning: implicit declaration of function ‘dbi_conn_new_r’ afsql.c:583: error: ‘dbi_instance’ undeclared (first use in this function) afsql.c:583: error: (Each undeclared identifier is reported only once afsql.c:583: error: for each function it appears in.) afsql.c: In function ‘afsql_dd_init’: afsql.c:1007: warning: implicit declaration of function ‘dbi_initialize_r’ afsql.c:1007: error: ‘dbi_instance’ undeclared (first use in this function)

4 14

Re: [syslog-ng] Mass text log files processing over network.
by Tomasz Wrona 09 Sep '10

09 Sep '10

> From: Joe McDonagh <joseph.e.mcdonagh(a)gmail.com> > Probably easier to just pipe the configs to logger, since you can set > the facility and log level with it. This is what I do for my apache > stuff, pretty much a standard practice among the guys I know. > > -- > Joe McDonagh Joe, You are right, piping logs to logger works fine for apache logs, but AFAIK it doesn't work for php error_logs ( php_value error_log "|logger [..]" ). > From: Baron Von Pietri <barnaclebob(a)gmail.com> > If you do use the file() source driver then what i have done in a > similar situation is use the program() flag to set what destination > filename the log data should be in. then in the destination i can > just use file("/var/log/customstuff/$PROGRAM.log"). > > Hope that helps. > -- > Karl Thank You Karl! It works for me. So I use parameter log_prefix("service1: ") as equivalent for program_override() on client side and then $PROGRAM.log on server side. However, it is still needed to configure individually each log file [set "prefix/program"] on client side but server process it smart to dedicated files. Regards Tom

1 0

syslog-ng is not reading level/facility properly
by Charlie Reddington 08 Sep '10

08 Sep '10

Hi List, After my last weeks note, I kept on doing some more digging. Perhaps some of this output will aid in finding a solution. I compiled from source, from the latest stable 2.0 line thinking maybe that was part of my problem since I installed via the epel repositories. Nope, still having the same problem. On a remote host, I have a nginx server, that is sending it's access logs to local5.notice and it's error logs to local5.crit. In the syslog.conf file I have the following entries. local5.notice @syslog-ng server local5.crit @syslog-ng server This data is definitely making it to the systlog server. I have done a tcpdump and it shows this with the following. Also I have it logging to local log files and they are working the way I expect. $ tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 16:16:40.725473 IP CLIENTIP.syslog > SYSLOGNGIP.syslog: SYSLOG local5.notice, length: 304 <-- coming in as local5.notice as I expect. Okay so they are hitting the server how I expect, but they aren't being filtered the way I expect. Below is my syslog-ng config. # Syslog-ng config options { use_dns(no); keep_hostname(yes); long_hostnames(off); chain_hostnames(off); sync(1); log_fifo_size(1024); create_dirs(yes); perm(0644); stats_freq(3600); mark_freq(600); }; # # Data Sources # # External Syslog source s_external { udp(ip("0.0.0.0") port(514) flags(no_parse)); }; # # Where to put Data # # Nginx / Lighttpd / Apache Access Logs destination d_access_log { file("/data01/syslog-ng/$HOST/ access.log"); }; # Nginx / Lighttpd / Apache Error Logs destination d_error_log { file("/data01/syslog-ng/$HOST/error.log"); }; # Everything else destination d_log { file("/data01/syslog-ng/$HOST/$FACILITY.$LEVEL"); }; # # Data Filters # # Facility local5 filter f_local5 { facility(local5); }; # Level info filter f_info { level(info); }; # Level notice filter f_notice { level(notice); }; # Level warn filter f_warn { level(warn); }; # Level crit filter f_crit { level(crit); }; # Level err filter f_err { level(err); }; # # Data Logging Locations # # Access Logs log { source(s_external); filter(f_local5); filter(f_notice); destination(d_access_log); }; # Error Logs log { source(s_external); filter(f_local5); filter(f_crit); destination(d_error_log); }; # Everything Else log { source(s_external); destination(d_log); }; Now the only place that ANY messages seem to go, is into /data01/ syslog-ng/$HOST/kern.emerg. Why is everything being marked as kern.emerg? And lastly here's some debug output that maybe able to shed some light to those who know more than I do. ./sbin/syslog-ng -d -v -e --cfgfile=./etc/syslog-ng.conf Running application hooks; hook='1' syslog-ng starting up; version='2.0.11' Incoming log entry; line='<156>logger: REMOTEIP www.domain.com - [08/ Sep/2010:17:17:44 -0400] "GET /2010/09/08-7912 HTTP/1.0" 200 83134 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) "\x0a' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' For some reason this isn't matching, but I'm officially out of experience on this to know what to do further. I should also note that I'm running splunk on this syslog-ng server and it's able to get my access logs properly. If anyone can throw me a bone here I'd really appreciate it, many thanks in advance. Charlie

3 3

pdbtool parse with success but syslog-ng.conf NOT
by otgovorete＠gmail.com 03 Sep '10

03 Sep '10

Hi guys, I have following situation - I can parse with pdbtool bellow message successfully but when I parse it through syslog-ng.conf and put .classifier.class in DB I got "unknown" in the field where is placed .classifier.class or nothing when put the value FailedLogin_USERNAME in the table. Please see my syslog-ng.conf file below. Please help me to find the mistake. Thank you. Kosta Values: MESSAGE=Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2 PROGRAM=ssh .classifier.class=violation .classifier.rule_id=ssh-failed FailedLogin_MONTH=Sep FailedLogin_DATE=13 FailedLogin_TIME=17:34:00 FailedLogin_SERVER=server1 FailedLogin_SERVICE.ID2=sshd[20981]: FailedLogin_USERNAME=dfgdf FailedLogin_SOURCE_IP=xxxx FailedLogin_SOURCE.PORT=3602 kosta@Kostadin:~$ /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/login.parser.new.xml -P "ssh" -M "Sep 13 17:34:00 server1 sshd[20981]: Failed keyboard-interactive/pam for invalid user dfgdf from xxxx port 3602 ssh2" <rule provider='balabit' id='ssh-failed' class='violation'> <patterns> <pattern>@ESTRING:FailedLogin_MONTH: @@ESTRING:FailedLogin_DATE: @@ESTRING:FailedLogin_TIME: @@ESTRING:FailedLogin_SERVER: @@ESTRING:FailedL$ </patterns> </rule> ################ # DESTINATIONS # ################ destination d_mssql_unix { sql(type(mssql) host("medea.mobiltel.bg") port("1433") username("syslog_ng") password("nglogp@ss") database("SysLog") table("SYSLOG_unix1")columns("Date varchar(40)", "Time varchar(16)", "SourceServer varchar(8)", "SendingProgram varchar(5)", "PID varchar(6)$ values("$DATE_", "$TIME_", "$HOST", "$PROGRAM", "$PID", "${.classifier.class}")); }; ############### # PARSER # ############### parser pattern_db { db_parser(file("/opt/syslog-ng/var/login.parser.new.xml")); }; parser DateTime { csv-parser(columns("DATE_", "TIME_") delimiters("T") flags(escape-none) template("${ISODATE}")); }; ############### # LOG # ############### log { source(s_net); parser (DateTime); parser(pattern_db); destination(d_mssql_unix); destination(d_messages_successful); };

4 3

Re: [syslog-ng] [PATCH] remove auto-generated files
by chris packham 03 Sep '10

03 Sep '10

On Sat, 2010-09-04 at 04:23 +1200, chris packham wrote: > On Fri, 2010-09-03 at 13:24 +0200, Balazs Scheidler wrote: > > On Fri, 2010-09-03 at 06:09 +1200, chris packham wrote: > > > INSTALL is generated by the build process. If the build host has a different > > > version of autotools then the contents of INSTALL can change which in turn > > > causes problems for developers working from the git repository. > > > > > > This patch removes the INSTALL file and adds it to .gitignore. > > > > So, in your case autotools generates an INSTALL file? That's funny, this > > is not the same as the autotools generated one (in fact it is a bit > > outdated, but I wrote that to document how to install syslog-ng). > > > > For me automake doesn't generate this file. Why does it for you? > > > > autogen.sh contains: > > > > > > aclocal > > sed -i -e 's/PKG_PROG_PKG_CONFIG(\[0\.16\])/PKG_PROG_PKG_CONFIG([0.14])/g' aclocal.m4 > > > > autoheader > > automake --foreign --add-missing > > autoconf > > > > > > as I understand automake will only be adding the INSTALL file if it doesn't exist. > > Are you using -f for automake? > > > > Actually our problem might be this > > $(call MESSAGE,"Running autogen.sh") > $(Q)cd $(SYSLOG_NG_GIT_DIR) && $(AUTORECONF) 2> /dev/null > $(Q)touch $@ > > So we're lying to our developers for a start. If I do run autogen.sh > instead I get this error > > configure.in:513: warning: macro `AM_PATH_GLIB_2_0' not found in library > configure.in:513: error: possibly undefined macro: AM_PATH_GLIB_2_0 > > which is probably the reason we didn't just use autogen.sh in the first > place. > > Looking at the expanded $(AUTORECONF) it expands to "/usr/bin/autoreconf > -f -i" (plus a lot of environment vars for our cross-compile > environment). I'll try fiddling with these options to see if some of > them changes the behaviour. Sure enough, removing the -f option stops it from overwriting INSTALL.

1 0

Solving parsing problem
by otgovorete＠gmail.com 03 Sep '10

03 Sep '10

Hi guys, I have done parser for the authentication logs of HPUX. Here is the situation: A)Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 B)Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 The parser i've made parses successfully A but not B. The problem is extra whitespace when the date is one digit. I am testing parsing with pdbtool of syslog-ng. The outcome of parsing of A is: PROGRAM=ssh .classifier.class=legitimate .classifier.rule_id=ssh-succeed SucceedLogin_MONTH=Sep SucceedLogin_DATE=22 SucceedLogin_TIME=13:14:24 SucceedLogin_SERVER=serverone SucceedLogin_SERVICE.ID=sshd[12934]: SucceedLogin_USER_NAME=username SucceedLogin_DESTINATION=xxxx SucceedLogin_SOURCE.PORT=1691 The outcome of parsing if B is: Matching part: Sep 2 13:14:24 serverone Values: MESSAGE=Sep 2 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2 PROGRAM=ssh .classifier.class=unknown Here is the parser from xml file: <patterns> <pattern>@ESTRING:SucceedLogin_MONTH: @@ESTRING:SucceedLogin_DATE: @@ESTRING:SucceedLogin_TIME: @@ESTRING:SucceedLogin_SERVER: @@ESTRING:SucceedLogin_SERVICE.ID: @Accepted keyboard-interactive/pam for @ESTRING:SucceedLogin_USER_NAME: @from @ESTRING:SucceedLogin_DESTINATION: @port @ESTRING:SucceedLogin_SOURCE.PORT: @ssh2</pattern> </patterns> and here is the command pdbtool: /opt/syslog-ng/bin/pdbtool match -D -c -p login.parser.new.xml -P "ssh" -M "Sep 22 13:14:24 serverone sshd[12934]: Accepted keyboard-interactive/pam for username from xxxx port 1691 ssh2" Please, advice me how to proceed. Thank you. Kosta

5 5

Re: [syslog-ng] [PATCH] remove auto-generated files
by chris packham 03 Sep '10

03 Sep '10

On Fri, 2010-09-03 at 13:24 +0200, Balazs Scheidler wrote: > On Fri, 2010-09-03 at 06:09 +1200, chris packham wrote: > > INSTALL is generated by the build process. If the build host has a different > > version of autotools then the contents of INSTALL can change which in turn > > causes problems for developers working from the git repository. > > > > This patch removes the INSTALL file and adds it to .gitignore. > > So, in your case autotools generates an INSTALL file? That's funny, this > is not the same as the autotools generated one (in fact it is a bit > outdated, but I wrote that to document how to install syslog-ng). > > For me automake doesn't generate this file. Why does it for you? > > autogen.sh contains: > > > aclocal > sed -i -e 's/PKG_PROG_PKG_CONFIG(\[0\.16\])/PKG_PROG_PKG_CONFIG([0.14])/g' aclocal.m4 > > autoheader > automake --foreign --add-missing > autoconf > > > as I understand automake will only be adding the INSTALL file if it doesn't exist. > Are you using -f for automake? > Actually our problem might be this $(call MESSAGE,"Running autogen.sh") $(Q)cd $(SYSLOG_NG_GIT_DIR) && $(AUTORECONF) 2> /dev/null $(Q)touch $@ So we're lying to our developers for a start. If I do run autogen.sh instead I get this error configure.in:513: warning: macro `AM_PATH_GLIB_2_0' not found in library configure.in:513: error: possibly undefined macro: AM_PATH_GLIB_2_0 which is probably the reason we didn't just use autogen.sh in the first place. Looking at the expanded $(AUTORECONF) it expands to "/usr/bin/autoreconf -f -i" (plus a lot of environment vars for our cross-compile environment). I'll try fiddling with these options to see if some of them changes the behaviour.

1 0

syslog-ng on Solaris 10
by Riaz, Javed (WW Global Network Services) 03 Sep '10

03 Sep '10

This the first time I am running syslog-ng 3.0.4 on Solaris 10. Following are the issues I am seeing: 1. Two processes of syslog-ng are running. Is this normal: root 15423 1 0 Aug 31 ? 0:00 /usr/local/sbin/syslog-ng root 15424 15423 0 Aug 31 ? 0:00 /usr/local/sbin/syslog-ng 2. I need to disable/enable syslog-ng multiple times to get syslog-ng online in Solaris SVCADM after updating conf file. 3. Also if I disable syslog-ng in Solaris SVCADM should the tw0 processes stop? Please help! # /usr/local/sbin/syslog-ng -V syslog-ng 3.0.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 Compile-Date: Sep 2 2009 06:15:53 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on # uname -a SunOS usplsbavisp007 5.10 Generic_118833-33 sun4u sparc SUNW,Sun-Fire-V445 Thanks,

2 1

feature request (parallel processing)
by syslogng＠feystorm.net 03 Sep '10

03 Sep '10

So, after months of work, we finally turned on our production environment for syslog collection. However, we hit one immediate snag. Currently were writing to the database, and the way the database works is that it collects enough data to fill a single block, and then it flushes out that block. Well every time it goes to flush the block out, the insert takes an extra couple milliseconds. Now when I'm doing about 220000 inserts a second, that millisecond delay is significant. So basically syslog has to pause on that log statement while it waits for the database to flush. (1 out of 10 messages was getting dropped) Now I tried to solve this by writing multiple destination drivers so that a second database thread could be processing while the first was flushing, but that didnt work as it appears syslog waits for the destination driver to complete before it hands data off to the second driver. Instead I managed to solve the problem by creating yet more syslog processes. So basically the master process listens for data from all the hosts. It then runs a match on the $PID and sends all even numbered PIDs to one syslog process, and all odd numbered PIDs to a second syslog process. This way both processes can be inserting to the database at the same time. It effectively cuts the amount of work each database thread does in half, so that when it has to pause to flush, it doesnt cause the syslog buffer to fill up. Ultimately my request is this, allow multiple destination drivers to work at the same time. I realize this is probably not a simple change, but seems like it would be a significant speed enhancement.

4 7

logic and duplicate suppression
by John Kristoff 03 Sep '10

03 Sep '10

I have a couple of scenarios where I'm looking to enhance how I handle and process some logs. I'm looking for suggestions on what my options are, but maybe these are potential feature requests? 1. In using a parser (cvs or the patterndb), I'd like to use some conditionals based on a resultant macro value. So for example, if I have an sshd authentication log message with a source address in a macro and that address is contained w/in a specific prefix, I'd like to handle that message differently. Perhaps not log it all or set another MACRO to a certain value. 2. I'd like to be able to suppress duplicate messages even if they are not necessarily contiguous at the destination. So for example, if I have a SSH client that generates a log of its SSH client protocol and software, I don't need to see that over and over again (e.g. as you might commonly see today in SSH brute force attacks). John

4 10