- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
by Worsham, Michael 29 Sep '10

29 Sep '10

I must be doing something wrong or the syntax is really dependant on the program name now. Orginally under the v2.1 build, the intent of the filter was to strip any incoming message going to the syslog-ng server from the data being sent from the syslog-ng client. There was no one specific program attached to the incoming data. It could come from the client's syslog-ng side via logger command, apache data, kernel, etc. Server-side syslog-ng.conf filter section: filter M_audit { not match("Auditing" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_repeat { not match("Repeating" value("last message repeated") flags(ignore-case)); }; filter M_stats { not match("Stats" value("Log statistics") flags(ignore-case)); }; On the client side, I would just do 'logger daemon' to send over the message to the syslog-ng server. On the server side, running syslog-ng -d reveals this output: Incoming log entry; line='<13>Sep 20 17:16:31 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='Audit daemon rotating log files' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_audit' Filter rule evaluation begins; filter_rule='M_repeat' No such value known; value='last message repeated' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_repeat' Filter rule evaluation begins; filter_rule='M_stats' No such value known; value='Log statistics' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_stats' Server-side syslog output being seen in the data file: Sep 20 17:16:31 drupal root: daemon ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

5 18

pub_date
by Peter Czanik 27 Sep '10

27 Sep '10

Hello, All of the pdb files in git have pub_date set to 2010-07-13. Should I also use this date, or it's just a coincidence and I can use the actual date? Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

[Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent
by bugzilla＠bugzilla.balabit.com 23 Sep '10

23 Sep '10

https://bugzilla.balabit.com/show_bug.cgi?id=93 Summary: filter() functionality between 2.1 to 3.0 not consistent Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: mworsham(a)scires.com Type of the Report: bug Estimated Hours: 0.0 In the older syslong-ng v2.1, this line works perfectly: filter M_audit { not match("Audit daemon rotating log files"); }; Under 3.0.8, none of the following are working (if added one line at a time) and the daemon restarted: filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY) flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files")); }; What I am looking to do is if any incoming data has the following keywords (i.e. "Audit daemon rotating log files") being detected, it should be filtered (i.e. dropped), and then not show up in the actual message log file. For example, if I go over on the syslog-ng client and do 'logger daemon': - The older v2.1 syslog-ng server detects the embedded keyword and drops the message. <-- This is correct - On the v3.0.8, the message passes through and found in the data file. <-- This is incorrect Attempted to use the following URLs to correct the formatting, but it still not working: - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui… - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui… -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

5 10

Converting filtering from 2.1 to 3.0?
by Worsham, Michael 20 Sep '10

20 Sep '10

We are running syslog-ng v2.1 in a multiple client-server configuration. Due to new encryption requirements, we planning to upgrade to v3.0.8 soon and use the TLS capabilities. Upon one of our test servers, I discovered that the following filtering works just fine under syslog-ng v2.1 and blocks the messages from reaching the remote syslog-ng server: filter M_audit { not match("Audit daemon rotating log files"); }; filter M_snmp { not match("snmpd"); }; filter M_repeat { not match("last message repeated"); }; filter M_stats { not match("Log statistics"); }; log { source(s_localhost); filter(M_audit); filter(M_snmp); filter(M_repeat); filter(M_stats); destination(d_general); # Remote Syslog server }; However under syslog-ng v3.0.8, these messages are getting passed over to the remote syslog server. What do I need to do to strip these messages from getting passed through? I tried using "not message" but that doesn't seem to work. -- Michael ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

2 1

[patterndb] classification
by Balazs Scheidler 20 Sep '10

20 Sep '10

Hi, As you probably know one goal for patterndb is to implement message classification. E.g. in addition to extracting information from log messages, it also associates a "class", later available in the "${.classifier.class}" value. Right now, syslog-ng doesn't really care what this string is. But the XML schema validating patterndb file lists the following four classes (taken from the logcheck project) violation - security violation security - other security events system - system information unknown - no rule matches One one hand, the tagging functionality (e.g. the ability to also associate tags with each message) is superior to classes. On the other hand, all tags are equivalent, thus if a message has 5 tags, then currently syslog-ng only provides functions to _filter_ based on tags, but not use it as a macro. So for example it is possible to do: file d_class_files { file("/var/log/messages.${.classifier.class}.log"); }; But it is difficult to do with tags (except for using filters and different destinations), as there's no such functionality. Another problem is that tags/classes are completely independent, in order to filter on the class of the message, one would have to use a match() filter like this: filter f_class { match("violation" value(".classifier.class")); }; My conclusion is that classes are better when used in templates, tags are better when filtering. The two should be merged somehow. So I'm thinking on how to move forward. Here are the alternatives I'm considering: 1) the class of the message is always a tag in some generated format (e.g. if a message has class XXX, then a tag named ".class.XXX" would be automatically associated with the message. This is somewhat cumbersome. 2) the class of the message is created as a tag as well, with the same name as the class. e.g. we'd have a tag named "violation", but that'd preclude the use of the "violation" name as a tag. 3) drop the class stuff and implement a macro trick that makes it possible to use tags in macro context One way to do this: file d_class_files { file("/var/log/messages.$(expand-tag-name violation security system unknown).log"); }; The "expand-tag-name" macro function would try to look for the tags listed as parameters, and if the message matches it'd expand to the tagname. This is not intuitive and if someone wants to use such an expansion in a lot of templates, it is also irritating and difficult to get right. On an independent matter, the set of classes may need some thought. As I said the original list is borrowed from logcheck, but I think it probably needs to be expanded. Last time I got patterns for DNS queries, and although I could shove them into "system", right now I feel that the point of classification is to categorize events by "importance", in a similar spirit to syslog severity, but one that works even if the application developer uses a bogus severity when sending syslog messages. So one email, two questions, feedback appreciated. Thanks. -- Bazsi

4 17

sql driver does'nt work on RHEL5 syslog-ng 3.0.7
by LEBRETON Philippe 20 Sep '10

20 Sep '10

Hello i want to use mysl driver. My configuration file is : destination d_mysql_drv { sql(type(mysql) host("l1-154901l333sng") port("4058") username("root") password("admin") database("Syslog") table("logs") columns("host", "facility", "priority", "pid", "tag", "datetime", "program", "msg") values( '$HOST', '$FACILITY_NUM', '$LEVEL_NUM', '$PID', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' )); }; but in system log i have this error : Sep 8 15:54:01 154901l333sng syslog-ng[15030]: Error establishing SQL connection; type='mysql', host='l1-154901l333sng', port='4058', username='root', database='Syslog', error='2003: Can\'t connect to MySQL server on \'l1-154901l333sng\' (111)' on shell prompt, i use : mysql -u admin -p -h l1-154901l333sng -P 4058 Syslog an i can connect to database. Where is the problem? a bug ? a bad configuration? Help me. Philippe ***************************************************** "Le contenu de ce courriel et ses eventuelles pièces jointes sont confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas violer le secret des correspondances, vous ne devez pas le transmettre à d'autres personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire. Attention : L'Organisme de l'émetteur du message ne pourra être tenu responsable de l'altération du présent courriel. Il appartient au destinataire de vérifier que les messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles ne reflètent pas la position de l'Organisme sauf s'il en est disposé autrement dans le présent courriel." ******************************************************

5 9

Log statistics
by Stefan Jakobs 20 Sep '10

20 Sep '10

Hello list, I'm using syslog-ng 3.1.1 on a single user system. I'm getting these statistics in my logfile: syslog-ng[1653]: Log statistics; processed='destination(newsnotice)=0', processed='destination(firewall)=1', processed='destination(newscrit)=0', processed='src.internal(src#0)=2', stamp='src.internal(src#0)=1284066773', processed='center(queued)=0', processed='destination(cron)=15', processed='destination(acpid)=0', processed='destination(d_freshclam)=13', processed='destination(messages)=175', processed='destination(localmessages)=24', processed='global(payload_reallocs)=0', processed='global(msg_clones)=0', processed='destination(d_clamd)=36', processed='destination(console)=13', processed='destination(mail)=52', processed='destination(netmgm)=0', processed='destination(xconsole)=13', processed='destination(warn)=16', processed='destination(newserr)=0', processed='source(chroots)=10', processed='global(sdata_updates)=0', processed='destination(mailerr)=0', processed='center(received)=0', processed='destination(devnull)=0', processed='destination(mailwarn)=1', processed='source(src)=242', processed='destination(mailinfo)=2' I'm wondering why processed='center(received) and processed='center(queued) are zero. The documentation says: "The center(received) entry shows the total number of messages received from every configured sources." Any ideas? Is it still true that center(received)=sum(source(*))+sum(src.internal(*))? In this example: center(received)=src.internal(src#0)+source(src). And is this also right: center(queued)=sum(destination(*))? And what is the meaning of stamp='src.internal(src#0)=1284066773', processed='global(payload_reallocs)=0', processed='global(msg_clones)=0' and processed='global(sdata_updates)=0' These are a lot of questions, I know. Would be nice if someone can help out. Thanks and best regards Stefan

2 1

Re: [syslog-ng] syslog-ng under HP-UX processing unwanted "padded" data
by Scott Rochford 20 Sep '10

20 Sep '10

(resending as my first attempt didn't make it to the list, apologies if there are duplicates) > > If you still think it is, you might want to use tusc to check if > > syslog-ng reads /dev/log in 2048 byte chunks (which is should if > > pad_size() is enabled and which it doesn't if it is not). > > Most of the time it does, but sometimes it doesn't: > > 7197: read(4, "< 3 8 > S e p 7 1 0 : 5 2 ".., 8192) = 2048 > 7197: read(4, 0 <---- snip -----> > 7197: read(4, "\0\0\0\0\0< 3 8 > S e p 7 ".., 8192) = 6149 > 7197: read(4, "< 2 2 > S e p 7 1 0 : 5 5 ".., 8187) = 8187 > 7197: read(4, "u s . n e ", 8173) = 5 > 7197: read(4, 0 <---- snip -----> I have since tried the syslog-ng 1.6.12 package from the HP-UX Porting and Archive Centre again. This version is a little flaky (core dumps on SIGHUP or .conf syntax errors) however when configured correctly it does appear to read the /dev/log device with the correct padding. Therefore it would appear to be more recent enhancements that have changed this behaviour - although I assume at least some people are running syslog-ng v3 successfully under HP-UX. I tried to use gdb to determine whether the padding value in the LogProto object had the expected value of 2048, but... I'm not sure that's the right variable to look at, or even whether I'm doing it correctly, being a gdb newbie. Is there any more information I can provide or tests I can perform to help identify the cause, as I would prefer to use a more recent and hopefully stable version than 1.6 if possible? Regards, Scott

2 1

where does this mean in log status
by fedora fedora 20 Sep '10

20 Sep '10

I have a syslog-ng server running and i noticed in the stats info there are some big number of log gets dropped, see below. Log statistics; dropped='pipe(/dev/xconsole)=542760' Can someone tell me what exactly is /dev/xconsole? what it drops so many logs? Thanks FD

2 1

Re: [syslog-ng] Matching against an embedded @
by Peter Mills 17 Sep '10

17 Sep '10

I have just found the answer to my own question by trial and error: @@ Peter ________________________________ From: Peter Mills Sent: 17 September 2010 09:56 To: syslog-ng(a)lists.balabit.hu. Cc: Sam Moore Subject: Matching against an embedded @ Since @ is a special character, how do you match against strings containing an @? Do you escape them e.g. \@ in order to parse them correctly? I am trying to skip over/match the string [wx-event(a)juniper.net and have had no success. Thanks Peter root@nagios-collector:/opt/syslog-ng/var <mailto:root@nagios-collector:/opt/syslog-ng/var> # /opt/syslog-ng/bin/pdbtool match -D -c -p /opt/syslog-ng/var/db/patterns/v1/test2.xml -M "- - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime="1284641119" metric="Primary Reg Server Unreachable" sev="major" type="sys"] REG: Self registration failed. IP=192.168.61.242" Pattern matching part: - - WXC-@IPv4:.wxc.host_ip=192.168.61.202 <mailto:WXC-@IPv4:.wxc.host_ip=192.168.61.202> @ @STRING:.wxc.process=register@ @NUMBER:.wxc.pid=1102@ @ANYSTRING:rest=[wx-event@juniper.net eventtime=1284641119 metric=Primary@ Matching part: - - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime=1284641119 metric=Primary Values: MESSAGE=- - WXC-192.168.61.202 register 1102 [wx-event(a)juniper.net eventtime=1284641119 metric=Primary .classifier.class=system .classifier.rule_id=1 .wxc.host_ip=192.168.61.202 .wxc.process=register .wxc.pid=1102 rest=[wx-event(a)juniper.net eventtime=1284641119 metric=Primary Imtech Telecom Global is a limited company registered in England and Wales. Registered number: GB04407184. Registered office: Viables 3, Jays Close, Basingstoke, Hampshire, RG22 4BS. ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________

1 0