- syslog-ng - lists.balabit.hu

Cisco ASA logging to syslog-ng and a weird extra-characters in new-lines
by Finnur Orn Gudmundsson 16 Aug '13

16 Aug '13

Hi all, I have a syslog-ng server: rpm -qi syslog-ng Name : syslog-ng Relocations: (not relocatable) Version : 3.2.5 Vendor: Fedora Project Release : 3.el6 Build Date: Sun 15 Jan 2012 07:49:04 PM GMT Install Date: Wed 06 Mar 2013 03:56:17 PM GMT Build Host: x86-14.phx2.fedoraproject.org Group : System Environment/Daemons Source RPM: syslog-ng-3.2.5-3.el6.src.rpm Size : 1594638 License: GPLv2+ Signature : RSA/8, Sun 15 Jan 2012 08:55:15 PM GMT, Key ID 3b49df2a0608b895 The syslog-ng package installed is built and distributed by Fedora EPEL but I was hoping I could post here. I replaced a old syslog-ng 2.x server to this one and almost everything works as expected (there is always this one thing that does not work after a migration:). There is a Cisco ASA box logging to this server. For some records I get this extra space in a new-line like this: Aug 16 15:35:56 10.X.X.X %ASA-6-302021: Teardown ICMP connection for faddr 10.X.X.X/9483 gaddr 10.X.X.X/0 laddr 10.X.X.X/0 Aug 16 15:35:56 10.X.X.X %ASA-5-304001: 10.X.X.X Accessed URL 46.X.X.X:/ < Aug 16 15:35:56 10.X.X.X %ASA-5-304001: 10.X.X.X Accessed URL 92.X.X.X:http://sphotos-g.ak.fbcdn.net/hphotos-X/XxX/11X_221X18801_1X554_n.… Aug 16 15:35:56 10.X.X.X %ASA-6-305011: Built dynamic TCP translation from inside:10.X.X.X/38697 to outside:194.X.X.X/38697 (the < char is in a new-line) And another one: Aug 16 15:35:56 10.Z.Z.Z %ASA-6-302013: Built outbound TCP connection 1809896329 for outside:54.Z.Z.Z/80 (54.Z.Z.Z/80) to inside:10.Z.Z.Z/38684 (194.Z.Z.Z/38684) Aug 16 15:35:56 10.Z.Z.Z %ASA-5-304001: 10.Z.Z.Z Accessed URL 10.Z.Z.Z:/ 1 Aug 16 15:35:56 10.Z.Z.Z %ASA-6-302014: Teardown TCP connection 1809896328 for dmz-lb-int:10.Z.Z.Z/58759 to inside:10.Z.Z.Z/80 duration 0:00:00 bytes 162 TCP FINs (the single 1 char is in a new-line). Does someone know if this is a known bug ? I am aware that I am running a somewhat old version. Bgrds, Finnzi

1 1

insider 2013-08: New releases; GSoC updates; Dedicated OSE developer; Observe, Hack, Make 2013;
by Peter Czanik 15 Aug '13

15 Aug '13

Dear syslog-ng users, This is the 26th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.4.3 and 3.3.11 are released --------------------------------------- Both the current (3.4) and previous (3.3) stable syslog-ng releases received maintenance updates. Sources are available from git or as tgz from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… For third-party binary packages for various Linux and UNIX platforms, visithttp://www.balabit.com/network-security/syslog-ng/opensource-logging-s… for 3^rd party binary packages for various distributions and UNIX variants. syslog-ng Agent for Windows 5.0 ------------------------------- There is a major new version of syslog-ng Agent available, version 5.0. Under the hood it builds now on the same code base as the UNIX version of syslog-ng. The new highlights include support for the Reliable Log Transfer Protocol (RLTP), flow control and support for the Windows Server 2012 and Windows 8 platforms. Long-time syslog-ng contributor Gergely Nagy now working full-time on syslog-ng Open Source Edition ----------------------------- If you've been following the mailing list, the community around syslog-ng or the commits in git you most certainly recognize the name of Gergely Nagy. He's been working at BalaBit for several years and we're happy to announce that from now on his dedicated role is to work on syslog-ng OSE full-time. You can read his thoughts about the change on his blog: https://algernon.blogs.balabit.com/2013/07/onto-greener-open-pastures/ syslog-ng GSoC: it's half-time! ------------------------------- The coding for Google Summer of Code is at half-time. Our students passed half-time evaluations and are working hard on a faster MySQL destination and on a brand new redis destination. You can follow their progress from their related blogs: http://petrovicsgyula.blogspot.com/ http://tichygsoc.blogspot.hu/ There is also a plan to provide test packages for those really adventurous in the coming weeks :-) syslog-ng JSON HowTo -------------------- JSON is gaining popularity, not just in Web 2.0 but in all fields of IT. As it's an easy way of storing and transmitting name value pairs, JSON emitting and parsing is now part of syslog-ng. The HowTo describing how it works in syslog-ng is available at http://asylum.madhouse-project.org/blog/2013/07/29/json-howto/ syslog-ng developers at Observe, Hack, Make 2013 ------------------------------------------------ Laser harp, Wikileaks, lockpicking, NSA, SIM card exploiting, LHC, counter-cryptanalysis, 24 pull requests, neutrino detectors, open source Bach, weird lights, quadrocopters, electric music cranked up to 11, workshops, lectures, discussions, and a lot more! We've been Observing, Hacking and Making! You can read more about it at https://athos.blogs.balabit.com/2013/08/observed-hacked-made/ SHORT NEWS: * CzP presenting log management and syslog-ng at Libre Software Meeting in Brussels: https://czanik.blogs.balabit.com/2013/08/czp-libre-software-meeting/ * What is Pseudonymous Data? An update on how upcoming EU regulations affect logging: https://jluby.blogs.balabit.com/2013/07/18/what-is-pseudonymous-data-in-the… NEW RELEASES * syslog-ng 3.3.11: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-June/000163.html * syslog-ng 3.4.3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-August/000164.ht… * syslog-ng Agent 5.0.1: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-May/000158.html * syslog-ng PE 4.0.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-August/000165.ht… * syslog-ng PE 4.2.4: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-June/000160.html Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

syslog-ng 3.4.3 has been released
by devel＠balabit.hu 13 Aug '13

13 Aug '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.3 SUMMARY : new stable release DATE : Aug 13, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.3) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.3 Tue, 13 Aug 2013 13:00:00 +0100 This is the second bug-fix release for the 3.4.x series, collecting significant fixes over the 3.4.2 release. Upgrading to this release is highly recommended. Bugfixes ======== * The pacct module has been fixed, and now compiles correctly. * Updated the configure.in script, and a few other places to fix issues with recent autotools. * Fixed an infinite loop when template expansion resulted in a zero-length string. * A considerable number of minor issues surrounding the build system have been eliminated. * The template_escape() setting is now respected by template functions that create templates themselves (such as the conditional functions) * Restore support for defaultless block{} arguments. [#230] * RFC5424 NILVALUE is now accepted in timestamps [#238] * The JSON parser will throw an error when the input is not an object (but, for example, a string) instead of crashing. * The program() source has been fixed, and now does not throw a syntax error at config parsing time, but works correctly again. * In case resolving a hostname fails, the IP address does not get truncated anymore. [#240 & #242] * A race condition in certain filter operations have been eliminated. Features ======== * The JSON parser now accepts generic parser related options, such as template(), so one can easily set the input for the parser. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andras Tim <tia(a)balabit.hu> Anton Koldaev <koldaevav(a)gmail.com> Attila Szalay <sasa(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Martin <bmartin(a)lavabit.com> Michael Sterrett <mr_bones_(a)gentoo.org> Vincent Brillault <spam(a)lerya.net> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 2

[Bug 238] New: syslog-protocol: allow NILVALUE for TIMESTAMP as specified in rfc5424
by bugzilla＠bugzilla.balabit.com 13 Aug '13

13 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=238 Summary: syslog-protocol: allow NILVALUE for TIMESTAMP as specified in rfc5424 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: spam(a)lerya.net Type of the Report: --- Estimated Hours: 0.0 RFC5424 states: - section 6: """ TIMESTAMP = NILVALUE / FULL-DATE "T" FULL-TIME """ - section 6.2.3 "" A syslog application MUST use the NILVALUE as TIMESTAMP if the syslog application is incapable of obtaining system time. "" But the 3.4 version of syslog-ng does not allow such a NILVALUE: https://github.com/balabit/syslog-ng-3.4/blob/master/modules/syslogformat/s… For example, the following message does not work: <6>1 - - netlog - - - [ 775.895618] /bin/ping[2936] UDP 10.0.2.15:33285 -> 192.168.0.1:53 (uid=0) Whereas this one works: <6>1 2003-10-11T22:14:15.003Z - netlog - - - [ 775.895618] /bin/ping[2936] UDP 10.0.2.15:33285 -> 192.168.0.1:53 (uid=0) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 13

[Bug 240] New: host() in filters fail to match if using use_dns(yes)
by bugzilla＠bugzilla.balabit.com 13 Aug '13

13 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=240 Summary: host() in filters fail to match if using use_dns(yes) Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bmartin(a)lavabit.com Type of the Report: bug Estimated Hours: 0.0 After upgrading from version 3.2.5 to 3.4.2 a filter using host() did not work anymore. The filter looks something like this: filter f_filter { host("172.12.0.1") or host("172.12.0.2"); }; Messages which would match this filter never reached their destination as they were used to with 3.2.5. I figured out that $HOST contained only the string '172'. My assumption is that host() matches against $HOST. In turn I set use_dns(no) and things started working again and now $HOST contains the whole IP address. used global options: options { threaded(yes); chain_hostnames(no); use_dns(no); stats_freq(43200); mark_freq(3600); }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 242] New: IP address stripped to first number with use_fqdn off
by bugzilla＠bugzilla.balabit.com 13 Aug '13

13 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=242 Summary: IP address stripped to first number with use_fqdn off Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: list0570(a)paradise.net.nz Type of the Report: bug Estimated Hours: 0.0 transport("udp") ip("0.0.0.0") port(514) ip-protocol(4) keep_timestamp(no) keep_hostname(no) # no = rewrite hostname use_dns(yes) # yes, no, persist_only use_fqdn(no) Regardless of what the use_dns setting is, if the host name is an IP address A.B.C.D, use_fqdn is no, and /etc/hosts has no name and the DNS lookup fails, then only "A" is logged as the host name. This is not useful at all - if a name lookup fails for an IP address, you have to keep the IP address and not truncate it to something useless. syslog-ng-3.4.2-1.5.1.i586 from openSUSE 12.3 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 243] New: patterndb rule should allow action upon context timeout
by bugzilla＠bugzilla.balabit.com 09 Aug '13

09 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=243 Summary: patterndb rule should allow action upon context timeout Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: enhancement Estimated Hours: 0.0 Currently in an event correlation configuration, when a certain context times out, the whole rule is thrown away. It would be very useful to be able to trigger an action when a rule's context-timeout is reached. Of course the ability to add a timeout_action for every rule would further enhance the idea. Example: <ruleset name='dummy_ruleset' id='03eb0142-4b0c-4226-ac98-6bcb03e59e00'> <pattern>dummy_program</pattern> <rules> <rule provider="dummy_provider" id="0cc9a000-2a4e-41f2-b30b-09d67af68ddc" class='dummy_class' context-timeout="300" context-scope="program" context-id="dummy_context"> <patterns> <pattern>First message has @ANYSTRING:dummy_string@</pattern> </patterns> <timeout_actions> <action> <message> <values> <value name="MESSAGE">Timeout: Failed to correlate ${dummy_string}@1 with anything</value> </values> </message> </action> </timeout_actions> </rule> <rule provider="dummy_provider" id="6fbefe59-3448-4b29-8c4f-7d9c1ab65a4c" class='dummy_class' context-scope="program" context-id="dummy_context"> <patterns> <pattern>Second message has @ANYSTRING:dummy_string@</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">Correlated ${dummy_string}@1 with ${dummy_string}@2</value> </values> </message> </action> </actions> </rule> </rules> </ruleset> -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Time Conversion and String delimiter
by Kyaw Kyi 07 Aug '13

07 Aug '13

Hi, I have a non-standard (not rfc 5414) log message format coming into syslog-ng client. I am trying to use csv-parser to parse into custom macros and send it off to syslog-ng host in the correct format. With no-parse option, how do I make syslog-ng handle this incoming timestamp: [July 30, 2013 10:19:06 AM] ? It would be so much easier to simply use the timestamp for the time message received in syslog-ng (e.g $DATE marco) but I have to extract the timestamp from the message itself somehow. Example: [July 30, 2013 10:19:06 AM EDT] DisplayType=CLEAR ClassName=Host.... Secondly, the documentation of syslog-ng seems to hint that I can use string delimiter in parser. "Currently only the csv-parser is implemented, which can separate columns based on delimiter characters and strings." but it also says that "If you specify multiple characters, every character will be treated as a delimiter." . Is there a way for me to use a string, not single characters, as delimiters in csv-parser? Thanks. ------------------------------------------------------------------------------ Please consider the environment before printing this email. Visit our website at http://www.nyse.com <http://www.nyse.com> ***************************************************************************** Note: The information contained in this message and any attachment to it is privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to the message, and please delete it from your system. Thank you. NYSE Euronext.

2 1

system() source problem.
by Nemeth Gyorgy 07 Aug '13

07 Aug '13

I compiled a syslog-ng 3.4 and when starting I get teh following error message: Error parsing source, source plugin system not found in /etc/syslog-ng/syslog-ng.conf at line 31, column 2: system(); The part of syslog-ng.conf is the following: source s_src { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel")); # use the following line if you want to receive remote UDP logging messages # (this is equivalent to the "-r" syslogd flag) # udp(); system(); }; According to the documentation the system() source is correct.. What is wrong here? When I comment the system() source out, the syslog-ng daemon starts without errors. # syslog-ng -V syslog-ng 3.4.2 Installer-Version: 3.4.2 Revision: Compile-Date: Aug 5 2013 11:00:20 Available-Modules: syslogformat,afsocket-notls,afsocket-tls,afsql,affile,afprog,afuser,afamqp,afmongodb,csvparser,confgen,system-source,basicfuncs,cryptofuncs,dbparser,afsocket Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: on -- Friczy

3 3

problem tagging with patterndb (syslog-ng 3.4.1)
by mailing lists 06 Aug '13

06 Aug '13

Hello all, I am having a problem understanding patterndb and tags, for the following rule the log line is matched but tags are not settled in pdbtool output. What am i missing here? <rule id="dad57bd5-6f9e-47b8-9e9f-401e3eb34334" provider="user" class="system"> <patterns> <pattern>@ESTRING:postfix.qid::@ to=@QSTRING:postfix.to:<>@, orig_to=@QSTRING:postfix.orig_to:<>@, relay=@ESTRING:postfix.relay.hostname:[@@ESTRING:postfix.relay.path:]@, delay=@ESTRING:postfix.delay:,@ delays=@ESTRING:postfix.delays.1:/@@ESTRING:postfix.delays.2:/@@ESTRING:postfix.delays.3:/@@ESTRING:postfix.delays.4:,@ dsn=@ESTRING:postfix.dsn:,@ status=@ESTRING:postfix.status: @@QSTRING:postfix.statusmsg:()@</pattern> <pattern>@ESTRING:postfix.qid::@ to=@QSTRING:postfix.to:<>@, orig_to=@QSTRING:postfix.orig_to:<>@, relay=@ESTRING:postfix.relay.hostname:[@@ESTRING:postfix.relay.path:]@:24, delay=@ESTRING:postfix.delay:,@ delays=@ESTRING:postfix.delays.1:/@@ESTRING:postfix.delays.2:/@@ESTRING:postfix.delays.3:/@@ESTRING:postfix.delays.4:,@ dsn=@ESTRING:postfix.dsn:,@ status=@ESTRING:postfix.status: @@QSTRING:postfix.statusmsg:()@</pattern> <pattern>@ESTRING:postfix.qid::@ to=@QSTRING:postfix.to:<>@, orig_to=@QSTRING:postfix.orig_to:<>@, relay=@ESTRING:postfix.relay.hostname:[@@ESTRING:postfix.relay.path:]@:24, delay=@ESTRING:postfix.delay:,@ delays=@ESTRING:postfix.delays.1:/@@ESTRING:postfix.delays.2:/@@ESTRING:postfix.delays.3:/@@ESTRING:postfix.delays.4:,@ dsn=@ESTRING:postfix.dsn:,@ status=@ESTRING:postfix.status: @(@ESTRING:postfix.status.code1: @@ESTRING:postfix.status.code2: @@EMAIL:postfix.status.recipient:<> @@ESTRING:postfix.status.qid: @Saved)</pattern> </patterns> <tags> <tag>postfix</tag> <tag>lmtp</tag> </tags> </rule> $ pdbtool match -P 'postfix' -p postfix.pdb -D -c -f mail.log Pattern matching part: @ESTRING:postfix.qid=B5BBAADB@ to=@QSTRING:postfix.to=user002@example.com@, orig_to=@QSTRING:postfix.orig_to=noreply@example.com@, relay=@ESTRING:postfix.relay.hostname=lmtp.example.com@@ESTRING:postfix.relay.path=10.180.242.142@:24, delay=@ESTRING:postfix.delay=0.07@ delays=@ESTRING:postfix.delays.1=0.04@@ESTRING:postfix.delays.2=0@@ESTRING:postfix.delays.3=0@@ESTRING:postfix.delays.4=0.03@ dsn=@ESTRING:postfix.dsn=2.0.0@ status=@ESTRING:postfix.status=sent@(@ESTRING:postfix.status.code1=250@@ESTRING:postfix.status.code2=2.0.0@@EMAIL:postfix.status.recipient=noreply@example.com@@ESTRING:postfix.status.qid=XP52K7Bp+1G/FAAAtCZERg@Saved) Matching part: B5BBAADB: to=<user002(a)example.com>, orig_to=<noreply(a)example.com>, relay=lmtp.example.com[10.180.242.142]:24, delay=0.07, delays=0.04/0/0/0.03, dsn=2.0.0, status=sent (250 2.0.0 <noreply(a)example.com> XP52K7Bp+1G/FAAAtCZERg Saved) Values: HOST=mailserver MESSAGE=B5BBAADB: to=<user002(a)example.com>, orig_to=<noreply(a)example.com>, relay=lmtp.example.com[10.180.242.142]:24, delay=0.07, delays=0.04/0/0/0.03, dsn=2.0.0, status=sent (250 2.0.0 <noreply(a)example.com> XP52K7Bp+1G/FAAAtCZERg Saved) PROGRAM=postfix/lmtp PID=29484 LEGACY_MSGHDR=postfix/lmtp[29484]: .classifier.class=system .classifier.rule_id=dad57bd5-6f9e-47b8-9e9f-401e3eb34334 postfix.qid=B5BBAADB postfix.to=user002(a)example.com postfix.orig_to=noreply(a)example.com postfix.relay.hostname=lmtp.example.com postfix.relay.path=10.180.242.142 postfix.delay=0.07 postfix.delays.1=0.04 postfix.delays.2=0 postfix.delays.3=0 postfix.delays.4=0.03 postfix.dsn=2.0.0 postfix.status=sent postfix.status.code1=250 postfix.status.code2=2.0.0 postfix.status.recipient=noreply(a)example.com postfix.status.qid=XP52K7Bp+1G/FAAAtCZERg TAGS=

2 1