August 2013 - syslog-ng - lists.balabit.hu

syslogng-web: a simple and fast log viewer
by Peter Czanik 23 Aug '13

23 Aug '13

Hello, It's always nice to learn about projects building on syslog-ng. "syslogng-web" is one of them, a simple and fast log viewer for syslog-ng logs collected into mongodb. You can read more about it at https://czanik.blogs.balabit.com/2013/08/syslog-ng-web-a-simple-and-fast-lo… If you know about any other projects with syslog-ng at their core, please let me know so I can check it out and spread the word about it! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[Bug 245] New: syslog-ng 3.4.3 / 3.4.2 add bogus characters to /var/log/ messages
by bugzilla＠bugzilla.balabit.com 17 Aug '13

17 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=245 Summary: syslog-ng 3.4.3 / 3.4.2 add bogus characters to /var/log/messages Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: himbeere(a)meine-oma.de Type of the Report: bug Estimated Hours: 0.0 Hello. Suddenly my syslog-ng adds bogus characters to /var/log/messages like this: Aug 16 21:40:56 hn2 syslog-ng[2009]: syslog-ng shutting down; version='3.4.2' Aug 16 21:41:44 hn2 syslog-ng[1994]: syslog-ng starting up; version='3.4.3' Aug 16 21:41:44 hn2 syslog-ng[1994]: Internal error, alarm_set() called while an alarm is still active; Aug 16 21:41:44 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ kernel: [ 0.000000] Initializing cgroup subsys cpuset Aug 16 21:41:44 hn2 kernel: [ 0.000000] Initializing cgroup subsys cpu Aug 16 21:41:44 hn2 kernel: [ 0.000000] Initializing cgroup subsys cpuacct Aug 16 21:41:44 hn2 kernel: [ 0.000000] Linux version 3.10.7 (root@hn2) (gcc version 4.6.3 (Gentoo 4.6.3 p1.13, pie-0.5.2) ) #2 SMP Fri Aug 16 21:17:31 CEST 2013 This leads to an unreadable file. Well i can open the file with vi and then it looks okay. But with less it looks like this: 00000000 41 75 67 20 31 36 20 32 31 3a 32 37 3a 34 36 20 |Aug 16 21:27:46 | 00000010 68 6e 32 20 73 79 73 6c 6f 67 2d 6e 67 5b 32 38 |hn2 syslog-ng[28| 00000020 38 31 5d 3a 20 73 79 73 6c 6f 67 2d 6e 67 20 73 |81]: syslog-ng s| 00000030 74 61 72 74 69 6e 67 20 75 70 3b 20 76 65 72 73 |tarting up; vers| 00000040 69 6f 6e 3d 27 33 2e 34 2e 32 27 0a 41 75 67 20 |ion='3.4.2'.Aug | 00000050 31 36 20 32 31 3a 32 38 3a 30 33 20 68 6e 32 20 |16 21:28:03 hn2 | 00000060 73 68 75 74 64 6f 77 6e 5b 32 38 39 35 5d 3a 20 |shutdown[2895]: | 00000070 73 68 75 74 74 69 6e 67 20 64 6f 77 6e 20 66 6f |shutting down fo| 00000080 72 20 73 79 73 74 65 6d 20 72 65 62 6f 6f 74 0a |r system reboot.| 00000090 41 75 67 20 31 36 20 32 31 3a 32 38 3a 30 33 20 |Aug 16 21:28:03 | 000000a0 68 6e 32 20 69 6e 69 74 3a 20 53 77 69 74 63 68 |hn2 init: Switch| 000000b0 69 6e 67 20 74 6f 20 72 75 6e 6c 65 76 65 6c 3a |ing to runlevel:| 000000c0 20 36 0a 41 75 67 20 31 36 20 32 31 3a 32 38 3a | 6.Aug 16 21:28:| Is there any solution to this? Thanks and best regards t. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Cisco ASA logging to syslog-ng and a weird extra-characters in new-lines
by Finnur Orn Gudmundsson 16 Aug '13

16 Aug '13

Hi all, I have a syslog-ng server: rpm -qi syslog-ng Name : syslog-ng Relocations: (not relocatable) Version : 3.2.5 Vendor: Fedora Project Release : 3.el6 Build Date: Sun 15 Jan 2012 07:49:04 PM GMT Install Date: Wed 06 Mar 2013 03:56:17 PM GMT Build Host: x86-14.phx2.fedoraproject.org Group : System Environment/Daemons Source RPM: syslog-ng-3.2.5-3.el6.src.rpm Size : 1594638 License: GPLv2+ Signature : RSA/8, Sun 15 Jan 2012 08:55:15 PM GMT, Key ID 3b49df2a0608b895 The syslog-ng package installed is built and distributed by Fedora EPEL but I was hoping I could post here. I replaced a old syslog-ng 2.x server to this one and almost everything works as expected (there is always this one thing that does not work after a migration:). There is a Cisco ASA box logging to this server. For some records I get this extra space in a new-line like this: Aug 16 15:35:56 10.X.X.X %ASA-6-302021: Teardown ICMP connection for faddr 10.X.X.X/9483 gaddr 10.X.X.X/0 laddr 10.X.X.X/0 Aug 16 15:35:56 10.X.X.X %ASA-5-304001: 10.X.X.X Accessed URL 46.X.X.X:/ < Aug 16 15:35:56 10.X.X.X %ASA-5-304001: 10.X.X.X Accessed URL 92.X.X.X:http://sphotos-g.ak.fbcdn.net/hphotos-X/XxX/11X_221X18801_1X554_n.… Aug 16 15:35:56 10.X.X.X %ASA-6-305011: Built dynamic TCP translation from inside:10.X.X.X/38697 to outside:194.X.X.X/38697 (the < char is in a new-line) And another one: Aug 16 15:35:56 10.Z.Z.Z %ASA-6-302013: Built outbound TCP connection 1809896329 for outside:54.Z.Z.Z/80 (54.Z.Z.Z/80) to inside:10.Z.Z.Z/38684 (194.Z.Z.Z/38684) Aug 16 15:35:56 10.Z.Z.Z %ASA-5-304001: 10.Z.Z.Z Accessed URL 10.Z.Z.Z:/ 1 Aug 16 15:35:56 10.Z.Z.Z %ASA-6-302014: Teardown TCP connection 1809896328 for dmz-lb-int:10.Z.Z.Z/58759 to inside:10.Z.Z.Z/80 duration 0:00:00 bytes 162 TCP FINs (the single 1 char is in a new-line). Does someone know if this is a known bug ? I am aware that I am running a somewhat old version. Bgrds, Finnzi

1 1

insider 2013-08: New releases; GSoC updates; Dedicated OSE developer; Observe, Hack, Make 2013;
by Peter Czanik 15 Aug '13

15 Aug '13

Dear syslog-ng users, This is the 26th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng 3.4.3 and 3.3.11 are released --------------------------------------- Both the current (3.4) and previous (3.3) stable syslog-ng releases received maintenance updates. Sources are available from git or as tgz from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… For third-party binary packages for various Linux and UNIX platforms, visithttp://www.balabit.com/network-security/syslog-ng/opensource-logging-s… for 3^rd party binary packages for various distributions and UNIX variants. syslog-ng Agent for Windows 5.0 ------------------------------- There is a major new version of syslog-ng Agent available, version 5.0. Under the hood it builds now on the same code base as the UNIX version of syslog-ng. The new highlights include support for the Reliable Log Transfer Protocol (RLTP), flow control and support for the Windows Server 2012 and Windows 8 platforms. Long-time syslog-ng contributor Gergely Nagy now working full-time on syslog-ng Open Source Edition ----------------------------- If you've been following the mailing list, the community around syslog-ng or the commits in git you most certainly recognize the name of Gergely Nagy. He's been working at BalaBit for several years and we're happy to announce that from now on his dedicated role is to work on syslog-ng OSE full-time. You can read his thoughts about the change on his blog: https://algernon.blogs.balabit.com/2013/07/onto-greener-open-pastures/ syslog-ng GSoC: it's half-time! ------------------------------- The coding for Google Summer of Code is at half-time. Our students passed half-time evaluations and are working hard on a faster MySQL destination and on a brand new redis destination. You can follow their progress from their related blogs: http://petrovicsgyula.blogspot.com/ http://tichygsoc.blogspot.hu/ There is also a plan to provide test packages for those really adventurous in the coming weeks :-) syslog-ng JSON HowTo -------------------- JSON is gaining popularity, not just in Web 2.0 but in all fields of IT. As it's an easy way of storing and transmitting name value pairs, JSON emitting and parsing is now part of syslog-ng. The HowTo describing how it works in syslog-ng is available at http://asylum.madhouse-project.org/blog/2013/07/29/json-howto/ syslog-ng developers at Observe, Hack, Make 2013 ------------------------------------------------ Laser harp, Wikileaks, lockpicking, NSA, SIM card exploiting, LHC, counter-cryptanalysis, 24 pull requests, neutrino detectors, open source Bach, weird lights, quadrocopters, electric music cranked up to 11, workshops, lectures, discussions, and a lot more! We've been Observing, Hacking and Making! You can read more about it at https://athos.blogs.balabit.com/2013/08/observed-hacked-made/ SHORT NEWS: * CzP presenting log management and syslog-ng at Libre Software Meeting in Brussels: https://czanik.blogs.balabit.com/2013/08/czp-libre-software-meeting/ * What is Pseudonymous Data? An update on how upcoming EU regulations affect logging: https://jluby.blogs.balabit.com/2013/07/18/what-is-pseudonymous-data-in-the… NEW RELEASES * syslog-ng 3.3.11: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-June/000163.html * syslog-ng 3.4.3: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-August/000164.ht… * syslog-ng Agent 5.0.1: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-May/000158.html * syslog-ng PE 4.0.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-August/000165.ht… * syslog-ng PE 4.2.4: https://lists.balabit.hu/pipermail/syslog-ng-announce/2013-June/000160.html Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> To read this newsletter on-line, visit: http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

syslog-ng 3.4.3 has been released
by devel＠balabit.hu 13 Aug '13

13 Aug '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.3 SUMMARY : new stable release DATE : Aug 13, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.3) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.3 Tue, 13 Aug 2013 13:00:00 +0100 This is the second bug-fix release for the 3.4.x series, collecting significant fixes over the 3.4.2 release. Upgrading to this release is highly recommended. Bugfixes ======== * The pacct module has been fixed, and now compiles correctly. * Updated the configure.in script, and a few other places to fix issues with recent autotools. * Fixed an infinite loop when template expansion resulted in a zero-length string. * A considerable number of minor issues surrounding the build system have been eliminated. * The template_escape() setting is now respected by template functions that create templates themselves (such as the conditional functions) * Restore support for defaultless block{} arguments. [#230] * RFC5424 NILVALUE is now accepted in timestamps [#238] * The JSON parser will throw an error when the input is not an object (but, for example, a string) instead of crashing. * The program() source has been fixed, and now does not throw a syntax error at config parsing time, but works correctly again. * In case resolving a hostname fails, the IP address does not get truncated anymore. [#240 & #242] * A race condition in certain filter operations have been eliminated. Features ======== * The JSON parser now accepts generic parser related options, such as template(), so one can easily set the input for the parser. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andras Tim <tia(a)balabit.hu> Anton Koldaev <koldaevav(a)gmail.com> Attila Szalay <sasa(a)balabit.hu> Balazs Scheidler <bazsi(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Martin <bmartin(a)lavabit.com> Michael Sterrett <mr_bones_(a)gentoo.org> Vincent Brillault <spam(a)lerya.net> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

3 2

[Bug 238] New: syslog-protocol: allow NILVALUE for TIMESTAMP as specified in rfc5424
by bugzilla＠bugzilla.balabit.com 13 Aug '13

13 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=238 Summary: syslog-protocol: allow NILVALUE for TIMESTAMP as specified in rfc5424 Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: spam(a)lerya.net Type of the Report: --- Estimated Hours: 0.0 RFC5424 states: - section 6: """ TIMESTAMP = NILVALUE / FULL-DATE "T" FULL-TIME """ - section 6.2.3 "" A syslog application MUST use the NILVALUE as TIMESTAMP if the syslog application is incapable of obtaining system time. "" But the 3.4 version of syslog-ng does not allow such a NILVALUE: https://github.com/balabit/syslog-ng-3.4/blob/master/modules/syslogformat/s… For example, the following message does not work: <6>1 - - netlog - - - [ 775.895618] /bin/ping[2936] UDP 10.0.2.15:33285 -> 192.168.0.1:53 (uid=0) Whereas this one works: <6>1 2003-10-11T22:14:15.003Z - netlog - - - [ 775.895618] /bin/ping[2936] UDP 10.0.2.15:33285 -> 192.168.0.1:53 (uid=0) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 13

[Bug 240] New: host() in filters fail to match if using use_dns(yes)
by bugzilla＠bugzilla.balabit.com 13 Aug '13

13 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=240 Summary: host() in filters fail to match if using use_dns(yes) Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bmartin(a)lavabit.com Type of the Report: bug Estimated Hours: 0.0 After upgrading from version 3.2.5 to 3.4.2 a filter using host() did not work anymore. The filter looks something like this: filter f_filter { host("172.12.0.1") or host("172.12.0.2"); }; Messages which would match this filter never reached their destination as they were used to with 3.2.5. I figured out that $HOST contained only the string '172'. My assumption is that host() matches against $HOST. In turn I set use_dns(no) and things started working again and now $HOST contains the whole IP address. used global options: options { threaded(yes); chain_hostnames(no); use_dns(no); stats_freq(43200); mark_freq(3600); }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 242] New: IP address stripped to first number with use_fqdn off
by bugzilla＠bugzilla.balabit.com 13 Aug '13

13 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=242 Summary: IP address stripped to first number with use_fqdn off Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: list0570(a)paradise.net.nz Type of the Report: bug Estimated Hours: 0.0 transport("udp") ip("0.0.0.0") port(514) ip-protocol(4) keep_timestamp(no) keep_hostname(no) # no = rewrite hostname use_dns(yes) # yes, no, persist_only use_fqdn(no) Regardless of what the use_dns setting is, if the host name is an IP address A.B.C.D, use_fqdn is no, and /etc/hosts has no name and the DNS lookup fails, then only "A" is logged as the host name. This is not useful at all - if a name lookup fails for an IP address, you have to keep the IP address and not truncate it to something useless. syslog-ng-3.4.2-1.5.1.i586 from openSUSE 12.3 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 243] New: patterndb rule should allow action upon context timeout
by bugzilla＠bugzilla.balabit.com 09 Aug '13

09 Aug '13

https://bugzilla.balabit.com/show_bug.cgi?id=243 Summary: patterndb rule should allow action upon context timeout Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: enhancement Estimated Hours: 0.0 Currently in an event correlation configuration, when a certain context times out, the whole rule is thrown away. It would be very useful to be able to trigger an action when a rule's context-timeout is reached. Of course the ability to add a timeout_action for every rule would further enhance the idea. Example: <ruleset name='dummy_ruleset' id='03eb0142-4b0c-4226-ac98-6bcb03e59e00'> <pattern>dummy_program</pattern> <rules> <rule provider="dummy_provider" id="0cc9a000-2a4e-41f2-b30b-09d67af68ddc" class='dummy_class' context-timeout="300" context-scope="program" context-id="dummy_context"> <patterns> <pattern>First message has @ANYSTRING:dummy_string@</pattern> </patterns> <timeout_actions> <action> <message> <values> <value name="MESSAGE">Timeout: Failed to correlate ${dummy_string}@1 with anything</value> </values> </message> </action> </timeout_actions> </rule> <rule provider="dummy_provider" id="6fbefe59-3448-4b29-8c4f-7d9c1ab65a4c" class='dummy_class' context-scope="program" context-id="dummy_context"> <patterns> <pattern>Second message has @ANYSTRING:dummy_string@</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">Correlated ${dummy_string}@1 with ${dummy_string}@2</value> </values> </message> </action> </actions> </rule> </rules> </ruleset> -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Time Conversion and String delimiter
by Kyaw Kyi 07 Aug '13

07 Aug '13

Hi, I have a non-standard (not rfc 5414) log message format coming into syslog-ng client. I am trying to use csv-parser to parse into custom macros and send it off to syslog-ng host in the correct format. With no-parse option, how do I make syslog-ng handle this incoming timestamp: [July 30, 2013 10:19:06 AM] ? It would be so much easier to simply use the timestamp for the time message received in syslog-ng (e.g $DATE marco) but I have to extract the timestamp from the message itself somehow. Example: [July 30, 2013 10:19:06 AM EDT] DisplayType=CLEAR ClassName=Host.... Secondly, the documentation of syslog-ng seems to hint that I can use string delimiter in parser. "Currently only the csv-parser is implemented, which can separate columns based on delimiter characters and strings." but it also says that "If you specify multiple characters, every character will be treated as a delimiter." . Is there a way for me to use a string, not single characters, as delimiters in csv-parser? Thanks. ------------------------------------------------------------------------------ Please consider the environment before printing this email. Visit our website at http://www.nyse.com <http://www.nyse.com> ***************************************************************************** Note: The information contained in this message and any attachment to it is privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to the message, and please delete it from your system. Thank you. NYSE Euronext.

2 1