- syslog-ng - lists.balabit.hu

Problem with csv-parser
by Tracy Dalzell 26 Sep '12

26 Sep '12

I am having a problem with the csv-parser. I expect the parser to not include the single quotes in the MACRO (column). It does not include the single quotes for the first column (id) but they are included for every column after the first one. Relevent details below: parser p_apistats { db-parser(file("/etc/syslog-ng/apilogs.xml"));}; parser p_apistats returns ${VALUES} = 'b8144eea3646a7bc47169e0b9cc5f575', '555.55.55.555', 'web07.myhost.com' parser p_apistatscsv { csv-parser(columns(id, ip, host) delimiters(",") flags(greedy,strip-whitespace) template("${VALUES}")); }; destination d_test { file("/var/log/splunk/$HOST" template("{ \"id\" : \"${id}\" \"ip\" : \"${ip}\" \"host\" : \"${host}\" }\n")); }; log { source(s_remote); parser(p_apistats); parser(p_apistatscsv); destination(d_test); }; Results: { "id" : "b8144eea3646a7bc47169e0b9cc5f575" "ip" : "'555.55.55.555'" "host" : "'web07.myhost.com'" }

1 0

[Bug 197] New: Characters lost when sending message over network that begins with {
by bugzilla＠bugzilla.balabit.com 25 Sep '12

25 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=197 Summary: Characters lost when sending message over network that begins with { Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bfield.wrk(a)gmail.com Type of the Report: --- Estimated Hours: 0.0 When sending a message that begins with { over the network the leading parenthesis and white space are lost. I've tested this on 3.3.4 with a manually compiled version and the version distributed with Ubuntu. Both are impacted by this bug. Here is a simple configuration to try: --------- syslog-ng.conf --------- source s_src { system(); }; source s_udp { udp(ip(127.0.0.1)); }; destination d_testfile { file("/var/log/test_json" template("$MSG\n")); }; destination d_udp_addparen { udp("127.0.0.1" template("{ second: { $MSG\n")); }; destination d_filedirect { file("/var/log/test_jsondirect" template("{ second: { $MSG\n")); }; log { source(s_src); destination(d_udp_addparen); destination(d_filedirect); }; log { source(s_udp); destination(d_testfile); }; -------- end syslog-ng.conf ------- # echo "{ json type message }" | logger # tail /var/log/test_jsondirect { second: { { json type message } # tail /var/log/test_json second: { { json type message } -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[PATCH] system-source: Correct a typo in the unix-dgram() part
by Gergely Nagy 25 Sep '12

25 Sep '12

From: Gergely Nagy <algernon(a)madhouse-project.org> The option is called perm(), not perms(). Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)madhouse-project.org> --- modules/system-source/system-source.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c index 9ba0480..929a5f8 100644 --- a/modules/system-source/system-source.c +++ b/modules/system-source/system-source.c @@ -41,7 +41,7 @@ system_sysblock_add_unix_dgram(GString *sysblock, const gchar *path, { g_string_append_printf(sysblock, "unix-dgram(\"%s\"", path); if (perms) - g_string_append_printf(sysblock, " perms(%s)", perms); + g_string_append_printf(sysblock, " perm(%s)", perms); g_string_append(sysblock, ");\n"); } -- 1.7.10

1 0

[Bug 196] New: syslog-ng fails to compile with json-c >= 0.10
by bugzilla＠bugzilla.balabit.com 25 Sep '12

25 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=196 Summary: syslog-ng fails to compile with json-c >= 0.10 Product: syslog-ng Version: 3.3.x Platform: All OS/Version: All Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: bug Estimated Hours: 0.0 The json-c 0.10 release changed how it serializes data, and that in turn broke syslog-ng, as we relied on the old not-all-that-public interface. Bumping the version requirement to >= 0.10 would not be wise, so syslog-ng should support both. I have a fix already, but that currently depends on other work I did starting from the 3.4 branch, so it needs to be backported to 3.3. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

Can't seem to do a simple parse
by Tracy Dalzell 21 Sep '12

21 Sep '12

I am trying to do a very simple parsing test to reformat a message using syslog-ng 3.3.6 and I can't seem to use the value of my macro (mystring): patterndb (test.xml): <patterndb version='4' pub_date='2012-09-21'> <ruleset name='test' id='123456678'> <pattern>test</pattern> <rules> <rule provider='me' id='182437592347598' class='system'> <patterns> <pattern>@STRING:mystring: @</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> pdbtool results: [root@loghost01 syslog-ng]# pdbtool match -c -D -v -p test.xml -P test -M "INSERT INTO Test (id, value) VALUES (1, .044212)" Pattern matching part: @STRING:mystring=INSERT INTO Test @ Matching part: INSERT INTO Test Values: MESSAGE=INSERT INTO Test (id, value) VALUES (1, .044212) PROGRAM=test .classifier.class=unknown TAGS=.classifier.unknown relevent syslog-ng.conf lines: source s_remote { tcp(); }; parser p_test { db-parser(file("/etc/syslog-ng/test.xml")); }; destination d_test { file("/var/log/splunk/$HOST" template("Here is the message: ${R_DATE} ${mystring}\n")); }; log { source(s_remote); parser(p_test); destination(d_test); };\n")); }; Results from logger "INSERT INTO Test (id, value) VALUES (1, .044212)" on client are: Here is the message: Sep 21 16:46:53 Value of ${mystring} is null. What am I missing?

2 2

syslog-ng Insider - September 2012
by Peter Czanik 20 Sep '12

20 Sep '12

Dear syslog-ng users, This is the 16th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng 3.3.6 is released --------------------------- There was a new release for the syslog-ng stable line: version 3.3.6 There are mostly bugfixes in this release but also some smaller but long waited features: * the @include statement now accepts globs * upstream ivykis is used * better systemd support The change to upstream ivykis introduced a bug on FreeBSD and some other problems were also discovered. Version 3.3.7 is scheduled to arrive at the end of October, and there is now a 3.3.6.90, aka 3.3.7RC available for testing. Announcement for 3.3.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-August/000146.ht… Announcement for 3.3.6.90: https://lists.balabit.hu/pipermail/syslog-ng/2012-September/019365.html syslog-ng 3.4 news ------------------ Although there aren't any new alpha releases yet, there are already plenty of news for the development branch aswell! * The MongoDB destination received a major speed boost, and work is underway to make it even more performant. * The $(format-json) function now supports nested structures * There's a MAC address parser available for patterndb * A couple of new template functions were introduced, $(md5), $(sha1) and others. * The unit tests were improved upon greatly, and many many bugs were squashed aswell. For the complete list of changes, you can browse the commit messages at https://github.com/balabit/syslog-ng-3.4/commits/master And while there, check out the sources and give it a try, feedback is appreciated, help us make syslog-ng even better! SSB 3 LTS: wildcard search in logspaces --------------------------------------- The third important new feature in SSB 3 LTS is the ability to search within log messages using wildcard characters. It is indeed a small and simple change on the surface, but brings a lot of new possibilities. Read about the details at http://gyp.blogs.balabit.com/2012/07/new-features-in-ssb-3-lts-wildcard-sea… OTHER SHORT NEWS * syslog-ng ports in FreeBSD explained: https://czanik.blogs.balabit.com/2012/09/syslog-ng-rc/ * syslog-ng will be presented at two conferences in October: http://fedoraproject.org/wiki/FUDCon:Paris_2012 and http://conference.opensuse.org/ NEW RELEASES: * syslog-ng OSE 3.3.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-August/000146.ht… * syslog-ng OSE 3.3.6.90: https://lists.balabit.hu/pipermail/syslog-ng/2012-September/019365.html CASE STUDY: * University of Victoria: http://www.balabit.com/support/documentation/syslog-ng_cs_uvic_en.pdf ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[PATCH 0/7] value-pairs dot notation walker
by Gergely Nagy 16 Sep '12

16 Sep '12

The patches that will follow this overview soon implement a 'walker' for value-pairs, a way to walk the internal flat structure syslog-ng uses as if it was a tree. This is important for two modules in particular: $(format-json) and the mongodb destination. The latter already supportes storing structured data, but it is using upserts (with all its downsides) to accomplish that. The former - up until now - was only able to format flat JSON. With these patches, both change: $(format-json) becomes able to emit structured data, and the mongodb destination starts to use inserts. The patches also merge the tfjson and jsonparser modules into a single json module, and implements our own JSON formatting for $(format-json). This means that json-glib support is completely removed, as it became unnecessary, and json-c is only used by the parser, but for practical reasons, required to build the plugin. That is, it is no longer possible to build $(format-json) only, but not json-parser, nor the other way around. The final patch is a bonus, that corrects a small error in jsonparser, and stops it from emitting an assertion warning on stderr in certain cases.

3 9

syslog-ng 3.3.7 RC1 (3.3.6.90) released
by Gergely Nagy 14 Sep '12

14 Sep '12

Hi! As promised earlier, I prepared a release candidate for the upcoming syslog-ng 3.3.7 release (due october 31), with the most pressing bugs already fixed. Testing this release candidate and reporting any and all issues found with it - be those regressions or anything else - would be greatly appreciated! The current state of the NEWS file is included below, and if anyone wants to track how the release is progressing and what needs to be fixed still before the final release, see the 3.3.7 target milestone on bugzilla: https://bugzilla.balabit.com/buglist.cgi?query_format=advanced&target_miles… The release is available from the following sources: GitHub ------ The release is tagged as v3.3.6.90 on the official repository at git://github.com/balabit/syslog-ng-3.3.git Tarball ------- A make dist tarball built from the same tag is available from my package repository: http://packages.madhouse-project.org/syslog-ng/3.3/3.3.6.90/syslog-ng-3.3.6… And finally, the NEWS file from the release: 3.3.6.90 (3.3.7 RC1) Thu, 06 Sep 2012 14:00:00 +0200 Bugfixes ======== * The bundled ivykis has been updated to fix two bugs found after the 3.3.6 release was tagged: * Fixed a Solaris-specific issue relating to TCP sources [#190] * Fixed a spinlock issue, triggered on at least FreeBSD [#193] * The glob-based configuration file inclusion was fixed to behave similarly to including a whole directory, to include files alphabetically. [#191] * Fixed a crash when trying to display the available modules in debug mode. [#189] * Fixed the building of afsocket-notls, so that it is correctly built without TLS support. [#188] * Fix compilation without spoof-source. [#192] Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Cy Schubert <cy(a)FreeBSD.org> Dave Reisner <dreisner(a)archlinux.org> Gergely Nagy <algernon(a)balabit.hu> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Lennert Buytenhek <buytenh(a)wantstofly.org> Marvin Nipper <marvin.nipper(a)stream.com> Peter Czanik <czanik(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> -- |8]

5 5

Syslog-ng configuration for network source (udp)
by Alex Clone 12 Sep '12

12 Sep '12

Hello, I am trying to "read" a log stream from interface eth0 and to save it in a file (/var/log/messages_tcp). I am newbie on the use of syslog-ng but I try to learn reading the official manual and a lot of configuration examples. Although, I have try a lot of different configurations, the result is always the same : Syslog-ng cannot read the streaming data. I hope that you can help me. That's the version of syslog-ng that I use: *~# /usr/local/syslog-ng/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Sep 6 2012 14:36:19 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: on Enable-Pacct: off* There is an incoming log stream in eth0 (udp), as you can see next : *~# tcpdump -i eth0 udp 'port 514' -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 06:05:58.669992 IP (tos 0x0, ttl 254, id 20209, offset 0, flags [none], proto UDP (17), length 82) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 54 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host WAN:172.30.254.4\0x0a 06:05:58.670134 IP (tos 0x0, ttl 254, id 52, offset 0, flags [none], proto UDP (17), length 85) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 57 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host outside:172.26.10.4\0x0a ………..* The syslog-ng.conf that I use is the following: * @version: 3.2 options { chain_hostnames(off); long_hostnames(off); use_dns(no); # We don't want DNS names, IP's only. flush_lines(0); stats_freq(43200); time_reopen(10); }; source s_network { # tcp(ip("172.30.6.70") port(514)); # udp(ip("172.30.6.70") port(514)); # syslog( # flags(no-multi-line) # ip(172.30.6.70) # keep-alive(yes) # keep_hostname(yes) # transport(udp) # TLS Options # tls() # ); tcp(); udp(); }; destination d_tcp { file("/var/log/messages_tcp" owner("root") group("adm") perm(0640)); }; log { source(s_network); destination(d_tcp); }; * When I start syslog-ng in debug mode (syslog-ng -Fevd) the results are the followings: *~# /usr/local/syslog-ng/sbin/syslog-ng -Fevdt Trying to open module; module='syslogformat', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libsyslogformat.so' Trying to open module; module='basicfuncs', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libbasicfuncs.so' Trying to open module; module='afsocket', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafsocket.so' Trying to open module; module='affile', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libaffile.so' Trying to open module; module='afprog', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafprog.so' Trying to open module; module='afuser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafuser.so' Trying to open module; module='dbparser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libdbparser.so' Trying to open module; module='csvparser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libcsvparser.so' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.2.4' ^CTermination requested via signal, terminating; syslog-ng shutting down; version='3.2.4' Closing log transport fd; fd='6' Running application hooks; hook='4'* As you can see there is no "Incoming log entry..:" line. I pressed Ctrl+C for the termination of program. Also listening to the network interface seems to be right: *~# netstat -tanpu | grep syslog tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 25950/syslog-ng udp 0 0 0.0.0.0:514 0.0.0.0:* 25950/syslog-ng * Please, can anyone help me with this problem? I don't know what could be blocking the packets, if it is because of the OS or because of syslog-ng configuration.What could it be wrong? I am waiting for your response. PS. Sorry for my "bad" English. Thanks is advance, Alex

3 4

6 identical servers, only 4 sending logs
by Brian Johnson 11 Sep '12

11 Sep '12

Hello all, I hope what I'm asking hasn't been covered previously, I tried some searches with no luck. If I'm duplicating something else, I apologize. My problem is, I have 6 DHCP servers with identical syslog-ng.conf and syslog.conf files on them. The set up is as so: dhcp-a-01 and dhcp-b-01 are a DHCP failover pair dhcp-a-02 and dhcp-b-02 are a DHCP failover pair dhcp-a-03 and dhcp-b-03 are a DHCP failover pair The 'dhcp-a' servers are in the A data center. 'dhcp-b' servers are in the B data center. Again, the syslog-ng.conf files on all of them are identical, checked with sha1sum. It is confirmed that all of them are using syslog-ng for logging. I have them all set to log to the same remote logging server. Logs from dhcp-[a,b]-01 and dhcp-[a,b]-03 are making it to the remote server with no issues. I can see it on the remote server and I can see it when doing a 'tcpdump port 514' on the servers themselves. For some reason, I'm not seeing any logs from dhcp-[a,b]-02 on the remote server and when I do 'tcpdump port 514' for a length of time, I get: dhcp-b-02:~# tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel when the other servers, done at the same time, show packets captured. I just did a "tail -f /var/log/syslog > /tmp/test" all of the servers between 11:43:26 and 11:45:38 (2m12s). In that time: dhcp-[a,b]-01 had roughly 2700 lines dhcp-[a-b]-02 had roughly 11000 lines dhcp-[a-b]-03 had roughly 1100 lines So to me it seems like there's some sort of throttling on the data that's able to be sent. There's ~5x more traffic on pair 2 than 1 (which will be rebalanced, just trying to get this working first) so that would make sense. The only thing that I could find that looks like it would help is the log_fifo_size option, but that doesn't seem to help -- I've made several adjustments to it, but it doesn't seem to make any difference. Can someone please let me know what I'm missing? Thanks! Brian

2 2