- syslog-ng - lists.balabit.hu

[Bug 201] New: system() does not work on FreeBSD: /dev/ klog is pretty much ignored
by bugzilla＠bugzilla.balabit.com 13 Oct '12

13 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=201 Summary: system() does not work on FreeBSD: /dev/klog is pretty much ignored Product: syslog-ng Version: 3.3.x Platform: All OS/Version: FreeBSD Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 While debugging another problem, I found a couple of issues with system() on FreeBSD: * follow-freq(0) doesn't appear to work, so /dev/klog gets ignored outright * Even with follow-freq(1), we never read from /dev/klog, because stat() will report its size to be 0, so we never end up reading from it. The second is easy to fix, adding an || !S_ISREG(st.st_mode) to the appropriate condition in log_reader_io_follow_file() does the right thing. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 200] New: logging does not work on FreeBSD
by bugzilla＠bugzilla.balabit.com 13 Oct '12

13 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=200 Summary: logging does not work on FreeBSD Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: FreeBSD Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: czanik(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 Using the latest syslog-ng 3.4 git, with Algernon sandbox patches applied (to be able to generate a tar.gz dist file), logging does not seem to work. Without command line arguments, nothing gets logged, except for version information at startup. When using "-vdF", logs coming from the network are logged. Tested on FreeBSD 9.0 with a slightly modified sysutils/syslog-ng-rc port (removed utmp magic from ports Makefile). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

cisco rewrite code
by Evan Rempel 12 Oct '12

12 Oct '12

Does anyone have a pre build set of patterns/rewrite rule to rewrite all cisco logs into something that is a little more compliant? We are trying to use a master pattern database to identify/classify messages, but the cisco logs don't have usable "program names" so the pattern database can't even get started :-( Thanks for any pointers. -- Evan

4 3

[PATCH (3.4)] afsocket: Use the "dgram" proto for dgram sockets
by Gergely Nagy 11 Oct '12

11 Oct '12

When applying a transport to a socket, don't just set the transport, but also set logproto_name properly: "dgram" for SOCK_DGRAM sockets, "text" for everything else. If this is not done, and our socket is SOCK_DGRAM, we'd use the text proto, which after the refactoring, would look for a terminating new line, instead of datagram boundary - and it wouldn't find it, which will result in partial messages accumulating in a buffer, to eventually be sent out as garbage. This patch simply makes it so that SOCK_DGRAM will use the "dgram" logproto, which does the right thing, and knows about datagram boundaries. Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afunix.c | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/modules/afsocket/afunix.c b/modules/afsocket/afunix.c index 1a1465f..8849927 100644 --- a/modules/afsocket/afunix.c +++ b/modules/afsocket/afunix.c @@ -172,12 +172,16 @@ afunix_sd_apply_transport(AFSocketSourceDriver *s) if (!self->super.bind_addr) self->super.bind_addr = g_sockaddr_unix_new(self->filename); + self->super.logproto_name = "text"; + if (self->super.sock_type == SOCK_DGRAM) - afsocket_sd_set_transport(&self->super.super.super, "unix-dgram"); + { + afsocket_sd_set_transport(&self->super.super.super, "unix-dgram"); + self->super.logproto_name = "dgram"; + } else afsocket_sd_set_transport(&self->super.super.super, "unix-stream"); - self->super.logproto_name = "text"; return TRUE; } -- 1.7.9

1 0

syslog-ng Insider - October 2012
by Peter Czanik 11 Oct '12

11 Oct '12

Dear syslog-ng users, This is the 17th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS New AMQP destination driver --------------------------- An important milestone in syslog-ng's history is the AMQP destination driver, which is the first complete module contributed by a third party, and was not originated from within BalaBit! The destination allows syslog-ng to publish messages (using various exchange types) to an AMQP server such as RabbitMQ using an intuitive method: sending the headers selected by value-pairs() as headers, thereby solving the problem of how to serialize the payload: don't. For more information about the driver, see the announcement at: https://lists.balabit.hu/pipermail/syslog-ng/2012-October/019483.html A comparison of PE diskbuffer and the AMQP driver: http://algernon.blogs.balabit.com/2012/10/syslog-ng-amqp-and-disk-buffers/ Vote for syslog-ng Store Box to become the Best Computer Forensics Tool ------------------------------------------------------------------ We’re proud to announce we entered our syslog-ng Store Box appliance into the SC Awards US for the Best Computer Forensic Tool. Voting is now open until Oct 26. (requires registration or subscription to SC Magazine) so we kindly ask you visit https://www.bigpulse.com/m20086/intro to vote for us. We appreciate your help. syslog-ng 3.3.7RC2 is released ------------------------------ Version 3.3.7 is scheduled to arrive at the end of October, and a second release candidate was released for it (called version 3.3.6.91). It has many fixes for FreeBSD, so it should compile with the bundled ivykis on FreeBSD 8 and 9. Patches to compile it on FreeBSD 7 are included in the updated sysutils/syslog-ng-rc port: http://www.freshports.org/sysutils/syslog-ng-rc/ Packages for all supported or development versions of openSUSE are also available: https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 Announcement for 3.3.6.91: https://lists.balabit.hu/pipermail/syslog-ng/2012-October/019480.html OTHER SHORT NEWS * syslog-ng will be presented at two conferences in October: http://fedoraproject.org/wiki/FUDCon:Paris_2012 and http://conference.opensuse.org/ NEW RELEASES: * syslog-ng OSE 3.3.6.91: https://lists.balabit.hu/pipermail/syslog-ng/2012- <https://lists.balabit.hu/pipermail/syslog-ng/2012-October/019480.html>October/019480.html <https://lists.balabit.hu/pipermail/syslog-ng/2012-October/019480.html> ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

challenges with SDATA
by Chris Johnson 09 Oct '12

09 Oct '12

I have successfully written parsers for unstructured logs, but would like to be able to leverage SDATA. Based on what I have researched so far, I have built the following skeleton conf file for writing to mongodb #####Filter##### filter f_session_close {message("RT_FLOW_SESSION_CLOSE");}; #####Destinations##### destination d_mongodb_close { mongodb( collection("close") value-pairs( scope(rfc5424 sdata) ) ); }; #####Log##### log { source(s_network); filter(f_session_close); destination(d_mongodb_close); }; With this in mind, I get the following logs with only the rfc5424 scope: "DATE" : "Oct 5 20:40:21", "FACILITY" : "user", "HOST" : "192.168.199.2", "MESSAGE" : "2012-10-05T20:40:20.444 SRX100 RT_FLOW - RT_FLOW_SESSION_CREATE [junos(a)2636.1.1.1.2.36 source-address=\"192.168.199.207\" source-port=\"55625\" destination-address=\"130.57.4.24\" destination-port=\"80\" service-name=\"junos-http\" nat-source-address=\"50.193.12.149\" nat-source-port=\"28178\" nat-destination-address=\"130.57.4.24\" nat-destination-port=\"80\" src-nat-rule-name=\"source-nat-rule\" dst-nat-rule-name=\"None\" protocol-id=\"6\" policy-name=\"trust-to-untrust\" source-zone-name=\"trust\" destination-zone-name=\"untrust\" session-id-32=\"61189\" username=\"N/A\" roles=\"N/A\" packet-incoming-interface=\"vlan.0\"]", "PRIORITY" : "info", "PROGRAM" : "1", "_id" : ObjectId("506fa825817f52de65000001") When using the SDATA scope, I only get an object id. None of the structured data is ever parsed. Is there something special I need to do to parse the structured portion of the messge? Chris

2 1

[Bug 151] New: syslog-ng opens /dev/ log device on every HUP without closing it first
by bugzilla＠bugzilla.balabit.com 08 Oct '12

08 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=151 Summary: syslog-ng opens /dev/log device on every HUP without closing it first Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: michael.hocke(a)nyu.edu Type of the Report: bug Estimated Hours: 0.0 On Solaris 10 the /dev/log device can be held open concurrently for up to only 16 instances. syslog-ng opens a new instance of /dev/log after HUPing the process, i.e. after the 16th HUP syslog-ng quits because the new configuration fails and so does the previous configuration. This has been working fine in 3.0. I cannot test if this started to happen in 3.1, 3.2, or if this is new for 3.3. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

Strange error in /var/log/message about GPF (ip protection)
by Sebastien Pasche 06 Oct '12

06 Oct '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello I have some strange messages in my var/log/message about ip protection. Oct 5 16:54:01 mgblcof01 kernel: [1284555.180325] syslog-ng[9735] general protection ip:36104616f2 sp:7fba6e3304a0 error:0 in libglib-2.0.so.0.3200.4[3610400000+11d000] Any idea what would make syslog-ng having trouble with GPF ? My version of syslog-ng [root@mgblcof01 ]# syslog-ng --version syslog-ng 3.3.6.90 Installer-Version: 3.3.6.90 Revision: Compile-Date: Sep 20 2012 13:34:34 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: afsocket-tls,dbparser,afuser,syslogformat,afprog,confgen,csvparser,affile,dummy,basicfuncs,afsocket,afmongodb,tfjson,afsocket-notls,afsql,convertfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on OS : [root@mgblcof01 ]# uname -a Linux mgblcof01 3.5.3-1.fc17.x86_64 #1 SMP Wed Aug 29 18:46:34 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Thank ! Sébastien -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQbvbbAAoJEE3IBph3MKVPeM0P/js5cILHxgbACshcmcPPlXUH ELiDuP50QKkOHTzQOprK8kpBrRuzE2El5PuI0Tyyvbo4pF/Ehtlbe0donSh1w2aF vWGdj6EVxCtUDLJ3L2ZGiIxICyqDW/EDeSz04FGMp5pMdX++iVg2vnMuMic4kHsu OSaUWTdQ3v4ntC4HaI26xtnqjw7+egKND34ll2XgYFCMstTBwHdHwdXYWAD3KPBj jX+nqXgk5ADJybjAXS0u3TAb7rmPZsZ3cNn41i+15ed+I6ZcPGwkefFRsMmxbzrN wrwg+Dc2gk2ezP/Cdk5uZzlyYJe9R5AMeTeGvwPCaoEzznyAuDPlC+v+tF38KGri t+pEJ1Vy9QL1uqVAgj6aYzOzGZhAEimNfwBo/0WqV7DL2+FfT+5IyvIR9goRP+sS 7odfN9OyCnR3cYPSQLmtC4Xqlq4XwHE3BpIV364WArSblmGNpiuE3uMUwiHpjvrO mkSHzTmZLjvLfuJfV+ECC+xwBqVVStvB8r/upotFsT05ArQCJb5y+wJOlofPP3cg bQ+pGDw1Wf1OdyGj6n6ITu18VLKV1jx4XIkv3yTqbggffbAvuZ7XaUXVQQQYsaFT 4hIA2/YrMqEXK8HVHb1/zSD36bYzI/zrmUSs3H2opjW+b5doqmZ0KCYDgmvvnIEo wTdz8m7ZiQT+AJ8Lp+Ng =4lxl -----END PGP SIGNATURE-----

3 2

Syslog stops writing logs with extra [SQL] destination
by Anton Koldaev 05 Oct '12

05 Oct '12

Hello, I have the same "expr" logs going to file AND sql. With file-only logging it works great. When I'm enabling SQL logging it stops writing both to file and to sql ~1-10min(from 10 000 to 200 000 lines) after start. Every time there are no dropped messages, but 1000 stored for SQL location. syslog-ng 3.3.6 - number of nodes: ~150 - max-connections(200) - log_iw_size(20000) - log_fetch_limit(100) - log_fifo_size (400000); - flush_lines(100) - flush_timeout(100) - flags(flow-control) # the same issue with and without flow control syslog-server ~ % sudo syslog-ng-ctl stats | sort -rnk6 -t ";" | grep expr dst.file;d_app_expr#0;/logs/app/expr.log;o;processed;30457 destination;d_app_expr;;a;processed;30457 destination;d_mysql_expr;;a;processed;30003 * dst.sql;d_mysql_expr#0;mysql,mysql-db-ip,port,syslog_production,expr_${HOST}_${R_Y;a;stored;1000 * dst.sql;d_mysql_expr#0;mysql,mysql-db-ip,port,syslog_production,expr_${HOST}_${R_Y;a;dropped;0 dst.file;d_app_expr#0;/logs/app/expr.log;o;stored;0 dst.file;d_app_expr#0;/logs/app/expr.log;o;dropped;0 I see no errors in syslog-ng debug logs or mysql error log. After restarting syslog-ng it writes both logs for some time and stops again. I will appreciate any help. Additional question: is there any mysqltuner.pl-like tool for syslog-ng? I don't feel happy to tune all those buffers and windows manually with no being sure it will help. -- Best regards, Koldaev Anton

1 1

Re: [syslog-ng] I need some help with Syslog-ng and the new json parser
by Sebastien Pasche 05 Oct '12

05 Oct '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thank a lot for the help. I tested with syslog-ng 3.4.0 alpha3 and its working but I use 3.3 :-( So I will use regex and move to json parser when 3.4 will be stable :) Thanks again ! Sébastien On 10/05/2012 12:00 PM, syslog-ng-request(a)lists.balabit.hu wrote: > Sebastien Pasche <braoru(a)gmail.com> writes: > >> > I will present to you what I want to do and what I actually have. >> > >> > I would like to extract a field from a json log arriving in this source : >> > >> > source s_collector_tcp_json { >> > tcp(ip(0.0.0.0) port(514) flags(no-multi-line) flags(no-parse)); >> > }; >> > >> > And replacing the Program field I use in my destination > [...] >> > from the field @type of this json log : >> > >> > { >> > "@source": "tcp://127.0.0.1:9999/client/127.0.0.1:57530", >> > "@type": "tomcat_logstash_raw_json", >> > "@tags": [ >> > "tomcat_site" >> > ], >> > "@fields": { >> > "priority": "INFO", >> > "logger_name": "com.zzz.user.UserData", >> > "thread": "TP-Processor7", >> > "class": >> > "org.apache.jsp.WEB_002dINF.jsp.user.ViewInvoiceDetail_jsp", >> > "file": "ViewInvoiceDetail_jsp.java:162", >> > "method": "_jspService", >> > "prop_userIp": "192.168.215.50", >> > "prop_userId": "1440704" >> > }, >> > "@source_host": "127.0.0.1:57530", >> > "@source_path": "com.leshop.user.UserData", >> > "@message": "order : {WAREHOUSE_TYPE=drive, OID=5693367, >> > ORDER_DATE=2012-10-03 08:49:17.41, SHIPPING_FRESH=0.0, >> > FROZEN_DEPOSIT=0.0, WAREHOUSE_ID=5, DUE_AMOUNT=0.0, TOTAL_CREDITS=0.0, >> > ADDRESS_NUMBER=, DELIV_HELPFUL_INDICATION=, DELIVERY_MODE=20:00, >> > DELIVERY_DATE=2012-10-03 00:00:00.0, TOTAL=134.75, ACTION_TOTAL=0.0, >> > ORDER_NUMBER=abc-014085706-xyz, TRACK_TRACE=, RETAILER_GROUP=0, ZIP=, >> > ORDER_STATE=3, PAYMENT_TYPE=7, DELIV_DOORCODE=, FROZEN_FEES=0.0, >> > ENV_CO2=0.0, NAME= , ENV_CO2_RETAIL=0.0, HIDE_BVR=false, ADDRESS=, >> > TOTAL_CREDIT=0.0, MODIFICATION_STATE=1, REMINDER_LEVEL=0, >> > SUBTOTAL=134.75, GRAND_TOTAL=134.75, BVR_REFERENCE=, CITY=, >> > DELIV_PHONE=, SHIPPING_FIXED=0.0}", >> > "@timestamp": "2012-10-03T06:49:23.373000Z" >> > } > [...] > > Assuming that the JSON arrives on a single line, something along these > lines should do the trick: > > parser p_tomcat_json { > json-parser(prefix("json.")); > }; > > rewrite rw_tomcat_site_logstash_json_program_name { > set("${json.type}", value("$PROGRAM")); > }; > > And then chain it together: > > log { > source(s_collector_tcp_json); > parser(p_tomcat_json); > rewrite(rw_tomcat_site_logstash_json_program_name); > destination(d_file_normal_r); > }; > > Hope that helps! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQbrE/AAoJEE3IBph3MKVPidkQALooeGZRm2FEIGPN7uqJVrRc 2yeQ0L68Nny0WLnR2mZU4nOy7EHMKLPufrYINAp1FyNM7U4xDOkLTxoVhkPtejis hKce2Z/djJX1cABlPl0qf3ClBsSnH7oe745BvSdXUnBXFGc7mBV7E10GF4tb4Ce/ ElUUEx2UyIfCByOcETOJkvZrY1WSky1MqJZLI5dX2BzjKsdBML5Bmi/pj1VVRCaa WUC6lkSnGUdoyAdVCF7hHzNwQbm1txTUjsdo1oUvdaly4ASDkHWuHhRXsNw9gugA 3TRNugj/69oeOzxZkt/eVukpcK2JbeJ6UGVWpYe8Vo3x5slHXT5tObOcIni4eCYU /DnGFs7UT1nGmQ8HYTkytgnDUa3dpAcVU60xdyLsjkx7WPUmLjKJcKD//cXdJXH/ uzI3WvEEqnPqb3oRJfNqrR0Ikn1tF0jOHpJxeW3lQIYMaadahdTUQ1dhccycUWrB ISEoy1tFSUJepOHcZf+RGRTopxJVxvpK7JondR3oV0BMDf00Vgu8SLmuWl/ok3aZ KFAZ019kMYX1Bq0sCAtOyJwWzIXUFEtMz8rEkc1V0fbAI032WY6IzUecWuLGM783 uZ/C2bDBR2Xgbs/oDMhZImKJzjouN8IGuNK9ZaRMH0Qx0dx4wjc8QORD9hB2kmmF lEjQIEQ8D8ADAavqwURp =pUIX -----END PGP SIGNATURE-----

1 0