- syslog-ng - lists.balabit.hu

I need some help with Syslog-ng and the new json parser
by Sebastien Pasche 04 Oct '12

04 Oct '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello :) I will present to you what I want to do and what I actually have. I would like to extract a field from a json log arriving in this source : source s_collector_tcp_json { tcp(ip(0.0.0.0) port(514) flags(no-multi-line) flags(no-parse)); }; And replacing the Program field I use in my destination : #Destination that uses received time as timestamp for logs destination d_file_normal_r {file("/var/log/leshop/leshop_log/$R_YEAR/$HOST/$PROGRAM/$R_YEAR-$R_MONTH-$R_DAY.log" template(t_d_default_r) group(users) dir_group(users) perm(0640) dir_perm(0750) flags(no-multi-line) frac_digits(6));}; #Templates for destination that uses received time as timestamp for logs template t_d_default_r { template("$R_ISODATE $HOST LEVEL=$LEVEL $MSGHDR $MSG\n"); }; from the field @type of this json log : { "@source": "tcp://127.0.0.1:9999/client/127.0.0.1:57530", "@type": "tomcat_logstash_raw_json", "@tags": [ "tomcat_site" ], "@fields": { "priority": "INFO", "logger_name": "com.zzz.user.UserData", "thread": "TP-Processor7", "class": "org.apache.jsp.WEB_002dINF.jsp.user.ViewInvoiceDetail_jsp", "file": "ViewInvoiceDetail_jsp.java:162", "method": "_jspService", "prop_userIp": "192.168.215.50", "prop_userId": "1440704" }, "@source_host": "127.0.0.1:57530", "@source_path": "com.leshop.user.UserData", "@message": "order : {WAREHOUSE_TYPE=drive, OID=5693367, ORDER_DATE=2012-10-03 08:49:17.41, SHIPPING_FRESH=0.0, FROZEN_DEPOSIT=0.0, WAREHOUSE_ID=5, DUE_AMOUNT=0.0, TOTAL_CREDITS=0.0, ADDRESS_NUMBER=, DELIV_HELPFUL_INDICATION=, DELIVERY_MODE=20:00, DELIVERY_DATE=2012-10-03 00:00:00.0, TOTAL=134.75, ACTION_TOTAL=0.0, ORDER_NUMBER=abc-014085706-xyz, TRACK_TRACE=, RETAILER_GROUP=0, ZIP=, ORDER_STATE=3, PAYMENT_TYPE=7, DELIV_DOORCODE=, FROZEN_FEES=0.0, ENV_CO2=0.0, NAME= , ENV_CO2_RETAIL=0.0, HIDE_BVR=false, ADDRESS=, TOTAL_CREDIT=0.0, MODIFICATION_STATE=1, REMINDER_LEVEL=0, SUBTOTAL=134.75, GRAND_TOTAL=134.75, BVR_REFERENCE=, CITY=, DELIV_PHONE=, SHIPPING_FIXED=0.0}", "@timestamp": "2012-10-03T06:49:23.373000Z" } I know I can do it with patterndb or directly with a regex like . #match and create a group with le type value filter f_bigip_http_vs_extract { match('"@type": "([^\"]+)",' value("MESSAGE") type("pcre") flags("store-matches" "ignore-case")); }; #replace program field with extracted value from le log line rewrite rw_tomcat_site_logstash_json_program_name { subst('.*', "${1}", value("PROGRAM"));}; But I would like to use the new json parser to keep a configuration as clean as possible. Anyone can help me to know where I need to start ? (not found anyting into the admin guide :/) My version of syslog-ng : [root@mgblcof01 192.168.217.205]# syslog-ng --version syslog-ng 3.3.6.90 Installer-Version: 3.3.6.90 Revision: Compile-Date: Sep 20 2012 13:34:34 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: afsocket-tls,dbparser,afuser,syslogformat,afprog,confgen,csvparser,affile,dummy,basicfuncs,afsocket,afmongodb,tfjson,afsocket-notls,afsql,convertfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on Thank ! Seb -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQbStAAAoJEE3IBph3MKVPpZgP/0fyydXA3dBwBm4/A66DsJCJ DfsGtQYVEXMgtQhit+nh47ruBvl6O+E6XPTnL2vKjObKr8J7A1gy3Zsf6CAzkBod Wxs74wPHOvy8UIRRmzDl2rU+xv0Ak84lN7ngFjgJZpqlqajlPyfj48dKPwINV80X 3hZLpFHXD0y0T8JoL30rm35nbDMVfRBqFnLd5U4J8dMBZ6O8Y0bnYvPZ65GR3M0S L8x3cc4suOBSg/PEF4FAL/j/QMCTAnGvntIFN4pYOU9qkXixqtwheJ2EenzI+W2M mcs8VPWjRAfgg2w3gzy3UOGMgogHS2yR9guqHowGOeTuVfKEO1D7ZaP1MRHEfATj AdJVNGuDilquJwuLCLUwinvp68BpcJ1cM5E2t5P7FwZJQxfX8MJrGwQ+4bF2pvhI YzIRGHDmB888CRNYoyyrFCoXkjZ3/Nd8FxSx2BBhb3PNRgEkNWgfbkuAr422W+Ni ZzzDrD5Rpl7E7Fzz7Xb2XAxJJS7E+tj5d/ukD3PGFFfr6kXNHwfQyxCCgQOt93+2 zuSYGg0QPpztyK4y83NXaM+L7dYsP75pG24FhnUQFvA164sOKqT/i4/4mgu6u8pF ZX1Ji//fa2CfqEw0nhmrZxYYwXi4HYZ8uUd6MpQSfE04J5ymyOcrWa7wVBN+jlev Ts/ZAjTBDJku7JYLBmSf =VpL2 -----END PGP SIGNATURE-----

2 1

how to convert multiline message to single line?
by Sergey V. Lobanov 03 Oct '12

03 Oct '12

Hello, I'd like to see multiline syslog messages as one line. Is it possible via syslog-ng 3.0? -- wbr, Sergey V. Lobanov Timezone: MSK(GMT+4)

4 6

[PATCH] afmongodb: Include libmongo-client/configure.gnu in the tarball
by Gergely Nagy 02 Oct '12

02 Oct '12

In order to install cleanly with an internal libmongo-client, we need that library to be compiled in statically, which is what the configure.gnu script (installed via autogen.sh from git) enforces. However, since configure.gnu is not part of libmongo-client, its make dist will not include it, so its parent must do that. For this reason, and similarly to how the same file in lib/ivykis is handled, include libmongo-client/configure.gnu in the EXTRA_DIST of afmongodb/Makefile.am. Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/Makefile.am | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/modules/afmongodb/Makefile.am b/modules/afmongodb/Makefile.am index d022912..eb6b27b 100644 --- a/modules/afmongodb/Makefile.am +++ b/modules/afmongodb/Makefile.am @@ -17,7 +17,7 @@ libafmongodb_la_LDFLAGS = $(MODULE_LDFLAGS) endif BUILT_SOURCES = afmongodb-grammar.y afmongodb-grammar.c afmongodb-grammar.h -EXTRA_DIST = $(BUILT_SOURCES) afmongodb-grammar.ym +EXTRA_DIST = $(BUILT_SOURCES) afmongodb-grammar.ym libmongo-client/configure.gnu include $(top_srcdir)/build/lex-rules.am -- 1.7.9

1 0

[PATCH 1/2] json: Include all sources in _SOURCES
by Gergely Nagy 02 Oct '12

02 Oct '12

With the renaming and such the list of sources required to build the plugin got out of sync with reality. This patch corrects that, by adding the two new headers, and removing an old, unused one. Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/json/Makefile.am | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/modules/json/Makefile.am b/modules/json/Makefile.am index 110b1b5..3338ff5 100644 --- a/modules/json/Makefile.am +++ b/modules/json/Makefile.am @@ -9,10 +9,11 @@ module_LTLIBRARIES = libjson-plugin.la libjson_plugin_la_SOURCES = \ format-json.c \ + format-json.h \ jsonparser.c \ + jsonparser.h \ jsonparser-grammar.y \ jsonparser-parser.c jsonparser-parser.h \ - json-plugin.h \ json-plugin.c libjson_plugin_la_CPPFLAGS = $(AM_CPPFLAGS) -- 1.7.9

1 1

Re: [syslog-ng] Syslog_NG
by Munh-Erdene E 02 Oct '12

02 Oct '12

Dears,,, Tnx for your quickly reply. :) I've 1 syslog-ng server and 8 clients in my network. Syslog-Ng server collecting all log of that clients. But syslog-ng can collecting only bash & sh shell logs. Can't collect all command log off csh, tcsh ... etc. other shells... BR--- Munkh-Erdene On Mon, Oct 1, 2012 at 10:47 PM, Gergely Nagy <algernon(a)balabit.hu> wrote: > Hi! > > I would like to ask you to write to the syslog-ng mailing list > (https://lists.balabit.hu/mailman/listinfo/syslog-ng) with syslog-ng > questions instead of contacting me directly. > > However, I will try to answer your mail here now, since I'm a little > confused about what you're trying to accomplish. > > Munh-Erdene E <munhluu(a)gmail.com> writes: > > > Syslog_NG works fine with bash & sh shells &&& writing clientlogs on > > server. > > > > But not working other shells, example: csh, ksh, tscsh ... > > > > In Client side: > > Add some config on *Vi /etc/bashrc * > > > > # log to syslog > > > > declare ME > > > > ME=$(awk -v val=$UID -F ":" '$3==val{print $1}' /etc/passwd) > > > > export PROMPT_COMMAND='{ msg=$(history 1 | { read x y; echo $y; }); > logger > > -p local5.info "CMDLOG PWD=$PWD ; USER=$ME ; COMMAND=$msg"; }' > > > > > > > > *2. Vi /etc/rsyslog.conf* > > > > > > > > *.info;mail.none;authpriv.none;cron.none @192.168.0.1 > > > > authpriv.* @192.168.0.1 > > > > Can you help me? pls > > I don't really see where syslog-ng comes into play here? Neither logger > nor rsyslog have much to do with syslog-ng itself. Nor should any shell > or whatsoever affect syslog-ng, as it's a daemon that should be started > at boot time, the shell you use need not have any effect on its > behaviour. > > -- > |8] > >

2 1

[Bug 199] New: syslog-ng 3.3.6. 90 sometimes exits on SIGHUP on Solaris 10 SPARC
by bugzilla＠bugzilla.balabit.com 28 Sep '12

28 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=199 Summary: syslog-ng 3.3.6.90 sometimes exits on SIGHUP on Solaris 10 SPARC Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: irwin(a)princeton.edu Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=66) --> (https://bugzilla.balabit.com/attachment.cgi?id=66) configure.out Version: syslog-ng 3.3.6.90 Platform: Solaris 10 on SPARC I've been running syslog-ng 3.0.6 fine for several years. I've been trying to upgrade to 3.3.latest. When I upgraded to 3.3.6, syslog-ng would sometimes quit and dump core unexpectedly when I sent it SIGHUP as part of log file rotation. I figured it might be an ivykis issue perhaps related to https://bugzilla.balabit.com/show_bug.cgi?id=190 , so I've just tried upgrading to 3.3.6.90. But the error I see remains the same. Log message: Sep 26 03:00:03 example.org supervise/syslog-ng[23280]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='131' Here's the core dump's backtrace: gdb /usr/local/sbin/syslog-ng /var/core/core.syslog-ng.0.23281 GNU gdb (GDB) 7.4 ... Reading symbols from /var/local/sbin/syslog-ng...done. [New LWP 1] [New LWP 2] [New LWP 3] [Thread debugging using libthread_db enabled] [New Thread 1 (LWP 1)] [New Thread 2 (LWP 2)] [New Thread 3 (LWP 3)] Core was generated by `/usr/local/sbin/syslog-ng --cfgfile /etc/syslog-ng/main.conf --pidfile /var/run'. Program terminated with signal 3, Quit. #0 0xfef4e068 in kill () from /lib/libc.so.1 (gdb) bt #0 0xfef4e068 in kill () from /lib/libc.so.1 #1 0xff2b4c24 in main_loop_reload_config_apply () at mainloop.c:518 #2 0xff2b4640 in main_loop_io_worker_sync_call (func=0xff2b4a20 <main_loop_reload_config_apply>) at mainloop.c:455 #3 0xff2d68c8 in iv_signal_event (_this=0xff30ddfc) at iv_signal.c:169 #4 0xff2d62b8 in iv_event_raw_got_event (_this=0xff30de20) at iv_event_raw.c:87 #5 0xff2d3d38 in iv_run_active_list (active=<optimized out>, st=<optimized out>) at iv_main.c:235 #6 iv_main () at iv_main.c:279 #7 0xff2b4474 in main_loop_run () at mainloop.c:735 #8 0x00011538 in main (argc=1, argv=0xffbffd4c) at main.c:263 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Problem with csv-parser
by Tracy Dalzell 26 Sep '12

26 Sep '12

I am having a problem with the csv-parser. I expect the parser to not include the single quotes in the MACRO (column). It does not include the single quotes for the first column (id) but they are included for every column after the first one. Relevent details below: parser p_apistats { db-parser(file("/etc/syslog-ng/apilogs.xml"));}; parser p_apistats returns ${VALUES} = 'b8144eea3646a7bc47169e0b9cc5f575', '555.55.55.555', 'web07.myhost.com' parser p_apistatscsv { csv-parser(columns(id, ip, host) delimiters(",") flags(greedy,strip-whitespace) template("${VALUES}")); }; destination d_test { file("/var/log/splunk/$HOST" template("{ \"id\" : \"${id}\" \"ip\" : \"${ip}\" \"host\" : \"${host}\" }\n")); }; log { source(s_remote); parser(p_apistats); parser(p_apistatscsv); destination(d_test); }; Results: { "id" : "b8144eea3646a7bc47169e0b9cc5f575" "ip" : "'555.55.55.555'" "host" : "'web07.myhost.com'" }

1 0

[Bug 197] New: Characters lost when sending message over network that begins with {
by bugzilla＠bugzilla.balabit.com 25 Sep '12

25 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=197 Summary: Characters lost when sending message over network that begins with { Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bfield.wrk(a)gmail.com Type of the Report: --- Estimated Hours: 0.0 When sending a message that begins with { over the network the leading parenthesis and white space are lost. I've tested this on 3.3.4 with a manually compiled version and the version distributed with Ubuntu. Both are impacted by this bug. Here is a simple configuration to try: --------- syslog-ng.conf --------- source s_src { system(); }; source s_udp { udp(ip(127.0.0.1)); }; destination d_testfile { file("/var/log/test_json" template("$MSG\n")); }; destination d_udp_addparen { udp("127.0.0.1" template("{ second: { $MSG\n")); }; destination d_filedirect { file("/var/log/test_jsondirect" template("{ second: { $MSG\n")); }; log { source(s_src); destination(d_udp_addparen); destination(d_filedirect); }; log { source(s_udp); destination(d_testfile); }; -------- end syslog-ng.conf ------- # echo "{ json type message }" | logger # tail /var/log/test_jsondirect { second: { { json type message } # tail /var/log/test_json second: { { json type message } -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[PATCH] system-source: Correct a typo in the unix-dgram() part
by Gergely Nagy 25 Sep '12

25 Sep '12

From: Gergely Nagy <algernon(a)madhouse-project.org> The option is called perm(), not perms(). Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)madhouse-project.org> --- modules/system-source/system-source.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c index 9ba0480..929a5f8 100644 --- a/modules/system-source/system-source.c +++ b/modules/system-source/system-source.c @@ -41,7 +41,7 @@ system_sysblock_add_unix_dgram(GString *sysblock, const gchar *path, { g_string_append_printf(sysblock, "unix-dgram(\"%s\"", path); if (perms) - g_string_append_printf(sysblock, " perms(%s)", perms); + g_string_append_printf(sysblock, " perm(%s)", perms); g_string_append(sysblock, ");\n"); } -- 1.7.10

1 0

[Bug 196] New: syslog-ng fails to compile with json-c >= 0.10
by bugzilla＠bugzilla.balabit.com 25 Sep '12

25 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=196 Summary: syslog-ng fails to compile with json-c >= 0.10 Product: syslog-ng Version: 3.3.x Platform: All OS/Version: All Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: bug Estimated Hours: 0.0 The json-c 0.10 release changed how it serializes data, and that in turn broke syslog-ng, as we relied on the old not-all-that-public interface. Bumping the version requirement to >= 0.10 would not be wise, so syslog-ng should support both. I have a fix already, but that currently depends on other work I did starting from the 3.4 branch, so it needs to be backported to 3.3. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5