August 2011 - syslog-ng - lists.balabit.hu

malformed syslog packets?
by Matt Zagrabelny 30 Aug '11

30 Aug '11

Hi! I've got a central log server running the OSE 3.1.3 version of syslog-ng: dpkg -l syslog-ng Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Description +++-========================================-========================================-================================================================================================ ii syslog-ng 3.1.3-3 Next generation logging daemon I have dns lookup turned on via: options { use_dns(yes); dns_cache(2000); dns_cache_expire(87600); }; And this seems to work just fine...except for a certain type of device on our network. We have a number of UPSes that log to our central log server and it seems that the dns look ups do not work for those (types of devices). % cd /var/log/syslog-ng/remote_clients % ls -d 10.* 10.25.32.4 10.25.5.15 10.25.5.19 10.25.5.26 10.25.5.35 10.25.5.4 10.25.5.44 10.25.5.51 10.25.5.6 10.25.5.65 10.25.5.69 10.25.5.76 10.25.5.1 10.25.5.16 10.25.5.2 10.25.5.27 10.25.5.36 10.25.5.40 10.25.5.49 10.25.5.52 10.25.5.60 10.25.5.66 10.25.5.7 10.25.5.79 10.25.5.10 10.25.5.17 10.25.5.20 10.25.5.28 10.25.5.37 10.25.5.41 10.25.5.5 10.25.5.55 10.25.5.61 10.25.5.67 10.25.5.72 10.25.5.81 10.25.5.14 10.25.5.18 10.25.5.23 10.25.5.3 10.25.5.38 10.25.5.43 10.25.5.50 10.25.5.58 10.25.5.62 10.25.5.68 10.25.5.75 10.25.5.9 When I look up those IP addresses, they are *all* APC batteries (UPSes). For instance: % dig -x 10.25.5.43 +short kplz246Abat1.d.umn.edu. Is it possible that they are sending some sort of munged data to the log server and syslog-ng is not able to perform the (reverse) name lookup? Any advice? Thanks, much! -Matt Zagrabelny

3 5

syslog-ng.persist file error
by Scott McLane Gardner 29 Aug '11

29 Aug '11

Hello, When I came in this morning, syslog-ng was not running, and the following error is produced when trying to start it with init script: >Error loading persistent configuration file; name='//var/syslog-ng.persist' I'm guessing that additional forward slash is not supposed to be there, but I have no idea how this is getting set or where to change it. Can anyone help me? Thanks, Scott

3 3

[PATCH] templates: Add a C_* family of timestamp macros.
by Gergely Nagy 29 Aug '11

29 Aug '11

The C_* family of timestamp macros work very similarly to R_ (received) and S_ (stamp), except C_ uses the current system time as of the macro expansion. The patch below uses cached_g_current_time() for performance reasons, but in case the minor caching is undesired, it's easy to convert to something else. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/templates.c | 39 +++++++++++++++++++++++++++++++++++++-- 1 files changed, 37 insertions(+), 2 deletions(-) diff --git a/lib/templates.c b/lib/templates.c index cb0c1b5..63000a3 100644 --- a/lib/templates.c +++ b/lib/templates.c @@ -86,6 +86,7 @@ enum M_RECVD_OFS = M_TIME_MACROS_MAX, M_STAMP_OFS = 2 * M_TIME_MACROS_MAX, + M_CSTAMP_OFS = 3 * M_TIME_MACROS_MAX, }; #define M_TIME_MACROS 15 @@ -171,6 +172,29 @@ LogMacroDef macros[] = { "S_TZ", M_STAMP_OFS + M_TZ }, { "S_UNIXTIME", M_STAMP_OFS + M_UNIXTIME }, + { "C_DATE", M_CSTAMP_OFS + M_DATE }, + { "C_FULLDATE", M_CSTAMP_OFS + M_FULLDATE }, + { "C_ISODATE", M_CSTAMP_OFS + M_ISODATE }, + { "C_STAMP", M_CSTAMP_OFS + M_STAMP }, + { "C_YEAR", M_CSTAMP_OFS + M_YEAR }, + { "C_YEAR_DAY", M_CSTAMP_OFS + M_YEAR_DAY }, + { "C_MONTH", M_CSTAMP_OFS + M_MONTH }, + { "C_MONTH_WEEK", M_CSTAMP_OFS + M_MONTH_WEEK }, + { "C_MONTH_ABBREV", M_CSTAMP_OFS + M_MONTH_ABBREV }, + { "C_MONTH_NAME", M_CSTAMP_OFS + M_MONTH_NAME }, + { "C_DAY", M_CSTAMP_OFS + M_DAY }, + { "C_HOUR", M_CSTAMP_OFS + M_HOUR }, + { "C_MIN", M_CSTAMP_OFS + M_MIN }, + { "C_SEC", M_CSTAMP_OFS + M_SEC }, + { "C_WEEKDAY", M_CSTAMP_OFS + M_WEEK_DAY_ABBREV }, /* deprecated */ + { "C_WEEK_DAY", M_CSTAMP_OFS + M_WEEK_DAY }, + { "C_WEEK_DAY_ABBREV",M_CSTAMP_OFS + M_WEEK_DAY_ABBREV }, + { "C_WEEK_DAY_NAME", M_CSTAMP_OFS + M_WEEK_DAY_NAME }, + { "C_WEEK", M_CSTAMP_OFS + M_WEEK }, + { "C_TZOFFSET", M_CSTAMP_OFS + M_TZOFFSET }, + { "C_TZ", M_CSTAMP_OFS + M_TZ }, + { "C_UNIXTIME", M_CSTAMP_OFS + M_UNIXTIME }, + { "SDATA", M_SDATA }, { "MSGHDR", M_MSGHDR }, { "SOURCEIP", M_SOURCE_IP }, @@ -421,7 +445,7 @@ log_macro_expand(GString *result, gint id, gboolean escape, LogTemplateOptions * gchar buf[64]; gint length; time_t t; - LogStamp *stamp; + LogStamp *stamp, sstamp; glong zone_ofs; if (id >= M_TIME_FIRST && id <= M_TIME_LAST) @@ -438,7 +462,18 @@ log_macro_expand(GString *result, gint id, gboolean escape, LogTemplateOptions * id -= M_STAMP_OFS; stamp = &msg->timestamps[LM_TS_STAMP]; } - else + else if (id >= M_TIME_FIRST + M_CSTAMP_OFS && id <= M_TIME_LAST + M_CSTAMP_OFS) + { + GTimeVal tv; + + id -= M_CSTAMP_OFS; + cached_g_current_time(&tv); + sstamp.tv_sec = tv.tv_sec; + sstamp.tv_usec = tv.tv_usec; + sstamp.zone_offset = -1; + stamp = &sstamp; + } + else { g_assert_not_reached(); break; -- 1.7.0.4

2 1

Need help with filtering messages - new user with syslog-ng
by Cosmin Neagu 29 Aug '11

29 Aug '11

Hello, First of all, i started to use syslog-ng on Ubuntu a few days ago and it seams a great syslog server. But today i stumble on a problem. I configured snmptrapd with TRAPDOPTS='-Lsd ' and this means that snmptrapd will send the trap received to syslog-ng. Now, syslog-ng puts those traps by default in /var/log/syslog because of this default configurations: source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); destination d_syslog { file("/var/log/syslog"); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; log { source(s_src); filter(f_syslog3); destination(d_syslog); }; What i want to acomplish is to have traps from diferent host put in diferent files, not all together in the same file like it happens now. At first i tried to filter based on the host's ip address that was sending the trap, but i realized that the snmptrapd process is the one that sends the trap to syslog-ng process, not the device directly: Aug 29 11:42:48 Dell *snmptrapd*[3801]: 2011-08-29 11:42:43 10.90.0.252 [UDP: [10.90.0.252]:49364->[192.168.53.151]]: iso.3.6.1.2.1.1.3.0 = Timeticks: (1563318974) 180 days, 22:33:09.74 iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.4.1.9.9.41.2.0.1 iso.3.6.1.4.1.9.9.41.1.2.3.1.2.31 = STRING: "LINK" iso.3.6.1.4.1.9.9.41.1.2.3.1.3.31 = INTEGER: 4 iso.3.6.1.4.1.9.9.41.1.2.3.1.4.31 = STRING: "UPDOWN" iso.3.6.1.4.1.9.9.41.1.2.3.1.5.31 = STRING: "Interface Serial0/0/0, changed state to down" iso.3.6.1.4.1.9.9.41.1.2.3.1.6.31 = Timeticks: (1563318974) 180 days, 22:33:09.74 So maibe you have done this - how can i filter based on the program that it sending the message (like snmptrapd). And also, can filters based on the text itself can be used? Like: - if the mesage contains "10.90.0.252 [UDP: [10.90.0.252]:XXXXX->[192.168.53.151]" put the mesage in "this" file - if the mesage contains "10.90.1.22 [UDP: [10.90.1.22]:XXXXX->[192.168.53.151]" put the mesage in "that" file Thanks --

2 1

[PATCH] tfjson: Typo (& build) fix.
by Gergely Nagy 28 Aug '11

28 Aug '11

My previous patch contained a typo (an extra . at the end of a line) which resulted in breaking compilation. This patch fixes that typo. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/tfjson/tfjson.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c index 3d54201..65daadd 100644 --- a/modules/tfjson/tfjson.c +++ b/modules/tfjson/tfjson.c @@ -160,7 +160,7 @@ const ModuleInfo module_info = { .canonical_name = "tfjson", .version = VERSION, - .description = "The tfjson module provides a JSON formatting template function for syslog-ng.",. + .description = "The tfjson module provides a JSON formatting template function for syslog-ng.", .core_revision = SOURCE_REVISION, .plugins = builtin_tmpl_func_plugins, .plugins_len = G_N_ELEMENTS(builtin_tmpl_func_plugins), -- 1.7.2.5

1 0

[PATCH] scl: Add support for GNU/kFreeBSD
by Gergely Nagy 28 Aug '11

28 Aug '11

GNU/kFreeBSD is GNU userland (including libc) on top of a FreeBSD kernel. It's very similar to FreeBSD, but not completely. On GNU/kFreeBSD, the system log source is /var/run/log, while the kernel messages go to /dev/klog, in correctly formatted legacy BSD format. Tested on Debian GNU/kFreeBSD (squeeze). Signed-off-by: Gergely Nagy <algernon(a)madhouse-project.org> --- scl/system/generate-system-source.sh | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/scl/system/generate-system-source.sh b/scl/system/generate-system-source.sh index fab7135..1088b63 100755 --- a/scl/system/generate-system-source.sh +++ b/scl/system/generate-system-source.sh @@ -75,6 +75,12 @@ unix-dgram("/var/run/logpriv" perm(0600)); file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse)); EOF ;; + GNU/kFreeBSD) + cat <<EOF +unix-dgram("/var/run/log"); +file("/dev/klog" follow-freq(0) program-override("kernel")); +EOF + ;; HP-UX) cat <<EOF pipe("/dev/log" pad_size(2048)); -- 1.7.2.5

2 1

[PATCH] tfjson: Add module_info.
by Gergely Nagy 28 Aug '11

28 Aug '11

In order for syslog-ng --module-registry to be able to display meaningful information about a module, module_info must be set in said modules. Add that to tfjson, as it was lacking one. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/tfjson/tfjson.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c index 39a6cdb..3d54201 100644 --- a/modules/tfjson/tfjson.c +++ b/modules/tfjson/tfjson.c @@ -155,3 +155,13 @@ tfjson_module_init(GlobalConfig *cfg, CfgArgs *args) plugin_register(cfg, builtin_tmpl_func_plugins, G_N_ELEMENTS(builtin_tmpl_func_plugins)); return TRUE; } + +const ModuleInfo module_info = +{ + .canonical_name = "tfjson", + .version = VERSION, + .description = "The tfjson module provides a JSON formatting template function for syslog-ng.",. + .core_revision = SOURCE_REVISION, + .plugins = builtin_tmpl_func_plugins, + .plugins_len = G_N_ELEMENTS(builtin_tmpl_func_plugins), +}; -- 1.7.2.5

2 1

[PATCH] lib/Makefile.am: Include cfg-grammar.h in the package headers too.
by Gergely Nagy 28 Aug '11

28 Aug '11

cfg-grammar.h is included by some of the public headers we install, therefore cfg-grammar.h should be installed alongside aswell. Signed-off-by: Gergely Nagy <algernon(a)madhouse-project.org> --- lib/Makefile.am | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/lib/Makefile.am b/lib/Makefile.am index d941a8d..6527bc8 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -24,6 +24,7 @@ pkginclude_HEADERS = \ block-ref-parser.h \ center.h \ cfg.h \ + cfg-grammar.h \ cfg-lexer.h \ cfg-parser.h \ children.h \ -- 1.7.2.5

2 1

[Bug 131] New: XML tags being stripped
by bugzilla＠bugzilla.balabit.com 28 Aug '11

28 Aug '11

https://bugzilla.balabit.com/show_bug.cgi?id=131 Summary: XML tags being stripped Product: syslog-ng Version: 3.0.x Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jsu2(a)emory.edu Type of the Report: bug Estimated Hours: 0.0 Is there a known bug where syslog-ng arbitrarily strips xml tags from log messages? That is, some of our logs output xml, and syslog-ng seems to be stripping some of these tags before writing them. In particular, we have syslog-ng client configured to read from a local log file, send the log file to the syslog-ng server. The server does some basic csv parsing before writing it out. If this problem is fixed in a newer version, please let me know which version. Here's what the original log file should look like: 2011-08-24 12:00:02,685 INFO [Thread-35674] - [MessageIdentificationSyncCommand] Processing message <?xml version="1.0" encoding="UTF-8"?> <PurchaseOrderMessage version="2.2"> <Header> <MessageId>3125206c-5063-4465-a020-0131fc850885</MessageId> <Timestamp>2011-08-24T12:00:02.183-04:00</Timestamp> <Authentication> <Identity>Emory</Identity> <SharedSecret>PO$xport</SharedSecret> </Authentication> </Header> <PurchaseOrder> <POHeader id="16972984"> <WorkflowStatus status="Completed" /> <SupplierStatus status="SentToSupplier"> <HasBackorderedItems>false</HasBackorderedItems> <HasCancelledItems>false</HasCancelledItems> <HasSupplierEdits>false</HasSupplierEdits> <HasShippedItems>false</HasShippedItems> </SupplierStatus> <APStatus status="Close" /> <RevisionNumber>1</RevisionNumber> <RevisionDate>2011-08-24T11:54:10.745-04:00</RevisionDate> <PONumber>S660540</PONumber> <AlternatePONumber /> <Requestor> <UserProfile username="nbonus"> <FirstName>Nancy</FirstName> <LastName>Doe</LastName> <Email>ndoe(a)mydomain.bogus</Email> <Phone> <TelephoneNumber> <CountryCode>1</CountryCode> <AreaCode>555</AreaCode> <Number>5552765</Number> </TelephoneNumber> </Phone> <Department>My Department</Department> But, it shows up on syslog-ng server as: intesbprod1 [10857]: Error processing log message: <PurchaseOrderMessage version="2.2"> intesbprod1 : intesbprod1 : intesbprod1 : 00:02.183-04:00</Timestamp> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : id="16972984"> intesbprod1 : status="Completed" /> intesbprod1 : status="SentToSupplier"> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : status="Close" /> intesbprod1 : intesbprod1 : 54:10.745-04:00</RevisionDate> intesbprod1 : intesbprod1 : /> intesbprod1 : intesbprod1 : username="ndoe"> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : My Department</Department> Here's how the client is configured: source s_RoutingService { file("/opt/openeai/dev/logs/RoutingService.log" program_override("EnterpriseServiceBus RoutingService.log dev ")); }; source s_axis2 { file("/var/log/jboss/axis2.log" program_override("JBoss axis2.log qa ")); }; ###################### # logging rules # ###################### log { source(s_RoutingService); destination(d_net);} ; log { source(s_axis2); destination(d_net);} ; Here's how the server is configured: filter f_apps { program("EnterpriseServiceBus" flags ("ignore-case")) or program("JBoss" flags ("ignore-case")) or program("SonicMQ" flags ("ignore-case")); }; template t_apps { template("$SHOST ${ESB.MSG}\n"); }; parser p_apps { csv-parser(columns("ESB.LOG","ESB.DIR","ESB.MSG") delimiters(" ") flags(greedy, escape-none)); }; destination d_apps { file("/var/UTS/actv/$R_YEAR-$R_MONTH-$R_DAY/apps/$PROGRAM/${ESB.DIR}/${ESB.LOG}" create_dirs(yes) template(t_apps)); }; log { source(s_net); parser(p_shost); log { filter(f_systems); . . . log { filter(f_apps); parser(p_apps); destination(d_apps); flags(final); }; log { destination(d_srvr_mesg); flags(final); }; }; }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 2

syslog-ng 3.4 repo updated
by Balazs Scheidler 27 Aug '11

27 Aug '11

Hi, I just wanted to let you know, that I've forward merged all changes in 3.3 into the 3.4 tree. compiled && "make check" ran fine. I'd like start merging new functionalities from the syslog-ng PE team in BalaBit and Algernon's work. If there's anything else I should focus on, please let me know. Thanks. -- Bazsi

1 0