- syslog-ng - lists.balabit.hu

Need help with filtering messages - new user with syslog-ng
by Cosmin Neagu 29 Aug '11

29 Aug '11

Hello, First of all, i started to use syslog-ng on Ubuntu a few days ago and it seams a great syslog server. But today i stumble on a problem. I configured snmptrapd with TRAPDOPTS='-Lsd ' and this means that snmptrapd will send the trap received to syslog-ng. Now, syslog-ng puts those traps by default in /var/log/syslog because of this default configurations: source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); destination d_syslog { file("/var/log/syslog"); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; log { source(s_src); filter(f_syslog3); destination(d_syslog); }; What i want to acomplish is to have traps from diferent host put in diferent files, not all together in the same file like it happens now. At first i tried to filter based on the host's ip address that was sending the trap, but i realized that the snmptrapd process is the one that sends the trap to syslog-ng process, not the device directly: Aug 29 11:42:48 Dell *snmptrapd*[3801]: 2011-08-29 11:42:43 10.90.0.252 [UDP: [10.90.0.252]:49364->[192.168.53.151]]: iso.3.6.1.2.1.1.3.0 = Timeticks: (1563318974) 180 days, 22:33:09.74 iso.3.6.1.6.3.1.1.4.1.0 = OID: iso.3.6.1.4.1.9.9.41.2.0.1 iso.3.6.1.4.1.9.9.41.1.2.3.1.2.31 = STRING: "LINK" iso.3.6.1.4.1.9.9.41.1.2.3.1.3.31 = INTEGER: 4 iso.3.6.1.4.1.9.9.41.1.2.3.1.4.31 = STRING: "UPDOWN" iso.3.6.1.4.1.9.9.41.1.2.3.1.5.31 = STRING: "Interface Serial0/0/0, changed state to down" iso.3.6.1.4.1.9.9.41.1.2.3.1.6.31 = Timeticks: (1563318974) 180 days, 22:33:09.74 So maibe you have done this - how can i filter based on the program that it sending the message (like snmptrapd). And also, can filters based on the text itself can be used? Like: - if the mesage contains "10.90.0.252 [UDP: [10.90.0.252]:XXXXX->[192.168.53.151]" put the mesage in "this" file - if the mesage contains "10.90.1.22 [UDP: [10.90.1.22]:XXXXX->[192.168.53.151]" put the mesage in "that" file Thanks --

2 1

[PATCH] tfjson: Typo (& build) fix.
by Gergely Nagy 28 Aug '11

28 Aug '11

My previous patch contained a typo (an extra . at the end of a line) which resulted in breaking compilation. This patch fixes that typo. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/tfjson/tfjson.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c index 3d54201..65daadd 100644 --- a/modules/tfjson/tfjson.c +++ b/modules/tfjson/tfjson.c @@ -160,7 +160,7 @@ const ModuleInfo module_info = { .canonical_name = "tfjson", .version = VERSION, - .description = "The tfjson module provides a JSON formatting template function for syslog-ng.",. + .description = "The tfjson module provides a JSON formatting template function for syslog-ng.", .core_revision = SOURCE_REVISION, .plugins = builtin_tmpl_func_plugins, .plugins_len = G_N_ELEMENTS(builtin_tmpl_func_plugins), -- 1.7.2.5

1 0

[PATCH] scl: Add support for GNU/kFreeBSD
by Gergely Nagy 28 Aug '11

28 Aug '11

GNU/kFreeBSD is GNU userland (including libc) on top of a FreeBSD kernel. It's very similar to FreeBSD, but not completely. On GNU/kFreeBSD, the system log source is /var/run/log, while the kernel messages go to /dev/klog, in correctly formatted legacy BSD format. Tested on Debian GNU/kFreeBSD (squeeze). Signed-off-by: Gergely Nagy <algernon(a)madhouse-project.org> --- scl/system/generate-system-source.sh | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/scl/system/generate-system-source.sh b/scl/system/generate-system-source.sh index fab7135..1088b63 100755 --- a/scl/system/generate-system-source.sh +++ b/scl/system/generate-system-source.sh @@ -75,6 +75,12 @@ unix-dgram("/var/run/logpriv" perm(0600)); file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse)); EOF ;; + GNU/kFreeBSD) + cat <<EOF +unix-dgram("/var/run/log"); +file("/dev/klog" follow-freq(0) program-override("kernel")); +EOF + ;; HP-UX) cat <<EOF pipe("/dev/log" pad_size(2048)); -- 1.7.2.5

2 1

[PATCH] tfjson: Add module_info.
by Gergely Nagy 28 Aug '11

28 Aug '11

In order for syslog-ng --module-registry to be able to display meaningful information about a module, module_info must be set in said modules. Add that to tfjson, as it was lacking one. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/tfjson/tfjson.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/modules/tfjson/tfjson.c b/modules/tfjson/tfjson.c index 39a6cdb..3d54201 100644 --- a/modules/tfjson/tfjson.c +++ b/modules/tfjson/tfjson.c @@ -155,3 +155,13 @@ tfjson_module_init(GlobalConfig *cfg, CfgArgs *args) plugin_register(cfg, builtin_tmpl_func_plugins, G_N_ELEMENTS(builtin_tmpl_func_plugins)); return TRUE; } + +const ModuleInfo module_info = +{ + .canonical_name = "tfjson", + .version = VERSION, + .description = "The tfjson module provides a JSON formatting template function for syslog-ng.",. + .core_revision = SOURCE_REVISION, + .plugins = builtin_tmpl_func_plugins, + .plugins_len = G_N_ELEMENTS(builtin_tmpl_func_plugins), +}; -- 1.7.2.5

2 1

[PATCH] lib/Makefile.am: Include cfg-grammar.h in the package headers too.
by Gergely Nagy 28 Aug '11

28 Aug '11

cfg-grammar.h is included by some of the public headers we install, therefore cfg-grammar.h should be installed alongside aswell. Signed-off-by: Gergely Nagy <algernon(a)madhouse-project.org> --- lib/Makefile.am | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/lib/Makefile.am b/lib/Makefile.am index d941a8d..6527bc8 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -24,6 +24,7 @@ pkginclude_HEADERS = \ block-ref-parser.h \ center.h \ cfg.h \ + cfg-grammar.h \ cfg-lexer.h \ cfg-parser.h \ children.h \ -- 1.7.2.5

2 1

[Bug 131] New: XML tags being stripped
by bugzilla＠bugzilla.balabit.com 28 Aug '11

28 Aug '11

https://bugzilla.balabit.com/show_bug.cgi?id=131 Summary: XML tags being stripped Product: syslog-ng Version: 3.0.x Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jsu2(a)emory.edu Type of the Report: bug Estimated Hours: 0.0 Is there a known bug where syslog-ng arbitrarily strips xml tags from log messages? That is, some of our logs output xml, and syslog-ng seems to be stripping some of these tags before writing them. In particular, we have syslog-ng client configured to read from a local log file, send the log file to the syslog-ng server. The server does some basic csv parsing before writing it out. If this problem is fixed in a newer version, please let me know which version. Here's what the original log file should look like: 2011-08-24 12:00:02,685 INFO [Thread-35674] - [MessageIdentificationSyncCommand] Processing message <?xml version="1.0" encoding="UTF-8"?> <PurchaseOrderMessage version="2.2"> <Header> <MessageId>3125206c-5063-4465-a020-0131fc850885</MessageId> <Timestamp>2011-08-24T12:00:02.183-04:00</Timestamp> <Authentication> <Identity>Emory</Identity> <SharedSecret>PO$xport</SharedSecret> </Authentication> </Header> <PurchaseOrder> <POHeader id="16972984"> <WorkflowStatus status="Completed" /> <SupplierStatus status="SentToSupplier"> <HasBackorderedItems>false</HasBackorderedItems> <HasCancelledItems>false</HasCancelledItems> <HasSupplierEdits>false</HasSupplierEdits> <HasShippedItems>false</HasShippedItems> </SupplierStatus> <APStatus status="Close" /> <RevisionNumber>1</RevisionNumber> <RevisionDate>2011-08-24T11:54:10.745-04:00</RevisionDate> <PONumber>S660540</PONumber> <AlternatePONumber /> <Requestor> <UserProfile username="nbonus"> <FirstName>Nancy</FirstName> <LastName>Doe</LastName> <Email>ndoe(a)mydomain.bogus</Email> <Phone> <TelephoneNumber> <CountryCode>1</CountryCode> <AreaCode>555</AreaCode> <Number>5552765</Number> </TelephoneNumber> </Phone> <Department>My Department</Department> But, it shows up on syslog-ng server as: intesbprod1 [10857]: Error processing log message: <PurchaseOrderMessage version="2.2"> intesbprod1 : intesbprod1 : intesbprod1 : 00:02.183-04:00</Timestamp> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : id="16972984"> intesbprod1 : status="Completed" /> intesbprod1 : status="SentToSupplier"> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : status="Close" /> intesbprod1 : intesbprod1 : 54:10.745-04:00</RevisionDate> intesbprod1 : intesbprod1 : /> intesbprod1 : intesbprod1 : username="ndoe"> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : My Department</Department> Here's how the client is configured: source s_RoutingService { file("/opt/openeai/dev/logs/RoutingService.log" program_override("EnterpriseServiceBus RoutingService.log dev ")); }; source s_axis2 { file("/var/log/jboss/axis2.log" program_override("JBoss axis2.log qa ")); }; ###################### # logging rules # ###################### log { source(s_RoutingService); destination(d_net);} ; log { source(s_axis2); destination(d_net);} ; Here's how the server is configured: filter f_apps { program("EnterpriseServiceBus" flags ("ignore-case")) or program("JBoss" flags ("ignore-case")) or program("SonicMQ" flags ("ignore-case")); }; template t_apps { template("$SHOST ${ESB.MSG}\n"); }; parser p_apps { csv-parser(columns("ESB.LOG","ESB.DIR","ESB.MSG") delimiters(" ") flags(greedy, escape-none)); }; destination d_apps { file("/var/UTS/actv/$R_YEAR-$R_MONTH-$R_DAY/apps/$PROGRAM/${ESB.DIR}/${ESB.LOG}" create_dirs(yes) template(t_apps)); }; log { source(s_net); parser(p_shost); log { filter(f_systems); . . . log { filter(f_apps); parser(p_apps); destination(d_apps); flags(final); }; log { destination(d_srvr_mesg); flags(final); }; }; }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 2

syslog-ng 3.4 repo updated
by Balazs Scheidler 27 Aug '11

27 Aug '11

Hi, I just wanted to let you know, that I've forward merged all changes in 3.3 into the 3.4 tree. compiled && "make check" ran fine. I'd like start merging new functionalities from the syslog-ng PE team in BalaBit and Algernon's work. If there's anything else I should focus on, please let me know. Thanks. -- Bazsi

1 0

Variable issue with syslog-ng
by Aldrich, Jamie S 25 Aug '11

25 Aug '11

I am trying to setup syslog-ng to use the DAY and MONTH variable on the source file input. What am I doing wrong? source s_file { file("/psfs_logs/APPSRV_${MONTH}${DAY}.LOG" flags(no-parse)); }; destination d_messages{ udp("10.13.33.11"); }; log { source(s_file); destination(d_messages); }; Any help would be greatly appreciated...

2 2

[PATCH] basicfuncs: Implement a $(now) template function.
by Gergely Nagy 25 Aug '11

25 Aug '11

This implements a new basic template function: $(now), which will return the current time (seconds since the epoch) as of calling the template function. For the rare case that the time of template expansion is needed, or cached_gmtime() granularity is not enough. Signed-Off-By: Gergely Nagy <algernon(a)balabit.hu> --- modules/basicfuncs/basic-funcs.c | 12 ++++++++++++ 1 files changed, 12 insertions(+), 0 deletions(-) diff --git a/modules/basicfuncs/basic-funcs.c b/modules/basicfuncs/basic-funcs.c index 5e1fd5e..abe8c68 100644 --- a/modules/basicfuncs/basic-funcs.c +++ b/modules/basicfuncs/basic-funcs.c @@ -19,6 +19,17 @@ tf_echo(LogMessage *msg, gint argc, GString *argv[], GString *result) TEMPLATE_FUNCTION_SIMPLE(tf_echo); +static void +tf_now(LogMessage *msg, gint argc, GString *argv[], GString *result) +{ + GTimeVal tval; + + g_get_current_time(&tval); + g_string_append_printf (result, "%lu", tval.tv_sec); +} + +TEMPLATE_FUNCTION_SIMPLE(tf_now); + typedef struct _TFCondState { FilterExprNode *filter; @@ -158,6 +169,7 @@ static Plugin basicfuncs_plugins[] = TEMPLATE_FUNCTION_PLUGIN(tf_echo, "echo"), TEMPLATE_FUNCTION_PLUGIN(tf_grep, "grep"), TEMPLATE_FUNCTION_PLUGIN(tf_if, "if"), + TEMPLATE_FUNCTION_PLUGIN(tf_now, "now"), }; gboolean -- 1.7.0.4

3 5

fsf address, embedded date
by Peter Czanik 24 Aug '11

24 Aug '11

Hello, While working on the openSUSE package, I ran into this problem. Post compile checks brought up this problem: syslog-ng.x86_64: W: incorrect-fsf-address /usr/share/syslog-ng/include/scl/system/plugin.conf syslog-ng.x86_64: W: incorrect-fsf-address /usr/share/syslog-ng/include/scl/syslogconf/plugin.conf syslog-ng.x86_64: W: incorrect-fsf-address /usr/share/syslog-ng/include/scl/pacct/plugin.conf syslog-ng.x86_64: W: incorrect-fsf-address /etc/syslog-ng/scl.conf syslog-ng.x86_64: W: incorrect-fsf-address /usr/share/syslog-ng/include/scl/system/generate-system-source.sh The Free Software Foundation address in this file seems to be outdated or misspelled. Ask upstream to update the address, or if this is a license file, possibly the entire file with a new copy available from the FSF. There is another one I usually fix with a patch: syslog-ng.x86_64: W: file-contains-date-and-time /sbin/syslog-ng Your file uses __DATE and __TIME__ this causes the package to rebuild when not needed The reasoning behind this is, that if a package is recompiled for some reason, but results in the binary same files, then dependent packages are not recompiled. But if date is embedded, then it changes on each build... For syslog-ng-3.3.0beta1 it was this patch: linux-1wrf:~/home:czanik:syslog-ng33/syslog-ng # cat 1/syslog-ng-3.3.0beta1-no-datetime.diff --- syslog-ng/main.c.orig 2011-05-24 17:32:35.696000107 +0200 +++ syslog-ng/main.c 2011-05-24 17:33:04.886000067 +0200 @@ -128,7 +128,9 @@ printf(PACKAGE " " VERSION "\n" "Installer-Version: %s\n" "Revision: " SOURCE_REVISION "\n" +#if 0 "Compile-Date: " __DATE__ " " __TIME__ "\n" +#endif "Default-Modules: %s\n" "Available-Modules: ", installer_version, Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2