- syslog-ng - lists.balabit.hu

[Bug 131] New: XML tags being stripped
by bugzilla＠bugzilla.balabit.com 28 Aug '11

28 Aug '11

https://bugzilla.balabit.com/show_bug.cgi?id=131 Summary: XML tags being stripped Product: syslog-ng Version: 3.0.x Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jsu2(a)emory.edu Type of the Report: bug Estimated Hours: 0.0 Is there a known bug where syslog-ng arbitrarily strips xml tags from log messages? That is, some of our logs output xml, and syslog-ng seems to be stripping some of these tags before writing them. In particular, we have syslog-ng client configured to read from a local log file, send the log file to the syslog-ng server. The server does some basic csv parsing before writing it out. If this problem is fixed in a newer version, please let me know which version. Here's what the original log file should look like: 2011-08-24 12:00:02,685 INFO [Thread-35674] - [MessageIdentificationSyncCommand] Processing message <?xml version="1.0" encoding="UTF-8"?> <PurchaseOrderMessage version="2.2"> <Header> <MessageId>3125206c-5063-4465-a020-0131fc850885</MessageId> <Timestamp>2011-08-24T12:00:02.183-04:00</Timestamp> <Authentication> <Identity>Emory</Identity> <SharedSecret>PO$xport</SharedSecret> </Authentication> </Header> <PurchaseOrder> <POHeader id="16972984"> <WorkflowStatus status="Completed" /> <SupplierStatus status="SentToSupplier"> <HasBackorderedItems>false</HasBackorderedItems> <HasCancelledItems>false</HasCancelledItems> <HasSupplierEdits>false</HasSupplierEdits> <HasShippedItems>false</HasShippedItems> </SupplierStatus> <APStatus status="Close" /> <RevisionNumber>1</RevisionNumber> <RevisionDate>2011-08-24T11:54:10.745-04:00</RevisionDate> <PONumber>S660540</PONumber> <AlternatePONumber /> <Requestor> <UserProfile username="nbonus"> <FirstName>Nancy</FirstName> <LastName>Doe</LastName> <Email>ndoe(a)mydomain.bogus</Email> <Phone> <TelephoneNumber> <CountryCode>1</CountryCode> <AreaCode>555</AreaCode> <Number>5552765</Number> </TelephoneNumber> </Phone> <Department>My Department</Department> But, it shows up on syslog-ng server as: intesbprod1 [10857]: Error processing log message: <PurchaseOrderMessage version="2.2"> intesbprod1 : intesbprod1 : intesbprod1 : 00:02.183-04:00</Timestamp> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : id="16972984"> intesbprod1 : status="Completed" /> intesbprod1 : status="SentToSupplier"> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : status="Close" /> intesbprod1 : intesbprod1 : 54:10.745-04:00</RevisionDate> intesbprod1 : intesbprod1 : /> intesbprod1 : intesbprod1 : username="ndoe"> intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : intesbprod1 : My Department</Department> Here's how the client is configured: source s_RoutingService { file("/opt/openeai/dev/logs/RoutingService.log" program_override("EnterpriseServiceBus RoutingService.log dev ")); }; source s_axis2 { file("/var/log/jboss/axis2.log" program_override("JBoss axis2.log qa ")); }; ###################### # logging rules # ###################### log { source(s_RoutingService); destination(d_net);} ; log { source(s_axis2); destination(d_net);} ; Here's how the server is configured: filter f_apps { program("EnterpriseServiceBus" flags ("ignore-case")) or program("JBoss" flags ("ignore-case")) or program("SonicMQ" flags ("ignore-case")); }; template t_apps { template("$SHOST ${ESB.MSG}\n"); }; parser p_apps { csv-parser(columns("ESB.LOG","ESB.DIR","ESB.MSG") delimiters(" ") flags(greedy, escape-none)); }; destination d_apps { file("/var/UTS/actv/$R_YEAR-$R_MONTH-$R_DAY/apps/$PROGRAM/${ESB.DIR}/${ESB.LOG}" create_dirs(yes) template(t_apps)); }; log { source(s_net); parser(p_shost); log { filter(f_systems); . . . log { filter(f_apps); parser(p_apps); destination(d_apps); flags(final); }; log { destination(d_srvr_mesg); flags(final); }; }; }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 2

syslog-ng 3.4 repo updated
by Balazs Scheidler 27 Aug '11

27 Aug '11

Hi, I just wanted to let you know, that I've forward merged all changes in 3.3 into the 3.4 tree. compiled && "make check" ran fine. I'd like start merging new functionalities from the syslog-ng PE team in BalaBit and Algernon's work. If there's anything else I should focus on, please let me know. Thanks. -- Bazsi

1 0

Variable issue with syslog-ng
by Aldrich, Jamie S 25 Aug '11

25 Aug '11

I am trying to setup syslog-ng to use the DAY and MONTH variable on the source file input. What am I doing wrong? source s_file { file("/psfs_logs/APPSRV_${MONTH}${DAY}.LOG" flags(no-parse)); }; destination d_messages{ udp("10.13.33.11"); }; log { source(s_file); destination(d_messages); }; Any help would be greatly appreciated...

2 2

[PATCH] basicfuncs: Implement a $(now) template function.
by Gergely Nagy 25 Aug '11

25 Aug '11

This implements a new basic template function: $(now), which will return the current time (seconds since the epoch) as of calling the template function. For the rare case that the time of template expansion is needed, or cached_gmtime() granularity is not enough. Signed-Off-By: Gergely Nagy <algernon(a)balabit.hu> --- modules/basicfuncs/basic-funcs.c | 12 ++++++++++++ 1 files changed, 12 insertions(+), 0 deletions(-) diff --git a/modules/basicfuncs/basic-funcs.c b/modules/basicfuncs/basic-funcs.c index 5e1fd5e..abe8c68 100644 --- a/modules/basicfuncs/basic-funcs.c +++ b/modules/basicfuncs/basic-funcs.c @@ -19,6 +19,17 @@ tf_echo(LogMessage *msg, gint argc, GString *argv[], GString *result) TEMPLATE_FUNCTION_SIMPLE(tf_echo); +static void +tf_now(LogMessage *msg, gint argc, GString *argv[], GString *result) +{ + GTimeVal tval; + + g_get_current_time(&tval); + g_string_append_printf (result, "%lu", tval.tv_sec); +} + +TEMPLATE_FUNCTION_SIMPLE(tf_now); + typedef struct _TFCondState { FilterExprNode *filter; @@ -158,6 +169,7 @@ static Plugin basicfuncs_plugins[] = TEMPLATE_FUNCTION_PLUGIN(tf_echo, "echo"), TEMPLATE_FUNCTION_PLUGIN(tf_grep, "grep"), TEMPLATE_FUNCTION_PLUGIN(tf_if, "if"), + TEMPLATE_FUNCTION_PLUGIN(tf_now, "now"), }; gboolean -- 1.7.0.4

3 5

fsf address, embedded date
by Peter Czanik 24 Aug '11

24 Aug '11

Hello, While working on the openSUSE package, I ran into this problem. Post compile checks brought up this problem: syslog-ng.x86_64: W: incorrect-fsf-address /usr/share/syslog-ng/include/scl/system/plugin.conf syslog-ng.x86_64: W: incorrect-fsf-address /usr/share/syslog-ng/include/scl/syslogconf/plugin.conf syslog-ng.x86_64: W: incorrect-fsf-address /usr/share/syslog-ng/include/scl/pacct/plugin.conf syslog-ng.x86_64: W: incorrect-fsf-address /etc/syslog-ng/scl.conf syslog-ng.x86_64: W: incorrect-fsf-address /usr/share/syslog-ng/include/scl/system/generate-system-source.sh The Free Software Foundation address in this file seems to be outdated or misspelled. Ask upstream to update the address, or if this is a license file, possibly the entire file with a new copy available from the FSF. There is another one I usually fix with a patch: syslog-ng.x86_64: W: file-contains-date-and-time /sbin/syslog-ng Your file uses __DATE and __TIME__ this causes the package to rebuild when not needed The reasoning behind this is, that if a package is recompiled for some reason, but results in the binary same files, then dependent packages are not recompiled. But if date is embedded, then it changes on each build... For syslog-ng-3.3.0beta1 it was this patch: linux-1wrf:~/home:czanik:syslog-ng33/syslog-ng # cat 1/syslog-ng-3.3.0beta1-no-datetime.diff --- syslog-ng/main.c.orig 2011-05-24 17:32:35.696000107 +0200 +++ syslog-ng/main.c 2011-05-24 17:33:04.886000067 +0200 @@ -128,7 +128,9 @@ printf(PACKAGE " " VERSION "\n" "Installer-Version: %s\n" "Revision: " SOURCE_REVISION "\n" +#if 0 "Compile-Date: " __DATE__ " " __TIME__ "\n" +#endif "Default-Modules: %s\n" "Available-Modules: ", installer_version, Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

Announcing Debian packages for syslog-ng 3.3
by Gergely Nagy 23 Aug '11

23 Aug '11

Hi! As I've hinted about it in an earlier mail, I've been working on packaging syslog-ng 3.3beta2 (mostly to scratch an own itch: I want 3.3 packaged, and split into smaller parts :P), and as of this writing, packages are slowly flowing in to the newly created repository: # For Debian unstable & testing: deb http://packages.madhouse-project.org/debian unstable syslog-ng deb-src http://packages.madhouse-project.org/debian unstable syslog-ng # For Debian stable (squeeze): # (this also contains libmongo-client and libevtlog backported to # squeeze) deb http://packages.madhouse-project.org/debian stable syslog-ng deb-src http://packages.madhouse-project.org/debian stable syslog-ng For now, only amd64 binaries are built, if there's a need, I can build i386 and powerpc packages aswell. These Debian packages are NOT the stock debian syslog-ng package updated for 3.3: the packaging was pretty much redone from scratch (and they're still a work in progress). Nevertheless, I believe they're in a usable shape, and could be useful for people on Debian wishing to play with syslog-ng 3.3. Anyway, the repositories above contain the following packages: * syslog-ng This was turned into a meta-package, that doesn't really have any contents, but depends on the rest of the syslog-ng packages. It is provided both as a clear upgrade path from the stock Debian packages, and as a way to install every plugin syslog-ng has. * syslog-ng-core This is what used to be the syslog-ng package in stock Debian: the standard feature set, minus a few plugins, which were split out. See below! * syslog-ng-module-sql Contains the SQL destination module. Split out into a separate package, so that the core does not have to depend on libdbi. * syslog-ng-module-mongodb The MongoDB destination driver. * syslog-ng-module-json The $(format-json) (aka tfjson) plugin. * syslog-ng-dev Development files (headers, tools, etc) to build third-party syslog-ng modules. This part is non-functional at the moment, it will be fixed at some point in the not too distant future, but since this part is mainly useful for me alone, I didn't care that much. If there's any issues with the repository, or with the packages in them, please let me know, so I can fix them! I will keep updating the packages, to fix any bugs I - or anyone else - finds. -- |8]

1 1

libafsocket.so
by Peter Czanik 23 Aug '11

23 Aug '11

Hello, I'm working on the 3.3 beta2 package for openSUSE and I wonder, if there is any reason to keep libafsocket-notls.so in the package. -rwxr-xr-x 1 root root 117965 Aug 19 11:18 libafsocket-notls.so lrwxrwxrwx 1 root root 18 Aug 23 10:14 libafsocket.so -> libafsocket-tls.so -rwxr-xr-x 1 root root 117965 Aug 19 11:18 libafsocket-tls.so Only afsocket is referenced from modules.conf, which is a link to libafsocket-tls.so If I remove the link and the notls version and rename the tls to libafsocket.so, then I could get rid of a couple of post build check warnings (which are soon to be turned into errors), and also re-claim 100+k from my HDD. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

Problem with program_override in upgrade from 3.0.8 to 3.2.4
by Javi Polo 22 Aug '11

22 Aug '11

Hello there I've been using syslog-ng for a long time, no problems so far, till today ... :p I'm using Open Source Edition, upgrading from 3.0.8 to 3.2.4, installing from the .run file to /opt Today I wanted to update our syslog-ng's to the latest version and found that for some reason, when I override a program via program_override, the PROGRAM macro is empty when I send it to another loghost. program_override seems to be working, as locally writen files show so I did upgrade both syslog-ng, the client and the logserver When I switched back to the old version I found everything began working again Here's the conflicting config in the client: #CLIENT CONFIG source s_apache_access { file("/var/log/apache2/access.log" program_override ("apache_access")); }; destination d_logserver01 { tcp("logserver01"); }; destination d_tmp { file("/var/log/tmp.log" template("$HOST $PROGRAM $MESSAGE\n")); }; log { source(s_apache_error); destination(d_logserver01); destination(d_tmp); flags(final); }; And here's on the server: #LOGSERVER CONFIG filter f_tmp { host(web0[1-2]); }; filter f_apache_error { program(apache_error); }; destination d_apache_error { file("/array/logs/${S_YEAR}/${S_MONTH}/${S_DAY}/${HOST}/apache2/error.log" template("$ISODATE $HOST ${PROGRAM}: $MESSAGE\n") ); }; destination d_tmp { file("/array/logs/${S_YEAR}/${S_MONTH}/${S_DAY}/${HOST}/tmp.log" template("$HOST $PROGRAM $MSG\n")); }; log { source(s_tcp); filter(f_apache_error); destination(d_apache_error); flags(final); }; log { source(s_tcp); filter(f_tmp); destination(d_tmp); flags(final); }; After the update those are the results of tmp.log On client: web01 apache_error Aug 02 20:53:06 2011] [error] [client 192.168.0.5] client denied by server configuration: /var/www/asdadsasdas On server: web01 Aug 02 20:53:06 2011] [error] [client 192.168.50.65] client denied by server configuration: /var/www/asdadsasdas As you can see, the program_name has dissapeared I suspect is a bug, but I'm not sure as there might be some change in syslog-ng behaviour that I did miss :? Anybody could lend me a hand? thanks in advance :) -- Javi Polo Administrador de Sistemas Tel 93 734 97 70 Fax 93 734 97 71 jpolo(a)wtransnet.com

2 4

Quick question about libsyslog-ng.so.0
by Gergely Nagy 21 Aug '11

21 Aug '11

Hi! As part of packaging syslog-ng, and splitting it into smaller packages, I face the problem of having to create a devel package. Now, I could go the easy way (as I've done with the current set of Debian packages) and create a syslog-ng-core package, which would contain libsyslog-ng.so.0, the plugins and the rest, and a syslog-ng-dev package with the headers and other tools. This could work, but the downside is that on an automated buildd, one does not neccessarily want to replace the running syslogd (even if that's already syslog-ng, which it is in my case) just to be able to build a package. So I was thinking about splitting the libsyslog-ng.so library out into its own package. And herein lies the problem! The SONAME of the library is currently not set, thus, libtool defaults to libsyslog-ng.so.0. The problem with this, is that the SONAME will remain the same, even if the library breaks its ABI underneath. Which will be a problem for any third party module linked against this library: a new version of syslog-ng comes out, all of them will break at runtime, in various interesting ways, perhaps even after days of working, when the module runs over an incompatible change. This can be worked around on the packaging level, by ensuring that a new version of the libsyslog-ng package will conflict with any older versions, but... that's a pain to maintain, and very fragile. So instead, I'd like to propose a different way: use the -release flag of Libtool. This will make the library's SONAME contain syslog-ng's version aswell, thus, every upgrade will use a different soname (like libsyslog-ng-3.3.0beta2.so), and modules will break at startup time, which is much better, as we don't trip over unexpected changes. It is also tremendously easier to package a semi-private library this way. (Though, this has a few downsides as well from a packaging point of view, but considerably less.) So here I am, seeking input on the issue! By the way, a third option also exists: maintaining ABI and API compatibility, and only bumping the SONAME where something did really change incompatibly. This would be the best option, but this also takes a lot more work to maintain properly (I'm happy to help with this, though, I'm already doing this for libmongo-client; but on the other hand, libsyslog-ng really is something syslog-ng specific, and not a general purpose library, so it might not worth the trouble to go this far). -- |8]

2 2

A couple of notes regarding packagability
by Gergely Nagy 20 Aug '11

20 Aug '11

Hi! I started to play around with creating various packages of 3.3.0beta2, and noticed a couple of annoying bits and pieces (none of this is serious, I can workaround most if not all of them, but the packaging would be a lot simpler if I wouldn't have to): * I sent a patch back in May, which made it possible to use a system-wide libmongo-client installation, but it does not appear to be merged yet. It is attached to this email, diffed against current git head. * autogen.sh is not executable in a fresh checkout, and this repeatedly bites me over and over again. * autogen.sh will always try to call git submodule init when autogenerating the submodules. I'd like a way to skip this step, and just use whatever is in the subdirectories, assuming there IS something. The reason I want this, is because I'm building from a git checkout, not a make dist'd tarball, and my build chroots do not have network access (nor do I want to give them any). * It would also be nice if autogen.sh used the --copy flag for libtoolize and automake. * When the system-wide libmongo-client support patch is applied, and syslog-ng is built with --with-libmongo-client=system, make distclean will fail, as it tries to clean in modules/afmongodb/libmongo-client/, while that hasn't been configured in. I do not know an easy solution around this, except maybe overriding DIST_SUBDIRS with SUBDIRS for the distclean-recursive target in modules/afmongodb/ - but that sounds like a crude hack, and I haven't tested if it actually works. This is all, so far. I'll have another packaging sprint sometime tomorrow, and see if I find anything else. I'll try to find time to submit patches for the above, but I'm not sure if I'll be able to this weekend. It is very likely, that I'll submit a patch for system() to recognise GNU/kFreeBSD and create the appropriate entries. PS: syslog-ng 3.3.0beta2 packages for Debian squeeze & sid should be available by the end of the coming week, for i386, amd64 and kfreebsd-amd64 at least. I just have to iron out a few issues and add some finishing touches to my build farm. -- |8]

2 2