April 2011 - syslog-ng - lists.balabit.hu

[Bug 104] New: Add libtool option -module
by bugzilla＠bugzilla.balabit.com 23 Oct '11

23 Oct '11

https://bugzilla.balabit.com/show_bug.cgi?id=104 Summary: Add libtool option -module Product: syslog-ng Version: 3.2.x Platform: All OS/Version: All Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vinschen(a)redhat.com Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=30) --> (https://bugzilla.balabit.com/attachment.cgi?id=30) Patch to add libtool option -module Libtool 2.2.6 and later provides an option -module which should be used when creating shared libraries which are supposed to be used as modules which get loaded via dlopen. See the "Building modules to dlopen" chapter in `info libtool'. For Cygwin the additional benefit is that the modules are not copied into a parallel "bin" directory at installation time, which is the default otherwise to accommodate the DLL search algorithm on Windows. The attached patch just adds the -module option to each of the Makefile.am files in the modules subdir. Corinna -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 4

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Syslog-ng error while using TSL
by Pramod Pillai 04 Jun '11

04 Jun '11

Hi I am getting following error while trying to configure TSL in syslogng Error On Client Certificate validation failed; subject='C=IN, ST=KAR, O=orola, CN=12.168.50.192, emailAddress=a(a)d.com', issuer='C=Generic, ST=Generic, O=Generic, CN=Generic_Int_CA_1', error='unable to get local issuer certificate', depth='0' SSL error while writing stream; tls_error='SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed' I/O error occurred while writing; fd='4', error='Broken pipe (32)' Syslog connection broken; fd='4', server='AF_INET(10.232.165.128:5695)', time_reopen='60' Error on Server SSL error while reading stream; tls_error='SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca' Regards Pramod

4 8

Re: [syslog-ng] [PATCH RFC] Use cap_syslog when it is available (v3)
by Gergely Nagy 08 May '11

08 May '11

(forwarding the original mail, as it looks like it never made to the list) Serge Hallyn <serge.hallyn(a)canonical.com> writes: > (A-ha - great idea from Andrew - we *can* figure out whether the > kernel knows about CAP_SYSLOG, using the bounding set API) > > If cap_syslog exists, the kernel will complain (once) that we only > have cap_sys_admin. Additionally, using cap_syslog instead of > cap_sys_admin significantly lowers the unneeded privs we are > using. > > Changelog: > v2: At startup, detect whether libcap knows about CAP_SYSLOG. > (Thanks to Gergely Nagy for pointing out that case) > v3: Andrew Morgan pointed out a nice way to detect whether > the kernel has CAP_SYSLOG. Thanks, Andrew! > > Signed-off-by: Serge Hallyn <serge.hallyn(a)ubuntu.com> > --- > lib/gprocess.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++- > lib/gprocess.h | 2 +- > modules/affile/affile.c | 4 +++ > syslog-ng/main.c | 10 ++------ > 4 files changed, 55 insertions(+), 9 deletions(-) > > diff --git a/lib/gprocess.c b/lib/gprocess.c > index d9a2dfb..f9bda7b 100644 > --- a/lib/gprocess.c > +++ b/lib/gprocess.c > @@ -98,6 +98,7 @@ static gint startup_result_pipe[2] = { -1, -1 }; > static gint init_result_pipe[2] = { -1, -1 }; > static GProcessKind process_kind = G_PK_STARTUP; > static gboolean stderr_present = TRUE; > +static int have_capsyslog = FALSE; > > /* global variables */ > static struct > @@ -210,6 +211,14 @@ g_process_cap_modify(int capability, int onoff) > if (!process_opts.caps) > return TRUE; > > +#ifdef CAP_SYSLOG > + /* > + * if libcap or kernel doesn't support cap_syslog, then resort to > + * cap_sys_admin > + */ > + if (capability == CAP_SYSLOG && !have_capsyslog) > + capability = CAP_SYS_ADMIN; > +#endif > caps = cap_get_proc(); > if (!caps) > return FALSE; > @@ -416,7 +425,7 @@ g_process_set_working_dir(const gchar *cwd) > * capability set. The process will change its capabilities to this value > * during startup, provided it has enough permissions to do so. > **/ > -void > +static void > g_process_set_caps(const gchar *caps) > { > if (!process_opts.caps) > @@ -879,6 +888,43 @@ g_process_change_dir(void) > } > > /** > + * g_process_setup_caps(void) > + * > + * NOTE: polling /proc/kmsg requires cap_sys_admin, otherwise it'll always > + * indicate readability. Enabling/disabling cap_sys_admin on every poll > + * invocation seems to be too expensive. So I enable it for now. > + */ > +void > +g_process_setup_caps(void) > +{ > + int ret; > + gchar * capsstr = "cap_net_bind_service,cap_net_broadcast,cap_net_raw," > + "cap_dac_read_search,cap_dac_override,cap_chown,cap_fowner=p " > + "cap_sys_admin=ep"; > +#ifdef CAP_SYSLOG > + cap_t caps; > + > + /* Check whether cap_syslog exists. If not, get cap_sys_admin, else get > + * cap_syslog */ > + caps = cap_from_text("cap_syslog=p"); > + if (caps) { > + cap_free(caps); > + /* libcap knows it, does the kernel? */ > + ret = prctl(PR_CAPBSET_READ, CAP_SYSLOG); > + if (ret != -1) { > + /* kernel knows cap_syslog! Use it */ > + capsstr = "cap_net_bind_service,cap_net_broadcast,cap_net_raw," > + "cap_dac_read_search,cap_dac_override,cap_chown,cap_fowner=p " > + "cap_syslog=ep"; > + have_capsyslog = TRUE; > + } > + } > +#endif > + > + g_process_set_caps(capsstr); > +} > + > +/** > * g_process_send_result: > * @ret_num: exit code of the process > * > diff --git a/lib/gprocess.h b/lib/gprocess.h > index cda35b0..4ad0866 100644 > --- a/lib/gprocess.h > +++ b/lib/gprocess.h > @@ -66,7 +66,7 @@ void g_process_set_chroot(const gchar *chroot); > void g_process_set_pidfile(const gchar *pidfile); > void g_process_set_pidfile_dir(const gchar *pidfile_dir); > void g_process_set_working_dir(const gchar *cwd); > -void g_process_set_caps(const gchar *caps); > +void g_process_setup_caps(void); > void g_process_set_argv_space(gint argc, gchar **argv); > void g_process_set_use_fdlimit(gboolean use); > void g_process_set_check(gint check_period, gboolean (*check_fn)(void)); > diff --git a/modules/affile/affile.c b/modules/affile/affile.c > index a713bfc..358a083 100644 > --- a/modules/affile/affile.c > +++ b/modules/affile/affile.c > @@ -59,7 +59,11 @@ affile_open_file(gchar *name, gint flags, > if (privileged) > { > g_process_cap_modify(CAP_DAC_READ_SEARCH, TRUE); > +#ifdef CAP_SYSLOG > + g_process_cap_modify(CAP_SYSLOG, TRUE); > +#else > g_process_cap_modify(CAP_SYS_ADMIN, TRUE); > +#endif > } > else > { > diff --git a/syslog-ng/main.c b/syslog-ng/main.c > index 7e81f2f..5e62c1f 100644 > --- a/syslog-ng/main.c > +++ b/syslog-ng/main.c > @@ -167,13 +167,9 @@ main(int argc, char *argv[]) > > g_process_set_argv_space(argc, (gchar **) argv); > > - /* NOTE: polling /proc/kmsg requires cap_sys_admin, otherwise it'll always > - * indicate readability. Enabling/disabling cap_sys_admin on every poll > - * invocation seems to be too expensive. So I enable it for now. */ > - > - g_process_set_caps("cap_net_bind_service,cap_net_broadcast,cap_net_raw," > - "cap_dac_read_search,cap_dac_override,cap_chown,cap_fowner=p " > - "cap_sys_admin=ep"); > + /* Set up the minimal privilege we'll need */ > + g_process_setup_caps(); > + > ctx = g_option_context_new("syslog-ng"); > g_process_add_option_group(ctx); > msg_add_option_group(ctx); -- |8]

2 4

Solaris 10 UDP overflows, message drops
by Mishou Michael 03 May '11

03 May '11

All, I've done a lot of reading, and I can't figure out what I can do to this config in order to fix the UDP drops due to udpInOverflows on netstat -s. Here are some statistics relating to the amount of traffic we receive via syslog-ng, it's pretty busy but in reading I'm finding that some folks are doing much more. These stats are based on a ~30 second window of traffic during peak times, but variance due to time is not so much in our environment. I used tcpdump with a bpf to capture only inbound udp/514, so this is what the interface is seeing in the way of syslog. Elapsed: 00:00:34 Packets: 200000 Avg. packets/sec: 5836.546 Avg. packet size: 303.182 bytes Bytes: 60636477 Avg. bytes/sec: 1769537.884 Avg. MBit/sec: 14.156 So, about 6k messages per second. Here are the drop numbers over a time sample (done right after a process restart, you can see the buffer takes a moment to fill up [64 MB so_rcvbuf]): # while true; do echo -en "$(date) :: "; netstat -s | grep udpInOverflows | head -n 1 | sed 's|.*=||'; sleep 10; done Fri Apr 15 14:12:46 GMT 2011 :: 472517477 Fri Apr 15 14:12:56 GMT 2011 :: 472517477 Fri Apr 15 14:13:06 GMT 2011 :: 472517477 Fri Apr 15 14:13:16 GMT 2011 :: 472517477 Fri Apr 15 14:13:26 GMT 2011 :: 472543152 Fri Apr 15 14:13:36 GMT 2011 :: 472592800 Fri Apr 15 14:13:46 GMT 2011 :: 472638848 Fri Apr 15 14:13:56 GMT 2011 :: 472684407 So that's about 5k overflows a second, which jives with our calculations, suggesting we're getting only ~10% of our messages logged to disk. I inherited a config with _very_ many filter statements, but have decided to cut all that out to see if my performance problems in the way of udp drops continue (they do). I've attached a sanitized config to this message, all the stuff here concerns this config running (even though I thought eliminating the filters would really help, it didn't). We're running Solaris 10 SPARC. The syslog-ng version is: # /usr/local/sbin/syslog-ng -V syslog-ng 3.1.2 Installer-Version: 3.1.2 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainli ne--3.1#master#8bf13c304b6ab5fc1a372b49d55c78370efe14ca Compile-Date: Oct 25 2010 23:56:18 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on The following options are set for the OS: # ndd /dev/udp udp_max_buf 1073741824 # ndd /dev/udp udp_recv_hiwat 65536 Some options lines from the config based on what I've seen: * note the TCP stuff can be safely ignored, it's legacy from some testing but isn't currently seeing traffic * all 3 udp sources set with so_rcvbuf(67108864) (64 MB) options { # things I've changed/tweaked flush_lines(1000); flush_timeout(20); log_fifo_size (67108864); log_msg_size(8192); chain_hostnames(yes); # end my changes <snip> }; So I'm totally stumped. I can set the buffers with so_rcvbuf() to 1 GB, it still doesn't matter, they eventually fill up and I start losing packets. I'm hoping that someone can point me to some tweaks I can do to get the numbers of drops down or eliminated. Is it unreasonable to expect to be able to process this many messages per second via UDP? Maybe that's the problem. I might experiment some with default syslog to see if it can write this many messages without drops...this doesn't seem like an insane amount of traffic. But perhaps my expectations are unrealistic, that's what I'm hoping someone can tell me. Regards, --Mike

11 37

rewrite problem
by Micah Anderson 03 May '11

03 May '11

Hi, I've been trying to use the rewrite functionality using 3.1.3, but I am having some problems: 1. when having a rewrite setup, I can denial of service syslog-ng :( if I do this: rewrite r_ip { subst("([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}", "\[REDACTED\]", value("MESSAGE"), type("pcre"), flags("global")); }; filter f_mail { facility(mail) and not facility(local1) and not program("^postfix/"); }; destination df_mail { file("/var/log/mail.log"); }; log { source(s_src); filter(f_mail); rewrite(r_ip); destination(df_mail); flags(final); }; and then I do: $ logger -p mail.info hi syslog-ng starts going crazy, it eats 100% of the CPU and the memory. If you are lucky, the OOM killer kicks in and stops it before your system goes down. I originally thought that this was due to a bad regexp I had, but the above regexp works fine for normal mail syslog messages. However, in the process of trying to make a better rewrite rule, I found that I've created ones where normal mail messages do this too. 2. \. doesn't match literal periods Using the above rewrite, one would expect that the regexp would match three times a number containing 1-3 digits, each time followed by a literal period. Then a number containing 1-3 digits. A basic, and not very complete IPv4 regexp, that would match something like: 192.168.25.1. In fact, it does match that IPv4 address, but it is also matching other things that do not have periods in them. For example: rewrite r_ip { subst("([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}", "\[REDACTED\]", value("MESSAGE"), type("pcre"), flags("global")); }; $ logger -i -p mail.info 'Apr 21 20:38:58 algae postfix/smtp[14817]: EB4A2168: to=<482092-subscribe(a)bugs.debian.org>, relay=192.168.164.34[192.168.164.34]:587, delay=2.9, delays=0.19/0.14/2/0.58, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 6722214C119)' I would expect that the two IP addressesw in the above to be matched and replaced, but this is what happens: Apr 22 12:45:11 algae logger[6234]: Apr [REDACTED] algae postfix/smtp[14817]: EB4A2168: to=<482092-subscribe(a)bugs.debian.org>, relay=[REDACTED][[REDACTED]]:587, delay=2.9, delays=[REDACTED]/2/0.58, dsn=2.0.0, status=sent ([REDACTED] Ok: queued as [REDACTED]) Its matching the queue ID 'EB4A2168' and its also matching '6722214C119', neither of those have periods in them, and EB4A2168 is not 0-9, which is what :digit: should be matching. I tried to replace [[:digit:]] with \d, but that caused issue #1 to happen. micah --

3 7

[PATCH 0/3]: value-pairs() & related patches
by Gergely Nagy 01 May '11

01 May '11

Hi! For a multitude of reasons, ranging from the feature freeze looming over my head to hope that discrete patches are easier to review and merge than my awfully chaotic git tree, I'll be posting a couple of patches in a short while, as soon as I figure out what magic arguments I need to give to git send-email to make it send the batch the way I imagined. value-pairs() ============= The first one will be the value-pairs() implementation (also available on my integration/value-pairs branch), which is the base of the two others that will follow. While the value-pairs() implementation could still use some work, especially in the performance department, I believe it's good enough for initial use, and the benefits of having it (so that the mongodb driver can make use of it) far outweight the downsides. The implementation itself is reasonably light, it hardly touches anything outside it's own two files: the grammar and the makefile only, to hook itself in. If not used by a driver, then it does not change functionality at all. Therefore, I believe this could go in before the feature freeze. afmongodb value-pairs() port ============================ The next patch will be the port of the already existing mongodb destination driver to use value-pairs(). Also available on my integration/afmongodb-vp branch. This means that the driver will finally be able to do what it's good at: store arbitrary information in the database: whether that information comes via SDATA or patterndb, it doesn't matter, it will be stored in a structured manner. In my opinion, this is the major selling point of the mongodb destination driver at the moment (I have some other grand plans with it in the future, but that's more invasive, less important, and can easily wait 'till 3.4 ;). Without value-pairs(), afmongodb is pretty much useless. tfson() ======= For extra fun & profit, Balint Kovacs' format_json has been ported to use value-pairs(), and this will be the third patch in the series, and is also available on my integration/tfjson branch. This one's the least important, though it's dead simple, therefore I'd be very happy if it made past the feature freeze aswell. -- |8]

3 15

[PATCH] basicfuncs: Implement a $(substr START LEN STR) template function.
by Gergely Nagy 01 May '11

01 May '11

This implements the $(substr) template function - a fairly primitive one: it does error checking, but there's no error reporting present, therefore invalid function calls will just get ignored. Usage should be straightforward. Reported-By: Jakub Jankowski <shasta(a)toxcorp.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/basicfuncs/basic-funcs.c | 46 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 46 insertions(+), 0 deletions(-) diff --git a/modules/basicfuncs/basic-funcs.c b/modules/basicfuncs/basic-funcs.c index 44b8af3..542c2ec 100644 --- a/modules/basicfuncs/basic-funcs.c +++ b/modules/basicfuncs/basic-funcs.c @@ -4,6 +4,9 @@ #include "filter-expr-parser.h" #include "cfg.h" +#include <stdlib.h> +#include <errno.h> + static void tf_echo(LogMessage *msg, gint argc, GString *argv[], GString *result) { @@ -19,6 +22,48 @@ tf_echo(LogMessage *msg, gint argc, GString *argv[], GString *result) TEMPLATE_FUNCTION_SIMPLE(tf_echo); +static gboolean +tf_substr_parse_int(const gchar *s, long *d) +{ + gchar *endptr; + glong val; + + errno = 0; + val = strtoll(s, &endptr, 10); + + if ((errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) + || (errno != 0 && val == 0)) + return FALSE; + + if (endptr == s || *endptr != '\0') + return FALSE; + + *d = val; + return TRUE; +} + +static void +tf_substr(LogMessage *msg, gint argc, GString *argv[], GString *result) +{ + glong start, len; + + if (argc != 3) + return; + + if (!tf_substr_parse_int (argv[0]->str, &start)) + return; + if (!tf_substr_parse_int (argv[1]->str, &len)) + return; + + if (start > argv[2]->len || + start + len >= argv[2]->len) + return; + + g_string_append_len (result, argv[2]->str + start - 1, len); +} + +TEMPLATE_FUNCTION_SIMPLE(tf_substr); + typedef struct _TFCondState { FilterExprNode *filter; @@ -158,6 +203,7 @@ static Plugin basicfuncs_plugins[] = TEMPLATE_FUNCTION_PLUGIN(tf_echo, "echo"), TEMPLATE_FUNCTION_PLUGIN(tf_grep, "grep"), TEMPLATE_FUNCTION_PLUGIN(tf_if, "if"), + TEMPLATE_FUNCTION_PLUGIN(tf_substr, "substr"), }; gboolean -- 1.7.2.5

3 8

SSL certificate verification
by Peter Eckel 30 Apr '11

30 Apr '11

Hi all, I am currently setting up a remote logging facility for a customer who requires encryption and mutual authentication. After having run Syslog NG over Stunnel for several years, I am looking into the option to do the same withot stunnel, by using Syslog NG with the TLS option. My current version is 3.2.2, installed from the binary provided on the Syslog NG web site, on CentOS 5.5. My setup on the server side (stripped to the minimum): source s_remote { tcp( ip( 192.168.42.184 ) port( 601 ) tls( key_file("/opt/syslog-ng/etc/ssl/server_key.pem") cert_file("/opt/syslog-ng/etc/ssl/server_crt.pem") ca_dir( "/opt/syslog-ng/etc/ssl" ) peer_verify( required-trusted ) ) ); }; destination d_remote { file( "/var/log/remote" owner(root) group(root) perm(0600) ); }; log { source(s_remote); destination(d_remote); }; On the client side: source s_local { internal(); unix-stream( "/dev/log" ); file( "/proc/kmsg" ); }; destination d_remote { tcp( "192.168.42.184" port( 601 ) tls( key_file("/opt/syslog-ng/etc/ssl/client1_key.pem") cert_file("/opt/syslog-ng/etc/ssl/client1_crt.pem") ca_dir( "/opt/syslog-ng/etc/ssl" ) ) ); }; log { source(s_local); destination(d_remote); }; With the appropriate and valid client cert, it works perfrctly: [root@server1 ssl]# openssl x509 -noout -text < client1_crt.pem Certificate: Data: Version: 1 (0x0) Serial Number: 2 (0x2) Signature Algorithm: sha256WithRSAEncryption Issuer: O=xxx, OU=Client 1/emailAddress=xxx, L=xxx, ST=xxx, C=xx, CN=xxx Validity Not Before: Mar 8 15:53:24 2011 GMT Not After : Mar 7 15:53:24 2014 GMT Subject: C=xx, ST=xx, O=xx, OU=xx, CN=client1 Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (2048 bit) ... However, I copied over the cert of another client machine to client1, reconfigure Syslog NG to use it, and it works as well, though IMHO it shouldn't: [root@server1 ssl]# openssl x509 -noout -text < client2_crt.pem Certificate: Data: Version: 1 (0x0) Serial Number: 3 (0x3) Signature Algorithm: sha256WithRSAEncryption Issuer: O=xxx, OU=Client 2/emailAddress=xxx, L=xxx, ST=xxx, C=xx, CN=xxx Validity Not Before: Mar 8 15:54:35 2011 GMT Not After : Mar 7 15:54:35 2014 GMT Subject: C=xx, ST=xxx, O=xxx, OU=Client 2, CN=client2 Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (2048 bit) Modulus (2048 bit): ... Swapping certificates (client2's instead of server's in this case) on the server breaks trust (as expected): Mar 8 18:19:45 client1 syslog-ng[2991]: Syslog connection established; fd='10', server='AF_INET(192.168.42.184:601)', local='AF_INET(0.0.0.0:0)' Mar 8 18:20:13 client1 syslog-ng[2991]: Certificate subject does not match configured hostname; hostname='server', certificate='client2' Mar 8 18:20:13 client1 syslog-ng[2991]: SSL error while writing stream; tls_error='SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed' Mar 8 18:20:13 client1 syslog-ng[2991]: I/O error occurred while writing; fd='10', error='Broken pipe (32)' Doing the same on the client, however, does not. The Syslog NG server does not seem to validate the CN. Is this expected behaviour or a bug? Best regards, Peter.

4 10

filtering per device
by vijay 30 Apr '11

30 Apr '11

Hi All, I have 4 cisco devices having ip addresses for example 10.1.1.1 10.1.1.2 10.1.1.3 10.1.1.4 already i have filtering option such that only above informational level(level 6) will be captured.I have new requirement that to capture informational logs also of 10.1.1.4 not other devices.How to do that. thanks vijay

3 2