- syslog-ng - lists.balabit.hu

versioning and maintenance
by Balazs Scheidler 26 Apr '11

26 Apr '11

Hi, First of all, sorry for being absent from this forum for the past couple of weeks. I've tried to focus here too sometimes, but due to health issues, I sometimes simply have too little time to do too much. But anyway, I haven't stopped thinking about the way forward, and I've thought I'd share my thoughts about those with you. Please note that this is not an official announcement, but rather a way to gather feedback on my plans. Everything can be changed, and after we get to an agreement, an "official" announcement will be sent out on the conclusions. First of all, some status information: ====================================== * syslog-ng 3.0, 3.1 and 3.2 have all produced stable versions, and were all maintained in parallel. * syslog-ng 3.3 is in development, with the primary aim at scaling syslog-ng to multiple CPUs. It has produced two alpha releases so far. * syslog-ng team in BalaBit is busy merging the OSE and PE editions of syslog-ng, in order to use a common core. * the team is basing their work on OSE 3.3, which has been available as two alpha releases and the git tree, also their tree is also published as a git repository. * while they were porting the PE functionality, they were also busy porting the new core to new platforms and also of course fixing bugs, they've found. * I've been helping the team in case they've found something that affected the open source edition as well. Then some news about my future plans on syslog-ng versions: =========================================================== * I feel that maintaining 3 releases while doing development of a 4th one is too much for me. * I originally planned to employ the same stable/feature release structure that BalaBit's commercial products are using. There, a stable release is maintained for a longer time (similar to Ubuntu LTS), while incremental feature releases are published, with shorter support periods (e.g. like normal Ubuntu releases). * The stable/feature release breakdown would mean, that 3.0 is a long term release, and 3.1 & 3.2 are short term releases. * However, this doesn't match real-life expectations, distributions use newer versions (and I really applaud this), but this also means that dropping support for versions included in major Linux distributions would cause disruptions. * I'm really happy that distributions can move faster, and I'd really like to speed up syslog-ng development. Instead of sticking to an old release and stuffing in bugfixes and minor features; publish a major release more often, so that features don't cause problems in stable releases. * Therefore, I plan to abandon the feature/stable release breakdown altogether. One release will be the same as the other. I'm pledging support for major releases as long as: - there's only one release to be maintained in parallel to the development tree. - I plan to support a given major release for about a year. * This means, that syslog-ng support for OSE 3.0 & 3.1 will be discontinued, in the not too distant future. I'm planning on a 3 month period, in order to let people have time to prepare for the upgrade. E.g. I'd like to announce that these versions are going to be supported until 31th July, 2011. * The definition of "support" is of course vague, I can't provide SLAs or anything, but I'll do my best to help anyone who has issues with those versions. * Also, if a distribution which carries this version has an issue, I'll try to help, even past that EOL date, but general questions and problems found in those versions by users, will be advised to upgrade to a newer release, or to backport a specific fix themselves. Going forward: ============== * I plan to announce a feature freeze for 3.3 and publish a beta version. And at the same time open the 3.4 branch for integrating all the stuff which has accumulated on the mailing list. * Hopefully we can have a stable 3.3 by the time the previous two releases hit their end-of-life in July. What if you disagree: ===================== * Until now, I didn't really publish "hard" EOL dates for syslog-ng OSE, simply because there were not that many of the parallel branches. However with the increase in development pace, I simply cannot keep up, and I hate to do a substandard job. Backporting fixes to 3 branches, is simply too much effort. * I'm happy to help building up the proper infrastructure outside of BalaBit if someone wants to help in maintenance, and release engineering. But this requires real effort from members of the community. Now, having written this down, please give your feedback, and I'm trying to work down my backlog and start integrating patches. Thanks for listening. -- Bazsi

3 6

duplicated log lines
by Len Conrad 22 Apr '11

22 Apr '11

FreeBSD sl1.hctc.net 8.2-RELEASE syslog-ng 2.0.10 I filter the mail flow into per-mail-sever files. I start with the FreeBSD syslog-ng.conf which emulates the system syslog.conf and add my stuff. The problem is that all maillog files have duplicate lines. After modifying a bunch of stuff, no success. Either I get no maillog traffic at all, or the dupes remain. How to debug? thanks Len

2 1

Info about owner() options
by Cosmin Neagu 22 Apr '11

22 Apr '11

Hello, New around here, i'm trying to use the owner option when logging into file. My configs: source s_udp { udp();}; destination df_hosts { file("/home/cosmin/SYSLOG/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY$YEAR$MONTH$DAY" owner(cosmin) group(cosmin) perm(0600) dir_perm(0700) create_dirs(yes)); }; log { source(s_udp); destination(df_hosts); }; The only problem is that files, are owned by root, not by cosmin. cosmin@DellCosmin:~/SYSLOG$ sudo ls -l HOSTS/ total 8 drwx------ 3 root root 4096 2011-04-22 10:41 10.90.0.252 drwx------ 3 root root 4096 2011-04-22 10:41 192.168.156.100 So, can someone tell me if i'm doing something wrong? I would like the files to be created and owned by cosmin:cosmin not root:root The process is running as root: cosmin@DellCosmin:~/SYSLOG$ ps -ef | grep syslog-ng root 3448 1 0 10:41 ? 00:00:00 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid -- Cosmin Neagu NOC Team Leader Str. I. G. Duca nr. 36 Otopeni, Judetul Ilfov, 075100 Romania Tel: 021 303 3159 / 0732 669 193 www.omnilogic.ro

1 1

Cacti's syslog plugin - How to remove a host
by Wagner Pereira 19 Apr '11

19 Apr '11

Hi, folks. First, I'm sorry if this is not the right mailing list to handle this, but...here we go: I need to remove a host from my Select host(s) list, in the Cacti's syslog plugin tab. I'm using the CactiEZ solution. Regards, -- Wagner Pereira PoP-SP/RNP - Ponto de Presença da RNP em São Paulo CCE/USP - Centro de Computação Eletrônica da Universidade de São Paulo http://www.pop-sp.rnp.br Tel. +55 11 3091-8901 fone@RNP: 10158902

2 1

Re: [syslog-ng] [PATCH RFC] Use cap_syslog when it is available (v3)
by Gergely Nagy 18 Apr '11

18 Apr '11

Serge Hallyn <serge.hallyn(a)canonical.com> writes: > (A-ha - great idea from Andrew - we *can* figure out whether the > kernel knows about CAP_SYSLOG, using the bounding set API) > > If cap_syslog exists, the kernel will complain (once) that we only > have cap_sys_admin. Additionally, using cap_syslog instead of > cap_sys_admin significantly lowers the unneeded privs we are > using. > > Changelog: > v2: At startup, detect whether libcap knows about CAP_SYSLOG. > (Thanks to Gergely Nagy for pointing out that case) > v3: Andrew Morgan pointed out a nice way to detect whether > the kernel has CAP_SYSLOG. Thanks, Andrew! > > Signed-off-by: Serge Hallyn <serge.hallyn(a)ubuntu.com> Looks good to me! -- |8]

1 0

[PATCH RFC] Use cap_syslog when it is available
by Serge Hallyn 18 Apr '11

18 Apr '11

If cap_syslog exists, the kernel will complain (once per boot) that we only have cap_sys_admin. Ideally, we would detect whether the kernel knows about cap_syslog, and not use cap_sys_admin if it does. However, there is no good way to deduce that. We can only detect whether userspace knows about it, and kernel may not match userspace. I will work to get such polling into the kernel and, when successful, will provide a syslog-ng patch to use that, so that when CAP_SYSLOG is supported by the kernel, we can reduce the amount of privilege we needlessly take. Signed-off-by: Serge Hallyn <serge.hallyn(a)ubuntu.com> --- lib/gprocess.c | 35 ++++++++++++++++++++++++++++++++++- lib/gprocess.h | 2 +- modules/affile/affile.c | 3 +++ syslog-ng/main.c | 10 +++------- 4 files changed, 41 insertions(+), 9 deletions(-) diff --git a/lib/gprocess.c b/lib/gprocess.c index d9a2dfb..0884237 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -416,7 +416,7 @@ g_process_set_working_dir(const gchar *cwd) * capability set. The process will change its capabilities to this value * during startup, provided it has enough permissions to do so. **/ -void +static void g_process_set_caps(const gchar *caps) { if (!process_opts.caps) @@ -879,6 +879,39 @@ g_process_change_dir(void) } /** + * g_process_setup_caps(void) + * + * NOTE: polling /proc/kmsg requires cap_sys_admin, otherwise it'll always + * indicate readability. Enabling/disabling cap_sys_admin on every poll + * invocation seems to be too expensive. So I enable it for now. + * + * If CAP_SYSLOG exists, add that to our list of capabilities alongside + * CAP_SYS_ADMIN. Ideally we would *replace* it. However, we currently + * have no reliable way to check whether the kernel - which may not + * match userspace - supports it. If it doesn't, there is no harm in + * adding CAP_SYSLOG, but we of course still need CAP_SYS_ADMIN for + * /proc/kmsg. + * + * Once the kernel supports polling for the number of capabilities, we + * can drop CAP_SYSLOG after detecting whether CAP_SYSLOG exists. + */ +void +g_process_setup_caps(void) +{ +#ifdef CAP_SYSLOG + gchar * capsstr = "cap_net_bind_service,cap_net_broadcast,cap_net_raw," + "cap_dac_read_search,cap_dac_override,cap_chown,cap_fowner=p " + "cap_syslog,cap_sys_admin=ep"; +#else + gchar * capsstr = "cap_net_bind_service,cap_net_broadcast,cap_net_raw," + "cap_dac_read_search,cap_dac_override,cap_chown,cap_fowner=p " + "cap_sys_admin=ep"; +#endif + + g_process_set_caps(capsstr); +} + +/** * g_process_send_result: * @ret_num: exit code of the process * diff --git a/lib/gprocess.h b/lib/gprocess.h index cda35b0..4ad0866 100644 --- a/lib/gprocess.h +++ b/lib/gprocess.h @@ -66,7 +66,7 @@ void g_process_set_chroot(const gchar *chroot); void g_process_set_pidfile(const gchar *pidfile); void g_process_set_pidfile_dir(const gchar *pidfile_dir); void g_process_set_working_dir(const gchar *cwd); -void g_process_set_caps(const gchar *caps); +void g_process_setup_caps(void); void g_process_set_argv_space(gint argc, gchar **argv); void g_process_set_use_fdlimit(gboolean use); void g_process_set_check(gint check_period, gboolean (*check_fn)(void)); diff --git a/modules/affile/affile.c b/modules/affile/affile.c index a713bfc..b11b7ac 100644 --- a/modules/affile/affile.c +++ b/modules/affile/affile.c @@ -59,6 +59,9 @@ affile_open_file(gchar *name, gint flags, if (privileged) { g_process_cap_modify(CAP_DAC_READ_SEARCH, TRUE); +#ifdef CAP_SYSLOG + g_process_cap_modify(CAP_SYSLOG, TRUE); +#endif g_process_cap_modify(CAP_SYS_ADMIN, TRUE); } else diff --git a/syslog-ng/main.c b/syslog-ng/main.c index 7e81f2f..5e62c1f 100644 --- a/syslog-ng/main.c +++ b/syslog-ng/main.c @@ -167,13 +167,9 @@ main(int argc, char *argv[]) g_process_set_argv_space(argc, (gchar **) argv); - /* NOTE: polling /proc/kmsg requires cap_sys_admin, otherwise it'll always - * indicate readability. Enabling/disabling cap_sys_admin on every poll - * invocation seems to be too expensive. So I enable it for now. */ - - g_process_set_caps("cap_net_bind_service,cap_net_broadcast,cap_net_raw," - "cap_dac_read_search,cap_dac_override,cap_chown,cap_fowner=p " - "cap_sys_admin=ep"); + /* Set up the minimal privilege we'll need */ + g_process_setup_caps(); + ctx = g_option_context_new("syslog-ng"); g_process_add_option_group(ctx); msg_add_option_group(ctx); -- 1.7.4.1

2 1

Problem with Squid logs
by Matias Banchoff 15 Apr '11

15 Apr '11

Hello, I have a problem with remote logging for Squid logs. Our setup is the following: - syslog-ng server: syslog-ng 3.1.3. It is a dedicated server for logging. From now on, the server. - syslog-ng in Squid: syslog-ng 2.0.9. From now on, the client. The Squid process writes three log files: access.log, store.log and cache.log. I have configured the client syslog-ng to send those files to the log server. The problem is that the content of those files are sent only when syslog-ng starts on the client side. So: 1) The syslog-ng client writes all the information to the local files (access, cache and store). So, locally, it works. 2) The information is sent, but only when the client syslog-ng process restarts. So it is not a networking problem. 3) And, I've left the default config for all the other log stuff (like messages, syslog, etc.). That information is also sent to the log server. And, in this case, the information is sent constantly. I mean, I don't have to restart the syslog-ng client to make the Squid machine send the "messages", "syslog", "mail" and other logs. I've googled a while, but the only thing I found was some article about delayed logs, but it was due a bad timestamp. The thing here is Squid writes its logs with a strange timestamp: "1302699202.239", for example. Is there any way to translate that Squid timestamp into a more readable one, so I can probe if that is the problem? In addition, I've started syslog-ng client y debug mode (with -d -F), but nothing appears in the screen, although the logs are being written. Thank you for any answer or comment you might have :-) ----- CeSPI Centro Superior para el Procesamiento de la Información Universidad Nacional de La Plata ------------------------------------------------------------------------------- Proteja el Medioambiente. No imprima este mail si no es absolutamente necesario

4 4

Current stable version of syslog-ng
by Yu Watanabe 15 Apr '11

15 Apr '11

Hello all. I would like to ask a question regarding to the version of syslog-ng. Which version would be the most stable version for now? v 3.0.10 v 3.1.4 v 3.2.2 I asked the same question before but I didn't recieve a reply so I would like to ask again. Thanks, Yu Watanabe

3 2

syslog-ng Insider - April 2011
by Peter Czanik 14 Apr '11

14 Apr '11

Dear syslog-ng users, This is the second issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng OSE 3.3 development ----------------------------- The next syslog-ng release is developed at rocket speed. The second alpha release brought in support for MongoDB as destination, and threading for increased performance and scalability. Community involvement in the development process is increasing, with many patches and detailed bug reports, which helped to fix correlation problems. Direct SMTP support is also available as a not yet merged patch. There were also long discussions about JSON and name value pairs with value-pairs() and tfjson() available as patches. One can download the alpha2 release, or if impatient also check it out from git, which has regular updates at http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=summary Alpha2 release: https://bazsi.blogs.balabit.com/2011/03/the-2nd-alpha-for-syslog-ng-3-3-is-… Mailing list archive with many patches: https://lists.balabit.hu/pipermail/syslog-ng/ A blog, how to get started with coding: http://algernon.blogs.balabit.com/2011/01/howto-write-a-simple-template-fun… syslog-ng called open source project of the year ------------------------------------------------ IDG hold its first Open Source conference on the 24th of February in Hungary. A new award was established for open source projects, and it was first given to syslog-ng OSE. Blog series on web GUIs for syslog-ng ------------------------------------- Web based GUIs for syslog-ng is a hot topic recently, so a new blog series was started. Loganalyzer, Loggly and Logzilla are already covered and a few more reviews (ELSA, Octopussy) are already under way. If you have any suggestions what else to cover, please let us know! Logzilla (formerly php-syslog-ng): http://czanik.blogs.balabit.com/2011/04/logzilla-brings-syslog-ng-and-cisco… Loganalyzer: https://czanik.blogs.balabit.com/2011/02/loganalyzer/ Loggly: https://czanik.blogs.balabit.com/2011/03/logging-to-the-cloud/ Support the syslog-ng project with your testimonial! ---------------------------------------------------- We send syslog-ng T-Shirts for the best use-cases and testimonials. We are especially interested in how you use syslog-ng: your environment, performance metrics, the UNIX platforms that you use and anything that you deem interesting about your setup. This is your possibility to influence the syslog-ng development process and to help the syslog-ng project. http://www.balabit.com/testimonials/send OTHER SHORT NEWS * openSUSE 11.4 comes with syslog-ng 3.2: https://czanik.blogs.balabit.com/2011/03/opensuse-11-4-with-syslog-ng-and-s… * FreeBSD 8.2 comes with syslog-ng 3.2: https://czanik.blogs.balabit.com/2011/02/freebsd-8-2-released/ * Intrusion prevention based on syslog-ng and patterndb: https://bazsi.blogs.balabit.com/2011/02/another-nice-hack-with-syslog-ng/ * ELSA (Enterprise Log Search and Archive Project) had its first public release: http://code.google.com/p/enterprise-log-search-and-archive RECENT WHITEPAPERS * http://www.balabit.com/whitepaper?wp=8988187171444164966624785 This document aims at briefly demonstrating the technological and business processes that led to the revaluation of log information to such an extent that its collection and procession has become one of the most essential tasks of contemporary corporate IT. Furthermore, we attempt to identify those critical points of a central logging management project that determine its success the most. We hope that taking our six pieces of advice relating to these six points will substantially contribute to the successful professional activities of our readers. ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Openssl SSL3_WRITE_PENDING error correction
by Juhasz Viktor 12 Apr '11

12 Apr '11

Hi All! We've found a bug in the log_proto_framed_client. The bug is the following: - using ssl communication between the server and client. - using syslog protocol - If client send messages faster than the server can process messages, then the client go into the SSL3_WRITE_PENDING openssl error and close the connection. partial message handling correction: the flush is called only if proto have partially sent message after flush is called log_proto_text_client_post have to return because the post send one message per call it is important in case of framed messages diff --git a/lib/logproto.c b/lib/logproto.c index d821275..7f2f897 100644 --- a/lib/logproto.c +++ b/lib/logproto.c @@ -145,9 +145,13 @@ log_proto_text_client_post(LogProto *s, guchar *msg, gsize msg_len, gboolean *co g_assert(self->super.convert == (GIConv) -1); *consumed = FALSE; - rc = log_proto_flush(s); - if (rc == LPS_ERROR) - goto write_error; + if(self->partial) + { + rc = log_proto_flush(s); + if (rc == LPS_ERROR) + goto write_error; + return rc; + } /* OK, partial buffer empty, now flush msg that we just got */ @@ -1774,6 +1778,7 @@ log_proto_framed_client_new(LogTransport *transport) self->super.super.prepare = log_proto_text_client_prepare; self->super.super.post = log_proto_framed_client_post; + self->super.super.flush = log_proto_text_client_flush; self->super.super.transport = transport; self->super.super.convert = (GIConv) -1; return &self->super.super; Best Regards, Viktor

3 4

Current stable version
by Yu Watanabe 12 Apr '11

12 Apr '11

Hi all I would like to ask question about open source edition. Which version will be the current stable version? Thanks, Yu Watanabe

1 0

syslog-ng IPv6 problem ver 3.01
by Arya, Manish Kumar 10 Apr '11

10 Apr '11

I am getting syntax error on line udp6 or tcp6 in destination same works well as source in version 3.2.1 deb:/var/log# netstat -na | grep 514 tcp6 0 0 fd00:abcd:4321:2:20:514 :::* LISTEN udp6 0 0 fd00:abcd:4321:2:20:514 :::* here is my config for syslog client using syslog-ng 3.01 source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); udp(); }; destination ipv6_host { # udp6(ip(fd00:abcd:4321:2:20c:29ff:fea8:9671) port(514)); tcp6(ip(fd00:abcd:4321:2:20c:29ff:fea8:9671) port(514)); }; destination d_local { file("/var/log/messages"); }; log { source(s_local); destination(ipv6_host); destination(d_local); };

4 7

syslog-ng -s errors with catchall flag (1.6 to 3.1)
by Mishou Michael 09 Apr '11

09 Apr '11

Folks, I've done a lot of reading of the list archives through Google, but this is the first question that's really stumped me and also stumped the archives as well. I'm currently converting a 1.6 config to version 3.1, eliminating the warnings generated by -s and whatnot. One of the warnings I get is the following: Unknown log statement flag; flag='catchall' The configuration line this warning/error is referencing looks like this: log { source(network); destination(finalcatchall); destination(catchall_relay); flags(final,catchall); }; For some reason, the syslog-ng version I'm using doesn't like that flag option, but according to the 3.1 documentation, it is valid, as far as I can see. http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1- guide-admin-en.html/index.html-single.html#logpath I tried removing the final flag (thinking perhaps it was redundant) and just leaving the catchall in, but the error persists. Here is the version information for the syslog-ng I'm using, perhaps it is a bug? I'm not quite sure how to search the bug database, if anyone could point me in the right direction it would be appreciated. # /usr/local/sbin/syslog-ng -V syslog-ng 3.1.2 Installer-Version: 3.1.2 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainli ne--3.1#master#8bf13c304b6ab5fc1a372b49d55c78370efe14ca Compile-Date: Oct 25 2010 23:56:18 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on The version of syslog-ng I'm using is on Solaris 10 SPARC, pre-compiled packages I got from here: http://www.sunfreeware.com/programlistsparc10.html#syslogng If anyone has any ideas, please let me know. Thank you very much, --Mike

4 6

client -> server logging.. filters?
by Matt Wise 08 Apr '11

08 Apr '11

I'm doing some logging from clients -> servers via syslog-ng over a TCP session with stunnel. On my client, I'm seeing some log messages come in that are not making it to the server.. reliably. I don't have any filtering setup I don't think — is there some default setting somewhere that I don't know about that only passes certain types of messages to the remote dest? Do I need to specify somewhere 'send all'? Whats quite strange is that I have the hosts logging to 2 systems right now.. old syslog-ng server, and new one. The old one is getting all of the messages, but the new one is ignoring messages like these: Apr 8 18:21:43 xxx stunnel: LOG5[5459:1079724352]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket It seems to get all other messages.. but messages that have that LOG5 or LOG3 or LOG7 appended to them seem to get lost in translation... any ideas? —Matt

2 2

Using macro substring as another macro
by Jakub Jankowski 08 Apr '11

08 Apr '11

Hi, I'd like to implement simple hashing in file() destination, mainly to avoid ext3 32k directories limit, but also for convenience. Is it possible to do something simliar what this shell snippet does: $ NAME=value $ echo ${NAME[*]:0:1} v $ but for macros (especially those coming from parsers)? What I want is logs written to /var/log/h/ho/hos/hostname/service.log (with configurable level of subdirs) using a single destination(). Regards, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D

3 6

syslog-ng and IPv6 Interface Binding
by Reuben Farrelly 08 Apr '11

08 Apr '11

Hi, I've recently upgraded from syslog-ng 3.1 to syslog-ng 3.2.2 and I'm having limited success with my modified config in getting syslog-ng to bind to only specific IPv6 IP addresses. At the moment I have this in my config: source net { tcp6( port(601) ); udp6( port(601) ); }; However I'd like to configure this slightly differently - namely to increase the maximum connections from 10 to 25 and also to bind to only certain IPv6 addresses, as the box has multiple. I was hoping for a config like this (IPv6 address truncated somewhat): source net { tcp6( ip(2001::20) port(601) max_connections(25) ); udp6( ip(2001::20) port(601) max_connections(25) ); }; But seems to be a no-go: ----- Error parsing afsocket, syntax error, unexpected LL_NUMBER, expecting LL_IDENTIFIER or LL_STRING in /etc/syslog-ng/syslog-ng.conf at line 32, column 18: tcp6( ip(2001::20) port(601) max_connections(25) ); ^^^^ ----- Neither the max_connections() NOR the ip() options are accepted nor is the use of localip() for IPv6. http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gui… does not specifically mention if this command is meant to work under IPv6 or not, but it's reasonable I think to assume it would given the title of the page. Can someone please provide me with the relevant part of a working config in which I can specify the IPv6 interface address(es) I want syslog-ng to listen on, or have I run into a bug? The sample config files that ship with the package also seem a bit out of date, and the example at the bottom of s6.1.7 in the manual would be 100x more useful if it also included some IPv6 examples. Thanks, Reuben

4 5

stunnel through elb.. need packets sent semi-frequently
by Matt Wise 08 Apr '11

08 Apr '11

I'm using stunnel to pass some data through an Amazon ELB. Unfortunately, there are times when the session times out due to packets not passing for 60 seconds (an amazon ELB timeout). I'm wondering whether theres some way I can configure stunnel itself to pass some whitespaces or something back and forth every few seconds to make sure the session stays open and live? (and yes, I'm posing the same question to the app developer who's ap we're passing through.. but I'm looking at both avenues.. )

2 2

diskqueue framework patch
by Nagy Attila 08 Apr '11

08 Apr '11

Hi all, due to the need for disk queue usage in PE 4.1 we had to extend the syslog-ng sources, in order to be able write plugins that implement this feature. This patch contains these modifications. Basically it extends the 'class' hierarchy (and 'virtual' functions, see the new LogDestDriver) so that plugins can define their own queue mechanisms (as some do this in PE modules) and these special queues can be assigned to destinations. -Attila NAGY (syslog-ng-dev)

3 5

[Bug 118] New: syslog-ng -s errors with catchall flag (1.6 to 3.1)
by bugzilla＠bugzilla.balabit.com 07 Apr '11

07 Apr '11

https://bugzilla.balabit.com/show_bug.cgi?id=118 Summary: syslog-ng -s errors with catchall flag (1.6 to 3.1) Product: syslog-ng Version: unspecified Platform: Other OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: michael.mishou(a)csirc.irs.gov Type of the Report: bug Estimated Hours: 0.0 This is a bug report that is pretty much copied from the mailing list, verified by Gergely Nagy. I used this snapshot to find the bug: http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=snapshot;h=8bf13c304b6ab… But Gergely reports that the problem exists in multiple 3.x version (and also likely 2.x versions as well). I was getting the error Unknown log statement flag; flag='catchall' when doing a 'syslog-ng -s -f FILE' on a 2.x configuration file using version 3.1.2 of syslog-ng. All the documentation (2.x and 3.x) refers to log path flags: catchall, fallback, final, flow-control. But in center.c the actual strcmp functions are looking for: catch-all, fallback, final, flow_control OR flow-control gint log_connection_lookup_flag(const gchar *flag) { if (strcmp(flag, "catch-all") == 0) return LC_CATCHALL; else if (strcmp(flag, "fallback") == 0) return LC_FALLBACK; else if (strcmp(flag, "final") == 0) return LC_FINAL; else if (strcmp(flag, "flow_control") == 0 || strcmp(flag, "flow-control") == 0) return LC_FLOW_CONTROL; msg_error("Unknown log statement flag", evt_tag_str("flag", flag), NULL); return 0; } I'm guessing it should probably be something like this instead: gint log_connection_lookup_flag(const gchar *flag) { if (strcmp(flag, "catch-all") == 0 || strcmp(flag, "catchall") == 0) return LC_CATCHALL; else if (strcmp(flag, "fallback") == 0) return LC_FALLBACK; else if (strcmp(flag, "final") == 0) return LC_FINAL; else if (strcmp(flag, "flow_control") == 0 || strcmp(flag, "flow-control") == 0) return LC_FLOW_CONTROL; msg_error("Unknown log statement flag", evt_tag_str("flag", flag), NULL); return 0; } Gergely's notes/recommendations for fix: It's the same in 3.2 and 3.3 too - good catch! Whether this is a documentation or code bug, if the docs say it's 'catchall' ever since 2.0, then the code should support that too, along with 'catch-all'. I'd say this is a bug in both: the documentation should be updated to say 'catch-all', which has been working in past versions, and the code should be updated to support what the documentation has been advertising in the past years. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Convert UTC to local time zone using syslog-ng
by Pramod Pillai 07 Apr '11

07 Apr '11

Hi I receive syslog from remote note with timezone set as UTC. I need to convert these timestamp to local time zone Can some one suggest how can i achive it. Regards Pramod

4 5

[RFC]: afsql improvement plans
by Gergely Nagy 03 Apr '11

03 Apr '11

Hi! I'm about to blow up^W^Wrefactor the afsql destination to increase both performance and functionality. To do this properly, and to satisfy most needs, I'd like to ask the greater community (especially those of you who have more experience with logging to SQL than I do!) about the plans laid out on the paragraphs below. Those of you who do log into SQL one way or the other, but aren't interested in technical mumbo-jumbo, please skip to the end of this e-mail, I'd have a few questions, for which all responses would be most appreciated! Where, when, how?? ================== There are multiple goals to accomplish, which I believe are not exclusive: we want to keep the sql() statement in the configuration file, and maintain backwards compatibility aswell. On the other hand, we want to improve performance too, and that is best accomplished by using the native database libraries. To this end, I propose we introduce new database types: "mysql-native", "postgresql-native", "sqlite-native", and so on and so forth. The first step will be to rearrange the current afsql driver, so that it has a generic configuration interface, but can have multiple backends (dbi - the only existing backend right now; and one for each native db client). I'd change the AFSqlDestDriver structure to only contain the configurable data and nothing else. The backends would subclass this type (much like how AFSqlDestDriver subclasses LogDriver) and add their own. The reason I want to do this, making the AFSqlDestDriver a config-only interface instead of keeping larger chunks of code there (for example, the threading code) is because I do not want to limit my possibilities: I can imagine an SQL driver utilizing multiple writer threads easily. However, neither model works for all databases (concurrent writes to SQLite are usually not a good idea, for example). And that's about the generic interface! Lets dive into the more interesting parts: the native drivers. Since MySQL is the most used server out there, that will be the first I tackle. My goal with that is to implement a driver that is not only faster than our current destinations, but one that is also faster than the perl scripts people have been using until now, when they wanted top performance. All that while maintaining reasonable flexibility. An ambitious plan, but hopefully not impossible. I've been reading the MySQL documentation for the past few hours to explore my options, and found a few ways to increase our insert speed: mysql_list_tables() ------------------- Instead of doing a full select to verify that a table exists, we can just try to list the table. This is probably not a significant increase, but every bit counts. Prepared statements ------------------- We'll use prepared statements wherever possible - I expect this alone will result in a measurable performance increase, since the server will not have to parse each and every insert - only once, we'll bind variables thereafter. LOCK TABLES / transactions -------------------------- Currently, syslog-ng can do bulk-inserts, wrapped in a transaction. But that only works if the MySQL table we insert to is on a storage engine that supports transactions (InnoDB). However, MyISAM has been the default for quite a while, and that does not support transactions to the best of my knowledge. In this case, insert performance can be increased by locking the tables for a few hundred inserts at a time, and unlocking after: poor man's transactions, to some extent. This, however, is more of a workaround than a real improvement. Insertion plans =============== There's a couple of alternative ways to improve insert speed by changing how we actually do them. All have their pros and cons, which I will try to enumerate below. LOAD DATA INFILE ---------------- The fastest way by far is LOAD DATA INFILE, which is what people use in the various perl scripts, as far as I remember. The downside is that - as far as I saw so far - this does need a temporary file, which has its downsides, like writing to disk (unless put on tmpfs or similar). An option which I haven't tried would be to use a pipe(), so that the data never hits the disk. But nevertheless, the mysql-native db type will have an option to use LOAD DATA INFILE. * Pros: + The fastest way to insert * Cons: - Needs a temporary file, with all its drawbacks, some of which can be worked around. - Serialisation is costlier than simple INSERTs. - No fine-grained error checking: if something goes wrong, the whole LOAD will have to be dropped (unless we add code that splits bulks up into smaller chunks, and retries - but that's something I want to avoid if possible) For LOAD DATA to work, we will have to format the data to insert into a suitable format, and if there's an error in the serialisation somewhere, we'll have a hard time to figure out where the problem is, and syslog-ng will just give up. Also, beating our data into a suitable format isn't exactly cheap, either (think escaping: processing large amounts of string data, possibly changing strings - there's both processing and memory ramifications involved). The need for a file is also a (minor) issue. I suppose named pipes would work, that way data never hits the disk between syslog-ng and mysql. However, that still involves copying data around. We'll see if these affect LOAD DATA's performance badly enough (they probably don't). INSERT INTO ... VALUES (...), (...), ... ---------------------------------------- This is considerably slower than LOAD DATA, but still faster than using INSERTs (even if we use transactions / locking for those). The downside is that - like with LOAD DATA - serialisation is still costlier than INSERTs. * Pros: + Reasonably fast + Does not require neither a temporary file, nor gross hacks * Cons: - Slower than LOAD DATA - Serialisation can be costly. - No fine-grained error checking Serialisation is a little less costly than in LOAD DATA's case, but this can be worked around a little with prepared statements (with a fallback to parsed ones, in case we have to insert less data than originally planned). The same issues with error handling persist, though. INSERT ... DELAYED ------------------ The DELAYED strategy works best if we have multiple writer threads, as in this case, INSERT becomes a fire & forget call, and we can return to our business far sooner. This does not work all that well with bulk inserts, though. * Pros: + Reasonably fast with multiple writers + Simple & fast serialisation * Cons: - No error checking at all (except for connection and syntax errors) - Probably still slower than LOAD DATA The advantage of this is that we can get the data from syslog-ng to the mysql server faster than any of the other methods, and then the work is off-loaded to the server. This has the potential of being faster than any of the above, but with the cost of hardly any possibilities to recover from an error, as we get no feedback after inserting: did it succeed? did it not? We'll have no idea. The Plan ======== The plan is to implement all the strategies above, and measure them against each other and external scripts. This will take a little time, because I want to get the architecture right, and figure out whether we want a single or multiple writer threads (per-destination, of course), or if I want to make that a configurable option aswell. Once done, and people are happy with it, I'll move on to the next database (PostgreSQL, to scratch my own itch too). The Questions ============= What I'd like to know most, is, that would YOU expect from a native mysql destination? Apart from speed, what would you want from it, that the current driver does not support, but which is possible via external scripts? If you happened to read through my wall of text above, did I miss anything important? Cheers, -- |8]

4 10

More on httpry_logger
by Martin Holste 03 Apr '11

03 Apr '11

I Just thought I'd mention to the list that I posted a write-up on analyzing the effects of the lizamoon SQL injection campaign by looking at logs from httpry_logger through ELSA (both available at http://code.google.com/p/enterprise-log-search-and-archive/) on my blog at ossectools.blogspot.com. httpry_logger can be used stand-alone with whatever log system you're using now. In addition to syslog, it outputs to a number of other formats including basic DBI handle, MongoDB, and flat file. --Martin

1 0

Proper configuration for Syslog-ng PE v4.0 disk-based buffering feature is required...
by Prashant Badhe 02 Apr '11

02 Apr '11

Hi, I am evaluating Syslog-ng PE v4.0 disk-based buffering feature on Linux, but it doesn’t seems to be working. Below is the configuration I added to my sysylog-ng.conf file: destination d_syslog { tcp("myserver" port(1468) log_disk_fifo_size(102400) ); file("/var/log/messages"); }; When the connection to the server is up then I receive the forwarded syslogs properly. When the network connection goes down, syslog-ng stores these messages at location /opt/syslog-ng/var, (.qf files). But when the network connection reestablishes, these stored messages never sent to the server. Can anybody please guide me if there is some missing configuration or wrong configuration I have done to make this feature work? Thanks in advance, Prashant Badhe

2 1

Blog post on ELSA
by Martin Holste 01 Apr '11

01 Apr '11

I just put up an entry on my blog (http://ossectools.blogspot.com) describing the Enterprise Log Search and Archive Project (http://code.google.com/p/enterprise-log-search-and-archive) I've been working on which uses Syslog-NG >= 3.1 and pattern-db at its core. There are a lot of other open-source log collection frameworks out there that are easier to install, such as Logzilla (php-syslog-ng), but if you're trying to log > 1k messages/sec (common in large orgs) and need something GPL licensed, installing ELSA will probably be worth your while. We're using it to index 15k messages/sec with basic hardware. It's currently storing tens of billions logs, and full-text, ad-hoc queries complete in about 1/2 to 2 seconds, including group-by queries on arbitrary fields for reporting. I put a few screenshots and a feature list in the post. The documentation is pretty basic right now, but I'm happy to assist if you run into issues. ELSA is also open to plugin creation, so if you find ELSA useful and create plugins, please let me know and I can add them to the project. Also, patterns for the pattern-db are more than welcome! I've included patterns for Cisco FWSM connections and denies, Snort logs, Windows logs from Eventlog-to-Syslog as well as Snare, and URL's from my httpry wrapper, which is available on the project site as well as in the tarball/source code. Comments and feedback are welcome! Thanks, Martin

6 8

how to push sulogs/utmpx/wtmpx to central logserver
by Tony MacDoodle 31 Mar '11

31 Mar '11

Hello All, I would like to push only the following log files to a central syslog-ng server. All clients are Solaris 10 and the syslog-ng server is also Solaris 10. IS it possible to push only this information? wtmpx utmpx sulog Thanks

4 3