- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Message suppression not working
by Balazs Scheidler 02 Oct '08

02 Oct '08

On Thu, 2008-10-02 at 08:39 -0400, Kevin Cruse wrote: > Bazsi, > > I am trying to suppress messages sent to the database and sendmail. I > have configured suppress(900) globally and within my destinations and it > doesn't seem to work. Does anyone have this working? I am getting > flooded with emails when there are problems on my firewalls or routers. > > destination d_mysql { > pipe("/var/log/mysql.pipe" > template("INSERT INTO logs > (host, facility, priority, level, tag, datetime, program, msg) > VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', > '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', > '$PROGRAM', '$MSG' );\n") template-escape(yes) suppress(900)); > }; > > Or > > destination mail-alert-perl { program("/usr/local/bin/mail_logger.pl" > suppress(900)); }; > > also tried > > destination mail-alert-perl { program("/usr/local/bin/mail_logger.pl") > suppress(900); }; > > Any help would be appreciated. Thanks. suppress() only filters duplicates, and is not a general ratelimit. -- Bazsi

1 0

syslog-ng craches when sending from syslog-ng to another loghost (that have syslog-ng)
by Mattias Ryrlén 02 Oct '08

02 Oct '08

Hi, Im running syslog-ng-2.1beta2-1 on 2 of my servers. i have the same configuration on both the servers except that i want server A to forward messages to server B All works fine for a couple of minutes, then server A:s syslog-ng service craches without any information. running CentOS 5.0 on both servers with syslog-ng installed as rpm packages from balabit. i have also tried with udp() instead of tcp() with the same result. options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); dns_cache(yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_msg_size(8192); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; source s_udp { udp(); }; source s_tcp { tcp(max-connections(200)); }; destination d_loghost { tcp("192.168.1.150" port(514)); }; log { source(s_sys); source(s_tcp); source(s_udp); destination(d_loghost); }; Any sugestions? -- Vänliga hälsningar / Best Regards Mattias Ryrlén __________________________ op5 AB Första Långgatan 19 SE-413 27 Göteborg Mobil: +46 735-17 70 99 Support: +46 31-774 09 24 www.op5.com

2 2

syslog-ng OSE git tree opened
by Balazs Scheidler 02 Oct '08

02 Oct '08

Dear syslog-ng users, I'm proud to announce that I've opened the syslog-ng OSE 3.0 branch and pushed a backport of the result of the latest Premium Edition development cycle. The size of the patch is enourmous, it took me a while to integrate it back to OSE 3.0, that's why this announcement is coming so late. The diffstat is as follows: 175 files changed, 21537 insertions(+), 7377 deletions(-) The list of changes of interest are: * various performance improvements, syslog-ng can do about 120-130k messages/sec in case no destination is written to (not very useful though), and about 75k msg/sec when writing to a plain text file * TLS encrypted channels, works the same as the PE version, documentation is: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch03s09.html * support for the new IETF syslog protocols, in order to use this, you need to use the syslog() driver either as a source or a destination. It behaves quiet closely to tcp/udp, except it uses the new protocol * log message classification using a database, scales up to tens of thousands of messages per second, the pattern database also serves as a means to parse variable parts of messages. No real documentation so far, there's an XML Schema file in the tarball, it can be used using the db_parser() parser. * rewrite parts of the message using regexps, and simple string search and replace. Here's a post on the syslog-ng mailing list that has some info on usage: http://marc.info/?l=syslog-ng&m=122051372205555&w=2 * parse parts of the syslog message, makes it possible to parse fields in an Apache log for instance. Also the parsed values can be used as macros. Again, the post above has some info: http://marc.info/?l=syslog-ng&m=122051372205555&w=2 * generalize the log message paths, instead of a flat list of source->destination connections, log statements can be organized into a tree, making the configuration simpler and the performance better, here is a description: http://bazsi.blogs.balabit.com/2008/02/redesigning-syslog-ng-internals.html * PCRE and glob filters (in addition to POSIX regexps), use the type("pcre") option in the filter, like this: host("something" type("pcre")) * support for capabilities under Linux, this running syslog-ng as non-root is possible, also with reload support, see the documentation of Linux capabilities in capabilities(7), for the syntax of the --caps option, see cap_from_text(3) * program sources, simply use program("cat /dev/xconsole") in your configuration, the program will be restarted automatically if that exits. * syslog-ng got the ability to restart itself if it crashes, see the --process-mode command line option * new statistics framework that can be queried using UNIX domain sockets, provides per-source-host and per-destination-host statistics. In order to use it, possibly set the stats_level() option and write "STATS\n" to /var/run/syslog-ng.ctl which is a UNIX domain socket. This mass amount of changes also mean some incompatibilities between 2.1 and 3.0, however 3.0 will work in compatibility mode unless it is told to use the 3.0 configuration file format. It'll also report warnings against your configuration file you should change in order to use all the features and best performance. The most important incompatibilities include: * the $MSG macro will not include the "program[pid]:" string anymore, you should prepent an $MSGHDR macro in order to have that, in 2.1 mode the MSG will expand as before, but please note that this value is calculated everytime you reference the macro, thus it affects performance. * some defaults were changed (chain_hostname, template-escape, follow-freq for file sources), in 2.1 mode the old defaults are used but a warning is emitted * the input path does not remove NL characters anymore by default, but you can ask for this behaviour by specifying the "no-multi-line" flag. The documentation for the new features is not yet completely finished, I hope to get a snapshot of the current state to the website soon. In the meanwhile either use the source :) or ask direct questions on the mailing list. I'll also try to make some posts on my blog about various features in more detail and configuration examples. -- Bazsi

1 0

Re: [syslog-ng] Log to postgres
by ml＠bortal.de 02 Oct '08

02 Oct '08

Hello, I just tried to get syslog-ng 2.1.1 working by doing: #/usr/local/src/libdbi8-0.8.2bb2# make install --> Libraries have been installed in: /usr/local/lib #/usr/local/src/libdbi8-drivers-0.8.2bb3# ./configure --with-pgsql --with-dbi-libdir=/usr/local/lib --disable-docs --> configure: error: Invalid libdbi directory - include files not found. What am i doing wrong? Thanks, Mario

2 1

Log to postgres
by ml＠bortal.de 02 Oct '08

02 Oct '08

Hello List, is there a open-source way to log to a postgres database nativly? We have problems using a pipe. Is is beeing described on page 107 of the syslog-ng Admin Guid but i am not sure if this is only available for the commercial version. Thanks, Mario

2 1

Reload does not free up RAM
by Evan Rempel 02 Oct '08

02 Oct '08

We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before it is able to flush the data to the destination. Currently, the only way to "free up" that RAM is to restart syslog-ng, which means that some log messages are lost during the restart window. I open for discussion the idea that on a reload that syslog-ng flush and free all buffer space and essentially start from a "just started" state again. Comments?

2 1

Re: [syslog-ng] escaping \[ not respected
by Len Conrad 02 Oct '08

02 Oct '08

>Hello, > >> targeted string is "unknown[a.b.c.d]" >> >> my filter: >> >> filter f_conn_from_unk_private { not match >> ("unknown\[(10\.1\.|10\.2\.|10\.10\.5\.|192\.168\.200)"); }; >> >> error: >> >> Error compiling regular expression; >> re='[(10.1.|10.2.|10.10.5.|192.168.200)', error='brackets ([ >]) not balanced' > >I can't confirm this behaviour, as the following does work for me: > >filter f_internal_statistics { > match("^syslog-ng\[[[:digit:]]+.: STATS") or match ("^syslog-ng\[[[:digit:]]+\]: Log statistics"); >}; > >What syslog-ng version are you using? Mine is 2.0.9 Installed with FreeBSD pkg_add from freshports.org, pkg_info shows: "syslog-ng2-2.0.9_1 A powerful syslogd replacement" I conclude that I've found a bug in the parsing of the escape sequence "\[" , and will look for a work around. thanks, Len ______________________________________________ IMGate OpenSource Mail Firewall www.IMGate.net

5 5

$MSGONLY bad output
by Steven 01 Oct '08

01 Oct '08

Hi there, As I can't find this on the web, I'm asking here : I'm receiving the IIS servers logs through my LAN and syslog-ng writes everything to a local files, that is parsed during the night by awstats. The only thing is I can't have syslog-ng to write raw logs to files, it absolutely has to write something (either with $MSG or $MESSAGE). I even tried with $MSGONLY but now I get the opposite problem, it truncates data ! $MSG : IISWebLog 0 2008-09-30 14:02:40 GET /Localized/fr/Images/New2/tlogo.gif - 95.35.41.243 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.0;+SLCC1;+.NET+CLR+2.0.50727 ;+Media+Center+PC+5.0;+.NET+CLR+3.0.04506;+InfoPath.2) http://mypublicwebsite/home/regf 200 2185 With $MSGONLY, I'll get : 02:40 GET /Localized/fr/Images/New2/tlogo.gif - 95.35.41.243 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.0;+SLCC1;+.NET+CLR+2.0.50727 ;+Media+Center+PC+5.0;+.NET+CLR+3.0.04506;+InfoPath.2) http://mypublicwebsite/home/regf 200 2185 Here is my very simple template : template t_IIS { template("$MESSAGE\n"); template_escape(no); }; Destination and filter : destination df_ mypublicwebsite _IISWebLog { file("/data/syslog/mypublicwebsite/IISWebLog" template(t_IIS)); }; filter f_ mypublicwebsite _IISWebLog {(host("^192\.168\.117\.131$") or host("^192\.168\.117\.132$") or host("^192\.168\.117\.133$") or host("^192\.168\.117\.134$") or host("^192\.168\.117\.135$") ) and facility(local4) and program("IISWebLog"); }; log { source(s_net); filter(f_ mypublicwebsite _IISWebLog); destination(df_ mypublicwebsite _IISWebLog); }; # uname -a Linux TR-ADM 2.6.18-4-amd64 #1 SMP Fri May 4 00:37:33 UTC 2007 x86_64 GNU/Linux Anything I'm missing ? Any help would be greatly appreciated. Best regards Steven

3 4

separe files question
by Mártha Csaba 01 Oct '08

01 Oct '08

Hello I have a question, i get logfiles from facility0, with these matter: AppID(0-9a-Z)-ThreadID(0-9a-Z)-DATE-TIME-MESSAGE i created a filter, with this content: filter f_flash_msg { match ("[0-9a-zA-Z]+-[0-9a-zA-Z]-*"); }; and destination destination df_out { file("/var/log/client/flashlog/$1/$2/$YEAR/$MONTH/ $DAY/$TIME.txt"); }; i need separate folders/files. the log line contains these datas: 1234-ABCD-2008-09-30-16-20-35-FatalError AppID-Thread-Date-Time-MSG i need this structure: /var/log/syslog-ng/flashlogs/1234/ABCD/2008/09/30/16.txt what is my mistake? Best Regards Csaba

2 2

escaping \[ not respected
by Len Conrad 01 Oct '08

01 Oct '08

targeted string is "unknown[a.b.c.d]" my filter: filter f_conn_from_unk_private { not match ("unknown\[(10\.1\.|10\.2\.|10\.10\.5\.|192\.168\.200)"); }; error: Error compiling regular expression; re='[(10.1.|10.2.|10.10.5.|192.168.200)', error='brackets ([ ]) not balanced' Len ______________________________________________ IMGate OpenSource Mail Firewall www.IMGate.net

2 1