September 2008 - syslog-ng - lists.balabit.hu

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng client messages delayed
by mark peters 22 Mar '10

22 Mar '10

Hi, We are running into a issue where syslog-ng clients who are configured to forward to a central syslog-ng destination server that are reasonably chatty are getting there messages delayed at the source (ie. 'tcpdump'ing the client shows it is sending messages from N minutes ago, where N is anything from 5 to 60 minutes on average sometimes more). We have also seen a small percentage of loss at the destination. On these 'delayed' clients we also see the resident memory footprint on the growing. Clients are running 1.6.12 (have tried 2.0.8 also - same symptons). The central syslog-ng server is running 2.0.8. relevant config entries are: syslog-ng client: options { sync (1); time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; destination d_central { tcp("log2.xyz.com" port(5140)); }; filter f_local5 { facility(local5); }; log { source(s_sys); filter(f_local5); destination(d_central); flags(final); }; syslog-ng central server: options { time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_expire (28800); use_fqdn (no); create_dirs (yes); dir_perm (0755); perm (0644); keep_hostname (no); use_time_recvd (no); stats_freq(600); flush_timeout(16000); flush_lines(256); time_sleep(80); }; destination d_server { file("/logs/server.$LEVEL.log"); }; filter f_server { facility(local5) and program("server"); }; log { source(s_sys); filter(f_server); destination(d_server); flags(final); }; to give a rough idea of message volume the server is receiving approximately 50 million log lines per day on average any help/information would be appreciated thanks -------------------------------------- Easy + Joy + Powerful = Yahoo! Bookmarks x Toolbar http://pr.mail.yahoo.co.jp/toolbar/

3 4

[Bug 21] New: call shutdown() before closing a socket
by bugzilla＠bugzilla.balabit.com 30 Apr '09

30 Apr '09

https://bugzilla.balabit.com/show_bug.cgi?id=21 Summary: call shutdown() before closing a socket Product: syslog-ng Version: 2.1.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bazsi(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 syslog-ng should call shutdown() before closing a socket to ensure the contents of the socket buffer are emptied. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 23] New: MySQL default configuration causes error on 'pid' column
by bugzilla＠bugzilla.balabit.com 30 Apr '09

30 Apr '09

https://bugzilla.balabit.com/show_bug.cgi?id=23 Summary: MySQL default configuration causes error on 'pid' column Product: syslog-ng Version: 2.1.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: mark(a)la-croix.net Type of the Report: bug Estimated Hours: 0.0 When configuring a MySQL destination, the default 'pid' column type is INT. However, if a message comes through with no PID, the SQL command tries to insert an empty quoted string, which causes an error in MySQL 5.0. Example error: Error running SQL query; type='mysql', host='!MY_DB_HOST!', port='', user='syslog-ng', database='syslog-ng', error='1366: Incorrect integer value: \'\' for column \'pid\' at row 1', query='INSERT INTO !MY_LOG_CLIENT_HOST!_2008_09_23 (date, facility, level, host, program, pid, message) VALUES (\'2008-09-23 12:24:35\', \'1\', \'5\', \'!MY_LOG_CLIENT!\', \'root\', \'\', \'test\')' The message is coming from a TCP source from which a large percentage of messages have no PID associated. The workaround is to explicitly set the PID column to VARCHAR so it can handle empty strings, but it would be nice to see the empty PID case gracefully handled. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 9

[patch] Add follow_freq_ms option
by Joe Shaw 16 Jan '09

16 Jan '09

Hi, As I mentioned in the last thread, I've found myself wanting to have better than one second resolution for the follow_freq() option. Attached is a patch which adds a follow_freq_ms option, which takes its value in milliseconds rather than full seconds. follow_freq() continues to work as it did previously. Thanks, Joe

3 7

[PATCH] anonymizing filter
by Micah Anderson 06 Nov '08

06 Nov '08

Hello, A couple years ago this patch was submitted to the list for consideration for inclusion into syslog-ng. I am writing this email again to request that it be considered again. The patch provides a simple replace which enables you to strip out IP addresses from your logs before they are written to disk. The patch has been included in the Debian stable distribution, and currently is included in both Debian Sid and Lenny (unstable and testing). It has had a very wide testing base and is non-intrusive, it has existed since 2004 and has been adapted to work with the newer syslog-ng. The goal of this patch is to give an organization the means to implement site logging policies, by allowing for easy control over exactly what data is retained in the logfiles. When I first requested consideration for inclusion the reactions were some suggestions for improvement (which were done), some side discussions about the various states of data retention laws, and a general agreement that this patch is non-intrusive and had a valid use case (at least in the U.S., but also likely in other countries as well[0]). The side-discussions about data-retention laws were mostly around specific geographic localities that were considering laws that would make stripping of addresses illegal, or had already mandated such things. Although these were interesting discussions, as EU data retention laws would prohibit many people from making such configuration changes to their syslog-ng.conf, they were tangential to the point because this patch does not cause those to break such laws. On the other side of the pond, in the U.S., the EFF[1] has made it very clear that this mechanism of anonymizing logs is perfectly (a) legal in the U.S., and (b) advisable. There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain. The current climate in the U.S. makes this problem so much more important now than it was many years ago. Having the ability to implement a site-policy that enables an organization to decide if the trade-off between privacy and analysis is worthwhile. This patch allows organizations to have that choice if they feel that it is more important to avoid retaining sensitive data rather than having a full history of everything logged. Please accept this patch[2], Micah [0] EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html [1] The EFF is the major civil liberties internet watchdog in the US, their "Best Practices for Online Service Providers" can be found here: http://www.eff.org/osp, they explicitly link to our patch as a recommendation [2] The latest patch can be found at https://code.autistici.org/trac/privacy/browser/trunk/syslog-ng

6 11

syslog-ng 2.1: upstream libdbi does NOT provide pkgconfig .pc file
by Wolfram Schlich 07 Oct '08

07 Oct '08

Hi, syslog-ng 2.1 depends on libdbi >= 0.8.0 with --enable-sql. Unfortunately, it assumes that a pkg-config .pc file for libdbi is in place which the upstream libdbi just does not include. It seems that there is a balabit-modified version of libdbi available (libdbi8_0.8.2bb2-2.tar.gz) which was patched to include a dbi-0.8.pc file. What's the reason for that? Was upstream asked to include that .pc file in their regular releases? What other changes were made to the upstream libdbi to form the balabit version? This is kinda maintenance and dependency hell so far :( Please clear up the situation so I can take care of building a Gentoo package for 2.1 *including* SQL support :) Thanks! -- Regards, Wolfram Schlich <wschlich(a)gentoo.org> Gentoo Linux * http://dev.gentoo.org/~wschlich/

2 4

$MSGONLY bad output
by Steven 01 Oct '08

01 Oct '08

Hi there, As I can't find this on the web, I'm asking here : I'm receiving the IIS servers logs through my LAN and syslog-ng writes everything to a local files, that is parsed during the night by awstats. The only thing is I can't have syslog-ng to write raw logs to files, it absolutely has to write something (either with $MSG or $MESSAGE). I even tried with $MSGONLY but now I get the opposite problem, it truncates data ! $MSG : IISWebLog 0 2008-09-30 14:02:40 GET /Localized/fr/Images/New2/tlogo.gif - 95.35.41.243 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.0;+SLCC1;+.NET+CLR+2.0.50727 ;+Media+Center+PC+5.0;+.NET+CLR+3.0.04506;+InfoPath.2) http://mypublicwebsite/home/regf 200 2185 With $MSGONLY, I'll get : 02:40 GET /Localized/fr/Images/New2/tlogo.gif - 95.35.41.243 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.0;+SLCC1;+.NET+CLR+2.0.50727 ;+Media+Center+PC+5.0;+.NET+CLR+3.0.04506;+InfoPath.2) http://mypublicwebsite/home/regf 200 2185 Here is my very simple template : template t_IIS { template("$MESSAGE\n"); template_escape(no); }; Destination and filter : destination df_ mypublicwebsite _IISWebLog { file("/data/syslog/mypublicwebsite/IISWebLog" template(t_IIS)); }; filter f_ mypublicwebsite _IISWebLog {(host("^192\.168\.117\.131$") or host("^192\.168\.117\.132$") or host("^192\.168\.117\.133$") or host("^192\.168\.117\.134$") or host("^192\.168\.117\.135$") ) and facility(local4) and program("IISWebLog"); }; log { source(s_net); filter(f_ mypublicwebsite _IISWebLog); destination(df_ mypublicwebsite _IISWebLog); }; # uname -a Linux TR-ADM 2.6.18-4-amd64 #1 SMP Fri May 4 00:37:33 UTC 2007 x86_64 GNU/Linux Anything I'm missing ? Any help would be greatly appreciated. Best regards Steven

3 4

escaping \[ not respected
by Len Conrad 01 Oct '08

01 Oct '08

targeted string is "unknown[a.b.c.d]" my filter: filter f_conn_from_unk_private { not match ("unknown\[(10\.1\.|10\.2\.|10\.10\.5\.|192\.168\.200)"); }; error: Error compiling regular expression; re='[(10.1.|10.2.|10.10.5.|192.168.200)', error='brackets ([ ]) not balanced' Len ______________________________________________ IMGate OpenSource Mail Firewall www.IMGate.net

2 1

Trouble with logging to mysql
by Stallmann, Andreas 30 Sep '08

30 Sep '08

Hi there, I'm trying to let syslog-ng log to a mysql database, which is somehow not working (anymore). All my network traffic is supposed to be written into a separate logfile as well as into a mysql database. I followed the HowTo on http://gentoo-wiki.com/HOWTO_setup_PHP-Syslog-NG#Syslog-ng_INIT_Script, and altered the configuration to fit my needs. Here it comes: ############################################ source net { udp(ip("0.0.0.0") port(514)); }; destination d_mysql { program("/usr/bin/mysql --reconnect -f -T --user=syslogfeeder --password=PASSWORD syslog >> /var/log/db_log.log 2>&1" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC','$PROGRAM', '$MSG' );\n") template-escape(yes)); }; destination netmessages { file("/var/log/net.log"); }; log { source(net); destination(netmessages); }; log { source(net); destination(d_mysql); }; ############################################## Thing is, logging to the file works nicely. Logging to the database worked up to some date, but since a while (and no one knows exactly since when, because nobody had a look at the database for some time) it doesn't. This might be connected with a system update. I can provide you with information on the versions (the OS is SLES 10.0, all packages come from the standard source; no self-compiled packages installed) of syslog and mysql as soon as I have access to the customers machine again. I'm looking for a sensible way to debug syslog. Running syslog-ng in the foreground didn't help, I can't see any messages related to logging to mysql. It seems that the destination is never touched. Isn't it possible to log on source to two destinations (anymore)? (I think commented out logging to netmessages, but it still didn't work.) Any comments and hints will be helpful. Even RTFM if you point me to a comprehendible one. :-) TNX, Andreas -- CONET Solutions GmbH Andreas Stallmann, Senior Berater Theodor-Heuss-Allee 19, 53773 Hennef Tel.: +49 2242-939-677, Fax: +49 2242-939-393 Internet: http://www.conet.de, mailto: astallmann(a)conet.de ----------------------------------- CONET Solutions GmbH, Theodor-Heuss-Allee 19, 53773 Hennef Registergericht/Registration Court: Amtsgericht Siegburg (HRB Nr. 9136) Geschaftsfuhrer/Managing Directors: Dipl.-Inform. Rudiger Zeyen (Sprecher/Chairman), Dipl.-Betriebsw. Wilfried Putz und Dipl.-Inform. Jurgen Zender Vorsitzender des Aufsichtsrates/Chairman of the Supervisory Board: Dipl.-Math. Hans-Jurgen Niemeier

4 9