January 2008 - syslog-ng - lists.balabit.hu

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[PATCH] anonymizing filter
by Micah Anderson 06 Nov '08

06 Nov '08

Hello, A couple years ago this patch was submitted to the list for consideration for inclusion into syslog-ng. I am writing this email again to request that it be considered again. The patch provides a simple replace which enables you to strip out IP addresses from your logs before they are written to disk. The patch has been included in the Debian stable distribution, and currently is included in both Debian Sid and Lenny (unstable and testing). It has had a very wide testing base and is non-intrusive, it has existed since 2004 and has been adapted to work with the newer syslog-ng. The goal of this patch is to give an organization the means to implement site logging policies, by allowing for easy control over exactly what data is retained in the logfiles. When I first requested consideration for inclusion the reactions were some suggestions for improvement (which were done), some side discussions about the various states of data retention laws, and a general agreement that this patch is non-intrusive and had a valid use case (at least in the U.S., but also likely in other countries as well[0]). The side-discussions about data-retention laws were mostly around specific geographic localities that were considering laws that would make stripping of addresses illegal, or had already mandated such things. Although these were interesting discussions, as EU data retention laws would prohibit many people from making such configuration changes to their syslog-ng.conf, they were tangential to the point because this patch does not cause those to break such laws. On the other side of the pond, in the U.S., the EFF[1] has made it very clear that this mechanism of anonymizing logs is perfectly (a) legal in the U.S., and (b) advisable. There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain. The current climate in the U.S. makes this problem so much more important now than it was many years ago. Having the ability to implement a site-policy that enables an organization to decide if the trade-off between privacy and analysis is worthwhile. This patch allows organizations to have that choice if they feel that it is more important to avoid retaining sensitive data rather than having a full history of everything logged. Please accept this patch[2], Micah [0] EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html [1] The EFF is the major civil liberties internet watchdog in the US, their "Best Practices for Online Service Providers" can be found here: http://www.eff.org/osp, they explicitly link to our patch as a recommendation [2] The latest patch can be found at https://code.autistici.org/trac/privacy/browser/trunk/syslog-ng

6 11

Using Regex information for destination
by James Whitt 11 Jul '08

11 Jul '08

I had asked this question about a year ago, but was told to wait for the new 2.0 version. I'm currently running the latest syslog-ng 2.0.7. What I would like to do is put the line in a specific file based on part of the hostname. For example: Hostname: subdomain.domain.tld destination customer_firewall_logs { file("/logs/$TLD/$DOMAIN/$SUBDOMAIN/$R_YEAR-$R_MONTH-$R_DAY.log"); }; WHERE $TLD is the tld of the hostname, $DOMAIN is the domain from the hostname, and $SUBDOMAIN is teh subdomain from the hostname. I would like to be able to regex this information out of the hostname to use in the destination. Is this possible and any tips on getting it to work correctly would be greatly appreciated.

3 6

What packages need?
by hi ro 13 Apr '08

13 Apr '08

Hi, What packages does it need to compile syslog-ng? Is it the same package when to make RPM? (bison, flex, gcc-c++, pkgconfig, glib2-devel, libevtlog-devel) (It needs only libc to compile eventlog, doesn't it?) And if it's not necessary to connect database, does it need to install libdbi8? Thanks hiro _________________________________________________________________ 【MSNビデオ】超貴重！驚きの大物対談が実現。作家村上龍が話題のあの人に迫る http://video.msn.co.jp/rvr/default.htm

4 14

[Bug 3] New: udp flood
by bugzilla＠bugzilla.balabit.com 13 Apr '08

13 Apr '08

https://bugzilla.balabit.com/show_bug.cgi?id=3 Summary: udp flood Product: syslog-ng Version: 2.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: manoo(a)manoo.hu Type of the Report: bug Estimated Hours: 0.0 I installed syslog-ng 2.0.5 (because there was no .rpm for a RHEL3 server I made it with rpmbuild). In the config file I placed two destination lines for the same filter, first for place the log locally and an other to send it to a remote loghost (udp/514). I had a typo in the path of the local logfile (started with "/vavr" instead of "/var") and syslog-ng cannot create/write the specified file (I made no "create_dirs(yes)") when I started the logging (with logger - that was an apache logging system) the result was the following: it did not checked the writability of the file but could start itself (resart of syslog-ng was successfull) and started to sends this errormessage to the loghost with full speed: Dec 13 09:56:07 192.168.0.2 syslog-ng[19315]: Error opening file for writing; filename='/vavr/log/apache/apache.log', error='No such file or directory (2)' within 20 minutes it sends cca 700mb of clear text to the loghost. There is a leased line between the two hosts via internet and this flooded the communication channel. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

AIX startup
by Brian Loe 27 Mar '08

27 Mar '08

Okay, my syslog-ng instance died at some point last night and because I don't have it setup properly for respawning... Can anyone running syslog-ng on AIX 5.* give me some tips for this? I'm feeding pipes so i have to start instances of SEC as soon as I start syslog-ng - but the SEC scripts appear to "close" so if they're configured to respawn I'll have 2000 instances of them in an hour.

7 21

incorrect firewall logs with syslogng-2.0.0 [security error?]
by Jacek Kalinski 12 Mar '08

12 Mar '08

Hello, After upgradeing from syslogng-1.6.11 to 2.0.0 I've got a strange messages in /var/log/messages file: Nov 29 08:14:05 denise WINDOW=16985 RES=0x00 ACK URGP=0 Nov 29 11:10:19 denise PT=34536 WINDOW=16985 RES=0x00 ACK URGP=0 Nov 29 16:42:16 denise W=0 RES=0x00 RST URGP=0 Nov 29 21:37:15 denise 116 ID=34901 DF PROTO=TCP SPT=3584 DPT=80 WINDOW=65535 RES=0x00 ACK FIN URGP=0 These are truncated iptables logs. Because in firewall.log file are simmilar entries to truncated one, I think full file should be: Nov 29 11:10:19 denise firewallp=INVALID:1 a=DROP IN=eth1 OUT= MAC=00:30:4f:36:2b:dc:00:04:9a:2c:7f:20:08:00 SRC=193.41.230.81 DST=xxx.xxx.xxx.x LEN=576 TOS=0x00 PREC=0x00 TTL=117 ID=3495 PROTO=TCP SPT=443 DPT=34536 WINDOW=16985 RES=0x00 ACK URGP=0 (xxx.xxx.xxx.x is a real server IP address - one 'x' is one digit) Config file with interesting lines: options { dir_perm(0700); perm(0600); chain_hostnames(no); stats(1200); keep_hostname(yes); dir_owner(root); dir_group(root); owner(root); group(root); }; source local { unix-stream("/dev/log" max_connections(300)); file("/proc/kmsg"); # udp(ip(0.0.0.0) port(514)); internal(); }; # ..... # authpriv.none;cron.none;mail.none;news.none -/var/log/messages filter f_2 { not facility(news); }; filter f_3 { not facility(local5); }; filter f_4 { not facility(local6); }; filter f_5 { not facility(authpriv); }; filter f_6 { not facility(cron); }; filter f_7 { not facility(mail); }; filter f_notlocal7 { not facility(local7); }; filter f_notfirewall { not match("firewallp="); }; destination d_2 { file("/var/log/messages" create_dirs(yes)); }; log { source(local); filter(f_2); filter(f_4); filter(f_5); filter(f_6); filter(f_7); filter(f_notimapd); filter(f_notpopa3d); filter(f_notnamed); filter(f_notfirewall); destination(d_2); }; #..... # firewallp.* -/var/log/firewall.log filter f_16 { level(warning) and match("firewallp="); }; destination d_12 { file("/var/log/firewall.log" create_dirs(yes)); }; log { source(local); filter(f_16); destination(d_12); }; #..... Do you know where could be an error? It is a production server so I can't debug syslog-ng on this machine (sorry). NOTE: the only truncated lines I can find are iptables (AFAIR they are logged by /proc/kmesg) entries. All other lines are complete. And it is probably a critical error in security. PS: simple statistics line from syslog-ng: Nov 30 00:00:06 denise syslog-ng[32682]: Log statistics; dropped='pipe(/dev/tty12)=0', processed='center(queued)=758289', processed='center(received)=369020', processed='destination(d_maillogimapd)=4245', processed='destination(d_5)=1', processed='destination(d_4)=0', processed='destination(d_3)=14259', processed='destination(d_2)=14259', processed='destination(d_13)=0', processed='destination(d_1)=6000', processed='destination(d_12)=6000', processed='destination(d_11)=369020', processed='destination(d_10)=36', processed='destination(d_8)=0', processed='destination(d_maillogpopa3d)=103224', processed='destination(d_7)=241245', processed='destination(d_6)=0', processed='source(local)=369020' I think you see that it could be a little hard to debug (with so many messages parsed by syslog-ng) ;) Jacek

3 6

time_sleep() and inadvertent admission control
by John Morrissey 22 Feb '08

22 Feb '08

About a year ago, I noticed heavy CPU consumption during certain workloads (many processes sending a small number of log messages per process, such as a busy Postfix machine) due to io_iter spinning very rapidly on poll(). We kludged around it by adding the time_sleep() directive, to add an artificial delay at the end of the io_iter loop and prevent the loop from rolling over too quickly: http://marc.info/?l=syslog-ng&m=114009552929622&w=2 We started using time_sleep(30) across all of our machines, since that delay value didn't seem to cause any problems for our workloads and we wanted to keep the configuration uniform. We noticed recently that time_sleep() exhibits some inadvertent admission control behavior. When poll() indicates that the listener socket has activity (new connections), syslog-ng seems to accept() only once on it, allowing one new connection per poll(). As a result, it only allows: 1000 / time_sleep() connections per second. Accordingly, with time_sleep(30), only 33 connections would be allowed every second. This became problematic when we added a logger into our qmail-pop3d chain. Normally, qmail-pop3d is invoked thusly: qmail-popup $HOSTNAME auth_pop qmail-pop3d ./Maildir/ qmail-popup starts, pops its args, and then exec()s what's left over. By default, qmail's POP3 daemon doesn't log user information, so we added a very simple syslogging script to this exec chain, like so: qmail-popup $HOSTNAME auth_pop qmail-loguser qmail-pop3d ./Maildir/ [jwm@periscope:pts/0 ~> cat qmail-loguser #!/bin/sh logger -t $(basename $0) -- "${HOME##*/} from $TCPREMOTEIP on $TCPLOCALIP" exec "$@" This was disastrous under load, since the admission control behavior caused by time_sleep() made logger(1) block, quickly causing the machine to run out of memory. I realize time_sleep() is a kludge, but more advanced dynamic tuning to replace it seems like it would be error-prone and difficult to troubleshoot when things go wrong. I would also argue that time_sleep() is meant to prevent spinning on poll() and its effect was not intended for *new connections*, since new connections did not cause the original problem of excessive CPU consumption. time_sleep() unnecessarily throttles new connections, which in workloads like qmail's are critical to scalability. For example, syslog-ng still consumes only ~2% of a dual 2.8GHz Xeon machine's CPUs even with this rapid-connection qmail workload. Instead, would it make sense for syslog-ng to accept() on the listener FD (it would need to be marked as non-blocking; I haven't looked into the ramifications if it isn't already) until there are no more new connections waiting, perhaps with a hard ceiling if starvation of other FDs is a worry? john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

4 7

Why syslog-ng'config permission change ?
by hi ro 22 Feb '08

22 Feb '08

Hi, I installed syslog-ng. At that time, syslog-ng.conf's permission was 644. I let A user (not root) to update syslog-ng.conf, so I changed permission to 666. But I reboot the Server, then syslog-ng.conf's permission returned 644. Why ? Thanks hiro _________________________________________________________________ 出会いがない、と嘆いているあなたにぴったりの恋人探し！まずは無料検索！ http://match.jp.msn.com/channel/index.aspx?avo=1&trackingid=1034447

4 4

[Bug 12] New: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=87927
by bugzilla＠bugzilla.balabit.com 08 Feb '08

08 Feb '08

https://bugzilla.balabit.com/show_bug.cgi?id=12 Summary: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=87927 Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=87927 OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'd like to be able to include a log source within another log source, e.g.: source local { internal(); unix-stream("/dev/log"); }; source all { source("local"); udp(ip(1.2.3.4) port(514)); }; -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2