February 2008 - syslog-ng - lists.balabit.hu

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[PATCH] anonymizing filter
by Micah Anderson 06 Nov '08

06 Nov '08

Hello, A couple years ago this patch was submitted to the list for consideration for inclusion into syslog-ng. I am writing this email again to request that it be considered again. The patch provides a simple replace which enables you to strip out IP addresses from your logs before they are written to disk. The patch has been included in the Debian stable distribution, and currently is included in both Debian Sid and Lenny (unstable and testing). It has had a very wide testing base and is non-intrusive, it has existed since 2004 and has been adapted to work with the newer syslog-ng. The goal of this patch is to give an organization the means to implement site logging policies, by allowing for easy control over exactly what data is retained in the logfiles. When I first requested consideration for inclusion the reactions were some suggestions for improvement (which were done), some side discussions about the various states of data retention laws, and a general agreement that this patch is non-intrusive and had a valid use case (at least in the U.S., but also likely in other countries as well[0]). The side-discussions about data-retention laws were mostly around specific geographic localities that were considering laws that would make stripping of addresses illegal, or had already mandated such things. Although these were interesting discussions, as EU data retention laws would prohibit many people from making such configuration changes to their syslog-ng.conf, they were tangential to the point because this patch does not cause those to break such laws. On the other side of the pond, in the U.S., the EFF[1] has made it very clear that this mechanism of anonymizing logs is perfectly (a) legal in the U.S., and (b) advisable. There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain. The current climate in the U.S. makes this problem so much more important now than it was many years ago. Having the ability to implement a site-policy that enables an organization to decide if the trade-off between privacy and analysis is worthwhile. This patch allows organizations to have that choice if they feel that it is more important to avoid retaining sensitive data rather than having a full history of everything logged. Please accept this patch[2], Micah [0] EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html [1] The EFF is the major civil liberties internet watchdog in the US, their "Best Practices for Online Service Providers" can be found here: http://www.eff.org/osp, they explicitly link to our patch as a recommendation [2] The latest patch can be found at https://code.autistici.org/trac/privacy/browser/trunk/syslog-ng

6 11

Using Regex information for destination
by James Whitt 11 Jul '08

11 Jul '08

I had asked this question about a year ago, but was told to wait for the new 2.0 version. I'm currently running the latest syslog-ng 2.0.7. What I would like to do is put the line in a specific file based on part of the hostname. For example: Hostname: subdomain.domain.tld destination customer_firewall_logs { file("/logs/$TLD/$DOMAIN/$SUBDOMAIN/$R_YEAR-$R_MONTH-$R_DAY.log"); }; WHERE $TLD is the tld of the hostname, $DOMAIN is the domain from the hostname, and $SUBDOMAIN is teh subdomain from the hostname. I would like to be able to regex this information out of the hostname to use in the destination. Is this possible and any tips on getting it to work correctly would be greatly appreciated.

3 6

What packages need?
by hi ro 13 Apr '08

13 Apr '08

Hi, What packages does it need to compile syslog-ng? Is it the same package when to make RPM? (bison, flex, gcc-c++, pkgconfig, glib2-devel, libevtlog-devel) (It needs only libc to compile eventlog, doesn't it?) And if it's not necessary to connect database, does it need to install libdbi8? Thanks hiro _________________________________________________________________ 【MSNビデオ】超貴重！驚きの大物対談が実現。作家村上龍が話題のあの人に迫る http://video.msn.co.jp/rvr/default.htm

4 14

[Bug 3] New: udp flood
by bugzilla＠bugzilla.balabit.com 13 Apr '08

13 Apr '08

https://bugzilla.balabit.com/show_bug.cgi?id=3 Summary: udp flood Product: syslog-ng Version: 2.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: manoo(a)manoo.hu Type of the Report: bug Estimated Hours: 0.0 I installed syslog-ng 2.0.5 (because there was no .rpm for a RHEL3 server I made it with rpmbuild). In the config file I placed two destination lines for the same filter, first for place the log locally and an other to send it to a remote loghost (udp/514). I had a typo in the path of the local logfile (started with "/vavr" instead of "/var") and syslog-ng cannot create/write the specified file (I made no "create_dirs(yes)") when I started the logging (with logger - that was an apache logging system) the result was the following: it did not checked the writability of the file but could start itself (resart of syslog-ng was successfull) and started to sends this errormessage to the loghost with full speed: Dec 13 09:56:07 192.168.0.2 syslog-ng[19315]: Error opening file for writing; filename='/vavr/log/apache/apache.log', error='No such file or directory (2)' within 20 minutes it sends cca 700mb of clear text to the loghost. There is a leased line between the two hosts via internet and this flooded the communication channel. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 19] New: syslog-ng 2.0. x can not follow a change of IP at target syslog server
by bugzilla＠bugzilla.balabit.com 12 Apr '08

12 Apr '08

https://bugzilla.balabit.com/show_bug.cgi?id=19 Summary: syslog-ng 2.0.x can not follow a change of IP at target syslog server Product: syslog-ng Version: 2.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: enhancement Estimated Hours: 0.0 If the IP address of the destination for syslogging is changed, then all sending syslog-ng servers must be reloaded to figure out that the address has changed, even if the IP name has not changed. We recently moved one of our centralized syslog servers to a different building, resulting in a different network address. We took great pains (it was actually easy, but required planning) to maintain the IP name of the syslog server, but all of the sending hosts required a syslog-ng reload to resolve the IP name in the configuration file to the new IP address. The "reopen" logic should be changed to resolve the IP address of the hostname provided in the configuration file. Hopefully the reopen process should be relatively rare, and no significant load will be placed on the DNS servers during normal operation. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

5 6

AIX startup
by Brian Loe 27 Mar '08

27 Mar '08

Okay, my syslog-ng instance died at some point last night and because I don't have it setup properly for respawning... Can anyone running syslog-ng on AIX 5.* give me some tips for this? I'm feeding pipes so i have to start instances of SEC as soon as I start syslog-ng - but the SEC scripts appear to "close" so if they're configured to respawn I'll have 2000 instances of them in an hour.

7 21

Repeated log messages?
by John Morrissey 17 Mar '08

17 Mar '08

Recently, we've noticed a few machines are filling up their log filesystems with duplicate log entries. At first, I thought this behavior was caused by running out of disk space (i.e., the machine runs out of disk and syslog-ng does some sort of buffering, and as disk space oscellates between a few hundred bytes available and completely full, syslog-ng is writing this buffered log data out to disk but never removing the log entries from its buffer), but I can't reliably reproduce it. I also noticed that sometimes this would happen when the remote syslog sub was unavailable, but I can't reliably reproduce this behavior by blocking UDP syslog traffic directed at the remote syslog hub. I'd also wonder how syslog-ng would know that UDP syslog traffic is being dropped, unless the nature of the traffic block is such that an ICMP message (host/port unreachable, etc.) is returned to the sending host. I've poked and prodded at syslog-ng, attempting to reliably reproduce this behavior, but haven't been able to. I'm not sure if either of these two events (out of disk space, loss of network access to the syslog hub) are simple coincidences or actually cause/contribute to the behavior. The odd part is that the duplicate log entries seem to be logged forever, such as if syslog-ng was in an infinite loop. Additionally, each duplicate log entry has an additional space each time it's duplicated. For example: Original log entries: Feb 12 06:28:12 rdr01 su[21942]: Successful su for nobody by root Feb 12 06:28:12 rdr01 su[21942]: + ??? root:nobody First round of duplicates, with a single trailing space: Feb 12 06:28:12 localhost su[21942]: Successful su for nobody by root Feb 12 06:28:12 localhost su[21942]: + ??? root:nobody Second round of duplicates, with two trailing spaces: Feb 12 06:28:12 localhost su[21942]: Successful su for nobody by root Feb 12 06:28:12 localhost su[21942]: + ??? root:nobody Third round of duplicates, with three trailing spaces: Feb 12 06:28:12 localhost su[21942]: Successful su for nobody by root Feb 12 06:28:12 localhost su[21942]: + ??? root:nobody [and so on] This is with syslog-ng 2.0.0-1etch1. john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

3 4

incorrect firewall logs with syslogng-2.0.0 [security error?]
by Jacek Kalinski 12 Mar '08

12 Mar '08

Hello, After upgradeing from syslogng-1.6.11 to 2.0.0 I've got a strange messages in /var/log/messages file: Nov 29 08:14:05 denise WINDOW=16985 RES=0x00 ACK URGP=0 Nov 29 11:10:19 denise PT=34536 WINDOW=16985 RES=0x00 ACK URGP=0 Nov 29 16:42:16 denise W=0 RES=0x00 RST URGP=0 Nov 29 21:37:15 denise 116 ID=34901 DF PROTO=TCP SPT=3584 DPT=80 WINDOW=65535 RES=0x00 ACK FIN URGP=0 These are truncated iptables logs. Because in firewall.log file are simmilar entries to truncated one, I think full file should be: Nov 29 11:10:19 denise firewallp=INVALID:1 a=DROP IN=eth1 OUT= MAC=00:30:4f:36:2b:dc:00:04:9a:2c:7f:20:08:00 SRC=193.41.230.81 DST=xxx.xxx.xxx.x LEN=576 TOS=0x00 PREC=0x00 TTL=117 ID=3495 PROTO=TCP SPT=443 DPT=34536 WINDOW=16985 RES=0x00 ACK URGP=0 (xxx.xxx.xxx.x is a real server IP address - one 'x' is one digit) Config file with interesting lines: options { dir_perm(0700); perm(0600); chain_hostnames(no); stats(1200); keep_hostname(yes); dir_owner(root); dir_group(root); owner(root); group(root); }; source local { unix-stream("/dev/log" max_connections(300)); file("/proc/kmsg"); # udp(ip(0.0.0.0) port(514)); internal(); }; # ..... # authpriv.none;cron.none;mail.none;news.none -/var/log/messages filter f_2 { not facility(news); }; filter f_3 { not facility(local5); }; filter f_4 { not facility(local6); }; filter f_5 { not facility(authpriv); }; filter f_6 { not facility(cron); }; filter f_7 { not facility(mail); }; filter f_notlocal7 { not facility(local7); }; filter f_notfirewall { not match("firewallp="); }; destination d_2 { file("/var/log/messages" create_dirs(yes)); }; log { source(local); filter(f_2); filter(f_4); filter(f_5); filter(f_6); filter(f_7); filter(f_notimapd); filter(f_notpopa3d); filter(f_notnamed); filter(f_notfirewall); destination(d_2); }; #..... # firewallp.* -/var/log/firewall.log filter f_16 { level(warning) and match("firewallp="); }; destination d_12 { file("/var/log/firewall.log" create_dirs(yes)); }; log { source(local); filter(f_16); destination(d_12); }; #..... Do you know where could be an error? It is a production server so I can't debug syslog-ng on this machine (sorry). NOTE: the only truncated lines I can find are iptables (AFAIR they are logged by /proc/kmesg) entries. All other lines are complete. And it is probably a critical error in security. PS: simple statistics line from syslog-ng: Nov 30 00:00:06 denise syslog-ng[32682]: Log statistics; dropped='pipe(/dev/tty12)=0', processed='center(queued)=758289', processed='center(received)=369020', processed='destination(d_maillogimapd)=4245', processed='destination(d_5)=1', processed='destination(d_4)=0', processed='destination(d_3)=14259', processed='destination(d_2)=14259', processed='destination(d_13)=0', processed='destination(d_1)=6000', processed='destination(d_12)=6000', processed='destination(d_11)=369020', processed='destination(d_10)=36', processed='destination(d_8)=0', processed='destination(d_maillogpopa3d)=103224', processed='destination(d_7)=241245', processed='destination(d_6)=0', processed='source(local)=369020' I think you see that it could be a little hard to debug (with so many messages parsed by syslog-ng) ;) Jacek

3 6

Syslog-ng losing messages on solaris 10
by AndyH＠nominet.org.uk 12 Mar '08

12 Mar '08

Hi, I've been running 2.0.6 on Solaris 10 and it all seemed to be working fine. On a new application I have found that it loses messages. This application sends 3-7000 messages/sec. Part of these messages contains a unique ID which increments. When I run the syslogd as supplied with Solaris 10 then all messages get logged, but when I use syslog-ng then it loses messages. On a Sun V210 I see these messages message overflow on /dev/log minor #6 -- is syslogd(1M) running? message overflow on /dev/log minor #6 -- is syslogd(1M) running? message overflow on /dev/log minor #6 -- is syslogd(1M) running? on the console, and eventually there is a hardware reset and the server reboots. I have tried running syslogd to just pass messages on to syslog-ng and this doesn't cause the hardware reset, but still loses some messages. I have also tried the time_sleep and log_fetch_limit options with no success. I have noticed that syslogd is multi-threaded but syslog-ng is only single threaded. Is this what is causing these messages to get missed? Are there any plans for a threaded version? Anyone have any ideas as to how to get all messages logged with syslog-ng? Regards Andy

2 10