July 2008 - syslog-ng - lists.balabit.hu

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng client messages delayed
by mark peters 22 Mar '10

22 Mar '10

Hi, We are running into a issue where syslog-ng clients who are configured to forward to a central syslog-ng destination server that are reasonably chatty are getting there messages delayed at the source (ie. 'tcpdump'ing the client shows it is sending messages from N minutes ago, where N is anything from 5 to 60 minutes on average sometimes more). We have also seen a small percentage of loss at the destination. On these 'delayed' clients we also see the resident memory footprint on the growing. Clients are running 1.6.12 (have tried 2.0.8 also - same symptons). The central syslog-ng server is running 2.0.8. relevant config entries are: syslog-ng client: options { sync (1); time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; destination d_central { tcp("log2.xyz.com" port(5140)); }; filter f_local5 { facility(local5); }; log { source(s_sys); filter(f_local5); destination(d_central); flags(final); }; syslog-ng central server: options { time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_expire (28800); use_fqdn (no); create_dirs (yes); dir_perm (0755); perm (0644); keep_hostname (no); use_time_recvd (no); stats_freq(600); flush_timeout(16000); flush_lines(256); time_sleep(80); }; destination d_server { file("/logs/server.$LEVEL.log"); }; filter f_server { facility(local5) and program("server"); }; log { source(s_sys); filter(f_server); destination(d_server); flags(final); }; to give a rough idea of message volume the server is receiving approximately 50 million log lines per day on average any help/information would be appreciated thanks -------------------------------------- Easy + Joy + Powerful = Yahoo! Bookmarks x Toolbar http://pr.mail.yahoo.co.jp/toolbar/

3 4

[Bug 21] New: call shutdown() before closing a socket
by bugzilla＠bugzilla.balabit.com 30 Apr '09

30 Apr '09

https://bugzilla.balabit.com/show_bug.cgi?id=21 Summary: call shutdown() before closing a socket Product: syslog-ng Version: 2.1.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bazsi(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 syslog-ng should call shutdown() before closing a socket to ensure the contents of the socket buffer are emptied. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[PATCH] anonymizing filter
by Micah Anderson 06 Nov '08

06 Nov '08

Hello, A couple years ago this patch was submitted to the list for consideration for inclusion into syslog-ng. I am writing this email again to request that it be considered again. The patch provides a simple replace which enables you to strip out IP addresses from your logs before they are written to disk. The patch has been included in the Debian stable distribution, and currently is included in both Debian Sid and Lenny (unstable and testing). It has had a very wide testing base and is non-intrusive, it has existed since 2004 and has been adapted to work with the newer syslog-ng. The goal of this patch is to give an organization the means to implement site logging policies, by allowing for easy control over exactly what data is retained in the logfiles. When I first requested consideration for inclusion the reactions were some suggestions for improvement (which were done), some side discussions about the various states of data retention laws, and a general agreement that this patch is non-intrusive and had a valid use case (at least in the U.S., but also likely in other countries as well[0]). The side-discussions about data-retention laws were mostly around specific geographic localities that were considering laws that would make stripping of addresses illegal, or had already mandated such things. Although these were interesting discussions, as EU data retention laws would prohibit many people from making such configuration changes to their syslog-ng.conf, they were tangential to the point because this patch does not cause those to break such laws. On the other side of the pond, in the U.S., the EFF[1] has made it very clear that this mechanism of anonymizing logs is perfectly (a) legal in the U.S., and (b) advisable. There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain. The current climate in the U.S. makes this problem so much more important now than it was many years ago. Having the ability to implement a site-policy that enables an organization to decide if the trade-off between privacy and analysis is worthwhile. This patch allows organizations to have that choice if they feel that it is more important to avoid retaining sensitive data rather than having a full history of everything logged. Please accept this patch[2], Micah [0] EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html [1] The EFF is the major civil liberties internet watchdog in the US, their "Best Practices for Online Service Providers" can be found here: http://www.eff.org/osp, they explicitly link to our patch as a recommendation [2] The latest patch can be found at https://code.autistici.org/trac/privacy/browser/trunk/syslog-ng

6 11

OpenBSD + Syslog-NG 2.x Users ?
by Brian A. Seklecki 01 Aug '08

01 Aug '08

Please ping me on or off-list. ~BAS

1 1

Stripping numerals in the destination
by Cliff Fogle 01 Aug '08

01 Aug '08

I have several server types, like web001-web100 and thumb001-thumb010 or db001-db004 I want each server type to log to the same file and I can't for the life of me figure out a regex for $HOST in the destination statement to strip numerals. Where the first $HOST is below I want to do something like a sed "s/[0-9]*$//" destination d_host { file("/log1/syslog/$HOST/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY" template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n") template_escape(no) ); }; Thanks for any help you can provide. I've been banging my head against google for hours now...

2 1

AI_ADDRCONFIG missing from netdb.h on OpenBSD
by Brian A. Seklecki 31 Jul '08

31 Jul '08

obsd43# grep -ir AI_ADDRCONFIG /usr/include/* obsd43# fbsd63$ grep -ir AI_ADDRCONFIG /usr/include/* /usr/include/lwres/netdb.h:#define AI_ADDRCONFIG 0x00000020 Seems to be fine if you just cram it into: w-syslog-ng-2.0.9/syslog-ng-2.0.9/tests/loggen/loggen.c I suppose that we'll need to maintain it as a local patch. l8* -lava (Brian A. Seklecki - Pittsburgh, PA, USA) http://www.spiritual-machines.org/ "Guilty? Yeah. But he knows it. I mean, you're guilty. You just don't know it. So who's really in jail?" ~Maynard James Keenan

2 1

Example 3.4. Script to feed a growing logfile into syslog-ng
by Chris Pratt 31 Jul '08

31 Jul '08

Followed http://www.syslog.org/syslog-ng/v2/#id2535001 and created some tail logger combos to feed some growing logfiles into syslog-ng. Issue that I am need to get around is that while I haven't missed capturing any messages this way the time stamps in syslog-ng seem to be when logger actually gets the line into syslog rather than that of the tailed files line. I have a very quick-filling file (logged 143 messages yesterday all at 14:15:18) and when I view syslog-ng these are stamped up to 14:15:36 and no longer in any order. Does anyone have any suggestions? Thanks.

4 5

The best Front end interface to syslog-ng 2.0.9
by VPN Cluster 30 Jul '08

30 Jul '08

What is the best and easy to install web interface to syslog-ng 2.0.9 open source.

1 0

Re: [syslog-ng] Example 3.4. Script to feed a growing logfileinto syslog-ng
by Alan Edmonds 25 Jul '08

25 Jul '08

This is how I capture garbage collections with tomcat. On the tomcat host in the startup.sh script. Create a named pipe, catalinia.gc Run logger to read named pipe and forward to syslog-ng host. I use a "tag" (or program) to identify it as a gc.out. I created an $INSTANCE for the different tomcats running on a single host. .. rm -f "$CATALINA_BASE"/logs/catalina.gc mkfifo "$CATALINA_BASE"/logs/catalina.gc ... "$_RUNJAVA" $JAVA_OPTS $CATALINA_OPTS \ -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" -classpath "$CLASSPATH" \ -Dcatalina.base="$CATALINA_BASE" \ -Dcatalina.home="$CATALINA_HOME" \ -Djava.io.tmpdir="$CATALINA_TMPDIR" \ -Xloggc:"$CATALINA_BASE"/logs/catalina.gc -XX:-DisableExplicitGC \ org.apache.catalina.startup.Bootstrap "$@" start 2>&1 | logger -p local2.info -t tomcat$INSTANCE & if [ ! -z "$CATALINA_PID" ]; then echo $! > $CATALINA_PID fi fi logger -p local3.info -t GC$INSTANCE < "$CATALINA_BASE"/logs/catalina.gc & .. On the syslog-ng server, it sorts it out like this: filter f_gc { facility(local3); }; destination host-program-out { file("/logs/portal/$HOST/$PROGRAM.out" template ("$MSG\n") template_escape(no) ); }; log { source(local); filter(f_gc); destination(host-program-out); }; you get a GC$INSTANCE.out file in $HOST/ on the syslog-ng server. But, you have to strip the GC$INSTANCE: from each line of the .out file before running it through HPJtune, etc. you get the idea. Alan Edmonds -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: 25 July 2008 08:35 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Example 3.4. Script to feed a growing logfileinto syslog-ng On Thu, 2008-07-24 at 16:10 -0400, Chris Pratt wrote: > Thanks for your suggestion. This is for Tomcat. Unfortunately I am > stuck reading from the log because the syslog appender in the log4j > version on our app does not capture stack traces while the file > appenders do. In other cases we want to capture garbage collections > which are only written to stdout. I tried sending my file appends to > /dev/log but that didn't work so I am stuck trying to read from files. > Only my central log server runs syslog-ng. All of my Tomcat hosts > just use syslogd and send *.* @mylogserver. Otherwise I'd probably > try # source s_tail { file("log.log" follow_freq(1) flags(no-parse)); > }; to see if it kept it in order any better. > > What I could really use is a logger that has the ability to read and > forward to syslog the existing timestamp... syslog-ng could that if it was running on the client hosts too. but otherwise what about this dirty trick: tail -f <logfile> | sed -e 's/^/<133>/' | nc syslog-server 514 If the timestamps are otherwise in syslog-ng processable format, the above scriptlet would work. The sed sets local6.notice as a priority. However the difficult part is in doing the "tail -f" part robust enough, but I guess you achieved that already. -- Bazsi ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html T-Mobile International UK Limited Company Registered Number: 3951860 Registered Office Address: Hatfield Business Park, Hatfield, Hertfordshire, AL10 9BW Registered in England and Wales NOTICE AND DISCLAIMER This email (including attachments) is confidential. If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose

1 0