June 2008 - syslog-ng - lists.balabit.hu

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng client messages delayed
by mark peters 22 Mar '10

22 Mar '10

Hi, We are running into a issue where syslog-ng clients who are configured to forward to a central syslog-ng destination server that are reasonably chatty are getting there messages delayed at the source (ie. 'tcpdump'ing the client shows it is sending messages from N minutes ago, where N is anything from 5 to 60 minutes on average sometimes more). We have also seen a small percentage of loss at the destination. On these 'delayed' clients we also see the resident memory footprint on the growing. Clients are running 1.6.12 (have tried 2.0.8 also - same symptons). The central syslog-ng server is running 2.0.8. relevant config entries are: syslog-ng client: options { sync (1); time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; destination d_central { tcp("log2.xyz.com" port(5140)); }; filter f_local5 { facility(local5); }; log { source(s_sys); filter(f_local5); destination(d_central); flags(final); }; syslog-ng central server: options { time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_expire (28800); use_fqdn (no); create_dirs (yes); dir_perm (0755); perm (0644); keep_hostname (no); use_time_recvd (no); stats_freq(600); flush_timeout(16000); flush_lines(256); time_sleep(80); }; destination d_server { file("/logs/server.$LEVEL.log"); }; filter f_server { facility(local5) and program("server"); }; log { source(s_sys); filter(f_server); destination(d_server); flags(final); }; to give a rough idea of message volume the server is receiving approximately 50 million log lines per day on average any help/information would be appreciated thanks -------------------------------------- Easy + Joy + Powerful = Yahoo! Bookmarks x Toolbar http://pr.mail.yahoo.co.jp/toolbar/

3 4

[Bug 21] New: call shutdown() before closing a socket
by bugzilla＠bugzilla.balabit.com 30 Apr '09

30 Apr '09

https://bugzilla.balabit.com/show_bug.cgi?id=21 Summary: call shutdown() before closing a socket Product: syslog-ng Version: 2.1.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bazsi(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 syslog-ng should call shutdown() before closing a socket to ensure the contents of the socket buffer are emptied. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[PATCH] anonymizing filter
by Micah Anderson 06 Nov '08

06 Nov '08

Hello, A couple years ago this patch was submitted to the list for consideration for inclusion into syslog-ng. I am writing this email again to request that it be considered again. The patch provides a simple replace which enables you to strip out IP addresses from your logs before they are written to disk. The patch has been included in the Debian stable distribution, and currently is included in both Debian Sid and Lenny (unstable and testing). It has had a very wide testing base and is non-intrusive, it has existed since 2004 and has been adapted to work with the newer syslog-ng. The goal of this patch is to give an organization the means to implement site logging policies, by allowing for easy control over exactly what data is retained in the logfiles. When I first requested consideration for inclusion the reactions were some suggestions for improvement (which were done), some side discussions about the various states of data retention laws, and a general agreement that this patch is non-intrusive and had a valid use case (at least in the U.S., but also likely in other countries as well[0]). The side-discussions about data-retention laws were mostly around specific geographic localities that were considering laws that would make stripping of addresses illegal, or had already mandated such things. Although these were interesting discussions, as EU data retention laws would prohibit many people from making such configuration changes to their syslog-ng.conf, they were tangential to the point because this patch does not cause those to break such laws. On the other side of the pond, in the U.S., the EFF[1] has made it very clear that this mechanism of anonymizing logs is perfectly (a) legal in the U.S., and (b) advisable. There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain. The current climate in the U.S. makes this problem so much more important now than it was many years ago. Having the ability to implement a site-policy that enables an organization to decide if the trade-off between privacy and analysis is worthwhile. This patch allows organizations to have that choice if they feel that it is more important to avoid retaining sensitive data rather than having a full history of everything logged. Please accept this patch[2], Micah [0] EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html [1] The EFF is the major civil liberties internet watchdog in the US, their "Best Practices for Online Service Providers" can be found here: http://www.eff.org/osp, they explicitly link to our patch as a recommendation [2] The latest patch can be found at https://code.autistici.org/trac/privacy/browser/trunk/syslog-ng

6 11

Using Regex information for destination
by James Whitt 11 Jul '08

11 Jul '08

I had asked this question about a year ago, but was told to wait for the new 2.0 version. I'm currently running the latest syslog-ng 2.0.7. What I would like to do is put the line in a specific file based on part of the hostname. For example: Hostname: subdomain.domain.tld destination customer_firewall_logs { file("/logs/$TLD/$DOMAIN/$SUBDOMAIN/$R_YEAR-$R_MONTH-$R_DAY.log"); }; WHERE $TLD is the tld of the hostname, $DOMAIN is the domain from the hostname, and $SUBDOMAIN is teh subdomain from the hostname. I would like to be able to regex this information out of the hostname to use in the destination. Is this possible and any tips on getting it to work correctly would be greatly appreciated.

3 6

Re: [syslog-ng] loop caused by syslog-ng filter
by JUNG, Christian 07 Jul '08

07 Jul '08

Okay. Don't know if I get the things right, but: All scripts you've mailed to the list write something to a file or send a mail if they're called and terminate directly after that action. syslog-ng will respawn (neu starten) them directly. The program started by the program destination should not terminate itself. It has to listen on STDIN for a log message given from syslog-ng. Afterwards it can do something useful and then it has to listen for the next message. In a shell script you can do this with a "while read LINE; do ...; done". Try this: ---8<--- #!/bin/bash while read LINE; do echo $(date) $LINE >> /tmp/schrott done ---8<--- Execute this script on the command line, enter some random stuff and look into /tmp/schrott: user@box:~> ./test-script bla bla bla bla Terminate this script by pressing CTRL-D. You should see something like this in /tmp/schrott: Thu Mar 20 15:13:16 CET 2008 bla bla bla Thu Mar 20 15:13:18 CET 2008 bla If not your script doesn't work :-) See for typos. If this works, put it in your syslog-ng conf. It should do the same (only prio 1 messages should be visible with the date prepended). If this works well, try this script first on the command line: ---8<--- #!/bin/bash while read LINE; do mail -s "High Priority Snort Alert" Sub-Zero(a)xxx.de <<-EOF Alert, Priority 1 $LINE EOF done ---8<--- You should receive for every given input line exactly one mail. If this works put it in your syslog-ng.conf. Now you should be done. The thread you mentioned is about two running scripts where only one should run. This shouldn't be the case here. But you can have a look at the output of "ps fax". If you see multiple processes under syslog-ng then you might have the problem. bye Chris > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu > [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of Adam Richter > Sent: Thursday, March 20, 2008 2:46 PM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] loop caused by syslog-ng filter > > > Hi! > > Not working! Syslog-ng filters for exactly the string: [Priority: 1] > and not as it is piped by the mail script: #priority 1# note the ":" ! > Anyway I used your script --> same fault!!! > I have also used following script: > > > #!/bin/sh > > echo AAA >> /tmp/schrott > date >> /tmp/schrott > > There is no output like [Priority: 1]!!! > Then I did following: tail -f /tmp/schrott and got a loop > too! The loop starts when syslog-ng recognices the first > machting string [Priority: 1] and loops till I stop syslog-ng!!! > > > Sensor1:~# /etc/init.d/syslog-ng stop > Stopping system logging: syslog-ng. > Sensor1:~# tail -f /tmp/schrott > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > > > I think it has something in common with this thread: > > https://lists.balabit.hu/pipermail/syslog-ng/2006-October/009454.html > > Any other ideas? It´s very important! > > German: Es handelt sich hier um meine Abschlussprüfung, und > dies ist der letzte Fehler der Auftritt, ansonsten läuft das Projekt. > > bye, Adam / Sub-Zero ! > > > -------- Original-Nachricht -------- > > Datum: Thu, 20 Mar 2008 10:32:31 +0100 > > Von: "JUNG, Christian" <christian.jung(a)saarstahl.com> > > An: "Syslog-ng users\' and developers\' mailing list" > <syslog-ng(a)lists.balabit.hu> > > Betreff: Re: [syslog-ng] loop caused by syslog-ng filter > > > Hi Adam, > > > > syslog-ng does the right thing :-). > > > > It starts the program/script once and pipes on STDIN every > log-message > > which matches the filter. > > > > If your script is started, it will call mail and pipe > "Alert, priority 1" > > to its STDIN and then exits. syslog-ng sees this and > restarts it (version > > 2.0 or higher behave that way, see > > > <http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ > ch09s02.html#reference_destination_program>). > > > > For you purpose this would be better: > > > > ---8<--- > > #!/bin/bash > > > > while read LINE; do > > cat <<-EOF | mail -s "High Priority Snort Alert" > Sub-Zero(a)xxx.de > > Alert, Priority 1 > > $LINE > > EOF > > done > > ---8<--- > > > > > > bye > > Chris > > > > > > > > -----Original Message----- > > > From: syslog-ng-bounces(a)lists.balabit.hu > > > [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of > Adam Richter > > > Sent: Thursday, March 20, 2008 9:23 AM > > > To: syslog-ng(a)lists.balabit.hu > > > Subject: [syslog-ng] loop caused by syslog-ng filter > > > > > > > > > Hi! > > > > > > First off, sorry for my poor english! > > > I have a problem with a loop caused by syslog-ng v. 2.0.8. I > > > have set up Snort as an IDS System. Snort writes its messages > > > in unified-format to /var/log/snort/snort.alert and > > > /var/log/snort/snort.log. There are two Barnyard processes > > > which read the unified files and convert it to messages that > > > syslog and MySQL understand. Syslog-ng writes the messages > > > to /var/log/auth.log. All this is working fine. Now, I want > > > to set up a filter for Priority 1 alerts. This alert should > > > be send to the Administrator. > > > > > > I used following filter for syslog-ng: > > > > > > source src {unix-stream("/dev/log"); internal();}; > > > destination email{program("/usr/local/bin/alert_mail.sh");}; > > > filter high {match("[Priority: 1]");}; > > > log {source(src);filter(high); destination(email);}; > > > > > > > > > The alert_mail.sh: > > > > > > #!/bin/sh > > > cat << EOF | mail -s "High Priority Snort Alert" Sub-Zero(a)xxx.de > > > Alert, Priority 1 > > > EOF > > > > > > > > > Then I use Nessus to cause some alerts with Priority 1. I can > > > see 4 alerts with the Priority 1 with BASE and in > /var/log/auth.log. > > > > > > Syslog-ng recognises the alert with Priority 1 and activates > > > the script /usr/local/bin/alert_mail.sh > > > > > > All this is working, but the script is restarted by syslog-ng > > > again an again. > > > > > > Extract from /var/log/messages: > > > > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Closing log writer > > > fd; fd='11' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > … > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='256' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Starting > > > destination program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='256' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Starting > > > destination program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; > > > > > > … > > > > > > > > > I get thousands of mails per minute till I stop syslog-ng. > > > > > > Output of /var/log/auth.log(so y see that syslog-ng writes > > > snort/barnyard messages correctly to auth.log): > > > > > > Mar 19 13:56:54 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:4758 -> 172.28.100.10:137 > > > Mar 19 13:57:13 src@Sensor1 barnyard: [1:1446:8] SMTP vrfy > > > root [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:1085 -> 172.28.100.10:25 > > > Mar 19 13:57:13 src@Sensor1 barnyard: [1:660:11] SMTP expn > > > root [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:1085 -> 172.28.100.10:25 > > > Mar 19 13:57:22 src@Sensor1 barnyard: [1:12626:2] Snort Alert > > > [1:12626:0] [Classification: Decode of an RPC Query] > > > [Priority: 2] {UDP} 172.25.1.152:1146 -> 172.28.100.10:111 > > > Mar 19 13:57:22 src@Sensor1 barnyard: [1:585:9] RPC portmap > > > sadmind request UDP [Classification: Decode of an RPC Query] > > > [Priority: 2] {UDP} 172.25.1.152:1146 -> 172.28.100.10:111 > > > Mar 19 13:57:24 src@Sensor1 barnyard: [1:566:6] POLICY > > > PCAnywhere server response [Classification: Misc activity] > > > [Priority: 3] {UDP} 172.25.1.152:1155 -> 172.28.100.10:5632 > > > Mar 19 15:11:27 src@Sensor1 barnyard: [122:1:0] portscan: TCP > > > Portscan [Classification: Unknown] [Priority: 3] {PROTO255} > > > 172.25.1.152 -> 172.28.100.10 > > > Mar 19 15:11:58 src@Sensor1 barnyard: [1:1420:13] SNMP trap > > > tcp [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:162 > > > Mar 19 15:11:58 src@Sensor1 barnyard: [1:1418:13] SNMP > > > request tcp [Classification: Attempted Information Leak] > > > [Priority: 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:161 > > > Mar 19 15:12:05 src@Sensor1 barnyard: [1:1421:13] SNMP > > > AgentX/tcp request [Classification: Attempted Information > > > Leak] [Priority: 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:705 > > > Mar 19 15:12:16 src@Sensor1 barnyard: [122:1:0] portscan: TCP > > > Portscan [Classification: Unknown] [Priority: 3] {PROTO255} > > > 172.25.1.152 -> 172.28.100.10 > > > Mar 19 15:12:19 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > Mar 19 15:12:20 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > Mar 19 15:12:22 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > > > > > > > I think it has something in common with this topic: > > > > > > https://lists.balabit.hu/pipermail/syslog-ng/2006-October/009454.html > > > > Thanks in advance! > > > > Sub-Zero > > > > > > -- > > GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! > > Jetzt dabei sein: http://www.shortview.de/?mc=sv_ext_mf@gmx > > ______________________________________________________________ > > ________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > -- Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games! http://games.entertainment.gmx.net/de/entertainment/games/free ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

4 10

migrate over to PCRE?
by Balazs Scheidler 04 Jul '08

04 Jul '08

Dear syslog-ng users, The following text was written on my blog, but I'm reposting it here for a wider audience to read: As of now the development of the generic rewrite feature has been completed in one of my private git repositories. The new code uses PCRE and I'm somewhat undecided how to move forward with PCRE. For those who might not know PCRE is an implementation of regular expressions and is an acronym for "Perl Compatible Regular Expressions". PCRE adds a lot more features and seems to perform better than its POSIX equivalent. So the situation is as follows: * various filters use POSIX regexps * rewrite uses PCRE This is not a very consistent combination, thus I'm planning to add PCRE support for filters too. The only question is whether it is needed to have two independent regexp styles in syslog-ng in the long run. If I decide that one of them is enough, then I'd deprecate POSIX style regexps in filters and wouldn't implement POSIX in rewrite rules. This combination would yield a syslog-ng that would give warnings when POSIX-style regular expressions are in use and in a forthcoming release I'd change the default regexp style to PCRE, and yet another syslog-ng release later, I'd phase out POSIX completely. If the decision is to keep them both in the long run, it would mean that I'd need to implement POSIX style regexps for rewrite rules as well. This would probably the least intrusive for users, but also a lot more work. Also, this would allow adding other filtering options like globbing or prefix search. What do you think? Is the addition of modular search algorithms worth it? -- Bazsi

3 3

logging facility in log file
by Jing CDL Sun 30 Jun '08

30 Jun '08

Hi, ALL I am using syslog-ng to do remote logging for facility of "local6" and priority of "warn", the log server is Linux system(SLES10 SP1), it works well with the message content, only one problem is the keyword of "local6" is not logged in the log file, please see the details below: on c890mgrs4(Linux) Jun 25 20:53:34 c890f12ec07 root: sunjing test 1 Jun 25 20:53:38 c890f12ec07 root: sunjing test 2 I did the same thing on AIX log server, while it could log the keyword of "local6" in log file: on c890mgrs3(AIX) Jun 25 20:53:34 c890f12ec07 local6:warn|warning c890f12ec07 root: sunjing test 1 Jun 25 20:53:38 c890f12ec07 local6:warn|warning c890f12ec07 root: sunjing test 2 So I am guessing that I did not enable this facility logging in syslog-ng, maybe syslog-ng has a certain configuration that could let the "facility" logged in message file, am I right? or could you point me to the correct method? I need this "facility" information to develop my applications. Thank you. Best Regards, ----------------------------- Sun Jing(孙靖) IBM China Software Development Laboratory Tel: (86-10) 82453625 E-mail: sjing(a)cn.ibm.com Address:5F Deshi Building No.9 Shangdi East Road Haidian District,Beijing City P.R.China 100085, P.R.China.

4 6

syslog-ng 2.1beta2 has been released
by devel＠balabit.hu 30 Jun '08

30 Jun '08

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 2.1beta2 SUMMARY : new stable release DATE : Jun 30, 2008 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (2.1beta2) has been released. For latest fixes in the 2.1.x branch you are recommended to upgrade to this version. CHANGES: 2.1beta2 Fri, 13 Jun 2008 14:37:08 +0200 Bugfixes: * Create /var/lib/syslog-ng directory from RPM packages. * Fixed a 100% CPU usage problem triggered by the pipe() destination. This is a regression in 2.1beta1 and does not affect earlier versions. * Removed test code that injected "Connection timed out" errors into local file writes. * Set the current directory to the root once syslog-ng daemonized itself. * Fixed the $PID macro to only extract information from the syslog message header and not from the message contents. Other changes: * Instead of using O_APPEND for file writes, add an lseek before writing which performs much better with the NFS client in BSD kernels. * Added support for suppressing duplicate messages. This feature was contributed by Chris Packham. * The default linking mode is changed to dynamic linking, as static linking is more difficult for users to cope with. Those who use NFS mounted /usr on machines where syslog-ng runs on will probably need to explicitly ask for the previous behaviour. BINARY DOWNLOAD: If you have a binary subscription, you can download the latest binaries from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… OR, if you have a platform that is supported by apt-get, use the following apt sources to fetch the latest releases: Debian GNU/Linux ---------------- sarge: deb https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ debian-sarge/syslog-ng-2.1 syslog-ng etch: deb https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ debian-etch/syslog-ng-2.1 syslog-ng RedHat Enterprise Linux ----------------------- RHEL-4 rpm https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ rhel-4/syslog-ng-2.1 syslog-ng SUSE 10 ------- SUSE 10.0 rpm https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ suse-10.0/syslog-ng-2.1 syslog-ng SUSE 10.1 rpm https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ suse-10.1/syslog-ng-2.1 syslog-ng HTTP can also be used in the place of HTTPS If your version of apt-get does not support the HTTPS protocol. When using plain HTTP, the username and password will not be encrypted. SOURCE DOWNLOAD: The latest versions of syslog-ng in source format can be found at: http://www.balabit.com/downloads/files/syslog-ng/sources/2.1/src/

1 0

syslog-ng suddenly stops logging
by Richard Vigeant 30 Jun '08

30 Jun '08

Hi, I have a configuration where several nodes send all log messages to a central server. The applications on remote nodes send their logs locally either via UDP or a unix socket. The syslog-ng running on remote nodes simply pick up all log messages from all sources, i.e. TCP, UDP, /proc/kmsg, /dev/log and internal, and transmit all messages to the central server uisng TCP. The remote node's config file follows. We've been having intermittent problems where the central server would suddenly stop logging messages from certain nodes. We noticed that very often restarting syslog-ng on the central server would fix the condition and logging would carry on. Howver I discovered a new rare case where restarting the central syslog-ng didn't work. I found out by doing a tcpdump that the remote syslog-ng was not sending the log messages. I have done an strace on the remote syslog-ng and it shows that nothing happens after a message has been "recvfrom()" or "read()". Then I have restarted syslog-ng and things went back to normal. In the 2nd strace we can see that there is a "write()" after the "read()". I'm using syslog-ng 2.0.9. ********************* Remote node's Config file: ********************* options { chain_hostnames(no); log_msg_size(8192); log_fifo_size(50000); log_iw_size(250); log_fetch_limit(25); flush_lines(10); flush_timeout(10); }; source local { pipe("/proc/kmsg"); unix-stream("/dev/log" max-connections(50)); internal(); }; source network { udp(localport(514)); tcp(localport(514)); }; destination spotxdeoam { tcp("spotxdeoam" destport(514)); }; destination local { tcp("localhost" destport(514)); }; # To change the local log path for modules, change the following statement. destination local_Modules { file("/opt/spotxde/software/var/SPOTxdeServer/logs/$HOST/$PROGRAM/ messages" create_dirs(yes) template("$FULLDATE, $HOST, $MSG\n") template_escape(no)); }; # To change the local log path for node, change the following statement. destination local_Nodes { file("/opt/spotxde/software/var/SPOTxdeServer/logs/$HOST/messages" create_dirs(yes) template("$FULLDATE, $HOST, $MSG\n") template_escape(no)); }; filter Modules { program("^OAM") or program("^TRX") or program("^VA1") or [root@c15 Rich]# more /opt/spotxde/software/etc/syslog_ng/syslog- ng_trx.conf options { chain_hostnames(no); log_msg_size(8192); log_fifo_size(50000); log_iw_size(250); log_fetch_limit(25); flush_lines(10); flush_timeout(10); }; source local { pipe("/proc/kmsg"); unix-stream("/dev/log" max-connections(50)); internal(); }; source network { udp(localport(514)); tcp(localport(514)); }; destination spotxdeoam { tcp("spotxdeoam" destport(514)); }; destination local { tcp("localhost" destport(514)); }; # To change the local log path for modules, change the following statement. destination local_Modules { file("/opt/spotxde/software/var/SPOTxdeServer/logs/$HOST/$PROGRAM/ messages" create_dirs(yes) template("$FULLDATE, $HOST, $MSG\n") template_escape(no)); }; # To change the local log path for node, change the following statement. destination local_Nodes { file("/opt/spotxde/software/var/SPOTxdeServer/logs/$HOST/messages" create_dirs(yes) template("$FULLDATE, $HOST, $MSG\n") template_escape(no)); }; filter Modules { program("^OAM") or program("^TRX") or program("^VA1") or program("^VA2") or program("^VA3") or program("^VA4") or program("^MM1") or program("^MM3") or program("^MM4") or program("^MM7") or program("^WAP") or program("^STI") or program("^HCA") or program("^RPT") or program("^PRF") or program("^SVW") or program("^MNG") or program("^CDR") or program("^EAIF") or program("^TEST") or program("^MMSC") or program("^SSMA") or program("^SMTP") or program("^ICAP") or program("^BIDA") or program("^SMPP") or program("^LTRX") or program("^MCDN") or program("^RMSS") or program("^EPGS"); }; log { source(local); source(network); filter(Modules); destination(spotxdeoam); # Remove the comment on the next statement to log locally on the TRX nodes the Modules related logs # destination(local_Modules); flags(final); }; log { source(local); source(network); destination(spotxdeoam); # Remove the comment on the next statement to log locally on the TRX nodes the Node relatedlogs # destination(local_Nodes); flags(fallback); }; ********************* ********************* Following is an strace of syslog-ng that doesn't work. There were 2 attemps in this one: one using UDP and on using Unix socket. ********************* [root@c15 Rich]# strace -p 8259 Process 8259 attached - interrupt to quit gettimeofday({1213891616, 539195}, NULL) = 0 gettimeofday({1213891616, 539246}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 8, 22650) = 0 gettimeofday({1213891639, 190435}, NULL) = 0 time(NULL) = 1213891639 time(NULL) = 1213891639 gettimeofday({1213891639, 190507}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 8, 30000) = 0 gettimeofday({1213891669, 191715}, NULL) = 0 time(NULL) = 1213891669 time(NULL) = 1213891669 gettimeofday({1213891669, 191778}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN, revents=POLLIN}], 8, 30000) = 1 gettimeofday({1213891676, 326051}, NULL) = 0 recvfrom(3, "<30>Jun 19 12:07:56 RMSS: /syslo"..., 8192, 0, {sa_family=AF_INET, sin_port=htons(39166), sin_addr=inet_addr("127.0.0.1")}, [16]) = 100 gettimeofday({1213891676, 326294}, NULL) = 0 time(NULL) = 1213891676 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1252, ...}) = 0 time(NULL) = 1213891676 gettimeofday({1213891676, 326621}, NULL) = 0 recvfrom(3, 0x8131ef8, 8192, 0, 0xffffd560, 0xffffd55c) = -1 EAGAIN (Resource temporarily unavailable) gettimeofday({1213891676, 326794}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 8, 22865) = 0 gettimeofday({1213891699, 193074}, NULL) = 0 time(NULL) = 1213891699 time(NULL) = 1213891699 gettimeofday({1213891699, 193277}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 8, 30000) = 0 gettimeofday({1213891729, 195368}, NULL) = 0 time(NULL) = 1213891729 time(NULL) = 1213891729 gettimeofday({1213891729, 195569}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 8, 30000) = 0 gettimeofday({1213891759, 196643}, NULL) = 0 time(NULL) = 1213891759 time(NULL) = 1213891759 gettimeofday({1213891759, 196842}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 8, 30000) = 0 gettimeofday({1213891789, 197931}, NULL) = 0 time(NULL) = 1213891789 time(NULL) = 1213891789 gettimeofday({1213891789, 198129}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 8, 30000) = 0 gettimeofday({1213891819, 199219}, NULL) = 0 time(NULL) = 1213891819 time(NULL) = 1213891819 gettimeofday({1213891819, 199421}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 8, 30000) = 0 gettimeofday({1213891849, 200510}, NULL) = 0 time(NULL) = 1213891849 time(NULL) = 1213891849 gettimeofday({1213891849, 200707}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN, revents=POLLIN}, {fd=3, events=POLLIN}], 8, 30000) = 1 gettimeofday({1213891867, 753947}, NULL) = 0 accept(6, {sa_family=AF_FILE, path = "??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????"}, [2 ]) = 11 setsockopt(11, SOL_SOCKET, SO_KEEPALIVE, [0], 4) = 0 fcntl64(11, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(11, F_SETFL, O_RDWR|O_NONBLOCK) = 0 fcntl64(11, F_GETFD) = 0 fcntl64(11, F_SETFD, FD_CLOEXEC) = 0 brk(0) = 0x8ae1000 brk(0x8b02000) = 0x8b02000 gettimeofday({1213891867, 754658}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=11, events=POLLIN, revents=POLLIN|POLLHUP}, {fd=6, events=POLLIN}], 9, 11446) = 1 gettimeofday({1213891867, 754830}, NULL) = 0 read(11, "<30>Jun 19 12:11:07 RMSS: /syslo"..., 8192) = 100 gettimeofday({1213891867, 755001}, NULL) = 0 time(NULL) = 1213891867 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1252, ...}) = 0 gettimeofday({1213891867, 755271}, NULL) = 0 read(11, "", 8192) = 0 close(11) = 0 gettimeofday({1213891867, 755520}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}], 8, 11445) = 0 gettimeofday({1213891879, 201791}, NULL) = 0 time(NULL) = 1213891879 time(NULL) = 1213891879 gettimeofday({1213891879, 201987}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}], 8, 30000) = 0 gettimeofday({1213891909, 203098}, NULL) = 0 time(NULL) = 1213891909 time(NULL) = 1213891909 gettimeofday({1213891909, 203299}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN|POLLOUT}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}], 8, 30000) = 0 gettimeofday({1213891939, 205377}, NULL) = 0 time(NULL) = 1213891939 time(NULL) = 1213891939 gettimeofday({1213891939, 205580}, NULL) = 0 poll( ******************* And now the strace sequence when it works: We can see that after the read there is a write to transmit the message to the central log server. ******************* [root@c15 Rich]# strace -p 15310 Process 15310 attached - interrupt to quit gettimeofday({1213900598, 12321}, NULL) = 0 gettimeofday({1213900598, 12361}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=5, events=POLLIN}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=3, events=POLLIN}], 7, 2069) = 0 gettimeofday({1213900600, 83036}, NULL) = 0 time(NULL) = 1213900600 time(NULL) = 1213900600 gettimeofday({1213900600, 83101}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=5, events=POLLIN}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=6, events=POLLIN, revents=POLLIN}, {fd=3, events=POLLIN}], 7, 30000) = 1 gettimeofday({1213900610, 338336}, NULL) = 0 accept(6, {sa_family=AF_FILE, path=@}, [2]) = 10 setsockopt(10, SOL_SOCKET, SO_KEEPALIVE, [0], 4) = 0 fcntl64(10, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0 fcntl64(10, F_GETFD) = 0 fcntl64(10, F_SETFD, FD_CLOEXEC) = 0 gettimeofday({1213900610, 338937}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=5, events=POLLIN}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=3, events=POLLIN}, {fd=10, events=POLLIN, revents=POLLIN|POLLHUP}, {fd=6, events=POLLIN}], 8, 19745) = 1 gettimeofday({1213900610, 339123}, NULL) = 0 read(10, "<30>Jun 19 14:36:50 RMSS: /syslo"..., 8192) = 100 gettimeofday({1213900610, 339320}, NULL) = 0 time(NULL) = 1213900610 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1252, ...}) = 0 gettimeofday({1213900610, 339627}, NULL) = 0 read(10, "", 8192) = 0 close(10) = 0 gettimeofday({1213900610, 339888}, NULL) = 0 poll([{fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=5, events=POLLIN}, {fd=9, events=POLLIN}, {fd=4, events=POLLIN}, {fd=3, events=POLLIN}, {fd=6, events=POLLIN}], 7, 10) = 0 gettimeofday({1213900610, 351199}, NULL) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1252, ...}) = 0 write(8, "<30>Jun 19 14:36:50 c15 RMSS: /s"..., 103) = 103 gettimeofday({1213900610, 351494}, NULL) = 0 poll( <unfinished ...> Process 15310 detached Richard Vigeant www.vantrix.com

2 5