May 2008 - syslog-ng - lists.balabit.hu

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng client messages delayed
by mark peters 22 Mar '10

22 Mar '10

Hi, We are running into a issue where syslog-ng clients who are configured to forward to a central syslog-ng destination server that are reasonably chatty are getting there messages delayed at the source (ie. 'tcpdump'ing the client shows it is sending messages from N minutes ago, where N is anything from 5 to 60 minutes on average sometimes more). We have also seen a small percentage of loss at the destination. On these 'delayed' clients we also see the resident memory footprint on the growing. Clients are running 1.6.12 (have tried 2.0.8 also - same symptons). The central syslog-ng server is running 2.0.8. relevant config entries are: syslog-ng client: options { sync (1); time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; destination d_central { tcp("log2.xyz.com" port(5140)); }; filter f_local5 { facility(local5); }; log { source(s_sys); filter(f_local5); destination(d_central); flags(final); }; syslog-ng central server: options { time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_expire (28800); use_fqdn (no); create_dirs (yes); dir_perm (0755); perm (0644); keep_hostname (no); use_time_recvd (no); stats_freq(600); flush_timeout(16000); flush_lines(256); time_sleep(80); }; destination d_server { file("/logs/server.$LEVEL.log"); }; filter f_server { facility(local5) and program("server"); }; log { source(s_sys); filter(f_server); destination(d_server); flags(final); }; to give a rough idea of message volume the server is receiving approximately 50 million log lines per day on average any help/information would be appreciated thanks -------------------------------------- Easy + Joy + Powerful = Yahoo! Bookmarks x Toolbar http://pr.mail.yahoo.co.jp/toolbar/

3 4

[Bug 21] New: call shutdown() before closing a socket
by bugzilla＠bugzilla.balabit.com 30 Apr '09

30 Apr '09

https://bugzilla.balabit.com/show_bug.cgi?id=21 Summary: call shutdown() before closing a socket Product: syslog-ng Version: 2.1.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bazsi(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 syslog-ng should call shutdown() before closing a socket to ensure the contents of the socket buffer are emptied. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[PATCH] anonymizing filter
by Micah Anderson 06 Nov '08

06 Nov '08

Hello, A couple years ago this patch was submitted to the list for consideration for inclusion into syslog-ng. I am writing this email again to request that it be considered again. The patch provides a simple replace which enables you to strip out IP addresses from your logs before they are written to disk. The patch has been included in the Debian stable distribution, and currently is included in both Debian Sid and Lenny (unstable and testing). It has had a very wide testing base and is non-intrusive, it has existed since 2004 and has been adapted to work with the newer syslog-ng. The goal of this patch is to give an organization the means to implement site logging policies, by allowing for easy control over exactly what data is retained in the logfiles. When I first requested consideration for inclusion the reactions were some suggestions for improvement (which were done), some side discussions about the various states of data retention laws, and a general agreement that this patch is non-intrusive and had a valid use case (at least in the U.S., but also likely in other countries as well[0]). The side-discussions about data-retention laws were mostly around specific geographic localities that were considering laws that would make stripping of addresses illegal, or had already mandated such things. Although these were interesting discussions, as EU data retention laws would prohibit many people from making such configuration changes to their syslog-ng.conf, they were tangential to the point because this patch does not cause those to break such laws. On the other side of the pond, in the U.S., the EFF[1] has made it very clear that this mechanism of anonymizing logs is perfectly (a) legal in the U.S., and (b) advisable. There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain. The current climate in the U.S. makes this problem so much more important now than it was many years ago. Having the ability to implement a site-policy that enables an organization to decide if the trade-off between privacy and analysis is worthwhile. This patch allows organizations to have that choice if they feel that it is more important to avoid retaining sensitive data rather than having a full history of everything logged. Please accept this patch[2], Micah [0] EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html [1] The EFF is the major civil liberties internet watchdog in the US, their "Best Practices for Online Service Providers" can be found here: http://www.eff.org/osp, they explicitly link to our patch as a recommendation [2] The latest patch can be found at https://code.autistici.org/trac/privacy/browser/trunk/syslog-ng

6 11

Using Regex information for destination
by James Whitt 11 Jul '08

11 Jul '08

I had asked this question about a year ago, but was told to wait for the new 2.0 version. I'm currently running the latest syslog-ng 2.0.7. What I would like to do is put the line in a specific file based on part of the hostname. For example: Hostname: subdomain.domain.tld destination customer_firewall_logs { file("/logs/$TLD/$DOMAIN/$SUBDOMAIN/$R_YEAR-$R_MONTH-$R_DAY.log"); }; WHERE $TLD is the tld of the hostname, $DOMAIN is the domain from the hostname, and $SUBDOMAIN is teh subdomain from the hostname. I would like to be able to regex this information out of the hostname to use in the destination. Is this possible and any tips on getting it to work correctly would be greatly appreciated.

3 6

Re: [syslog-ng] loop caused by syslog-ng filter
by JUNG, Christian 07 Jul '08

07 Jul '08

Okay. Don't know if I get the things right, but: All scripts you've mailed to the list write something to a file or send a mail if they're called and terminate directly after that action. syslog-ng will respawn (neu starten) them directly. The program started by the program destination should not terminate itself. It has to listen on STDIN for a log message given from syslog-ng. Afterwards it can do something useful and then it has to listen for the next message. In a shell script you can do this with a "while read LINE; do ...; done". Try this: ---8<--- #!/bin/bash while read LINE; do echo $(date) $LINE >> /tmp/schrott done ---8<--- Execute this script on the command line, enter some random stuff and look into /tmp/schrott: user@box:~> ./test-script bla bla bla bla Terminate this script by pressing CTRL-D. You should see something like this in /tmp/schrott: Thu Mar 20 15:13:16 CET 2008 bla bla bla Thu Mar 20 15:13:18 CET 2008 bla If not your script doesn't work :-) See for typos. If this works, put it in your syslog-ng conf. It should do the same (only prio 1 messages should be visible with the date prepended). If this works well, try this script first on the command line: ---8<--- #!/bin/bash while read LINE; do mail -s "High Priority Snort Alert" Sub-Zero(a)xxx.de <<-EOF Alert, Priority 1 $LINE EOF done ---8<--- You should receive for every given input line exactly one mail. If this works put it in your syslog-ng.conf. Now you should be done. The thread you mentioned is about two running scripts where only one should run. This shouldn't be the case here. But you can have a look at the output of "ps fax". If you see multiple processes under syslog-ng then you might have the problem. bye Chris > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu > [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of Adam Richter > Sent: Thursday, March 20, 2008 2:46 PM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] loop caused by syslog-ng filter > > > Hi! > > Not working! Syslog-ng filters for exactly the string: [Priority: 1] > and not as it is piped by the mail script: #priority 1# note the ":" ! > Anyway I used your script --> same fault!!! > I have also used following script: > > > #!/bin/sh > > echo AAA >> /tmp/schrott > date >> /tmp/schrott > > There is no output like [Priority: 1]!!! > Then I did following: tail -f /tmp/schrott and got a loop > too! The loop starts when syslog-ng recognices the first > machting string [Priority: 1] and loops till I stop syslog-ng!!! > > > Sensor1:~# /etc/init.d/syslog-ng stop > Stopping system logging: syslog-ng. > Sensor1:~# tail -f /tmp/schrott > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > AAA > Do 20. MÃ¤r 14:29:12 CET 2008 > > > I think it has something in common with this thread: > > https://lists.balabit.hu/pipermail/syslog-ng/2006-October/009454.html > > Any other ideas? It´s very important! > > German: Es handelt sich hier um meine Abschlussprüfung, und > dies ist der letzte Fehler der Auftritt, ansonsten läuft das Projekt. > > bye, Adam / Sub-Zero ! > > > -------- Original-Nachricht -------- > > Datum: Thu, 20 Mar 2008 10:32:31 +0100 > > Von: "JUNG, Christian" <christian.jung(a)saarstahl.com> > > An: "Syslog-ng users\' and developers\' mailing list" > <syslog-ng(a)lists.balabit.hu> > > Betreff: Re: [syslog-ng] loop caused by syslog-ng filter > > > Hi Adam, > > > > syslog-ng does the right thing :-). > > > > It starts the program/script once and pipes on STDIN every > log-message > > which matches the filter. > > > > If your script is started, it will call mail and pipe > "Alert, priority 1" > > to its STDIN and then exits. syslog-ng sees this and > restarts it (version > > 2.0 or higher behave that way, see > > > <http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ > ch09s02.html#reference_destination_program>). > > > > For you purpose this would be better: > > > > ---8<--- > > #!/bin/bash > > > > while read LINE; do > > cat <<-EOF | mail -s "High Priority Snort Alert" > Sub-Zero(a)xxx.de > > Alert, Priority 1 > > $LINE > > EOF > > done > > ---8<--- > > > > > > bye > > Chris > > > > > > > > -----Original Message----- > > > From: syslog-ng-bounces(a)lists.balabit.hu > > > [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of > Adam Richter > > > Sent: Thursday, March 20, 2008 9:23 AM > > > To: syslog-ng(a)lists.balabit.hu > > > Subject: [syslog-ng] loop caused by syslog-ng filter > > > > > > > > > Hi! > > > > > > First off, sorry for my poor english! > > > I have a problem with a loop caused by syslog-ng v. 2.0.8. I > > > have set up Snort as an IDS System. Snort writes its messages > > > in unified-format to /var/log/snort/snort.alert and > > > /var/log/snort/snort.log. There are two Barnyard processes > > > which read the unified files and convert it to messages that > > > syslog and MySQL understand. Syslog-ng writes the messages > > > to /var/log/auth.log. All this is working fine. Now, I want > > > to set up a filter for Priority 1 alerts. This alert should > > > be send to the Administrator. > > > > > > I used following filter for syslog-ng: > > > > > > source src {unix-stream("/dev/log"); internal();}; > > > destination email{program("/usr/local/bin/alert_mail.sh");}; > > > filter high {match("[Priority: 1]");}; > > > log {source(src);filter(high); destination(email);}; > > > > > > > > > The alert_mail.sh: > > > > > > #!/bin/sh > > > cat << EOF | mail -s "High Priority Snort Alert" Sub-Zero(a)xxx.de > > > Alert, Priority 1 > > > EOF > > > > > > > > > Then I use Nessus to cause some alerts with Priority 1. I can > > > see 4 alerts with the Priority 1 with BASE and in > /var/log/auth.log. > > > > > > Syslog-ng recognises the alert with Priority 1 and activates > > > the script /usr/local/bin/alert_mail.sh > > > > > > All this is working, but the script is restarted by syslog-ng > > > again an again. > > > > > > Extract from /var/log/messages: > > > > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Closing log writer > > > fd; fd='11' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='0' > > > Mar 18 15:46:16 Sensor1 syslog-ng[5191]: Starting destination > > > program; cmdline='/usr/local/bin/alert_mail.sh ' > > > … > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='256' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Starting > > > destination program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; cmdline='/usr/local/bin/alert_mail.sh ', > > > status='256' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Starting > > > destination program; cmdline='/usr/local/bin/alert_mail.sh ' > > > Mar 18 15:42:00 Sensor1 syslog-ng[17354]: Child program > > > exited, restarting; > > > > > > … > > > > > > > > > I get thousands of mails per minute till I stop syslog-ng. > > > > > > Output of /var/log/auth.log(so y see that syslog-ng writes > > > snort/barnyard messages correctly to auth.log): > > > > > > Mar 19 13:56:54 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:4758 -> 172.28.100.10:137 > > > Mar 19 13:57:13 src@Sensor1 barnyard: [1:1446:8] SMTP vrfy > > > root [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:1085 -> 172.28.100.10:25 > > > Mar 19 13:57:13 src@Sensor1 barnyard: [1:660:11] SMTP expn > > > root [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:1085 -> 172.28.100.10:25 > > > Mar 19 13:57:22 src@Sensor1 barnyard: [1:12626:2] Snort Alert > > > [1:12626:0] [Classification: Decode of an RPC Query] > > > [Priority: 2] {UDP} 172.25.1.152:1146 -> 172.28.100.10:111 > > > Mar 19 13:57:22 src@Sensor1 barnyard: [1:585:9] RPC portmap > > > sadmind request UDP [Classification: Decode of an RPC Query] > > > [Priority: 2] {UDP} 172.25.1.152:1146 -> 172.28.100.10:111 > > > Mar 19 13:57:24 src@Sensor1 barnyard: [1:566:6] POLICY > > > PCAnywhere server response [Classification: Misc activity] > > > [Priority: 3] {UDP} 172.25.1.152:1155 -> 172.28.100.10:5632 > > > Mar 19 15:11:27 src@Sensor1 barnyard: [122:1:0] portscan: TCP > > > Portscan [Classification: Unknown] [Priority: 3] {PROTO255} > > > 172.25.1.152 -> 172.28.100.10 > > > Mar 19 15:11:58 src@Sensor1 barnyard: [1:1420:13] SNMP trap > > > tcp [Classification: Attempted Information Leak] [Priority: > > > 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:162 > > > Mar 19 15:11:58 src@Sensor1 barnyard: [1:1418:13] SNMP > > > request tcp [Classification: Attempted Information Leak] > > > [Priority: 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:161 > > > Mar 19 15:12:05 src@Sensor1 barnyard: [1:1421:13] SNMP > > > AgentX/tcp request [Classification: Attempted Information > > > Leak] [Priority: 2] {TCP} 172.25.1.152:4482 -> 172.28.100.10:705 > > > Mar 19 15:12:16 src@Sensor1 barnyard: [122:1:0] portscan: TCP > > > Portscan [Classification: Unknown] [Priority: 3] {PROTO255} > > > 172.25.1.152 -> 172.28.100.10 > > > Mar 19 15:12:19 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > Mar 19 15:12:20 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > Mar 19 15:12:22 src@Sensor1 barnyard: [1:1394:8] SHELLCODE > > > x86 NOOP [Classification: Executable code was detected] > > > [Priority: 1] {UDP} 172.25.1.152:137 -> 172.28.100.10:137 > > > > > > > > > I think it has something in common with this topic: > > > > > > https://lists.balabit.hu/pipermail/syslog-ng/2006-October/009454.html > > > > Thanks in advance! > > > > Sub-Zero > > > > > > -- > > GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! > > Jetzt dabei sein: http://www.shortview.de/?mc=sv_ext_mf@gmx > > ______________________________________________________________ > > ________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > -- Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games! http://games.entertainment.gmx.net/de/entertainment/games/free ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

4 10

Re: [syslog-ng] Filtering duplicate messages
by chris packham 09 Jun '08

09 Jun '08

Updated patch - back to using suppress, fix potential segfault. Implement "Last message repeated N times" functionality. This behaviour can be enabled by adding the new "suppress(<num>)" option to an output configuration e.g. destination tologfile { file("/var/log/messages" template(t) suppress(30)); }; As a log message is added to the queue it is remembered as the last message seen. When a new message is added to the queue it is checked against the last message. If its contents are the same the message is dropped and a counter incremented. A message summary indicating the value of the last message counter and a snippet of the message will be inserted into the log queue if a new message that differs is seen or if the configurable timeout period expires. --- src/cfg-grammar.y | 3 +- src/cfg-lex.l | 3 +- src/logmsg.c | 19 ++++++++++ src/logmsg.h | 2 +- src/logwriter.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++- src/logwriter.h | 5 +++ src/messages.c | 4 +-- 7 files changed, 130 insertions(+), 7 deletions(-) diff --git a/src/cfg-grammar.y b/src/cfg-grammar.y index 39ff172..734daf1 100644 --- a/src/cfg-grammar.y +++ b/src/cfg-grammar.y @@ -62,7 +62,7 @@ gint last_addr_family = AF_INET; %token KW_USER KW_DOOR KW_SUN_STREAMS KW_PROGRAM /* option items */ -%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE +%token KW_FSYNC KW_MARK_FREQ KW_STATS_FREQ KW_FLUSH_LINES KW_SUPPRESS KW_FLUSH_TIMEOUT KW_LOG_MSG_SIZE KW_FILE_TEMPLATE KW_PROTO_TEMPLATE %token KW_CHAIN_HOSTNAMES KW_NORMALIZE_HOSTNAMES KW_KEEP_HOSTNAME KW_CHECK_HOSTNAME KW_BAD_HOSTNAME %token KW_KEEP_TIMESTAMP @@ -671,6 +671,7 @@ dest_writer_option : KW_FLAGS '(' dest_writer_options_flags ')' { last_writer_options->options = $3; } | KW_LOG_FIFO_SIZE '(' NUMBER ')' { last_writer_options->fifo_size = $3; } | KW_FLUSH_LINES '(' NUMBER ')' { last_writer_options->flush_lines = $3; } + | KW_SUPPRESS '(' NUMBER ')' { last_writer_options->suppress= $3; } | KW_FLUSH_TIMEOUT '(' NUMBER ')' { last_writer_options->flush_timeout = $3; } | KW_TEMPLATE '(' string ')' { last_writer_options->template = cfg_lookup_template(configuration, $3); diff --git a/src/cfg-lex.l b/src/cfg-lex.l index 5c5f54f..0190278 100644 --- a/src/cfg-lex.l +++ b/src/cfg-lex.l @@ -6,7 +6,7 @@ /*************************************************************************** * - * Copyright (c) 1999 Balzs Scheidler + * Copyright (c) 1999 Balázs Scheidler * Copyright (c) 1999-2007 BalaBit IT Ltd. * * This program is free software; you can redistribute it and/or modify @@ -86,6 +86,7 @@ static struct keyword keywords[] = { { "flush_timeout", KW_FLUSH_TIMEOUT }, { "sync_freq", KW_FLUSH_LINES }, /* obsolete */ { "sync", KW_FLUSH_LINES }, /* obsolete */ + { "suppress", KW_SUPPRESS }, { "fsync", KW_FSYNC }, { "long_hostnames", KW_CHAIN_HOSTNAMES }, { "chain_hostnames", KW_CHAIN_HOSTNAMES }, diff --git a/src/logmsg.c b/src/logmsg.c index 05354b5..9219a01 100644 --- a/src/logmsg.c +++ b/src/logmsg.c @@ -653,6 +653,25 @@ log_msg_new(gchar *msg, gint length, GSockAddr *saddr, guint flags, regex_t *bad } /** + * log_msg_new_internal: + * @prio: message priority (LOG_*) + * @msg: message text + * @flags: parse flags (LP_*) + * + * This function creates a new log message for messages originating + * internally to syslog-ng + **/ +LogMessage * +log_msg_new_internal(gint prio, gchar *msg, guint flags) +{ + gchar *buf = g_strdup_printf("<%d> syslog-ng[%d]: %s\n", prio, getpid(), msg); + LogMessage *self = log_msg_new(buf, strlen(buf), NULL, flags, NULL); + g_free(buf); + + return self; +} + +/** * log_msg_new_mark: * * This function returns a new MARK message. MARK messages have the LF_MARK diff --git a/src/logmsg.h b/src/logmsg.h index 420a711..d780282 100644 --- a/src/logmsg.h +++ b/src/logmsg.h @@ -100,7 +100,7 @@ void log_msg_unref(LogMessage *m); LogMessage *log_msg_new(gchar *msg, gint length, GSockAddr *saddr, guint flags, regex_t *bad_hostname); LogMessage *log_msg_new_mark(void); - +LogMessage *log_msg_new_internal(gint prio, gchar *msg, guint flags); void log_msg_ack_block_inc(LogMessage *m); void log_msg_ack_block_start(LogMessage *m, LMAckFunc func, gpointer user_data); void log_msg_ack_block_end(LogMessage *m); diff --git a/src/logwriter.c b/src/logwriter.c index bb82b43..f831d93 100644 --- a/src/logwriter.c +++ b/src/logwriter.c @@ -179,10 +179,104 @@ log_writer_watch_new(LogWriter *writer, FDWrite *fd) } static void +log_writer_last_msg_release(LogWriter *self) +{ + if(self->last_msg_timerid) + g_source_remove(self->last_msg_timerid); + + if(self->last_msg) + log_msg_unref(self->last_msg); + + self->last_msg = NULL; + self->last_msg_count = 0; + self->last_msg_timerid = 0; +} + +static void +log_writer_last_msg_flush(LogWriter *self) +{ + gchar *msg; + LogMessage *m; + + msg = g_strdup_printf("Last message '%.20s' repeated %d times\n", + self->last_msg->msg.str, + self->last_msg_count); + + m = log_msg_new_internal(self->last_msg->pri, msg, 0); + + g_string_assign(&m->host,self->last_msg->host.str ); + g_string_assign(&m->host_from,self->last_msg->host_from.str ); + g_queue_push_tail(self->queue, m); + g_queue_push_tail(self->queue, GUINT_TO_POINTER(0x80000000)); + g_free(msg); + + log_writer_last_msg_release(self); +} + +static gboolean +last_msg_timer(gpointer pt) +{ + LogWriter *self = (LogWriter *)pt; + + if(self->last_msg_count) + log_writer_last_msg_flush(self); + else + log_writer_last_msg_release(self); + + return FALSE; +} + +static void +log_writer_last_msg_record(LogWriter *self, LogMessage *lm) +{ + if(self->last_msg) + log_msg_unref(self->last_msg); + + log_msg_ref(lm); + self->last_msg = lm; + self->last_msg_count = 0; + self->last_msg_timerid = g_timeout_add(self->options->suppress * 1000, last_msg_timer, self); +} + +static gboolean +log_writer_last_msg_check(LogWriter *self, LogMessage *lm, gint path_flags) +{ + if (self->options->suppress <= 0) + return FALSE; + + if(self->last_msg) + { + if(strcmp(self->last_msg->msg.str, lm->msg.str) == 0 && + strcmp(self->last_msg->host.str, lm->host.str) == 0) + { + if (self->dropped_messages) + (*self->dropped_messages)++; + msg_debug("Dropping duplicate message", + NULL); + self->last_msg_count++; + log_msg_drop(lm, path_flags); + return TRUE; + } + + if (self->last_msg_count) + log_writer_last_msg_flush(self); + else + log_writer_last_msg_release(self); + } + + log_writer_last_msg_record(self, lm); + + return FALSE; +} + +static void log_writer_queue(LogPipe *s, LogMessage *lm, gint path_flags) { LogWriter *self = (LogWriter *) s; - + + if(log_writer_last_msg_check(self, lm, path_flags)) + return; + if ((self->queue->length / 2) == self->options->fifo_size) { /* drop incoming message, we must ack here, otherwise the sender might @@ -415,6 +509,7 @@ log_writer_free(LogPipe *s) log_msg_unref(lm); } g_queue_free(self->queue); + log_writer_last_msg_release(self); g_free(self); } @@ -462,6 +557,9 @@ log_writer_new(guint32 flags, LogPipe *control, LogWriterOptions *options) self->queue = g_queue_new(); self->flags = flags; self->control = control; + self->last_msg = NULL; + self->last_msg_count = 0; + self->last_msg_timerid = 0; return &self->super; } @@ -476,6 +574,7 @@ log_writer_options_defaults(LogWriterOptions *options) options->ts_format = -1; options->zone_offset = -1; options->frac_digits = -1; + options->suppress = 0; } void diff --git a/src/logwriter.h b/src/logwriter.h index 284899b..cf6074b 100644 --- a/src/logwriter.h +++ b/src/logwriter.h @@ -66,6 +66,7 @@ typedef struct _LogWriterOptions gshort ts_format; glong zone_offset; gshort frac_digits; + gint suppress; } LogWriterOptions; typedef struct _LogWriter @@ -79,6 +80,10 @@ typedef struct _LogWriter gint partial_pos; LogPipe *control; LogWriterOptions *options; + LogMessage *last_msg; + guint32 last_msg_count; + guint last_msg_timerid; + } LogWriter; void log_writer_set_options(LogWriter *self, LogPipe *control, LogWriterOptions *options); diff --git a/src/messages.c b/src/messages.c index 6c69f48..206206a 100644 --- a/src/messages.c +++ b/src/messages.c @@ -56,10 +56,8 @@ msg_send_internal_message(int prio, const char *msg) if (G_LIKELY(internal_msg_queue)) { - buf = g_strdup_printf("<%d> syslog-ng[%d]: %s\n", prio, getpid(), msg); - m = log_msg_new(buf, strlen(buf), NULL, LP_INTERNAL | LP_LOCAL, NULL); + m = log_msg_new_internal(prio, msg, LP_INTERNAL | LP_LOCAL); g_queue_push_tail(internal_msg_queue, m); - g_free(buf); } } } -- 1.5.4.5

2 1

syslog_ng and NFS
by Chris Wagner 09 Jun '08

09 Jun '08

We run syslog-ng (both 1.x and now 2.0.9) on FreeBSD writing log files to a NetApp NFS server. At a very low load we start dropping inbound packets (udp, large socket buffers). A quick tcpdump showed a pathological interaction with NFS revolving around how NFS (at least on BSD) deals with APPEND mode. The upshot is that for each write, it flushes the attributes and actually re-reads the partial file block back from the server! A simple change to the source to remove O_APPEND had great results we can handle 10x the number of inbound messages. Our .conf file has every logging host going to a separate file, and with a quick scan of the code it sure looked like syslog-ng won¹t open the same file more than once anyway (I could be wrong here). So, a couple questions (I scanned the last few months of archives but didn¹t see any discussion about this) - what breaks if anything by turning off APPEND, is this interesting to others/how might I get this into future releases ( I can see this being a per-destination option). thanks. -- It is the mark of an instructed mind to rest satisfied with the degree of precision which the nature of a subject admits, and not to seek exactness when only an approximation of the truth is possible. Aristotle

2 3

Flow-control behaviour and Cached lines timestamping, plus a suspend error in 2.1beta1
by Behal, Pavel 05 Jun '08

05 Jun '08

Dear Bazsi, I have found two "features" and one bug I would like to discuss a little: 1) We use the syslog-ng 2.1alpha1 as a root system syslog. The problem is, we have had two destinations setup. First destination was the traditional write of system messages to the file like "/var/log/messages" etc. The second destination path was to the TCP stream to the central syslog server. And the problem is, we have used the flow-control flag on the central TCP path. But, when the central TCP syslog-ng becomes unavailable, after all the buffers filled up, the local system syslog-ng stopped processing any system messages. It did not write anything through the local file destinations until the TCP communication was re-established. Is this a correct behaviour? We have found, that to allow smooth processing of local system sources, we can not use flow-control flag on any destination paths it used.. 2) When messages are in syslog-ng buffer, they do not contain any timestamp. We have realized, that the timestamp is added at the time the syslog-ng flushes all the buffers to the disk. This is not very good in case you need to debug the messages ages and all of them have same and non-related timestamp. Is it possible to modify the logic of syslog-ng to timestamp the messages as soon as possible - even before the buffer, at the collection time? We have found it important for log integrity reasons. 3) Directly in the 2.1beta1 there has been introduced some error with the "suspend writing a destination file when an I/O error ..." patch. During our tests we have found it continously thinks the destination is unavailable and writes strange messages into to the internal log: May 29 15:00:02 europa syslog-ng[16798]: I/O error occurred while writing; fd='7', error='Connection timed out (110)' May 29 15:00:02 europa syslog-ng[16798]: Connection broken; time_reopen='60' May 29 15:00:02 europa syslog-ng[16798]: Suspending write operation because of an I/O error; fd='7', time_reopen='60' the lsof -p 16798: COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME syslog-ng 16798 root cwd DIR 253,0 4096 2 / syslog-ng 16798 root rtd DIR 253,0 4096 2 / syslog-ng 16798 root txt REG 253,0 114584 671841 /sbin/syslog-ng syslog-ng 16798 root mem REG 253,0 106397 820372 /lib/ld-2.3.4.so syslog-ng 16798 root mem REG 253,0 1454546 820373 /lib/tls/libc-2.3.4.so syslog-ng 16798 root mem REG 253,0 28504 2919694 /usr/lib/libwrap.so.0.7.6 syslog-ng 16798 root mem REG 253,0 93985 820376 /lib/tls/libpthread-2.3.4.so syslog-ng 16798 root mem REG 253,0 47671 820386 /lib/tls/librt-2.3.4.so syslog-ng 16798 root mem REG 253,0 505200 2930064 /usr/lib/libglib-2.0.so.0.400.7 syslog-ng 16798 root mem REG 253,0 95148 820384 /lib/libnsl-2.3.4.so syslog-ng 16798 root mem REG 253,0 36639 2920956 /usr/local/lib/libevtlog.so.0.0.0 syslog-ng 16798 root 0r CHR 1,3 1619 /dev/null syslog-ng 16798 root 1w CHR 1,3 1619 /dev/null syslog-ng 16798 root 2w CHR 1,3 1619 /dev/null syslog-ng 16798 root 3r REG 0,3 0 4026531850 /proc/kmsg syslog-ng 16798 root 4u unix 0xc1be7ba0 351802 /dev/log syslog-ng 16798 root 5u IPv4 351804 TCP europa.training.hp.com:37205->europa.training.hp.com:5140 (ESTABLISHED) syslog-ng 16798 root 6w REG 253,0 4802 1655155 /var/log/messages syslog-ng 16798 root 8w REG 253,0 525 1655159 /var/log/boot.log This happens during standard system syslog processing on RH ELinux 4 and Solaris 9. You have the syslog-ng confs attached. Could you pleae check it? Best Regards, Pavel

3 3

syslog-ng to trigger rancid
by Mladen Tsvetkov 04 Jun '08

04 Jun '08

Hi, Guys, I am new to syslog-ng. I have successfully installed it and integrated with mysql and php. My question is how can I make syslog-ng to start rancid when a syslog configuration message is received from a cisco device: 112578: 1564643: May 15 09:05:19.141 EEST: %SYS-5-CONFIG_I: Configured from console by ******** on vty0 (10.10.10.10). So far I know that I have to use the program() directive. But how can I pass the hostname and other parameters to rancid? Thanks in advance, Longman

3 2