- syslog-ng - lists.balabit.hu

Message aggregation: or, 'Last message repeated umpteen times'.
by G.W. Haywood 28 Aug '07

28 Aug '07

Hi there, On Mon, 15 Jul 2002 12:33:17 +0200 Balazs Scheidler wrote: > On Tue, Jul 09, 2002 at 07:02:19PM -0400, Kevin Welch wrote: > > I know this issue has been discussed before, but I never saw an answer other > > than 'syslog-ng doesnt do that'. > > > > Perhaps someone could provide insight as to why syslog-ng doesnt aggregate > > messages received? Is this on a feature list for future releases? > > It might be implemented once I have time, though IMHO message aggregation > makes log analysis more difficult, especially if you have multiple cascaded > log relays. Balazs, I read the manpage, Googled and grepped the source and I believe I know the answer, but I'd like to have your confirmation. Is it still the case that syslog-ng does no message aggregation? (I'm not asking because I want it. I'm asking because I _don't_ want it. :) -- 73, Ged. PS: I also found this about not aggregating messages if the output is a pipe http://lists.hellug.gr/pipermail/linux-greek-users/2007-April/068169.html

2 1

File reopen
by Szeti, Balazs 28 Aug '07

28 Aug '07

Hi! Syslog-ng reopens destination files, if I send a SIGHUP to the process, which is a common way of handling logrotates in Linux. If I don't write in to a file (idle), syslog-ng will close the file according to time_reap() value. If the file was deleted from the directory, will it create the file next time it logs to the deleted destination file? I've found, that it doesn't, can I config it somehow? (My log files are read by different users, and sometimes they delete it, so it should be recreated again.) Is it a soultion, if I send a SIGHUP to syslog-ng every minute? I was looking for a solution in the archive, but I haven't found a clear answer. Thanks, Balazs Szeti

2 1

syslog-ng unable to starts
by mailing list 26 Aug '07

26 Aug '07

syslog-ng 2.0.4, Solaris 10, attempting to replace syslogd on Solaris. I stopped syslogd before starting syslog-ng on my system and I keep getting the below error messages in /var/adm/messages: (ps -ef shows syslog-ng is running) Aug 23 14:39:52 server1 syslog-ng[7764]: Connection failed; error='Connection refused (146)' Aug 23 14:39:52 server1 syslog-ng[7764]: Initiating connection failed, reconnecting; time_reopen='10' I've edited /etc/inet/services to include port 5140 and stop/start inetd via Solaris smf (svcadm). Was there something else I need to make the port open or perhaps something in syslog-ng.conf? Additionally, I used the config file from http://www.campin.net/syslog-ng/solaris-conf.txt and only make changes to the "destination loghost" is there anything specific to Solaris 10 that I need to add/remove in the configuration file? Thank you for any pointers. Regards, -- Mike ____________________________________________________________________________________ Fussy? Opinionated? Impossible to please? Perfect. Join Yahoo!'s user panel and lay it on us. http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7

2 3

Revisiting: problem with filter behavior in 2.0.4
by aero1967＠fastmail.fm 23 Aug '07

23 Aug '07

Hello. I posted the below message on 13 July to the list, but still received no response. I'm wondering if I'm seeing a bug in syslog-ng, or there's some configuration detail I've overlooked. Balazs, could you comment on this please? Since I posted the original message, I'm seeing all kinds of messages in the "critical.log" file that shouldn't belong, according to my understanding of the filtering, e.g. messages from facility+priority combination "user.info" without the string "crit" anywhere within the message (though oddly enough in that example, the word "subscription" is present in the message, which differs from "crit" by the addition of the "p" before the "t", but I'm not sure if that's anything more than coincidence). Anyway, I'd appreciate it if someone could help me look into this. -- aero1967(a)fastmail.fm -- http://www.fastmail.fm - A no graphics, no pop-ups email service ----- Original message ----- I'm seeing some results with filtering that I don't expect / understand. I have the following statements in syslog-ng.conf (I am just giving the relevant configuration lines): source src { unix-stream("/dev/log"); internal(); file("/proc/kmsg"); }; filter kern { facility(kern); }; filter kern_critical { filter(kern) and priority(emerg .. err); }; filter critical { filter(kern_critical) or match("crit*") or priority(emerg .. crit); }; template verbose_msg_fmt { template("$FULLDATE $FACILITY.$PRIORITY $MSG\n"); template_escape(no); }; destination root_tty { usertty("root"); }; destination critical_logfile { file("/var/log/critical.log" template(verbose_msg_fmt)); }; log { source(src); filter(critical); destination(root_tty); destination(critical_logfile); }; There are no other configuration lines which reference the critical_logfile destination , and no destination has a "final" flag. Now, when I check /var/log/critical.log, I get messages from the facility.priority combination of daemon.warning that do not contain any string starting with "crit". I also get messages from daemon.err, which also shouldn't match the filter. For example: 2007 Jul 12 10:51:07 daemon.warning <message, not containing the string crit anywhere> 2007 Jul 11 20:33:57 daemon.err <message, not containing the string crit anywhere> IIRC, I was using these same lines in a syslog-ng 1.6.x installation without these kinds of results.

2 1

cisco tcp syslog weirdness/merkwuerdigkeit
by Blurry 22 Aug '07

22 Aug '07

When a certain cisco router is set to UPD syslog delivery it creates a unique message for each unique message, duh ! But when this same router is set to TCP syslog, it takes all messages and tacks them back to back until syslog-ng runs out of buffer space in one line. I kept increasing the message log size, but the real problem is that the messages would have to be parsed out of this massive long line. My router guy says he can't make the cisco router behave any differently. How do I handle this problem ? Ideas ? Thanks

3 7

Custom Syslog-ng logs and remote syslog-ng
by Corey Bobb 22 Aug '07

22 Aug '07

Greetings, I am trying to accomplish something and am requiring some assistance. I have custom logs that write needed output to this directory: /production/server/20070822/xxxx.log This log file rotates every day so the next date will be 20070823 and so on. How can I direct a custom destination file to log remotely that log file to the central server and have it pick it up everyday? So for example . . . I have a central syslog server that writes all its input to /remote/logs/<date>/<HOST> On the server sending the remote logs I have this parameter: destination { file("/production/server/$YYYY$MM$DD/xxxx.log"); }; For some reason I think its not working properly due to the $DATE variables, I tried also using % to no avail. All the necessary logs do log remotely as I wish them to. Is this the proper way to configure this scenario? What I want to do is log the custom logging @ xxxx.log to the correct server $hostname on the central server This would be the idea: Custom.log -> local syslog-ng -> central-syslog-ng -> /remote/logs/<date>/<HOST> I am currently logging all the servers logging locally and remotely, but I cannot get syslog-ng to log that custome log file to the remote server. I hope this makes sense! Thanks. Regards, Corey M. Bobb Data Center Manager 850 South Concourse Suite 150 Maitland, FL 32751 D: 321.445-2150 M: 321-262-2845 F: 407.333.7701 cbobb(a)cygnus.com www.cygnus.com

1 0

Multiple woes with sles10sp1 / syslog-ng-1.6.8-20.18
by Robin P. Blanchard 22 Aug '07

22 Aug '07

# syslog-ng -d -f /etc/syslog-ng/syslog-ng.conf [snip] Cannot open file /export/syslog/10.10.0.2102007/08/21/messages for writing (Permission denied) # ls -ald /export/ ; ls -ald /export/syslog/ drwxr-xr-x 5 root root 4096 Aug 21 10:16 /export/ drwxrwsr-x 2 root root 4096 Aug 21 09:41 /export/syslog/ # fgrep "\$HOST" /etc/syslog-ng/syslog-ng.conf |fgrep messages destination localmessages { file("/export/syslog/$HOST/$YEAR/$MONTH/$DAY/localmessages"); }; destination messages { file("/export/syslog/$HOST/$YEAR/$MONTH/$DAY/messages"); }; #destination allmessages { file("/export/syslog/$HOST/$YEAR/$MONTH/$DAY/allmessages"); }; So...it would apppear that: 1) macro expansion is not working ? (I can hand hack-in an extra "/" to force that path to look correct) 2) syslog-ng cannot write to its output dir (regardless if I hack in that extra "/" Ideas ? # uname -a Linux auth 2.6.16.46-0.14-default #1 Thu May 17 14:00:09 UTC 2007 i686 i686 i386 GNU/Linux # rpm -qa |fgrep -i syslog syslog-ng-1.6.8-20.18 --------------------------------------- Robin P. Blanchard Systems Integration Specialist The University of Georgia Center for Continuing Education Conference Center & Hotel fon: 706.542.2404 < > fax: 706.542.1587 ---------------------------------------

3 3

REGEX rewrites on packet body possible?
by Eli Stair 21 Aug '07

21 Aug '07

I've got a problem with some network devices that is leading me to need to find some way to do regex rewriting of portions of the message body of syslog messages. Problem: I've got a bunch of Foundry devices that put their hostname followed by a comma in the body of the message (and some that do not). Some of these look like this: { 2007-08-16:2007-08-16T09:50:16-07:00 hostname [hostname.local7.notice] hostname, Linecard Module 13 temperature 50.0 C degrees is normal } # template("$R_ISODATE $HOST [$PROGRAM.$FACILITY.$PRIORITY] $MSG\n") As you can see from the template, the second hostname reported with the comma is part of the MSG body. For reasons of properly searching/indexing this data I need to strip this out. I've seen mention of a tool called 'syslog-mailer' that sounds like it would do the job somewhat. Additionally, I've seen blog chatter about potentially adding full regex rewrite capability to syslog-ng in the recent past. I can't find evidence of either of these methods however. My first try at solving this using an external program showed that when passing data OUT of syslog-ng to a defined program, only the message body is sent and before application of a template, the other information is dropped. Thus it's not possible to do processing of the whole payload externally, re-import the data via a socket and finish writing because the facility and HOST information is all gone! I'm looking at writing a log proxy using Net::Dev::Tools::Syslog in perl to handle listening, rewriting if necessary, and forwarding full messages on to syslog-ng after. I'd just like to know if there are any better suggestions, or if this has been done before successfully in another way? Cheers, and thanks for any insight. /eli

2 2

syslog-ng feature request
by Andreas Kurz 17 Aug '07

17 Aug '07

Hello, I am currently integrating syslog-ng as an HA cluster resource, because the remote syslog logging (MySQL error logfile on shared filesystem) for one resource should be moved around the cluster with the resource. I encounterd one syslog-ng feature that would be nice to have: A configureable path for the persistent state file (like the config file or pid file) to allow several instances of syslog-ng per node and to place a persistant state file also on a shared file system. Regards, Andreas

1 0

syslog-ng 2.0.5 does not remember last sent log line position for file() source
by Andreas Kurz 17 Aug '07

17 Aug '07

Hello all, I've installed syslog-ng 2.0.5 (compiled from src) on a CentOS 4.5 server and wanted to include a MySQL logfile via a file() source directive: ... source s_db_com { file("/var/lib/mysql/community/community.err" log_prefix("mysql-community: ") follow_freq(1) flags(no-parse)); }; filter f_db_com { match("mysql-community"); }; log { source(s_db_com); filter(f_db_com); destination(ha_local6); destination(ha_tcp); flags(final); }; ... But when I restart syslog-ng the complete MySQL error file is read and logged again by syslog-ng. Is there a special option that has to be set to get this feature working? Where does syslog-ng store its persistant data to get the information after a restart? Regards, Andreas

2 2