August 2008 - syslog-ng - lists.balabit.hu

Charset problem
by Kostyantyn Gushtin 18 Aug '08

18 Aug '08

Hello, there is an encoding problem if using syslog-ng with windows agent. Logs from Linux hosts are written to file as UTF-8, while the logs from windows agent are written to file as cp1252. The problem is visible if you have a message with chars out of ASCII set, like Ö. As a result we have a file that is both unreadable and useless for automation. Is there any way to fix this? Or do you plan to fix this? >From my point of view syslog-ng should write file using one charset. Preferably configurable. -- Kostyantyn Gushtin Software engineer N-iX LLC Solutions for Your success http://www.n-ix.com

2 1

Re: [syslog-ng] problem with logger -t and $TAG
by JUNG, Christian 14 Aug '08

14 Aug '08

The TAG macro is the facility/priority information in a syslog message (logger option -p) not the tag you define with logger (-t). Try the PROGRAM (should be the string you've defined with -t) and MSGONLY (rest of the message) macros. bye Chris > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu > [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of Nilshar > Sent: Thursday, August 14, 2008 11:59 AM > To: syslog-ng(a)lists.balabit.hu > Subject: [syslog-ng] problem with logger -t and $TAG > > > Hello everyone. > > Ok first sorry if this question has already been answered in an older > thread, but I found no way to search easily through the archives :/ > > So my problem is : > > I'm telling apache to log access logs to logger like that : > >> TransferLog "| /usr/bin/logger -p local0.info -t apache_access" > > so using the -t to define a custom tag, so I can easily filter it > using match() in filters. > > as a destination in syslog-ng, I want a file with usual apache format, > so using : > >> destination df_apache_access { > >> file("/var/log/apache/access.log" > >> template("$MESSAGE\n") ); > >> }; > > > But in the file I got the apache_access: tag before the $MESSAGE. I > was thinking it would go to $TAG, which seems not be the case. > > It's probably more a problem with logger... but wondering if anyone is > able to point me a way to fix that issue.. maybe using something else > than logger ? > > Thanks. > Nilshar. > ______________________________________________________________ > ________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

2 1

problem with logger -t and $TAG
by Nilshar 14 Aug '08

14 Aug '08

Hello everyone. Ok first sorry if this question has already been answered in an older thread, but I found no way to search easily through the archives :/ So my problem is : I'm telling apache to log access logs to logger like that : >> TransferLog "| /usr/bin/logger -p local0.info -t apache_access" so using the -t to define a custom tag, so I can easily filter it using match() in filters. as a destination in syslog-ng, I want a file with usual apache format, so using : >> destination df_apache_access { >> file("/var/log/apache/access.log" >> template("$MESSAGE\n") ); >> }; But in the file I got the apache_access: tag before the $MESSAGE. I was thinking it would go to $TAG, which seems not be the case. It's probably more a problem with logger... but wondering if anyone is able to point me a way to fix that issue.. maybe using something else than logger ? Thanks. Nilshar.

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 39, Issue 20
by Anurag Agarwal 12 Aug '08

12 Aug '08

Hello Bazsi, I did some more observation and I have following more precise inputs 1) Same config file is working correctly in my lab Solaris v240 machine. Filtered syslog sent to destined user only. 2) In production the destined user is receiving logs twice ( 2 copies). It means one copy for itself because it is destination user and 2nd copy is because all users are copied. 3) If you telnet to server and don't enter login/password even, still you start receiving a copy of message. ï»¿ Please help ! Anurag aggarwal -----Original Message----- From: Anurag Agarwal Sent: Thursday, July 24, 2008 4:54 PM To: 'syslog-ng(a)lists.balabit.hu' Subject: RE: syslog-ng Digest, Vol 39, Issue 20 Thanks Bazsi for reply. Relevant portion is below ************************************************************** destination console { usertty("wimaxnoc"); }; filter ps_nt_re { match("PS NOT REACHABLE"); }; log { source(network); filter(ps_nt_re); destination(console); }; ************************************************************** Please help. Anurag aggarwal -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Thursday, July 24, 2008 3:30 PM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 39, Issue 20 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: Re : Re : Re : Syslogd + Syslog-ng (Christopher Cashell) 2. Re: Re : Re : Re : Syslogd + Syslog-ng (Leandro Ferreira da Silva) 3. Re: Using regexp in match() (Balazs Scheidler) 4. Re: Usertty sending messages to all user and without login also (Balazs Scheidler) ---------------------------------------------------------------------- Message: 1 Date: Wed, 23 Jul 2008 10:04:34 -0500 From: Christopher Cashell <ChristopherCashell(a)solutionary.com> Subject: Re: [syslog-ng] Re : Re : Re : Syslogd + Syslog-ng To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <48874882.8070904(a)solutionary.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Leandro Ferreira da Silva did thus speak on 7/23/2008 7:14 AM: > The configuration is the standart, I only do the follow changes. > > # sources > source src { unix-dgram("/var/run/log"); > unix-dgram("/var/run/logpriv" perm(0600)); > udp(); internal(); file("/dev/klog"); }; > > I add this, > source r_src { udp(ip("*client.domain*") port(514)); }; Here's your problem. For src, you define 'udp()' as one of your log sources, without including any specific options for it. This will cause syslog-ng to go with its defaults for udp(), which is bind to *all* IP addresses assigned to the box, on port 514. Then, for r_src, you are telling syslog-ng to bind to a specific IP address on the box and port 514. When syslog-ng attempts to start, it will fail with the error message you received because that ip/port is already in use (by the source src, which is bound to all IP's). Unless you have a need to bind to a specific IP address on the box, I'd recommend removing 'udp();' from src, and replacing the current 'udp(ip("*client.domain*") port(514));' in r_src with 'udp();'. -- Christopher Cashell ------------------------------ Message: 2 Date: Wed, 23 Jul 2008 14:39:03 -0300 From: Leandro Ferreira da Silva <ferreira(a)iqm.unicamp.br> Subject: Re: [syslog-ng] Re : Re : Re : Syslogd + Syslog-ng To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <48876CB7.7080909(a)iqm.unicamp.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Christopher Cashell wrote: > Leandro Ferreira da Silva did thus speak on 7/23/2008 7:14 AM: >> The configuration is the standart, I only do the follow changes. >> >> # sources >> source src { unix-dgram("/var/run/log"); >> unix-dgram("/var/run/logpriv" perm(0600)); >> udp(); internal(); file("/dev/klog"); }; >> >> I add this, >> source r_src { udp(ip("*client.domain*") port(514)); }; > > Here's your problem. For src, you define 'udp()' as one of your log > sources, without including any specific options for it. This will > cause syslog-ng to go with its defaults for udp(), which is bind to > *all* IP addresses assigned to the box, on port 514. > > Then, for r_src, you are telling syslog-ng to bind to a specific IP > address on the box and port 514. When syslog-ng attempts to start, it > will fail with the error message you received because that ip/port is > already in use (by the source src, which is bound to all IP's). > > Unless you have a need to bind to a specific IP address on the box, > I'd recommend removing 'udp();' from src, and replacing the current > 'udp(ip("*client.domain*") port(514));' in r_src with 'udp();'. > It worked!! The real problem was udp () at src. Thank you very much for all the help... Now I'll go complete my rules.. See you!! =P ------------------------------ Message: 3 Date: Thu, 24 Jul 2008 10:16:27 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] Using regexp in match() To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Cc: documentation(a)balabit.com Message-ID: <1216887387.8130.5.camel(a)bzorp.balabit> Content-Type: text/plain; charset=UTF-8 On Tue, 2008-07-22 at 09:45 +0200, Jan Kreps wrote: > > ------------ P?vodn? zpr?va ------------ > > Od: Jan Kreps <krepsj(a)seznam.cz> > > P?edm?t: Re: [syslog-ng] Using regexp in match() > > Datum: 10.7.2008 10:29:35 > > ---------------------------------------- > > > > > > I'm trying to setup central syslog-ng server for my Exchange > > > > servers. On > > > windows servers I use Epilog agent (brother of Snare) forwarding > > > tracking logs to central syslog. > > > Obviously I have used bad regexp. Exchange tracking log uses tabs as delimiters. > > But when I saved $MSG string to text log, tabs was changed to '\011'. > > > > So now I changed my regexp to use tabs as delimiters: > > > > filter f_parsing { > > match("([^\t]*)\t([^\t]*)\t"); > > }; > > > > This works like charm and saves first two tab delimited fields (date > > and time in this case) to $1 and $2. > > > > Some more remarks to subject. > > Syslog-ng Administrator guide 1.1.1 Edition from May 2008 says on page 111: > > "The regular expressions can use up to 255 regexp matches ($1 ... $255)." > > and on page 112: > > "Regarding braces around macro names, the following two formats are equivalent "$MSG" and "${MSG}"." > > Fact is that for regexp matches greater than 9 it MUST be in ${} format. If you use for instance $12 it resolves as content of $1 + character '2'. I found that I have to use ${} syntax in ChangeLog remark for patch 137. I guess that should be stated in Admin Guide more clearly. In this respect, statement "The regular expressions can use up to 255 regexp matches ($1 ... $255)" is not true and should be corrected. > right, this should be fixed in the documentation. > And I have one question about file creation. > > When something is logged a file is created according to destination(). But when I delete the file, it's not created anymore, until restart of syslog-ng. Is that correct/expected behaviour? > > I run Debian Etch and syslog-ng 2.0.9-1 from unstable branch (in > stable is still version 2.0.0) Yes, you need to send the HUP signal to syslog-ng in order to reopen log files. This is the way UNIX works. (the application does not know about the deletion of a logfile) -- Bazsi ------------------------------ Message: 4 Date: Thu, 24 Jul 2008 10:28:06 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] Usertty sending messages to all user and without login also To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1216888086.8130.10.camel(a)bzorp.balabit> Content-Type: text/plain On Wed, 2008-07-23 at 13:39 +0530, Anurag Agarwal wrote: > Hello everybuddy, > > I am using syslog-ng and used usertty in destination. I am using a user > 'test' as destination. But other users can also see the message on their > console e.g root and other users. Please help as this is creating a > problem and NG not working as desired. > OS: Solaris 10 > It does work for me. Can you show us the relevant portions of your configuration file? I tried with this: source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; destination all { file("/var/log/messages"); }; log { source(local); destination(all); }; destination all_users { usertty("bazsi"); }; filter alert_emerg { level(err..emerg); }; log { source(local); filter(alert_emerg); destination(all_users); }; It correctly printed err..emerg messages on my tty, and not on others. The relevant portion of the code is here: #if HAVE_MODERN_UTMP if (ut->ut_type == USER_PROCESS && ((self->username->len == 1 && self->username->str[0] == '*') || (self->username->len <= sizeof(ut->ut_user) && memcmp(self->username->str, ut->ut_user, self->username->len) == 0))) #else if ((self->username->len == 1 && self->username->str[0] == '*') || (self->username->len <= sizeof(ut->ut_name) && memcmp(self->username->str, ut->ut_name, self->username->len) == 0)) #endif I don't see how this condition could match for everyone. Even if the wtmp format is different and ut->ut_name would contain a bogus value, the expected breakage would be to not display anywhere, not to display everywhere. -- Bazsi ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 39, Issue 20 *****************************************

1 0

destination drivers
by hi ro 12 Aug '08

12 Aug '08

Hi To listen on the specified UDP port for incoming messages over IPv6, must I use "udp6()" destination driver? Can I use "udp()" destination driver? Thanks _________________________________________________________________ あなたも総裁候補に？世界に一つだけのオリジナルブログパーツを作ろう！ http://questionbox.jp.msn.com/soudantou/

2 1

individual files
by Ryan Eldridge 11 Aug '08

11 Aug '08

Hello all. After playing around with syslog-ng all afternoon, needless to say I'm a little confused and frustrated. I have managed to get it running on most of my Linux boxes and it to accept data from a couple of them still running syslog which is what I want. The issue I'm facing is the config file I found do to the wonders of Google is splitting the information based on host but not into separate files like I'm use to with syslog aka cron, messages, syslog etc.. what i'm looking for output is the following: ls -shl /var/log/HOSTS/peter.domain.com/2008/08 4.0K -rw------- 1 root root 866 2008-08-10 20:46 auth.log 0 -rw-r--r-- 1 root root 0 2008-08-10 17:22 cron.log 0 -rw-r--r-- 1 root root 0 2008-08-10 17:22 daemon.log 4.0K -rw------- 1 root root 2.8K 2008-08-10 18:26 debug 8.0K -rw-r--r-- 1 root root 6.6K 2008-08-10 14:36 dmesg 0 -rw-r----- 1 root root 0 2002-04-06 19:13 maillog 12K -rw------- 1 root root 8.2K 2008-08-10 20:05 messages 0 -rw-r--r-- 1 root root 0 2008-08-10 18:22 ntpd 0 -rw-r----- 1 root root 0 1994-05-09 03:06 secure 0 -rw-r----- 1 root root 0 2002-03-09 00:29 spooler 12K -rw------- 1 root root 8.6K 2008-08-10 20:05 syslog Currently i'm getting the following: ls -shl /var/log/HOSTS/peter.domain.com/2008/08 12K -rw------- 1 root root 11K 2008-08-10 21:31 10 which contains all the results. I'll post the file I'm using below any help would be useful ############################################################### # First, set some global options. options { use_fqdn(yes); use_dns(yes); dns_cache(yes); keep_hostname(yes); long_hostnames(off); sync(1); log_fifo_size(1024); }; ############################################################### # # Logs may come from unix stream, and UDP:514 # source src { pipe("/proc/kmsg"); unix-stream("/dev/log"); internal(); udp(ip("192.168.2.3") port(514)); tcp(ip("192.168.2.3") port(514) keep-alive(yes)); }; ############################################################### # First some standard logfile # destination authlog { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination user { file("/var/log/user.log"); }; #destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; ########################################## # Here's the filter options. With this rules, we can set which # message go where. filter f_authpriv { facility(auth, authpriv); }; filter f_syslog { not facility(auth, authpriv) and not facility(mail); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; #filter f_mail { facility(mail); }; filter f_user { facility(user); }; #filter f_news { facility(news); }; #filter f_debug { not facility(auth, authpriv, news, mail); }; filter f_messages { level(info .. warn) and not facility(auth, authpriv, cron, daemon, mail, news); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; ############################################################### # # log statements actually send logs somewhere, to a file, across the network, etc # log { source(src); filter(f_authpriv); destination(authlog); }; log { source(src); filter(f_authpriv); destination(std); }; log { source(src); filter(f_syslog); destination(std); }; log { source(src); filter(f_cron); destination(std); }; log { source(src); filter(f_daemon); destination(std); }; log { source(src); filter(f_daemon); destination(std); }; log { source(src); filter(f_kern); destination(std); }; log { source(src); filter(f_user); destination(std); }; #log { source(src); filter(f_debug); destination(std); }; log { source(src); filter(f_messages); destination(std); }; ## set up logging to loghost #destination loghost { # tcp("10.0.0.1" port(514)); #}; # send everything to loghost, too #log { # source(src); # destination(loghost); #}; # # automatic host sorting (usually used on a loghost) # # set it up destination std { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY_$HOST_$YEAR_$MONTH_$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; # log it log { source(src); destination(std); };

2 1

Logrotate signal instead of HUP
by Jonas Eriksson 11 Aug '08

11 Aug '08

Hello The thing that i miss the most in modern syslog daemons is the ability to only release the local files without also releasing sockets for syslog servers. I also saw that this was mentioned earlier on this list[1] but no one seems to have written the patches for this until now. I have however created patches for this, that may be found at [2]. There are however one problem: In order to figure out which LogPipes that should be closed is now done by comparing logdriver->super.init with the affile_dd_init-function pointer. If it's a match, the LogDriver is a AFFileDestDriver. A more elegant solution could be to require some larger changes. My suggestion is to include a gboolean "logrotate_release" in the LogDriver, or similar. I have not done any of this, since this would be best to discuss first. I am also not sure what to do with write_hash in AFFileDestDriver. Should these be reinited as well? If that is the case, i can write the patch required for that as well. I have spent alot of time reading the code and trying to make the patch fit in with the style of the rest of the code. I hope I succeeded at this. I also have a git repos with the changes available at [3] and patches for the Ubuntu syslog-ng packages for 6.06 (dapper) and 7.04 (hardy) if it's of interest to anyone. I have been testing these patches by throwing alot of both USR1 and HUP signals on the running deamon and it seems to behave very well. I'm thinking about rolling them out in production next week. Your opinions are welcome. With regards, Jonas, HPC2N [1] https://lists.balabit.hu/pipermail/syslog-ng/2002-November/004063.html [2] http://www.acc.umu.se/~zqad/syslog-ng-2.0-git.diff [3] /afs/hpc2n.umu.se/home/z/zqad/Public/repos/syslog-ng-2.0/ ------------------------------------- - Jonas Eriksson, zqad(a)hpc2n.umu.se - - sysadmin @ {acc,hpc2n}.umu.se - -------------------------------------

2 1

OpenBSD + Syslog-NG 2.x Users ?
by Brian A. Seklecki 01 Aug '08

01 Aug '08

Please ping me on or off-list. ~BAS

1 1

Stripping numerals in the destination
by Cliff Fogle 01 Aug '08

01 Aug '08

I have several server types, like web001-web100 and thumb001-thumb010 or db001-db004 I want each server type to log to the same file and I can't for the life of me figure out a regex for $HOST in the destination statement to strip numerals. Where the first $HOST is below I want to do something like a sed "s/[0-9]*$//" destination d_host { file("/log1/syslog/$HOST/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY" template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n") template_escape(no) ); }; Thanks for any help you can provide. I've been banging my head against google for hours now...

2 1