- syslog-ng - lists.balabit.hu

how to fetch logs from linux, windows, and devices (firewall, router)
by senthil kumar 21 Aug '08

21 Aug '08

hi pals i am new to syslog-ng. it is a very nice software. i want to used opensource edition of syslog-ng i want to log all linux, windows, sonicwall firewall, watchguard firewall, cisco router, dlink router, DSL router. is it possible to log all above things just using opensource edition of syslog-ng. and i want the detailed notes for "HOW TO INSTALL SYSLOG-NG SERVER IN LINUX" and i want configuration details for "GETTING LOGS FROM LINUX, WINDOWS AND DEVICES" pls help me. Regards N.Senthil Kumar Mobile +91 9787150175 Unlimited freedom, unlimited storage. Get it now, on http://help.yahoo.com/l/in/yahoo/mail/yahoomail/tools/tools-08.html/

1 0

destination program is killed due to broken pipe
by midnight codder 19 Aug '08

19 Aug '08

I am trying to redirect all log messages to a destination program (written in java). This did not do much so I redirected all messages to a single catch-all log file AND also to a bash shell script that runs an endless loop, reads a line and spit it out to a text file. The shell script is killed by syslog-ng after the 5th message. The catch-all file, however, keeps getting all of the messages (that's how I know who is killing who). Here is the config file for syslog-ng: options { ts_format(iso); frac_digits(3); }; template bds_template{ template("<pri=$PRI> <date=$ISODATE> <host=$HOST> <msg=$MSG>\n"); template_escape(no); }; source src{ internal(); unix_stream("/dev/log"); unix-dgram("/dev/log"); file("/proc/kmsg" log_prefix("kernel:")); udp(); tcp(); }; destination test_adapter{ program("bash /usr/local/bds/logger/syslogAdapterTest.sh" template(bds_template)); }; log { source(src); destination(test_adapter); }; destination catchAll{ file("/var/log/syslog-ng-catch-all.log", template(bds_template)); }; log { source(src); destination(catchAll); }; Here is the bash script: #!/bin/bash file="/root/syslogAdapterTest" counter=1 limit=3 while true do read line echo "$line" > $file.$counter ((counter++)) done and finally the content of the catch-all file, in blue are the log messages that the shell script still captures and spits to text files, in red are the messages are the messages the shell does not get: <pri=45> <date=2008-08-19T15:13:56.191-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Server license found, starting in server mode; customer='University of Illinois', serial='49544f30-573e-11dd-9892-0007e9195011', limit='25'> <pri=46> <date=2008-08-19T15:13:56.191-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Starting destination program; cmdline='bash /usr/local/bds/logger/syslogAdapterTest.sh'> <pri=45> <date=2008-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: syslog-ng starting up; version='2.1.11a'> <pri=46> <date=2008-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Initializing destination file writer; template='/var/log/syslog-ng-catch-all.log', filename='/var/log/syslog-ng-catch-all.log'> <pri=45> <date=2008-08-19T15:13:56.193-05:00> <host=oracle-test> <msg=syslog-ng[27971]: License validity expires in one week, logging service will terminate at that point; error='License outside the validity period (option=syslog-ng, not-before=2008/07/21, not-after=2008/08/21)'> *<pri=43> <date=2008-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: I/O error occurred while writing; fd='11', error='Broken pipe (32)'> <pri=46> <date=2008-08-19T15:13:56.198-05:00> <host=oracle-test> <msg=syslog-ng[27971]: Sending child a TERM signal; child_pid='27972'>* Any idea?

1 0

Store and forward for ~800 hosts
by Josh Rivel 19 Aug '08

19 Aug '08

Hello. I have roughly 800 OpenSolaris hosts running syslog-ng 1.6.11, logging back to another OpenSolaris host running 2.1beta2. Most of the remote locations are over slow DSL lines, so the client has requested that we only do log collection between 04:00 and 06:00 (for example) What's the best method for this? I thought about just logging everything locally at the remote locations and rsync'ing the log files back to the syslog-ng server during those hours, but I' not sure if that two hour window will be enough to gather all the data. If the commercial version of syslog-ng supports such a configuration, then I'm sure that the client would be consider purchasing it, so if that's an option let me know. Ideally there should be a way to only send logs remotely between a certain time window from the remote machines, all other times log locally, and then dump the remaining logs during that time window. I've searched the mailing list archives and couldn't find anything applicable. Thanks in advance, Josh

1 0

Syslog-ng wont start after centos updates
by John Duraisw 19 Aug '08

19 Aug '08

Hello, I did a centos update yesterday and now my syslog-ng wont start. Please help. [root@central-logger init.d]# /etc/init.d/syslog-ng.init start Starting syslog-ng: syntax error at 189 Starting Kernel Logger: [FAILED] Thanks, -john

2 2

Charset problem
by Kostyantyn Gushtin 18 Aug '08

18 Aug '08

Hello, there is an encoding problem if using syslog-ng with windows agent. Logs from Linux hosts are written to file as UTF-8, while the logs from windows agent are written to file as cp1252. The problem is visible if you have a message with chars out of ASCII set, like Ö. As a result we have a file that is both unreadable and useless for automation. Is there any way to fix this? Or do you plan to fix this? >From my point of view syslog-ng should write file using one charset. Preferably configurable. -- Kostyantyn Gushtin Software engineer N-iX LLC Solutions for Your success http://www.n-ix.com

2 1

Re: [syslog-ng] problem with logger -t and $TAG
by JUNG, Christian 14 Aug '08

14 Aug '08

The TAG macro is the facility/priority information in a syslog message (logger option -p) not the tag you define with logger (-t). Try the PROGRAM (should be the string you've defined with -t) and MSGONLY (rest of the message) macros. bye Chris > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu > [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of Nilshar > Sent: Thursday, August 14, 2008 11:59 AM > To: syslog-ng(a)lists.balabit.hu > Subject: [syslog-ng] problem with logger -t and $TAG > > > Hello everyone. > > Ok first sorry if this question has already been answered in an older > thread, but I found no way to search easily through the archives :/ > > So my problem is : > > I'm telling apache to log access logs to logger like that : > >> TransferLog "| /usr/bin/logger -p local0.info -t apache_access" > > so using the -t to define a custom tag, so I can easily filter it > using match() in filters. > > as a destination in syslog-ng, I want a file with usual apache format, > so using : > >> destination df_apache_access { > >> file("/var/log/apache/access.log" > >> template("$MESSAGE\n") ); > >> }; > > > But in the file I got the apache_access: tag before the $MESSAGE. I > was thinking it would go to $TAG, which seems not be the case. > > It's probably more a problem with logger... but wondering if anyone is > able to point me a way to fix that issue.. maybe using something else > than logger ? > > Thanks. > Nilshar. > ______________________________________________________________ > ________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

2 1

problem with logger -t and $TAG
by Nilshar 14 Aug '08

14 Aug '08

Hello everyone. Ok first sorry if this question has already been answered in an older thread, but I found no way to search easily through the archives :/ So my problem is : I'm telling apache to log access logs to logger like that : >> TransferLog "| /usr/bin/logger -p local0.info -t apache_access" so using the -t to define a custom tag, so I can easily filter it using match() in filters. as a destination in syslog-ng, I want a file with usual apache format, so using : >> destination df_apache_access { >> file("/var/log/apache/access.log" >> template("$MESSAGE\n") ); >> }; But in the file I got the apache_access: tag before the $MESSAGE. I was thinking it would go to $TAG, which seems not be the case. It's probably more a problem with logger... but wondering if anyone is able to point me a way to fix that issue.. maybe using something else than logger ? Thanks. Nilshar.

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 39, Issue 20
by Anurag Agarwal 12 Aug '08

12 Aug '08

Hello Bazsi, I did some more observation and I have following more precise inputs 1) Same config file is working correctly in my lab Solaris v240 machine. Filtered syslog sent to destined user only. 2) In production the destined user is receiving logs twice ( 2 copies). It means one copy for itself because it is destination user and 2nd copy is because all users are copied. 3) If you telnet to server and don't enter login/password even, still you start receiving a copy of message. ï»¿ Please help ! Anurag aggarwal -----Original Message----- From: Anurag Agarwal Sent: Thursday, July 24, 2008 4:54 PM To: 'syslog-ng(a)lists.balabit.hu' Subject: RE: syslog-ng Digest, Vol 39, Issue 20 Thanks Bazsi for reply. Relevant portion is below ************************************************************** destination console { usertty("wimaxnoc"); }; filter ps_nt_re { match("PS NOT REACHABLE"); }; log { source(network); filter(ps_nt_re); destination(console); }; ************************************************************** Please help. Anurag aggarwal -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Thursday, July 24, 2008 3:30 PM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 39, Issue 20 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: Re : Re : Re : Syslogd + Syslog-ng (Christopher Cashell) 2. Re: Re : Re : Re : Syslogd + Syslog-ng (Leandro Ferreira da Silva) 3. Re: Using regexp in match() (Balazs Scheidler) 4. Re: Usertty sending messages to all user and without login also (Balazs Scheidler) ---------------------------------------------------------------------- Message: 1 Date: Wed, 23 Jul 2008 10:04:34 -0500 From: Christopher Cashell <ChristopherCashell(a)solutionary.com> Subject: Re: [syslog-ng] Re : Re : Re : Syslogd + Syslog-ng To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <48874882.8070904(a)solutionary.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Leandro Ferreira da Silva did thus speak on 7/23/2008 7:14 AM: > The configuration is the standart, I only do the follow changes. > > # sources > source src { unix-dgram("/var/run/log"); > unix-dgram("/var/run/logpriv" perm(0600)); > udp(); internal(); file("/dev/klog"); }; > > I add this, > source r_src { udp(ip("*client.domain*") port(514)); }; Here's your problem. For src, you define 'udp()' as one of your log sources, without including any specific options for it. This will cause syslog-ng to go with its defaults for udp(), which is bind to *all* IP addresses assigned to the box, on port 514. Then, for r_src, you are telling syslog-ng to bind to a specific IP address on the box and port 514. When syslog-ng attempts to start, it will fail with the error message you received because that ip/port is already in use (by the source src, which is bound to all IP's). Unless you have a need to bind to a specific IP address on the box, I'd recommend removing 'udp();' from src, and replacing the current 'udp(ip("*client.domain*") port(514));' in r_src with 'udp();'. -- Christopher Cashell ------------------------------ Message: 2 Date: Wed, 23 Jul 2008 14:39:03 -0300 From: Leandro Ferreira da Silva <ferreira(a)iqm.unicamp.br> Subject: Re: [syslog-ng] Re : Re : Re : Syslogd + Syslog-ng To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <48876CB7.7080909(a)iqm.unicamp.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Christopher Cashell wrote: > Leandro Ferreira da Silva did thus speak on 7/23/2008 7:14 AM: >> The configuration is the standart, I only do the follow changes. >> >> # sources >> source src { unix-dgram("/var/run/log"); >> unix-dgram("/var/run/logpriv" perm(0600)); >> udp(); internal(); file("/dev/klog"); }; >> >> I add this, >> source r_src { udp(ip("*client.domain*") port(514)); }; > > Here's your problem. For src, you define 'udp()' as one of your log > sources, without including any specific options for it. This will > cause syslog-ng to go with its defaults for udp(), which is bind to > *all* IP addresses assigned to the box, on port 514. > > Then, for r_src, you are telling syslog-ng to bind to a specific IP > address on the box and port 514. When syslog-ng attempts to start, it > will fail with the error message you received because that ip/port is > already in use (by the source src, which is bound to all IP's). > > Unless you have a need to bind to a specific IP address on the box, > I'd recommend removing 'udp();' from src, and replacing the current > 'udp(ip("*client.domain*") port(514));' in r_src with 'udp();'. > It worked!! The real problem was udp () at src. Thank you very much for all the help... Now I'll go complete my rules.. See you!! =P ------------------------------ Message: 3 Date: Thu, 24 Jul 2008 10:16:27 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] Using regexp in match() To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Cc: documentation(a)balabit.com Message-ID: <1216887387.8130.5.camel(a)bzorp.balabit> Content-Type: text/plain; charset=UTF-8 On Tue, 2008-07-22 at 09:45 +0200, Jan Kreps wrote: > > ------------ P?vodn? zpr?va ------------ > > Od: Jan Kreps <krepsj(a)seznam.cz> > > P?edm?t: Re: [syslog-ng] Using regexp in match() > > Datum: 10.7.2008 10:29:35 > > ---------------------------------------- > > > > > > I'm trying to setup central syslog-ng server for my Exchange > > > > servers. On > > > windows servers I use Epilog agent (brother of Snare) forwarding > > > tracking logs to central syslog. > > > Obviously I have used bad regexp. Exchange tracking log uses tabs as delimiters. > > But when I saved $MSG string to text log, tabs was changed to '\011'. > > > > So now I changed my regexp to use tabs as delimiters: > > > > filter f_parsing { > > match("([^\t]*)\t([^\t]*)\t"); > > }; > > > > This works like charm and saves first two tab delimited fields (date > > and time in this case) to $1 and $2. > > > > Some more remarks to subject. > > Syslog-ng Administrator guide 1.1.1 Edition from May 2008 says on page 111: > > "The regular expressions can use up to 255 regexp matches ($1 ... $255)." > > and on page 112: > > "Regarding braces around macro names, the following two formats are equivalent "$MSG" and "${MSG}"." > > Fact is that for regexp matches greater than 9 it MUST be in ${} format. If you use for instance $12 it resolves as content of $1 + character '2'. I found that I have to use ${} syntax in ChangeLog remark for patch 137. I guess that should be stated in Admin Guide more clearly. In this respect, statement "The regular expressions can use up to 255 regexp matches ($1 ... $255)" is not true and should be corrected. > right, this should be fixed in the documentation. > And I have one question about file creation. > > When something is logged a file is created according to destination(). But when I delete the file, it's not created anymore, until restart of syslog-ng. Is that correct/expected behaviour? > > I run Debian Etch and syslog-ng 2.0.9-1 from unstable branch (in > stable is still version 2.0.0) Yes, you need to send the HUP signal to syslog-ng in order to reopen log files. This is the way UNIX works. (the application does not know about the deletion of a logfile) -- Bazsi ------------------------------ Message: 4 Date: Thu, 24 Jul 2008 10:28:06 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] Usertty sending messages to all user and without login also To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1216888086.8130.10.camel(a)bzorp.balabit> Content-Type: text/plain On Wed, 2008-07-23 at 13:39 +0530, Anurag Agarwal wrote: > Hello everybuddy, > > I am using syslog-ng and used usertty in destination. I am using a user > 'test' as destination. But other users can also see the message on their > console e.g root and other users. Please help as this is creating a > problem and NG not working as desired. > OS: Solaris 10 > It does work for me. Can you show us the relevant portions of your configuration file? I tried with this: source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; destination all { file("/var/log/messages"); }; log { source(local); destination(all); }; destination all_users { usertty("bazsi"); }; filter alert_emerg { level(err..emerg); }; log { source(local); filter(alert_emerg); destination(all_users); }; It correctly printed err..emerg messages on my tty, and not on others. The relevant portion of the code is here: #if HAVE_MODERN_UTMP if (ut->ut_type == USER_PROCESS && ((self->username->len == 1 && self->username->str[0] == '*') || (self->username->len <= sizeof(ut->ut_user) && memcmp(self->username->str, ut->ut_user, self->username->len) == 0))) #else if ((self->username->len == 1 && self->username->str[0] == '*') || (self->username->len <= sizeof(ut->ut_name) && memcmp(self->username->str, ut->ut_name, self->username->len) == 0)) #endif I don't see how this condition could match for everyone. Even if the wtmp format is different and ut->ut_name would contain a bogus value, the expected breakage would be to not display anywhere, not to display everywhere. -- Bazsi ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 39, Issue 20 *****************************************

1 0

destination drivers
by hi ro 12 Aug '08

12 Aug '08

Hi To listen on the specified UDP port for incoming messages over IPv6, must I use "udp6()" destination driver? Can I use "udp()" destination driver? Thanks _________________________________________________________________ あなたも総裁候補に？世界に一つだけのオリジナルブログパーツを作ろう！ http://questionbox.jp.msn.com/soudantou/

2 1

individual files
by Ryan Eldridge 11 Aug '08

11 Aug '08

Hello all. After playing around with syslog-ng all afternoon, needless to say I'm a little confused and frustrated. I have managed to get it running on most of my Linux boxes and it to accept data from a couple of them still running syslog which is what I want. The issue I'm facing is the config file I found do to the wonders of Google is splitting the information based on host but not into separate files like I'm use to with syslog aka cron, messages, syslog etc.. what i'm looking for output is the following: ls -shl /var/log/HOSTS/peter.domain.com/2008/08 4.0K -rw------- 1 root root 866 2008-08-10 20:46 auth.log 0 -rw-r--r-- 1 root root 0 2008-08-10 17:22 cron.log 0 -rw-r--r-- 1 root root 0 2008-08-10 17:22 daemon.log 4.0K -rw------- 1 root root 2.8K 2008-08-10 18:26 debug 8.0K -rw-r--r-- 1 root root 6.6K 2008-08-10 14:36 dmesg 0 -rw-r----- 1 root root 0 2002-04-06 19:13 maillog 12K -rw------- 1 root root 8.2K 2008-08-10 20:05 messages 0 -rw-r--r-- 1 root root 0 2008-08-10 18:22 ntpd 0 -rw-r----- 1 root root 0 1994-05-09 03:06 secure 0 -rw-r----- 1 root root 0 2002-03-09 00:29 spooler 12K -rw------- 1 root root 8.6K 2008-08-10 20:05 syslog Currently i'm getting the following: ls -shl /var/log/HOSTS/peter.domain.com/2008/08 12K -rw------- 1 root root 11K 2008-08-10 21:31 10 which contains all the results. I'll post the file I'm using below any help would be useful ############################################################### # First, set some global options. options { use_fqdn(yes); use_dns(yes); dns_cache(yes); keep_hostname(yes); long_hostnames(off); sync(1); log_fifo_size(1024); }; ############################################################### # # Logs may come from unix stream, and UDP:514 # source src { pipe("/proc/kmsg"); unix-stream("/dev/log"); internal(); udp(ip("192.168.2.3") port(514)); tcp(ip("192.168.2.3") port(514) keep-alive(yes)); }; ############################################################### # First some standard logfile # destination authlog { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination user { file("/var/log/user.log"); }; #destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; ########################################## # Here's the filter options. With this rules, we can set which # message go where. filter f_authpriv { facility(auth, authpriv); }; filter f_syslog { not facility(auth, authpriv) and not facility(mail); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; #filter f_mail { facility(mail); }; filter f_user { facility(user); }; #filter f_news { facility(news); }; #filter f_debug { not facility(auth, authpriv, news, mail); }; filter f_messages { level(info .. warn) and not facility(auth, authpriv, cron, daemon, mail, news); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; ############################################################### # # log statements actually send logs somewhere, to a file, across the network, etc # log { source(src); filter(f_authpriv); destination(authlog); }; log { source(src); filter(f_authpriv); destination(std); }; log { source(src); filter(f_syslog); destination(std); }; log { source(src); filter(f_cron); destination(std); }; log { source(src); filter(f_daemon); destination(std); }; log { source(src); filter(f_daemon); destination(std); }; log { source(src); filter(f_kern); destination(std); }; log { source(src); filter(f_user); destination(std); }; #log { source(src); filter(f_debug); destination(std); }; log { source(src); filter(f_messages); destination(std); }; ## set up logging to loghost #destination loghost { # tcp("10.0.0.1" port(514)); #}; # send everything to loghost, too #log { # source(src); # destination(loghost); #}; # # automatic host sorting (usually used on a loghost) # # set it up destination std { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY_$HOST_$YEAR_$MONTH_$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; # log it log { source(src); destination(std); };

2 1