March 2003 - syslog-ng - lists.balabit.hu

RE: [syslog-ng]$MSG into different variables?
by McNutt, Justin M. 20 Mar '03

20 Mar '03

If you find a solution to this, *please please please* post it to the list! :-) Thanks! --J > -----Original Message----- > From: m(a)ervius.net [mailto:m@ervius.net] > Sent: Thursday, March 20, 2003 9:19 AM > To: syslog-ng(a)lists.balabit.hu > Subject: [syslog-ng]$MSG into different variables? > > > Hi, > > I'm trying to insert a variety of different logs into a MySql database > from syslog-ng. The logs are coming from cisco routers, squid > proxys, FW1 > etc. and have a varying number of fields. > > The database tables are created with fields that match the > different log > formats, - src IP, dest IP, protocol etc. > > Since I can match the incoming logs with regexps, it is possible to be > absolutely sure that a given log message will fit into a given table. > > What I then would like to do is to split the $MSG into its different > columns (separated by whitespace) and call the corresponding > destination > which would look something like this: > > destination cisco_icmp { > pipe("/tmp/icmp.pipe" > template("INSERT INTO router_icmp (host, priority, date, > time, program, > access_list, if, type_code ) VALUES ( '$HOST', '$PRIORITY', > '$YEAR-$MONTH-$DAY', $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG[6]', > '$MSG[8]', '$MSG[13]' );\n") template-escape(yes)); > }; > > Is this possible to do in syslog-ng? Or is there any other method of > getting the values from the $MSG variable? > > I could of course read it from the pipe with some awk or perl > script and > build the sql-syntax from there. But since I will have quite a few > different table formats it would then need a perl interpreter > running for > each table. Not very pretty IMO. > > Any suggestions? Have I missed something? > > Thanks! > > // Martin > > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > >

1 0

Urgent: Mangled messages in pipe() destinations in version 1.5.19
by Andreas Schulze 20 Mar '03

20 Mar '03

Hello Balazs, in version 1.5.19 (and in other versions between >1.5.7 and <=1.5.19, too) we observing obscure message mangling in pipe() destinations. Imagine the config below and the scenario: syslog-ng --write(2)--> pipe() --read(2)--> application All four log paths below, writes lines to the pipe '/tmp/pipe'. With 1.5.7 we read from the pipe exactly that, what syslog-ng (using the config below) writes to it. But with all newer versions (espec. 1.5.19) we read (with the same config): message_1 message_2 messagmessage_4 !!! ... message_n if we try to let syslog-ng write: message_1 message_2 message_3 message_4 ... message_n (The numbers and terms are only examples, because of the different sources and high message load.) Seems, that write(2) to fifo skips some message ends now. (espec. the terminating newline) > # example config: > # > source sys { sun-streams ("/dev/log" door("/var/run/syslog_door")); }; > source net { udp (ip(w.x.y.z) port(n)); }; > destination d_cisco { pipe("/tmp/pipe" template("@cisco: | $MSG\n")); }; > destination d_icmp { pipe("/tmp/pipe" template("@icmpd: | $MSG\n")); }; > log { source(net); filter(fa); destination(da); destination(d_cisco); }; > log { source(net); filter(fb); destination(db); destination(d_cisco); }; > log { source(sys); filter(f1); destination(d1); destination(d_icmp); }; > log { source(sys); filter(f2); destination(d2); destination(d_icmp); }; We are running a config similar to this above, with 1.5.7 since a lot of months very successfully in production ennvironment with very high message load. Since 1.5.1? the message mangling occurs. But now, some of your newer features becomes mission critical for us. Have you any idea, what happens? Versions <= 1.5.7 works well. Thanks a lot. -- Best regards --Andreas Schulze [phone: +49.5246.80.1275, fax: +49.5246.80.2275] | I believe, it was Dennis Ritchie who said something like: | "C is rarely the best language for a given task, | but it's often the second-best". | The implication being that: "[...]" | | sh# cat>$$.c<<EOT | main(l,a,n,d)char**a;{for(d=atoi(a[1])/10*80-atoi(a[2])/5-596;n="@NK\ | ACLCCGZAAQBEAADAFaISADJABBA^SNLGAQABDAXIMBAACTBATAHDBANZcEMMCCCCAAhE\ | IJFAEAAABAfHJETBdFLDAANEfDNBPHdBcBBBEA_AL H E L L O, W O R L D! " | [l++-3];)for(;n-->64;)putchar(!d+++33^l&1);} | EOT | gcc -o$$ $$.c;clear;./$$ 52 8;rm -f $$*

6 19

rpm
by Khristopher_Klaich＠nywd.uscourts.gov 20 Mar '03

20 Mar '03

Hello does anyone know if there is an rpm for redhat 7.3 available? Thanks- K.

2 1

buffer limitations and TCP compression
by Jason Haar 20 Mar '03

20 Mar '03

Hi there Can someone explain to me how the tcp/udp destination options handle network outages/congestion? If I set my fifo to 10000, does that mean if the remote server is down (I assume this only affects TCP - not UDP BTW), the "client" will hold up to 10000 records before dropping... the oldest? Just what does happen then? Also, does anyone think adding a compress option to the tcp destination option is a good idea? It could really cut down on the traffic (esp if you could force a *minimum* fifo queue size - to increase your chance of repeated data). It could make centralized logging over WANs much more attractive. Thanks Jason

3 5

2 questions: confusion + compile problem
by Brian C. Hill 20 Mar '03

20 Mar '03

Hello, Let me try this again. :) Is the stable release 1.6 or 1.9? I see a 1.9.1 and 1.6.0rc1. The older one seems to be a release candidate and the newer one seems to be full release. Am I missing something? Also, is olib needed for 1.9.1? The configure script didn't complain when it couldn't find it, but I did get a make error that getopt.h couldn't be found. I tried building against glib 2.2.1, which built and installed successfully. Thanks for any clues... Brian

2 1

2 questions: confusion + compile problem
by Brian C. Hill 20 Mar '03

20 Mar '03

Hello, Is the stable release 1.6 or 1.9? I see a 1.9.1, but a 1.6.0rc1. Should I go with the older release candidate the... 1.9.1 that doesn't seem to be a release candidate? Also, it olib needed for 1.9.1? The configure script didn't complain when it couldn't find it, but I did get an error that getopt.h couldn't be found. I tried building against glib 2.2.1, which built and installed successfully. Thanks for any clues... Brian

1 0

syslog-ng in a Solaris environment
by George Horvath 19 Mar '03

19 Mar '03

Hi All, I am working on a log centralization project in a Solaris environment and my initial search led me to syslog-ng. I signed up to this mailing list to get a feel for syslog-ng's performance and support issues. It seems like a good product which is recommended by a lot of people. I am wondering though if there are any other products that may be better, have better support etc.. I'm willing to pay for a commercial product with advantages over syslog-ng. Although I'm a huge fan of open source, I use a lot of os products how ever I am not a programmer so support is an issue for me. If you are running syslog-ng in a Solaris environment (especially Solaris 8/9) I would very much be interested hearing about your experiences. I would really be interested in positive comments from someone who is successfully employing syslog-ng. There were a few Solaris based questions recently but that's not a good indication of it being a problematic platform. Finally, I noticed there are frequent releases of syslog-ng. How easy/difficult/time consuming is it to upgrade an operational environment? TIA, George Horvath P.S. I am a Hungarian born, so it would be interesting for me to use syslog-ng

3 3

how stop syslog-ng from adding src@hostname part to log files?
by seberino＠spawar.navy.mil 19 Mar '03

19 Mar '03

My syslog-ng.conf is very tiny but functional... source src {unix-stream("/dev/log"); pipe("/proc/kmsg"); internal();}; destination dest {file("/var/log/messages");}; log {source(src); destination(dest);}; This makes /var/log/messages start every line with src@hostname where hostname is the name of my machine. How can I stop src@hostname from being printed??? It is messing up log checking programs! Thanks, Chris -- _______________________________________ Dr. Christian Seberino SPAWAR Systems Center San Diego Code 2872 49258 Mills Street, Room 158 San Diego, CA 92152-5385 U.S.A. Phone: (619) 553-9973 Fax: (619) 553-6521 Email: seberino(a)spawar.navy.mil _______________________________________

1 0

syslog-ng crash
by Chriss 19 Mar '03

19 Mar '03

Hello List, my syslog-ng "loghost" crash every 1-5 minute, so I did the update to 1.6.0rc1, but the problem still exist. The service died with a "Segmentation fault". A run in debug mode and/ore strace report: ----->8------schnipp-----8<----------------------------------- time(NULL) = 1047983908 time(NULL) = 1047983908 --- SIGSEGV (Segmentation fault) @ 0 (0) --- +++ killed by SIGSEGV +++ ----->8------schnapp-----8<----------------------------------- after the last logging entry. The os is a debian unstable on a Siemens Nixdorf 2x333Mhz server. Anybody with an idea? Bye Bye, Chriss mailto:syslog-ng@kordesee.de ---- Turn your Pentium into a Gameboy: Type WIN at C:\>

3 7

syslog-ng 1.4.17 / cron messages / RH 7.3
by Tim Sheets 19 Mar '03

19 Mar '03

Hello group, I have just installed syslog-ng 1.4.17 and playing with it on RH 7.3 Initially I was getting crond messages in my logs (was attempting to send them to a separate cron log file, but they were ending up in both /var/log/cron as well as the standard /var/log/messages files. I tried to change the filter (I hadn't matched the case, etc...) thinking that may help. The messages stopped going to both places. I have tried various things like changing it back, re-organizing the filter/log statements (grasping at straws), and added a log entry for filter(DEFAULT), but I still can't seem to get them to come back. Here is my conf file: options { sync(0); chain_hostnames(no); dir_perm(0755); perm(0644); time_reopen(10); log_fifo_size(1000); long_hostnames(off); use__dns(yes); use_fqdn(yes); create_dirs(yes); keep_hostname(yes); }; source s_sys { unix-stream("/dev/log"); internal(); udp(); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog"); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_hosts { file("/var/syslog/hosts/$YEAR/$MONTH/$HOST" owner(root) group(root) perm(0644) dir_perm(0700) create_dirs(yes)); }; destination d_mlal { usertty("*"); }; destination d_cron { file("/var/log/cron"); }; filter f_auth { facility(authpriv); }; filter f_cron { facility(CROND) or facility(cron); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_spooler { facility(uucp) or (facility(news) and level(crit)); }; filter f_boot { facility(local7); }; filter f_hosts { facility(local3) or facility(local4); }; # filter f_mesg { level(info) or facility(mail) or facility(authpriv) or facility(user) and not facility(cron); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_spooler); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_hosts); destination(d_hosts); }; # log { source(s_sys); filter(f_mesg); destination(d_mesg); }; log { source(s_sys); filter(DEFAULT); destination(d_mesg); }; Here is a sample of the /var/log/cron file that has a couple of the messages that aren't showing up: Mar 6 15:10:00 chumly CROND[6213]: (root) CMD (/home/mrtg/run-rrd-cfgs) Mar 6 15:15:00 chumly CROND[6232]: (root) CMD (/home/mrtg/run-rrd-cfgs) I have implemented the init.d file that is included in the contrib directory, and between config file changes, used service syslog-ng restart to kick it into re-reading the new conf file. Any ideas what I am doing wrong, or where to start looking? I am relatively new to the whole linux world, but trying very hard to get a handle on things. :-) Any help is appreciated. Tim _________________________________________________________________ Protect your PC - get McAfee.com VirusScan Online http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963

2 1