- syslog-ng - lists.balabit.hu

Re: [syslog-ng] Collecting Logs on Loghost
by peter.klausner＠systor.com 14 Apr '00

14 Apr '00

Hi Wolfi + Gero Your projects look interesting, because there seems to be not much publicly available information on that area. ... :o how much net load does syslog create? : ... :o how much data does a host generate? : The overhead of the syslog protocol is rather small. The problem is the volume of logging done by the hosts/applications. Normal operations cause no significant loads. But if things go wrong, then log info ranges from 0..infinity. I saw 1 gig /var filesystems filled in no time flat by syslogd. You don't want this in your DB, do you? So a serious architecture requires some preprocessing before sending messages to the central host. ... :o common tools : :i'm aware of several analyzing tools (swatch, logsurfer ... from : Not to forget the big commercial framework packages: CA Unicenter, HP OpenView, Tivoli TME, BMC Patrol + Command/Post. A smaller product with very good reputation is Netcool Omnibus. All these provide agents, which scan and preprocess arbitrary logfiles. Yes, with central configuration and pre-defined templates for standard log entries. No, long term analysis is not [yet] easy with these, but service reporting functionality is sprouting up everywhere and hopefully useful in the near future. ... :as said, i'd be very interested on your thoughts on the topic, your :experiences and maybe references to papers/tools. : Although everybody must work on this, there is not much public stuff on this AFAIK. Maybe www.summitonline.com has a bit. For analysis of complex event patterns, a rather new approach is to visualize the conceptual relationship of events. E.g.: Girardin98: A Visual Approach for Monitoring Logs (http://www.usenix.org/events/lisa98/girardin.html) Hellerstein99: EventBrowser: A Flexible Tool for Scalable Analysis of Event Data" (http://www.research.ibm.com/PM/perf_mgt.html) HTH Peter

1 0

Log analyzer/syslog-ng problems
by Bryan Stillwell 13 Apr '00

13 Apr '00

Hello everyone, I'm currently doing research at my job about writing a log analyzer. The first step we wish to accomplish is setting up a central log server. The server will be just for logs and will be secured so that it would be almost impossible for someone who has compromised another machine on the netowrk to modify the log files. The log server will be logging both internal and external machines, so we would really like for communication to be well encrypted. As I understand this option won't be implemented until the next version of syslog-ng. After this step is done we will set it up to automatically enter all logs into a postgresql database or mysql database for easier analysis. (I've been told postgresql has some native IP address data types that will come in handy.) However our number one item on the list is to make sure that the logs get to the server in a timely, consistant, and accurate manner. So I started doing some testing on various things that might cause problems during the transfering of the log data. Some of the things that I want to watch out for are: o What happens if the network is down for a day or two at a time? I want to make sure that they don't just disappear. o What if the network is being hit with a DOS attack and packets never reach the log server? or they reach the log server but the reply never gets back to the source computer? o What kind of verification can be done to make sure all the data made it to the log server ok? Should we log locally and remotely and compare the two at the end of each day? or can syslog-ng take care of it all? o What about if the main server dies and has to be restarted? client dies? Some of those questions will probably have the same answers, but I ran into the following problem when I started doing some stress tests between two computers on the internal lan. I've included the two syslog-ng.conf files to the end of this message. reptile is the client computer and testbox is the server. I ran the command `logger -f /usr/dict/linux.words` to test out how well it works. Apr 13 10:39:27 src@reptile/reptile logger: Aarhus Apr 13 10:39:27 src@reptile/reptile logger: Aaron ... Apr 13 10:39:38 src@reptile/reptile logger: strips Apr 13 10:39:38 src@reptile/reptile logger: striptease Apr 13 10:39:40 src@reptile/reptile logger: worked Apr 13 10:39:40 src@reptile/reptile logger: worker ... Apr 13 10:39:40 src@reptile/reptile logger: worn Apr 13 10:39:40 src@reptile/reptile logger: worried Apr 13 10:39:40 src@reptile/reptile logger<13>Apr 13 10:39:38 src@reptile logger: strive Apr 13 10:39:38 src@reptile/reptile logger: striven Apr 13 10:39:38 src@reptile/reptile logger: strives ... Apr 13 10:39:40 src@reptile/reptile logger: workbook Apr 13 10:39:40 src@reptile/reptile logger: workbooks Apr 13 10:34:54 reptile/reptile : worrier Apr 13 10:39:40 src@reptile/reptile logger: worriers Apr 13 10:39:40 src@reptile/reptile logger: worries ... Apr 13 10:39:40 src@reptile/reptile logger: Zulus Apr 13 10:39:40 src@reptile/reptile logger: Zurich So it works until it hits 'striptease' (purely coincidence btw) and then skips to 'worked'. When it hits 'worried' it switches back to where it left off and continues until it hits 'workbooks' where the previous group was supposed to be at. Other than that problem everything transferred fine. Any ideas on how to have that autofix itself, maybe I don't have the conf files setup correctly? Would it be better to use udp even though dropped packets will never be known about? I just want to maintain the upmost integrity in the log files. --- testbox(server) syslog-ng.conf --- options { long_hostnames(on); sync(0); log_fifo_size(100); }; source src { unix-dgram("/dev/log"); tcp(); internal(); }; destination local_log { file("/tmp/deleteme.log"); }; log { source(src); destination(local_log); }; --- testbox(server) syslog-ng.conf --- --- reptile(client) syslog-ng.conf --- options { sync(0); log_fifo_size(100000); }; source src { unix-dgram("/dev/log"); internal(); }; destination master_log_server { tcp("10.1.1.10" port(514)); }; log { source(src); destination(master_log_server); }; --- reptile(client) syslog-ng.conf --- Thank you, Bryan Stillwell -- http://www.tummy.com/ Consulting and Software for Linux and Unix KRUD - Kevin's Red Hat Uber Distribution - the Freshest Red Hat every month

1 0

Recommended server specs
by syslog-ng＠moe.co.uk 12 Apr '00

12 Apr '00

Hi All, Was just wondering what people recommend spec wise for a loghost? Currently have a P2-350, 320mb (ish) of ram and 18gb to store logs to. There are about 80 to 90 machines logging to this machine. I think it will run out of disk space before anything else. syslog-ng tends to use 3% to 7% cpu with a sync(5). Anyone have any interesting loghost information/experiences? Cheers, John

1 0

local[0-7], can't get them to write to file.
by i＠moe.co.uk 12 Apr '00

12 Apr '00

Hi All, I'm having a few minor problems writing local 6 and 7 to files. I'm most likely doing something incorrect in the config file, can some one help me out please. source src { unix-dgram("/var/run/log"); internal(); udp(ip(0.0.0.0) port(514)); }; destination local6 { file("/var/log/local6.log"); }; destination local7 { file("/var/log/local7.log"); } filter f_local6 { facility(local6); }; filter f_local7 { facility(local7); }; log { source(src); filter(f_local6); destination(local6); }; log { source(src); filter(f_local7); destination(local7); }; Cheers, John

2 3

RE: [syslog-ng] complex filtering
by Hamilton, Andrew Mr. 12 Apr '00

12 Apr '00

I am using 1.4.2 which I assume is still the latest version since I haven't heard of anything new lately and that being the latest on the web site. I will see if I can duplicate what I was doing and get the config to you and the logs I'm getting from it. Drew > -----Original Message----- > From: Leopold Toetsch [SMTP:lt@toetsch.at] > Sent: Thursday, April 06, 2000 12:17 PM > To: syslog-ng(a)venus.terrasoft.hu > Subject: Re: [syslog-ng] complex filtering > > > ----- Original Message ----- > From: Hamilton, Andrew Mr. <HamiltonA(a)hq.5sigcmd.army.mil> > To: <syslog-ng(a)venus.terrasoft.hu> > Sent: Donnerstag, 06. April 2000 07:10 > Subject: RE: [syslog-ng] complex filtering > > > > I've also had problems with the "and not" filtering in syslog-ng. I > ended > > up having to do a kind of a DeMorgan method and change the filtering > around > > a little. I got it to do what I wanted but not quite the way I > expected. > > Are you using the latest version? > There was a bug concerning 'not', which should be fixed now. > > leo > > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > http://lists.balabit.hu/mailman/listinfo/syslog-ng

2 1

write() failed (errno 32)
by Przemek Bak 07 Apr '00

07 Apr '00

Hi, I have following lines in syslog-ng.conf filter aaa { match ("aaa"); }; destination aaa { program ("/usr/bin/mail -s WARN user"); }; log { source(src); filter (aaa); destination(aaa); }; When I run logger: # logger aaaIIII in messages file I have: Apr 5 13:06:26 src@host syslog-ng[1627]: io.c: do_write: write() failed (errno 32), Broken pipe What does it mean ? It happen from time to time, not always. przemol

2 6

Re: tracelogs
by Balazs Scheidler 07 Apr '00

07 Apr '00

On Mon, Mar 27, 2000 at 04:33:18PM +0200, Gero Krueger wrote: > Hi Balazs ! > > HP has sent me a tracing tool. So I recorded a session > where syslog-ng listens on /dev/log and forwards that > to a remote host via tcp. > With the first logentry syslog-ng gets from /dev/log he > produces that huge unterminated output. > > I'm not really sure if this tracing-tool records the > information you need - just have a look at it and > let me know. > > > Thank you very much for your help. Sorry for the long time it took me to answer, but I'm really out of time. I've checked the trace you sent me, and the problem is that the info I need is missing: 3191: read(4 [/dev/log], 0x40019888, 1024) = 1024 The contents of the buffer is not displayed. It's strange that HPUX says that 1024 bytes could be successfully read, something should terminate that string, the question is what it is? -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 url: http://www.balabit.hu/pgpkey.txt

1 0

Expanding $PROGRAM and $FACILIITY macros with empty log messges
by Jason Edgecombe 06 Apr '00

06 Apr '00

hi all, i'm running syslog-ng-1.4.2 and I am having a problem with macro expansion an blank log message causes the problem. here is an excerpt from my logs: note, mathlog-pc is our central logging server, it is also running syslog-ng (same version) but is UDP forwarding all logs to the log server Apr 5 08:28:40 src@fret340f-pc/fret340f-pc Apr 5 08:28:40 src@mathlog-pc syslog-ng[1793]: Cannot open file /var/log/syslog-ng/2000/04/05/fret340f-pc/p/ for writing (Is a directory) Apr 5 08:28:40 src@fret340f-pc/fret340f-pc syslog-ng[4481]: Cannot open file /var/log/syslog-ng/2000/04/05/fret340f-pc/ for writing (Is a directory) Any help on this is appreciated. Thanks, Jason

1 0

RE: [syslog-ng] complex filtering
by Hamilton, Andrew Mr. 06 Apr '00

06 Apr '00

I've also had problems with the "and not" filtering in syslog-ng. I ended up having to do a kind of a DeMorgan method and change the filtering around a little. I got it to do what I wanted but not quite the way I expected. filter f_messages { level(info..warn) and not facility(local2,kern,auth); }; The above didn't work the way I expected it should have. I ended up doing filter f_messages { level(info..warn) and facility(mail,news,etc..); }; This seemed to work ok. I'm not sure if syslog-ng is evaluating the "and not" construct the way you'd expect. Of course it may be that I just wasn't using it correctly as well. Drew > -----Original Message----- > From: Balazs Scheidler [SMTP:bazsi@balabit.hu] > Sent: Wednesday, April 05, 2000 5:08 PM > To: syslog-ng(a)venus.terrasoft.hu > Subject: Re: [syslog-ng] complex filtering > > > > > Im currently building our loghost syslog-ng conf file and encounter a > problem. How can I filter message so that only not-alredy-routed line will > be routed/filtered again ? > > > > For example, I am trying to send ssh line to a dedicated log file, since > sshd send its message to the deamon facility and that ( generic) daemon > facility are send to a generic file, a given line appereas in both file > and that waste my disk space :-). > > > > I am trying to solve that by adding "and not filter(f_ssh)" in my dameon > filter but it does nothing ( not even syntax error message). > > > > Is there any error ? Is there another way ? > > > > Thanks for any help. > > The filter way should work, if it doesn't, it's a bug. There's a feature > you > may use here. You can define default log statements: > > log { source(src); filter(DEFAULT); destination(dst); }; > > This is a catchall statement, and should catch all messages which were not > accepted any of the previous statements. > > -- > Bazsi > PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C > 8EB1 > url: http://www.balabit.hu/pgpkey.txt > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > http://lists.balabit.hu/mailman/listinfo/syslog-ng

2 1

program
by Przemek Bak 05 Apr '00

05 Apr '00

Hi, I have following lines in syslog-ng.conf filter aaa { match ("aaa"); }; destination aaa { program ("/usr/bin/mail -s WARN user"); }; log { source(src); filter (aaa); destination(aaa); }; When I run logger: # logger aaa it doesn't always work. Why ? I just want to have lines of messages sent to me by mail. przemol

1 0