- syslog-ng - lists.balabit.hu

Missing remote logs
by fRANz 27 Sep '04

27 Sep '04

I've a syslogd that send his log remotly over a vpn tunnel to an openbsd3.4 with syslog-ng(port syslog-ng-1.5.24.tgz). I see a new log on syslogd, I see the packet go into vpn tunnel but syslog-ng not log it immediately, indeed, I not receive all logs, someone only. Can I miss something in syslog-ng config?! This is my syslog-ng.conf I'd like a realtime remote logging: I necessarily transport between 2 syslog-ng in tcp instead udp?! Thank you very much, fRANz # Copyright (c) 1999 anonymous # Copyright (c) 1999 Balazs Scheidler # $Id: syslog-ng.conf.sample,v 1.2 1999/11/15 12:30:41 bazsi Exp $ # options { long_hostnames(off); sync(0); }; source src { unix-stream("/dev/log"); internal(); }; source net { udp(ip(172.16.13.3)); }; destination authlog { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination kern { file("/var/log/kern.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination user { file("/var/log/user.log"); }; destination uucp { file("/var/log/uucp.log"); }; destination mail { file("/var/log/mail.log"); }; destination mailinfo { file("/var/log/mail.info"); }; destination mailwarn { file("/var/log/mail.warn"); }; destination mailerr { file("/var/log/mail.err"); }; destination newscrit { file("/var/log/news/news.crit"); }; destination newserr { file("/var/log/news/news.err"); }; destination newsnotice { file("/var/log/news/news.notice"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; destination console { usertty("root"); }; destination console_all { file("/dev/tty12"); }; destination xconsole { pipe("/dev/xconsole"); }; destination sshd { file("/var/log/secure"); }; destination gate-1 { file("/var/log/gate-1"); }; filter f_auth { facility(auth); }; filter f_authpriv { facility(auth, authpriv); }; filter f_syslog { not facility(authpriv, mail); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_user { facility(user); }; filter f_uucp { facility(cron); }; #filter f_ppp { facility(ppp); }; filter f_news { facility(news); }; filter f_debug { not facility(auth, authpriv, news, mail); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; filter f_sshd { program(sshd); }; filter f_gate-1 { host("172.16.13.254"); }; log { source(src); filter(f_authpriv); destination(authlog); }; log { source(src); filter(f_syslog); destination(syslog); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_daemon); destination(daemon); }; log { source(src); filter(f_kern); destination(kern); }; log { source(src); filter(f_lpr); destination(lpr); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_user); destination(user); }; log { source(src); filter(f_uucp); destination(uucp); }; log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); }; log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); }; log { source(src); filter(f_mail); filter(f_err); destination(mailerr); }; log { source(src); filter(f_news); filter(f_crit); destination(newscrit); }; log { source(src); filter(f_news); filter(f_err); destination(newserr); }; log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); }; log { source(src); filter(f_debug); destination(debug); }; log { source(src); filter(f_messages); destination(messages); }; log { source(src); filter(f_emergency); destination(console); }; #log { source(src); filter(f_ppp); destination(ppp); }; log { source(src); destination(console_all); }; log { source(src); filter(f_sshd); destination(sshd); }; log { source(net); filter(f_gate-1); destination(gate-1); }; #

1 0

RE: [syslog-ng]Corrupted messages in log
by Dmitri Smirnov 27 Sep '04

27 Sep '04

Thanks, Loic, some important parts of config, skipping filters: # source local { sun-streams("/dev/log"); internal(); udp(); }; options { use_fqdn(yes); use_dns(yes); dns_cache(yes); keep_hostname(yes); chain_hostnames(no); bad_hostname("^5.*"); sync(0); stats(0); log_fifo_size(1024); log_msg_size(2048); use_time_recvd(yes); dns_cache_expire(36000); dns_cache_expire_failed(3600); dns_cache_size(10000); }; destination syslog { file("/var/log/syslog" owner(root) group(other) perm(0644) template("$DATE $FULLHOST $FACILITY.$PRIORITY $MESSAGE\n") template_escape(no)); }; log { source(local); filter(filter1_not); filter(filter2_not); filter(filter3_not); filter(filter4_not); filter(filter5_not); destination(syslog); }; Dmitri -----Original Message----- From: Loïc Minier [mailto:lool+syslog@via.ecp.fr] Sent: Wednesday, September 22, 2004 3:19 AM To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng]Corrupted messages in log Dmitri Smirnov <Dmitri.Smirnov(a)fusepoint.com> - Tue, Sep 21, 2004: > syslog-ng 1.6.3, Solaris 8 with latest patches. > Destination to file using templates, network and local sources, > multiple filters. > Started to receive multiple syslog messages from different hosts > concatenated in one big syslog message .. Post your configuration, did you use unix-stream or unix-dgram in your sources? or sun-streams? -- Loïc Minier <lool(a)dooz.org> _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1

Which signal does reload sends ?
by Ali EL MOUSSAWI 24 Sep '04

24 Sep '04

Hi all Can you please tell me what happens when syslog-ng reloads (i.e. receives HUP signal) ? I need to know what signal does it send to the destination programs, and is it a good practice to catch this signal. In fact I don't want my destination program to die at reload, or at least I want to serialize his data structures before it dies, in order to retrieve it when it goes back. On the other hand, if a message arrives from an UDP source while syslog-ng is reloading, will it be lost ? Why the keep alive option is only for tcp source ? Thanks a lot for any help Ali

3 2

blow up a flea
by bedo.sandor＠ln.matav.hu 23 Sep '04

23 Sep '04

Hi All! I and my co-workers have had a trip around "STATS: dropped N" log messages in the last few days, and I am made to share my experience with you. I'm administrating a central log collector, archiver, retriever and analyser service. Of course its heart is syslog-ng. Syslog-ng stores the incoming data in separate files for each recognized subsystems in separate directories for every host. For example you can think about subsystems like cron, postfix, sshd, sendmail, exim, apache, su and sudo, netfilter, ipchains, kernel messages, grsec or execlog for special kernels, System, Security, Application, IIS, ActiveDirectory logs by a Windows source... The analyser part of the system runs a few scripts on the files containing yesterday arrived log messages, for example eximstats, or so. I have written a syslog-ng log analyser script just for fun. One step of the analyser is to collect those evil reputed "STATS: dropped N" messages and draw a graph, that shows the rate of dropped messages based on the last days crop. It was a little bit frustrating to see that graph for a few intranet log forwarder machines, because of more than 30 million lost lines per day! First I couldent believe! "Is that value measured in bytes?? Maybe in bits?" - that was the first idea of one of my co-worker, but the answer was crushing: it is measured in log lines! There have had plenty of agonized crack at the problem: increase the buffer size (certainly this was a silly idea), verify the network interfaces, drivers, cables (the idea was good and shed light on other problemes nevertheless solved the original), plans to upgrade to gigabit ethernet, eliminate every network devices and use only cross utp cables, use more computers for the analysis diffused on the intranet, etc., etc. After all there was an answer message on this list with a little mistake. It suggested that the number in the log lines can only be from tcp destinations. That's wrong! It can arise from other destination too. For example from program(), pipe() and maybe more... and there come the propitiation: f*ckin' debug pipes!!!! We have a local practice of pipe destinations. Chroot-ed environments often have a /var/log/messages pipe inside, and the syslog-ng running in the base system sends the data quarried from the jail back to that pipe for debugging reasons. If you are acting in the jail, you can "cat" the pipe and see the related events without exiting to the base system (where the data is stored in regular files). On the log forwarder machines there are pipes for debugging the incoming tcp and udp log traffic (for example it's nice to explore and locate unknown log sources, to see that remote logging is on, or to identify dummy systems using udp protocol). Pipes seemd to be practical because they didn't eat up the disk space even so ther was a point to access the log stream if it needed. Our solution is an ugly workaround. I have written a little C code that opens a pipe with O_NONBLOCK and sends data from standard input to the pipe. Because of non-blocking io syslog-ng cannot detect the data loss. So pipe destinations are replaced to program(".../sbin/pipecat .../var/log/pipe") destinations to eliminate false scaled syslog-ng statistics. (It is a pity that program("/bin/cat > /var/log/pipe"); generates the same wrong statistic, so my pipecat seems the one and only solution. Do You have any better idea?) I'm sorry if I was boring! I think promlems, their solutions and others experiences can be of service to anybody. Let these be a lesson for all of you out there! :) -- Have a nice day! bSanyI

1 0

SIGHUP not working (Plz Help)
by Shahid Shafi 23 Sep '04

23 Sep '04

Hi guys, I am in slight trouble. I installed syslog-ng, configured my stuff and everything is working fine. I am using newsyslog to rotate logs file every 2 MB. newsyslog is invoked via cron and send HUP to the syslog-ng after rotating. File is rotated and compressed but the new file never got any log data again. I then had to restart the syslog-ng process and everyhting works normally after that. Can somebody help here? I am using syslogng 1.6.5 with libol 0.3.14 thx Shahid

2 2

RE: [syslog-ng]problem with different timezones
by ARGEXT-HIDALGO, FRANCISCO 23 Sep '04

23 Sep '04

>the problem with this is that bufferring happens at a later stage, after >the message is already formatted. Maybe you could achieve the same by >putting the received timestamp at the receiving syslog-ng. No, that's just what I need, to know when originally was recieved from the router, before it's buffered. So the $S_DATE and $R_DATE macros are the solution I was looking for, thank you very much. >-- >Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

Re: [syslog-ng]Problems with syslog-ng remote logging on Sun/Linu x environment!
by Loïc Minier 23 Sep '04

23 Sep '04

Jörgen Ekstedt <jorgen.ekstedt(a)hiq.se> - Thu, Sep 23, 2004: > I tried to look up if the connection is beeing made with netstat and > tcpdump, and at the first glance it seem to be the problem. Now I am not > that kind of network guru to really say that IS doesnt connect. I suggest saving the output of tcpdump/snoop to a file and open it with Ethereal. Regards, -- Loïc Minier <lool(a)dooz.org>

1 0

problem with different timezones
by ARGEXT-HIDALGO, FRANCISCO 23 Sep '04

23 Sep '04

I have the following situation: I have several logging servers in different timezones and these send all their information with TCP to a central logging server. What I don't understand is that when I recieve a log in the central log server I recieve the exact same line that appears in the log server that sent the log. The date isnt updated no matter if I use or not the use_time_recvd() option. The only solution I found was to use a template with the $DATE macro. That would be my first doubt, why use_time_recvd() doesn't work? My second doubt is, is there any macro for the time sent?, so I can have in the same log (on the central logging) the time sent and the time received, so I know if there was any delay. For example if the link went down and they were retained in the original server for a while. (the servers would be sync'ed in time) Thank you.

2 1

unsubscribe
by Wyner, Peter 23 Sep '04

23 Sep '04

1 0

AIX log workaround
by bedo.sandor＠ln.matav.hu 23 Sep '04

23 Sep '04

Hi, In an earlier version of syslog-ng's changelog I have read about a patch that workarounds the AIX's "Message from xy:" and "Forwarded from xy:" idiocy. In the version 1.6.x it seems to be not fixed. I store my lod data in separate files with the name containing $HOST, but 1.6.x substitues $HOST to the string "Message" by the AIX log lines. Am I something misanderstanding? Had that patch gone about somthing else? Is there any possibility to separate my hosts log data? Thanks, bSanyI

2 1