- syslog-ng - lists.balabit.hu

stupid question: syslogging plaintext non-syslog-format messages unchanged (netconsole)
by Nix 12 Nov '09

12 Nov '09

So I'm trying to get the Linux kernel netconsole working, to diagnose some nasty persistent panics... and after some struggle I've got most of it working, but I can't make syslog-ng (3.0 from git as of tonight) cooperate. The problem seems to be that netconsole is handing syslog-ng completely bare text lines over UDP, and syslog-ng is persistently trying to linebreak them at apparently random space characters, even with the no-parse flag turned on. Some specifics. I'm receiving the netconsole messages with the following (trimmed down from the real 500-line-long config, all global options preserved in this trim, I'm pretty sure this is everything relevant but I haven't tested this on its own yet 'cos it would break syslogging for a fairly large network to actually replace the syslog-ng config with this): ,---- | options | { | keep_hostname(no); | create_dirs(1); | perm(0644); | dir_perm(0755); | log_fifo_size(1000); | use_dns(yes); | dns_cache(yes); | use_fqdn(no); | log_iw_size(5192); | flush_lines(100); | flush_timeout(1000); | stats_freq(3600); | chain_hostnames(yes); | }; | | source netconsole | { | udp(ip(0.0.0.0) port(24183) log_fetch_limit(100) flags(no-parse no-multi-line)); | }; | | # A bare template, for netconsole messages. | template bare | { | template("$MSG\n"); | template_escape(no); | }; | | destination netconsole | { | file("/var/log/$HOST-netconsole.log" | template(bare)); | }; | | log | { | source(netconsole); | destination(netconsole); | }; `---- but this is not enough. Most of the time the same message is received over an ordinary syslog source, too, so I can compare the two. If the syslogged message reads: ,---- | Oct 20 21:17:53 mutilate warning: : [ 6783.237435] pktcdvd: Fixed packets, 32 blocks, Mode-2 disc | Oct 20 21:17:53 mutilate warning: : [ 6783.238541] pktcdvd: write caching control failed | Oct 20 21:17:53 mutilate warning: : [ 6783.238547] pktcdvd: 55 10 00 00 00 00 00 00 14 00 00 00 - sense 05.00.00 (Illegal request) `---- the netconsole message that's logged using the configuration above reads ,---- | [ 6783.237435] pktcdvd: Fixed packets, | 32 blocks, | Mode-2 disc | [ 6783.238541] pktcdvd: write caching control failed | [ 6783.238547] pktcdvd: | 55 | 10 | 00 | 00 | 00 | 00 | 00 | 00 | 14 | 00 | 00 | 00 | - | sense 05.00.00 | (Illegal request) `---- This is obviously not sane. Removing the no-multi-line flag doesn't change anything at all (I only added it in desperation, because I *want* linebreaks in the original message to be reproduced in the output: I just don't want *more* linebreaks than are present in the original output), so that's not it either. I'm logging kernel messages using ,---- | source local | { | unix-stream("/dev/log" flags(no-multi-line)); | internal(); | file("/proc/kmsg" program_override("kernel: ") flags(kernel no-multi-line)); | }; `---- and things hanging off that, without incident: no random linebreaks or anything. So, what am I missing? It's probably incredibly obvious, but I've blown several hours hacking at configs and digging through the syslog-ng source and have seen nothing to indicate what's going on...

2 3

syslog-ng on solaris locks up after a while
by Igor Manassypov 12 Nov '09

12 Nov '09

Hello, I am having an issue with a solaris installation of the syslog-ng. It is configured such that all the logs are stored different per-ip folders. This is my centralized logging device, so it is fairly heavily loaded with receiving logs from a few dozen hosts. The syslog-ng process locks up every two to three weeks, with no messages logging to any of the files. The only way of getting it back is kill -9 the process and restart it. Is there any known issue of same sorts and is there any other way around it other than recycling the daemon every night? here is the version info: bash-3.00# syslog-ng --version syslog-ng 3.0.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 Compile-Date: Aug 11 2009 10:44:17 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: off Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on bash-3.00# uname -a SunOS prelude 5.10 Generic_137137-09 sun4v sparc SUNW,T5240 Thanks! -igor Igor Manassypov., M.Eng, P.Eng, CCIE 23032, CCVP Network Architect

3 11

Prefiltering
by Kārlis Repsons 12 Nov '09

12 Nov '09

People, is there currently some good way to filter source and then use it *filtered* for numerous "log" invocations in configuration file? If no, it takes writing the same thing many times...

4 4

syslog-ng 3.0 Administrator Guide, 9th Edition
by Robert Fekete 12 Nov '09

12 Nov '09

Dear syslog-ng users, A new version of the syslog-ng 3.0 Administrator Guide is available on our website. You can read about the changes in detail at the BalaBit Documentation Blog at http://robert.blogs.balabit.com/2009/11/syslog-ng-administrator-guide-9th.h… Kind Regards, Robert Fekete

2 1

rm /var/log/{dmesg,lastlog,wtmp,faillog,..?}?!
by Kārlis Repsons 11 Nov '09

11 Nov '09

If one has set up a dedicated syslog-ng configuration and wants to remove files like those, what would happen? Can anyone list the software entities responsible for writing to those? Has anyone configured a system, which doesn't needs them?

3 3

Using pipes for prefiltering?
by Kārlis Repsons 10 Nov '09

10 Nov '09

Hello, Recently I've been trying to squeeze out the most of syslog-ng 3.0 and one of problems pestering me is this: there are some messages, which I would like never to appear in any log file, so I created filters for them, directed real sources through them to pipe destination and created prefiltered sources pointing to those same pipes. Well and what happens is - some logs reach the final destinations, but some not. For example, nothing at all now gets logged form kernel through pipe, even if filters get thrown out! Its on Linux... Also I had to disable syslogng logs, because otherwise CPU was loaded 100% sometimes (one core). After all, how should I do prefiltering? Strange thing is, that even with pipes first of these works, while the other doesn't: log { source(s_compname_usp_f); source(s_compname_kern_f); destination(d_compname_vt12); }; log { source(s_compname_kern_f); filter(f__kern); destination(d_compname_kern); }; filter f__kern { facility(0); }; You can see pipes fragment below. Hope someone has suggestion... Best, Kārlis # prefiltering # destination d_compname_usp_f { pipe("/var/log/filtered-source"); }; destination d_compname_kern_f { pipe("/var/log/filtered-kernel-source"); }; # abusers/spammers dropping here # filter f_compname_antispam_usp { ( not ( program(upsd) and message("some shit there") ) ) and ( not program("syslog") ) and ( not ( program("cron") and ( message(".*rm \-f /var/spool/cron/lastrun/cron\..*") or message(".*test \-x /usr/sbin/run-crons \&\& /usr/sbin/run\-crons.*") ) ) ) and ( not ( program("acpid") and ( message(".*client connected from.*") or message(".*1 client rule loaded") or message(".*client .* has disconnected") ) ) ) ; }; filter f_compname_antispam_kern { ( not message(".*ata1: exception .*") ) and ( not message(".*ata1: .*connection status changed") ) and ( not message(".*Hangcheck\: hangcheck value past margin\!") ) ; }; log { source(s_compname_usp); filter(f_compname_antispam_usp); destination(d_compname_usp_f); }; log { source(s_compname_kern); destination(d_compname_kern_f); }; source s_compname_usp_f { pipe("/var/log/filtered-source"); }; source s_compname_kern_f { pipe("/var/log/filtered-kernel-source"); };

1 1

Re: [syslog-ng] Upper case $HOST
by Pallagi Zoltán 09 Nov '09

09 Nov '09

Tim Boyer írta: > Pallagi Zoltán wrote: >> Hi, >> >> Tim Boyer írta: > >>> Balazs - >>> >>> Looks like I spoke too soon. Something odd is happening. >>> >>> I put the option into the conf file: >>> >>> @version: 3.0 >>> # >>> # global options >>> # >>> >>> options { >>> normalize_hostnames(yes); >>> use_fqdn(no); >>> use_dns(yes); >>> dns_cache(yes); >>> keep_hostname(yes); >>> long_hostnames(off); >>> create_dirs(yes); >>> } >>> >>> > > <snip> > >> You should not use "keep_hostname" in the part of global options >> because this one will block rewriting of the hostname (see syslog-ng >> admin guide: >> http://www.balabit.hu/dl/html/syslog-ng-v3.0-guide-admin-en.html/ch08s09.ht…) >> > > Pallagi - > > Of course - that makes perfect sense; I'm asking it to do two > contradictory actions. Thanks very much... Anyway, I think this is also a problem so I'v just opened a bugzilla ticket: https://bugzilla.balabit.com/show_bug.cgi?id=60 > >

1 0

Upper case $HOST
by Tim Boyer 09 Nov '09

09 Nov '09

I'm running syslog-ng 3.03 on a RHEL5 system, sending logs to various files like so: # global log files destination deservers { file("/var/log/$HOST.log" owner(root) group(hobbit) perm(0640)); }; It's working fine on the Linux boxes. But I'm using Adiscon's EventLog on my Windows machines, and the $HOST name on them are coming up in all caps. [root@buran log]# ls -la /var/log/*.log ... -rw-r----- 1 root hobbit 282 Nov 4 14:37 /var/log/KANTECH.log -rw-r----- 1 root hobbit 535181 Nov 4 14:24 /var/log/PLCDATA.log Where's this $HOST macro get its data from? The DNS entry is lower case; the full computer name on the Windows box is lower case. I don't see where the upper is coming from, unless it's one of those weird Windows-to-Unix translation things. Not a big deal, but a bit of an annoyance. Thanks for any help. -- Tim Boyer Denman Tire Corporation

5 6

Macro question
by Michael J. Bauer 06 Nov '09

06 Nov '09

I am running syslog-ng 3.0.4 on RHEL 5.4. I have a log message that appears in my logfiles as Oct 28 16:41:22 juniper-router {wan-service-set}[FWNAT]: ASP_NAT_RULE_MATCH: proto 6 (TCP) application: any, ge-0/0/3.2:10.3.13.153:49818 -> 66.249.80.148:80, Match NAT rule-set: , rule: nat-outgoing, term: dynamic-nat There are no carriage returns in that entry; any that appear are artifacts of some mail server or client along the way. I want any log entry containing the string FWNAT to go into a separate file. match(FWNAT) on the filter does just that, but syslog-ng complains that I'm not using value() for efficiency. However, it's not at all clear which macro, if any, actually contains the string FWNAT. I can tell you five that don't: APPLICATION EVENT_SOURCE MESSAGE MSG PROGRAM What macro contains FWNAT? And where can I get a definitive list of macros and specifics on how the value in each is set for each line? The documentation is insufficiently detailed on this subject. Thanks, MJB

3 4

$HOST macro and relay host
by Jason Barrett 06 Nov '09

06 Nov '09

Hi all, I'm relaying log messages from one syslog-ng server to another. At the final stop, the only way I can get the $HOST macro to work is if I enable dns resolution on server 1. Is this by design? Here are the relevant configs: ---------------------------------------------------------------------- syslog-ng server 1 (relays to server 2): chain_hostnames(yes); keep_hostname(yes); use_dns(no); source s_udp { udp(port(514)); }; destination df_udpback { udp("192.168.1.157" port(514)); }; log { source(s_udp); destination(df_udpback); }; ---------------------------------------------------------------------- syslog-ng server 2: chain_hostnames(yes); keep_hostname(yes); use_dns(yes); source s_udp { udp(port(514)); }; destination df_udp { file("/var/log/ics/$HOST/$YEAR/$MONTH/$DAY"); }; log { source(s_udp); destination(df_udp); }; ---------------------------------------------------------------------- Sample log message on server 2: Oct 30 09:35:03 10.12.24.46/10.12.24.46 %ASA-5-111005: 10.28.22.55 end configuration: OK 10.12.24.46 is the correct IP address of the originating host, and $HOST resolves to this IP address. I would prefer $HOST to resolve to the hostname as it exists in the /etc/hosts file. Thanks!

3 6