August 2006 - syslog-ng - lists.balabit.hu

Pipe to MYSQL does not work
by Matthew Redel 25 Aug '06

25 Aug '06

Running 1.6.9 version of syslog-ng. libol version 3.9 OS is Redhat Fedora 4. mysql version is 4.1.12 . Below is my config: *options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); };* *source s_net { udp(ip(0.0.0.0) port(514)); };* *destination d_mysql { pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, f acility, priority, level, tag, date, time, program, msg) VALUES ('$HOST', '$FACI LITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$ PROGRAM', '$MSG');\n") template-escape(yes)); };* *log { source(s_net); destination(d_mysql); };* Here is the script used to pipe /tmp/mysql.pipe into databse: *#!/bin/bash* *if [ -e /tmp/mysql.pipe ]; then while [ -e /tmp/mysql.pipe ] do mysql -u syslog --password=password syslog < /tmp/mysql.pipe done else mkfifo /tmp/mysql.pipe fi* I have confirmed that I can redirect echo to the pipe, so I know the pipe is working. I've even chmod 777 the pipe. I have also confirmed that log messages are coming in over the network using tcpdump and when I use syslog-ng to log to file, it works just fine. Why doesn't syslog-ng output to the pipe, argh!? Any help would be greatly appreciated. Matt

2 1

RE: [syslog-ng] FQDN's in syslog-ng
by Stenglein, David 24 Aug '06

24 Aug '06

Russell and Steven, There are problems with trying to use macros, as you have found. There is a good bit of detail on this page (part of the FAQ, I believe) http://www.campin.net/syslog-ng/syslog.html. The problem I had was that I started to get strange directory names, like "cd .." and "::". All sorts of garbage. The reason? The Sun syslog service produces non-RFC compliant network packets. The packet is supposed to have the hostname twice, and then the ID field. Syslog-ng can deal with the second host not being there, but if it is missing the square brackets, there is no way for it to figure out where the message begins. So, you can use HOST_FROM and FULLHOST_FROM in your file output. Syslog-ng then logs everything very neatly, which is what the doc says. That's all well and good, but what about your remote hosts that have a forwarder in between? Sun syslog won't work for this anyway (doesn't handle the origin name correctly), so you use syslog-ng. Fine. But now you can't use _FROM macros, since that will be wrong too. What you will need to do is have the remote syslog-ng forwarders sending to the central loghost over a separate port (might as well be TCP). Last, you will need a separate parallel set of rules that use FULLHOST instead of FULLHOST_FROM in the output path when the source is a remote syslog-ng server (which will forward correctly formatted messages). Did I mention timestamps? ;-) -Dave -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Russell Fulton Sent: Thursday, August 24, 2006 4:18 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] FQDN's in syslog-ng There are a several of different options for dealing with host names and I have not managed to sort them all out yet. The single line explanations in the manual are not enough to work out the details. Be aware that the hostname can be set on either the central log server or on the system generating the logs. The default seems to be that the central server will keep the host name that is in the packet, if there is no hostname in the packet it will do a reverse lookup to determine it. We want to override this but have not spent a lot of time fiddling with the options to see what works. What you want to do is possible it just a matter of figuring out which options you need on the client and server. What is bothering us at the moment is that SUN monitoring crap called SRS manages to generate some syslog records with SRS in the hostname field. So on the central server these all turn up in the same file. I really must sort this out today, sigh... I'd be happy to hear from someone who has already sussed out how these options work! Russell Steven Hajducko wrote: > Hi everyone, > > We've currently got a syslog setup that centralizes our logs from many > of our different teirs into one location, using each system's prebuilt > syslog. It works at the moment, but we've got some issues. > > Our main one is that we name our hosts the same in different > environments, mainly because these environments are for moving code and > configs from 1 step to the next, so to make it easier on people so they > don't have to change configurations each time the code moves, hosts > share similar names. > > The problem is that when all the logs come back to the central syslog > server, they can't be seperated by host because of the similiarities. > So 'web5' in our QA is the same as 'web5' in production. > > Because of that, we were looking at syslog-ng and while I found a couple > references to using FQDN, I've been unable to tell whether or not this > is possible. > > Our machines DO have different FQDNs. For example, > web5.qa.location.domain.com vs. web5.prod.location.domain.com. If we > run our main central server on syslog-ng and replace all the syslog > daemon's on each system with it, can we force the FQDN to be given > rather then just the base hostname and have the central log server sort > the logs into different directories based on the FQDN? > > Thanks! > > -- > sh > > > ------------------------------------------------------------------------ > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

Bug? Dual destinations on Solaris
by Richard Whelan 24 Aug '06

24 Aug '06

Hi All, I've come across what looks like a bug. Whether it's syslog-ng, stunnel or Solaris I can't tell. I'm also going to report this to Michal (Stunnel). The setup/problem is as follows: I'm effectively re-creating a syslog-ng client/server setup, except, the traffic is being encrypted using Stunnel, and it's talking to two servers simultaneously for redundancy. In order for this to happen, I have stunnel set to listen to two separate ports on the client, and syslog-ng set to send to the two separate ports. The configuration, parts of which are below, are working fine with 6 or so Linux clients, and also appears to work fine with Solaris 8, but not under Solaris 9. The config for the stunnel side is: stunnel-4.15 client = yes cert = /usr/local/ssl/certs/syslog-ng-client.pem CAfile = /usr/local/ssl/certs/syslog-ng-server.pem chroot = /var/run/stunnel pid = /run/stunnel.pid setuid = stunnel setgid = stunnel verify = 3 debug = 7 output = /var/log/stunnel.log [5140] accept = 127.0.0.1:5140 connect = x.x.x.x:5140 TIMEOUTclose = 0 [5141] accept = 127.0.0.1:5141 connect = y.y.y.y:5140 TIMEOUTclose = 0 As you can see it's set to listen to ports 5140 and 5141 on the localhost. If you start this first, the process runs fine, and shows up with a netstat -a: localhost.5140 *.* 0 0 32768 0 LISTEN localhost.5141 *.* 0 0 32768 0 LISTEN The config for the syslog-ng process is: syslog-ng-1.6.11 options { sync (0); keep_hostname (yes); use_fqdn (yes); chain_hostnames (no); use_dns (no); time_reopen (10); time_reap (120); log_fifo_size (2048); }; source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; destination ....... destination d_stunnel { tcp ("127.0.0.1" port(5140)); }; destination d_stunnel2 { tcp ("127.0.0.1" port(5141)); }; Here showing that two of the destinations are to tcp ports 5140 and 5141 on the localhost. It seems reasonable enough, and as I mentioned does work on Linux. However as soon as the syslog-ng process starts, the stunnel process is either killed or just dies. The netstat then shows: localhost.64142 localhost.5141 32768 0 32768 0 CLOSE_WAIT localhost.5141 localhost.64142 32768 0 32768 0 FIN_WAIT_2 localhost.64143 localhost.5140 32768 0 32768 0 CLOSE_WAIT localhost.5140 localhost.64143 32768 0 32768 0 FIN_WAIT_2 and after a while you are just left with the CLOSE_WAIT lines. I've tried this on two separate Solaris 9 systems, both the same result. One actually producing a core dump of the stunnel process. Has anyone seen anything like this, or can shed any light as to why this may not be working. Thanks, Richard -- Richard Whelan Senior Systems & NMS Administrator Pipex Communications Tel: +44 (0) 1865 381568 Mob: +44 (0) 7786 276020 Web: http://www.pipex.com This e-mail is subject to: http://www.pipex.net/disclaimer.html

1 1

Cygwin bind_unix_socket(): bind failed /dev/log
by Alessio Massaro 24 Aug '06

24 Aug '06

Hi, On cygwin syslog-ng fails to start with the following in /var/log/syslog-ng.log io.c: bind_unix_socket(): bind failed /dev/log (Bad file descriptor) Error initializing configuration, exiting. Any help is greatly appreciated. Thanks I have a virgin intallation created by syslog-ng-config which creates the following /etc/syslog-ng.conf options { keep_hostname(yes); chain_hostnames(no); owner("system"); group("root"); perm(0664); sync(0); }; source applications { unix-dgram("/dev/log"); internal(); }; source kernel { file("/dev/kmsg", log_prefix("kernel: ")); }; destination messages { file("/var/log/messages"); }; log { source(applications); destination(messages); }; log { source(kernel); destination(messages); };

1 0

netmask
by Werner Thal 24 Aug '06

24 Aug '06

Hallo, I try to use the "netmask" filter option to generate seperate logfiles for different subnets. I tried this filter krzn_all { netmask("10.254.0.0/255.255.0.0"); }; this filter krzn_all { netmask(10.254.0.0/255.255.0.0); }; this filter krzn_all { netmask("10.254.0.0/16"); }; and this filter krzn_all { netmask(10.254.0.0/16); }; No errors have been reported starting syslog-ng but also no messages have been logged. I use syslog-ng 2.0rc1-2 on a debian Etch. Kind Regards W. Thal

2 2

PIX VPN logging
by Brian Loe 23 Aug '06

23 Aug '06

Anyone here have a complete list of VPN related syslog messages they'd like to share? I'm essentially wanting to monitor for site-to-site tunnels going down so that I can alert on them, but having a hell of a time finding exactly what I want to look for on the Cisco site. Part of the problem is that I won't have an example of such an event until it happens - and I've only just now implemented a syslog server capable of maintaining the logs.. At any rate, if anyone here is monitoring for this as well and you're willing to share...let me know!

2 4

RE: [syslog-ng] Performance tuning questions
by King, John (Greg) (LMIT-HOU) 23 Aug '06

23 Aug '06

Yeah the only other services than syslog are sshd and iptables, nothing else was installed, the only worry I have with moving from udp to tcp for such a large number of messages would be denial of service. I recall some years ago some mmorpg tried using tcp instead of udp for client data and it brought their system(s) to a halt. Since I am looking at (on avg) 26000 messages per minute I am worried about what this will do not only to the syslog server performance but to the agents as well (and thus the servers the agents reside on). I am still trying to get time to work on the lab syslog-ng system to convert redhat es 4 syslog configs into syslog-ng so the standard logging format for the system is not lost. Then create a rule for tcp based syslog and slowly ramp up traffic and see what happens in the lab to get an idea on what to expect. -Greg -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Kevin Sent: Wednesday, August 23, 2006 10:41 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Performance tuning questions Both TCP and UDP have risks and limitations. If message loss/spoofing are important to you, TCP is the way to go. (One key exception being logs from PIX firewalls :) Is the log server handling other services as well? Kevin _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

Config template for redhat es 4
by King, John (Greg) (LMIT-HOU) 22 Aug '06

22 Aug '06

Has anyone created a redhat es/as/ws 4 config file? I saw the Debian example and I was going to start working on converting it, but to save some time I was curious if anyone else has done this and would share their template. Thanks Greg

1 0

RE: [syslog-ng] Newbie Question
by William Bell 21 Aug '06

21 Aug '06

You should probably create multiple sources. Like: source remote { udp(ip("0.0.0.0") port(514)); }; Then send the remote files to a nice archive directive destination ala: destination remotelogs { file("/var/REMOTE/$HOST.log" owner(root) group(root) perm(0755) dir_perm(0755) create_dirs(yes)); }; Finish it up with its own log directive like: log { source(remote); destination(remotelogs); }; Adding any filters you want to the log directive above. I hope this helps. William L. Bell II CWIE Security Dept. _____ From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Donald Rush Sent: Monday, August 21, 2006 10:18 AM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] Newbie Question All, I am working on creating a logging host for some network hardware. Being new to lunix and syslog-ng I am a little confused to how to keep the remote syslog messages from showing up in the system log files. I had to rem out the messages and and localmessages do to they are filling to fast. Any advice would be greatly appreciated. Don R. rushd-linux:/home/rushd # cat /etc/syslog-ng/syslog-ng.conf # # /etc/syslog-ng/syslog-ng.conf # # Automatically generated by SuSEconfig on Thu Aug 17 16:58:59 PDT 2006. # # PLEASE DO NOT EDIT THIS FILE! # # you can modify /etc/syslog-ng/syslog-ng.conf.in instead # # # # File format description can be found in syslog-ng.conf(5) # and /usr/share/doc/packages/syslog-ng/syslog-ng.txt. # # # Global options. # options { long_hostnames(off); sync(0); perm(0640); stats(3600); }; # # 'src' is our main source definition. you can add # more sources driver definitions to it, or define # your own sources, i.e.: # #source my_src { .... }; # source src { # # include internal syslog-ng messages # note: the internal() soure is required! # internal(); # # the following line will be replaced by the # socket list generated by SuSEconfig using # variables from /etc/sysconfig/syslog: # unix-dgram("/dev/log"); # # uncomment to process log messages from network: # udp(ip("0.0.0.0") port(514)); }; # # Filter definitions # filter f_iptables { facility(kern) and match("IN=") and match("OUT="); }; filter f_console { level(warn) and facility(kern) and not filter(f_iptables) or level(err) and not facility(authpriv); }; filter f_newsnotice { level(notice) and facility(news); }; filter f_newscrit { level(crit) and facility(news); }; filter f_newserr { level(err) and facility(news); }; filter f_news { facility(news); }; filter f_mailinfo { level(info) and facility(mail); }; filter f_mailwarn { level(warn) and facility(mail); }; filter f_mailerr { level(err, crit) and facility(mail); }; filter f_mail { facility(mail); }; filter f_cron { facility(cron); }; filter f_local { facility(local0, local1, local2, local3, local4, local5, local6, local7); }; filter f_acpid { match('^\[acpid\]:'); }; filter f_netmgm { match('^NetworkManager:'); }; filter f_messages { not facility(news, mail) and not filter(f_iptables); }; filter f_warn { level(warn, err, crit) and not filter(f_iptables); }; filter f_alert { level(alert); }; # # Most warning and errors on tty10 and on the xconsole pipe: # destination console { file("/dev/tty10" group(tty) perm(0620)); }; log { source(src); filter(f_console); destination(console); }; destination xconsole { pipe("/dev/xconsole" group(tty) perm(0400)); }; log { source(src); filter(f_console); destination(xconsole); }; # Enable this, if you want that root is informed immediately, # e.g. of logins: # #destination root { usertty("root"); }; #log { source(src); filter(f_alert); destination(root); }; # # News-messages in separate files: # destination newscrit { file("/var/log/news/news.crit" owner(news) group(news)); }; log { source(src); filter(f_newscrit); destination(newscrit); }; destination newserr { file("/var/log/news/news.err" owner(news) group(news)); }; log { source(src); filter(f_newserr); destination(newserr); }; destination newsnotice { file("/var/log/news/news.notice" owner(news) group(news)); }; log { source(src); filter(f_newsnotice); destination(newsnotice); }; # # and optionally also all in one file: # (don't forget to provide logrotation config) # #destination news { file("/var/log/news.all"); }; #log { source(src); filter(f_news); destination(news); }; # # Mail-messages in separate files: # destination mailinfo { file("/var/log/mail.info"); }; log { source(src); filter(f_mailinfo); destination(mailinfo); }; destination mailwarn { file("/var/log/mail.warn"); }; log { source(src); filter(f_mailwarn); destination(mailwarn); }; destination mailerr { file("/var/log/mail.err" fsync(yes)); }; log { source(src); filter(f_mailerr); destination(mailerr); }; # # and also all in one file: # destination mail { file("/var/log/mail"); }; log { source(src); filter(f_mail); destination(mail); }; # # acpid messages in one file: # destination acpid { file("/var/log/acpid"); }; log { source(src); filter(f_acpid); destination(acpid); flags(final); }; # # NetworkManager messages in one file: # destination netmgm { file("/var/log/NetworkManager"); }; log { source(src); filter(f_netmgm); destination(netmgm); flags(final); }; # # Cron-messages in one file: # (don't forget to provide logrotation config) # #destination cron { file("/var/log/cron"); }; #log { source(src); filter(f_cron); destination(cron); }; # # Some boot scripts use/require local[1-7]: # ##destination localmessages { file("/var/log/localmessages"); }; ##log { source(src); filter(f_local); destination(localmessages); }; # # All messages except iptables and the facilities news and mail: # ##destination messages { file("/var/log/messages"); }; ##log { source(src); filter(f_messages); destination(messages); }; # # Firewall (iptables) messages in one file: # destination firewall { file("/var/log/firewall"); }; log { source(src); filter(f_iptables); destination(firewall); }; # # Warnings (except iptables) in one file: # destination warn { file("/var/log/warn" fsync(yes)); }; log { source(src); filter(f_warn); destination(warn); }; # # Enable this, if you want to keep all messages in one file: # (don't forget to provide logrotation config) # #destination allmessages { file("/var/log/allmessages"); }; #log { source(src); destination(allmessages); }; destination logip { file("/data/logs/HOSTS/$HOST_FROM/$FACILITY/$YEAR$MONTH/$FACILITY$YEAR$M ONTH$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(src); destination(logip); }; rushd-linux:/home/rushd #

2 1

Newbie Question
by Donald Rush 21 Aug '06

21 Aug '06

All, I am working on creating a logging host for some network hardware. Being new to lunix and syslog-ng I am a little confused to how to keep the remote syslog messages from showing up in the system log files. I had to rem out the messages and and localmessages do to they are filling to fast. Any advice would be greatly appreciated. Don R. rushd-linux:/home/rushd # cat /etc/syslog-ng/syslog-ng.conf # # /etc/syslog-ng/syslog-ng.conf # # Automatically generated by SuSEconfig on Thu Aug 17 16:58:59 PDT 2006. # # PLEASE DO NOT EDIT THIS FILE! # # you can modify /etc/syslog-ng/syslog-ng.conf.in instead # # # # File format description can be found in syslog-ng.conf(5) # and /usr/share/doc/packages/syslog-ng/syslog-ng.txt. # # # Global options. # options { long_hostnames(off); sync(0); perm(0640); stats(3600); }; # # 'src' is our main source definition. you can add # more sources driver definitions to it, or define # your own sources, i.e.: # #source my_src { .... }; # source src { # # include internal syslog-ng messages # note: the internal() soure is required! # internal(); # # the following line will be replaced by the # socket list generated by SuSEconfig using # variables from /etc/sysconfig/syslog: # unix-dgram("/dev/log"); # # uncomment to process log messages from network: # udp(ip("0.0.0.0") port(514)); }; # # Filter definitions # filter f_iptables { facility(kern) and match("IN=") and match("OUT="); }; filter f_console { level(warn) and facility(kern) and not filter(f_iptables) or level(err) and not facility(authpriv); }; filter f_newsnotice { level(notice) and facility(news); }; filter f_newscrit { level(crit) and facility(news); }; filter f_newserr { level(err) and facility(news); }; filter f_news { facility(news); }; filter f_mailinfo { level(info) and facility(mail); }; filter f_mailwarn { level(warn) and facility(mail); }; filter f_mailerr { level(err, crit) and facility(mail); }; filter f_mail { facility(mail); }; filter f_cron { facility(cron); }; filter f_local { facility(local0, local1, local2, local3, local4, local5, local6, local7); }; filter f_acpid { match('^\[acpid\]:'); }; filter f_netmgm { match('^NetworkManager:'); }; filter f_messages { not facility(news, mail) and not filter(f_iptables); }; filter f_warn { level(warn, err, crit) and not filter(f_iptables); }; filter f_alert { level(alert); }; # # Most warning and errors on tty10 and on the xconsole pipe: # destination console { file("/dev/tty10" group(tty) perm(0620)); }; log { source(src); filter(f_console); destination(console); }; destination xconsole { pipe("/dev/xconsole" group(tty) perm(0400)); }; log { source(src); filter(f_console); destination(xconsole); }; # Enable this, if you want that root is informed immediately, # e.g. of logins: # #destination root { usertty("root"); }; #log { source(src); filter(f_alert); destination(root); }; # # News-messages in separate files: # destination newscrit { file("/var/log/news/news.crit" owner(news) group(news)); }; log { source(src); filter(f_newscrit); destination(newscrit); }; destination newserr { file("/var/log/news/news.err" owner(news) group(news)); }; log { source(src); filter(f_newserr); destination(newserr); }; destination newsnotice { file("/var/log/news/news.notice" owner(news) group(news)); }; log { source(src); filter(f_newsnotice); destination(newsnotice); }; # # and optionally also all in one file: # (don't forget to provide logrotation config) # #destination news { file("/var/log/news.all"); }; #log { source(src); filter(f_news); destination(news); }; # # Mail-messages in separate files: # destination mailinfo { file("/var/log/mail.info"); }; log { source(src); filter(f_mailinfo); destination(mailinfo); }; destination mailwarn { file("/var/log/mail.warn"); }; log { source(src); filter(f_mailwarn); destination(mailwarn); }; destination mailerr { file("/var/log/mail.err" fsync(yes)); }; log { source(src); filter(f_mailerr); destination(mailerr); }; # # and also all in one file: # destination mail { file("/var/log/mail"); }; log { source(src); filter(f_mail); destination(mail); }; # # acpid messages in one file: # destination acpid { file("/var/log/acpid"); }; log { source(src); filter(f_acpid); destination(acpid); flags(final); }; # # NetworkManager messages in one file: # destination netmgm { file("/var/log/NetworkManager"); }; log { source(src); filter(f_netmgm); destination(netmgm); flags(final); }; # # Cron-messages in one file: # (don't forget to provide logrotation config) # #destination cron { file("/var/log/cron"); }; #log { source(src); filter(f_cron); destination(cron); }; # # Some boot scripts use/require local[1-7]: # ##destination localmessages { file("/var/log/localmessages"); }; ##log { source(src); filter(f_local); destination(localmessages); }; # # All messages except iptables and the facilities news and mail: # ##destination messages { file("/var/log/messages"); }; ##log { source(src); filter(f_messages); destination(messages); }; # # Firewall (iptables) messages in one file: # destination firewall { file("/var/log/firewall"); }; log { source(src); filter(f_iptables); destination(firewall); }; # # Warnings (except iptables) in one file: # destination warn { file("/var/log/warn" fsync(yes)); }; log { source(src); filter(f_warn); destination(warn); }; # # Enable this, if you want to keep all messages in one file: # (don't forget to provide logrotation config) # #destination allmessages { file("/var/log/allmessages"); }; #log { source(src); destination(allmessages); }; destination logip { file("/data/logs/HOSTS/$HOST_FROM/$FACILITY/$YEAR$MONTH/$FACILITY$YEAR$M ONTH$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); }; log { source(src); destination(logip); }; rushd-linux:/home/rushd #

1 0