December 2005 - syslog-ng - lists.balabit.hu

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

ignore src{}; contents at startup?
by Brian A. Seklecki 25 Oct '06

25 Oct '06

All: On FreeBSD, /dev/klog provides for output from the kernel (normally redirected to /dev/console before syslog starts). At startup, /etc/rc.d/dmesg copies the kernel boot hardware paramters to /var/run/dmesg.boot. If I have my src(); set to: source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ...at starup, syslog-ng drains the contents of /dev/klog. By default it tsends to them to user.notice. I like to log{} these to an SMS pager (so I know if a RAID is failing or other hardware problem). However, I'd like to avoid duplication of the efforts of /etc/rc.d/dmesg on behalf of syslog-ng(8) ...especially since I route user.notice to my pager, and kernel boot dmesg(8) can grow over 50 lines. /dev/klog really isn't a socket. It's a special char device. Is there anyway to tell syslog-ng(8) to ignore it's contents at startup? If it was tail(1), i'd say "tail -0 -f /file". Perhaps clear out the contents of /dev/klog somehow? TIA, ~BAS

2 2

RE: [syslog-ng]Pix problem
by SOLIS, ALEX 05 Jan '06

05 Jan '06

If you don't have too many assets to log from, I found that filtering by facility proved an effective logging method. 1. Setup your pix to send all its logs via a certain facility. For example local0. The pix command will look something like "logging facility 16". (note: Facilities have a corresponding ID in PIX IOS. Local0 starts at 16 and increments all the way to local7 which is 23.) 2. Setup the PIX to send its logs via UDP to the loghost server. Assuming Your loghost is on the inside of your PIX the command will look something like this: "logging host inside x.x.x.x". Be careful if you plan to use TCP for logging as the PIX will not allow communication between its interfaces if your syslog server fails! 3. Now modify the syslog-ng.conf file on you loghost with the appropriate destination, filter, and log statements. For example, to capture the PIX logs and send them to a private file we would have: destination pix { file("/var/log/pix"); }; filter f_pix { facility(local0); }; log { source(src); filter(f_pix); destination(pix); }; restart syslog-ng and you should be in business. If you don't see enough information (or too much) play with the PIX's logging levels. There are 7 of them with 7 being the most verbose. Hope that helps. Alex -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Andrew Meyer Sent: Tuesday, December 27, 2005 9:55 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng]Pix problem OK.....is there a way to log/define all 8 logging facilities in syslog-ng this is all new to me, and i'm following a debian tutorial. So any other info you can give me would be awsome. Thank you, Andrew --- Asher Yanich <ayanic01(a)cs.fiu.edu> wrote: > You will not want to setup TCP syslog from the PIX. > Even though the > PIX can connect and log via TCP to syslog, any error > will cause the > PIX to stop logging to syslog. > > Here are docs for setting up logging to syslog for > pix OS 7.0. I > believe the syntaz is the same for 5.x and 6.x but a > quick google > search should provide you with what you need. > > -asher > > > On 12/27/05, Andrew Meyer <andrewm659(a)yahoo.com> > wrote: > > How do I setup a PIX with syslog-ng? what are the > > values I can use? Do I need to specify the IP > addy of > > the PIX? What about the protocol I want to > capture, > > what if i want to capture UDP and TCP? How would > I > > set it up if i wanted all the data from my PIX to > go > > the syslog-ng server? > > > > Thank you, > > Andrew Meyer > > andrewm659(a)yahoo.com > > > > --- Damien Michau <d.michau(a)ag.com> wrote: > > > > > this is the server actual ip > > > ----- Original Message ----- > > > From: "Bill Nash" <billn(a)billn.net> > > > To: <syslog-ng(a)lists.balabit.hu> > > > Sent: Tuesday, May 10, 2005 7:20 PM > > > Subject: Re: [syslog-ng]Pix problem > > > > > > > > > > On Tue, 10 May 2005, Damien Michau wrote: > > > > > > > >> Hi All ! > > > >> I have some probleme ta log my pix's log into > my > > > syslog-ng server . > > > >> i have mounted a Syslog-ng server to store my > > > pix's log . But there is > > > >> nothing in my pix.log > > > >> i'have put this lines in my syslog-ng.conf > > > >> > > > >> > > > >> source pix { udp(ip(10.60.10.111) port());}; > > > >> destination pix { > file("/var/log/pix.log"); > > > }; > > > >> log { source(pix);destination(pix); }; > > > >> > > > > > > > > Your source declaration shouldn't be the IP of > > > your pix, it should be the > > > > IP on your syslog server you wish your udp > socket > > > to listen on. In most > > > > cases, this should just be 0.0.0.0 or your > servers > > > actual IP. > > > > > > > > - billn > > > > > _______________________________________________ > > > > syslog-ng maillist - > syslog-ng(a)lists.balabit.hu > > > > > > > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > Frequently asked questions at > > > http://www.campin.net/syslog-ng/faq.html > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > syslog-ng maillist - > syslog-ng(a)lists.balabit.hu > > > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > Frequently asked questions at > > > http://www.campin.net/syslog-ng/faq.html > > > > > > > > > > > > I'm worth a million in prizes > > With my torture film > > Drive a GTO > > Wear a uniform > > On a government loan. > > > > > > > > __________________________________________ > > Yahoo! DSL ï¿½ Something to write home about. > > Just $16.99/mo. or less. > > dsl.yahoo.com > > > > _______________________________________________ > > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Frequently asked questions at > http://www.campin.net/syslog-ng/faq.html > > > > > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at > http://www.campin.net/syslog-ng/faq.html > > I'm worth a million in prizes With my torture film Drive a GTO Wear a uniform On a government loan. __________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html <p class=MsoNormal><span style='font-size:8.5pt'>This e-mail contains Omaha Public Power District's confidential and proprietary information and is for use only by the intended recipient. Unless explicitly stated otherwise, this e-mail is not a contract offer, amendment, nor acceptance. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.</p>

6 11

SSL encapsulation for TCP transport?
by Bill Nash 03 Jan '06

03 Jan '06

Is it planned? I'm lookin around, and see previous mention of it, but nothing concrete. Yes, I know you can just tunnel the session, but I'd prefer native transport. - billn

3 2

rhost field
by ken.schweiker＠faa.gov 29 Dec '05

29 Dec '05

I am using syslog-ng for the first time. The initial setup is complete and appears to be working ok. However in my test environment, I am logging from a redhat desktop using syslog to a suse syslog-ng server. The output, after I deliberately input an incorrect password on my workstation doing a 'su', gets reported to the syslog-ng server as ....... Dec 23 17:50:12 suselog/suselog su(pam_unix)[13205]: authentication failure; logname=syss555 uid=500 euid=0 tty=pts/4 ruser=syss555 rhost= user=root How do I get it to display in the log the host(IP) the message came from? Thanks.

4 11

FC4 as Syslog-NG Server (to MySql)
by Vadim Pushkin 28 Dec '05

28 Dec '05

Hello; I am trying to get syslog-ng running as a centralized server, with data going to a mysql db. So far I've: 1. Installed mysql and have sucessfully created a db named syslogng. (confirmed with a show databases; and show tables;) 2. I have created a FIFO file as /tmp/mysql.pipe. 3. Am running mysql in the following manner: mysql -u username --password=userpassword syslogng </tmp/mysql.pipe I do not think that I've properly configured my syslog-ng.conf file to redirect all output to the DB. The FAQ, which has been very nice to me thus far, is somewhat confusing as to how to properly create the .conf file for usage this way. All test writes never appear within my DB, confirmed by running "mysql> select * from logs;". Ultimately, my goal is to log syslog data from routers, switches, Solaris/Sparc, Linux machines to this machine. The machine is within a trusted environment. Any help would be much appreciated, and I apologize in advance if I've misread the obvious somewhere... Vadim Anatoly Pushkin -- The Ukranian Stallion --

2 1

file written in the wrong place
by Paolo Supino 28 Dec '05

28 Dec '05

Hi I've installed syslog-ng 1.6.8 on a Linux FC4 system to act as a centeralized logging server. For a while everything was fine, until I tried to add another statement to one of the filters. After the addition syslog-ng started writing the destination files in the wrong places, writing the same file in 2 locations and ignoring messages sent from certain origins. Here are the relevant parts of my syslog-ng.conf: # Global options options { sync (0); time_reopen (1); log_fifo_size (999999); long_hostnames (yes); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); }; # source source s_net { udp(ip("192.168.1.100") port(514)); }; # destinations destination d_switch { file("/var/log/company/switches/$HOST.log" perm(0644)); }; destination d_edge { file("/var/log/company/edge_devices/$HOST.log" perm(0644)); }; destination d_fw { file("/var/log/company/firewalls/$HOST.log" perm(0644)); }; destination d_router { file("/var/log/company/routers/$HOST.log" perm(0644)); }; # filters filter f_edge { host("edge*") or host("10.10.*"); }; filter f_router { host("3600-primary") or host("backbone-3550") or host("secondary-3550"); }; filter f_switch { host("sw*") or host("backbone-5510") or host(pp8600*); }; filter f_fw { host("fw*"); }; # wrap everything up log { source(s_net); filter(f_switch); destination(d_switch); }; log { source(s_net); filter(f_router); destination(d_router); }; log { source(s_net); filter(f_edge); destination(d_edge); }; log { source(s_net); filter(f_fw); destination(d_fw); }; # end of configuration file when I run the command `find /var/log/company -type f -print` I get the following output: ./firewalls/fw-backup.log ./firewalls/edge-location1.log ./firewalls/fwmgmt.log ./edge_devices/edge-location2.log ./edge_devices/edge-location3.log ./edge_devices/edge-location4.log ./edge_devices/edge-location5.log ./edge_devices/edge-location6.log ./edge_devices/edge-location7.log ./edge_devices/edge-location8.log ./edge_devices/edge-location9.log ./edge_devices/edge-location1.log ./edge_devices/edge-location10.log ./edge_devices/edge-location11.log ./switches/edge-location3.log ./switches/fw-backup.log ./switches/edge-location6.log ./switches/edge-location7.log ./switches/edge-location11.log ./switches/edge-location12.log I tried to upgrade syslog-ng to 1.6.9 but it didn't resolve the problems. Any idea on why this happnes and how to fix this will be greatly appriciated. TIA Paolo PS - the systems sending the syslog messages are nortel switches, Checkpoint firewalls and edge devices. __________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/

4 22

backup syslog mysql database
by nikolay.nenchev＠rbb-sofia.raiffeisen.at 28 Dec '05

28 Dec '05

Dear list, I am using syslog-ng from some time and it is one realiable and easy for configuration software. But I have notice 2 issues in my configuration, which i have not solved yet. I'm using: # syslog-ng -V syslog-ng 1.6.5 on debian 3.1, uname -a: Linux logsrv 2.4.27-2-386. 1.I'm logging to mysql 4.0 db with pipe. The pipe is locate in /tmp/mysql.pipe. But it is not starting with system startup.I have tried to use script for initial startup script like: ################################################################## #!/bin/bash if [ -e /tmp/mysql.pipe ] then while [ -e /tmp/mysql.pipe ] do mysql -u loguser --password=XXXX syslog < /tmp/mysql.pipe done else mkfifo /tmp/mysql.pipe fi ################################################################## I have read somewhere that the pipe must be located in somewhere else and stuff like this but did not find a solution. Also what should be the order of starting syslog-ng, mysql and mysql.pipe? So if you have experience such a problem can you guide me through it? 2. and second issue is how to backup this mysql db (i know that it is a little bit of topic). Because the syslog db is nearly 5GB and like to extract and compress it. have try with mysqlhotcopy but it is locking database for time of cp db to another place which last time was around 10 minutes. Thanks in advance, Nikolay Nenchev

1 0

Pix problem
by Damien Michau 28 Dec '05

28 Dec '05

Hi All ! I have some probleme ta log my pix's log into my syslog-ng server . i have mounted a Syslog-ng server to store my pix's log . But there is nothing in my pix.log i'have put this lines in my syslog-ng.conf source pix { udp(ip(10.60.10.111) port());}; destination pix { file("/var/log/pix.log"); }; log { source(pix);destination(pix); }; Have you one idea ? Thx Damien Michau Paris

4 5

What is my incoming data
by David Anderson 26 Dec '05

26 Dec '05

Hi, I am trying to set up syslog-ng to process data from my router on port 514, but I don't know the format of the data to be able to build filters. Is there any way to collect all data on 514 to a file? My current filters, etc are: # # the draytek router # filter f_draytek { host("router") and not match("Local User") and not match("virtual\Server"); }; filter f_draytek_DSL_1 { host("router") and match("DSL"); }; filter f_draytek_DSL_2 { host("router") and match("ADSL_Status"); }; filter f_draytek_PPP { host("router") and match("PoE"); }; destination draytek { file("/var/log/draytek/draytek.log"); }; destination draytek_null { file("/dev/null"); }; destination draytek_dsl { file("/var/log/draytek/draytek_dsl.log"); }; destination draytek_ppp { file("/var/log/draytek/draytek_ppp.log"); }; log { source(src); filter(f_draytek); destination(draytek); }; log { source(src); filter(f_draytek_DSL_1); destination(draytek_dsl); }; log { source(src); filter(f_draytek_DSL_2); destination(draytek_dsl); }; log { source(src); filter(f_draytek_PPP); destination(draytek_ppp); }; The log files have no info, so I presume there is something wrong somewhere (the conf is from someone who had everything working (apart from the /dev/null). It would presumably be easier if I could see the raw data. Originally the 514 data was being blocked by the firewall, but that is now cleared. Any help would be appreciated. -- Best regards, David mailto:dma@pern.co.uk

3 2