Growing log files (i.e. access.log) and regex.
Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************
sed -n 's/\(.*\)logger:.*$/\1jboss: DEBUG/p' I use syslog2mysql.sh which came with the version of syslog-ng that i installed from http://www.phpwizardry.com - Ken Robert.Becker@motoristsgroup.com wrote:
Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar.
Here is an example of what I am doing.
This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
Then I send the log through logger to syslog-ng:
#!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info
The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
What I need is:
Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function?
-Rob Becker Systems Engineer Motorists Insurance
********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on.
********************************************************************** _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Perhaps I am missing something here, but I believe that the "logger:" can be easily replaced with "jboss:" using the -t flag for logger. So modify your script file as follows: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info -t jboss Another way to tackle this problem is to avoid logger completely by tailing the log into a pipe (mkfifo(1)) and use the pipe() source for syslog-ng. Thus, you could modify your script to be: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log > /dev/pipe_to_syslog Peter Nahas Software Engineer MRV Communications, InReach Division Ken Garland wrote:
sed -n 's/\(.*\)logger:.*$/\1jboss: DEBUG/p'
I use syslog2mysql.sh which came with the version of syslog-ng that i installed from http://www.phpwizardry.com
- Ken
Robert.Becker@motoristsgroup.com wrote:
Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar.
Here is an example of what I am doing.
This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
Then I send the log through logger to syslog-ng:
#!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info
The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
What I need is:
Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function?
-Rob Becker Systems Engineer Motorists Insurance
********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on.
********************************************************************** _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Why not use SEC (http://www.estpak.ee/~risto/sec/) to monitor the log file? I use SEC with my syslog-ng files as well as a few other key log files. You could probably then use SEC to look for specific data then call an external script to put it into your DB. Chris On 10/11/05, Peter Nahas <pnahas@mrv.com> wrote:
Perhaps I am missing something here, but I believe that the "logger:" can be easily replaced with "jboss:" using the -t flag for logger. So modify your script file as follows:
#!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info -t jboss
Another way to tackle this problem is to avoid logger completely by tailing the log into a pipe (mkfifo(1)) and use the pipe() source for syslog-ng. Thus, you could modify your script to be:
#!/bin/sh tail -f /opt/jboss/server/default/log/boot.log > /dev/pipe_to_syslog
Peter Nahas Software Engineer MRV Communications, InReach Division
Ken Garland wrote:
sed -n 's/\(.*\)logger:.*$/\1jboss: DEBUG/p'
I use syslog2mysql.sh which came with the version of syslog-ng that i installed from http://www.phpwizardry.com
- Ken
Robert.Becker@motoristsgroup.com wrote:
Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar.
Here is an example of what I am doing.
This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
Then I send the log through logger to syslog-ng:
#!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info
The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
What I need is:
Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function?
-Rob Becker Systems Engineer Motorists Insurance
********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on.
********************************************************************** _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
hello I use syslog-ng and it sends log in a local Mysql database via pipe and I have this syslog.conf options { flush_lines(0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); dns_cache(yes); dns_cache_size(2000); use_fqdn (yes); create_dirs (no); keep_hostname (yes); time_reopen (10); }; source src { unix-stream("/dev/log"); internal(); }; source src_firewall { file ("/proc/kmsg" log_prefix("firewall: ")); }; filter firewal_filter { match("REJECT"); }; destination d_mysql { pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL','$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; log { source(src); destination(d_mysql); }; log { source(src_firewall); destination(d_mysql); }; sometimes in the database i found this msg record syslog-ng[6130]: Log statistics; dropped='pipe(/tmp/mysql.pipe)=0', processed='center(queued)=57298', processed='center(received)=57298', processed='destination(d_mysql)=57298', processed='source(src_firewall)=51537', processed='source(src)=5761' I assume that this message advises that some logs are lost. I register the messages only local and that messages that i register include firewall's messages. It is possible solve this problem, which configuration I could use to solve this problem? tanks a lot Luigi
On Thu, 08 Mar 2007 11:29:25 +0100, Luigi Augello said:
sometimes in the database i found this msg record
syslog-ng[6130]: Log statistics; dropped='pipe(/tmp/mysql.pipe)=0', processed='center(queued)=57298', processed='center(received)=57298', processed='destination(d_mysql)=57298', processed='source(src_firewall)=51537', processed='source(src)=5761'
I assume that this message advises that some logs are lost.
Why do you think any were lost? 57,298 came in, 57,298 went into d_mysql, 0 were dropped. Also, the inbounds were 57,537+5,761 = 57,298, so the sum of your two sources is equal to the number that went out the destination. Is there some external piece of data (such as a network trace proving that 100K msgs were sent, and 43% of them evaporated?) that I'm not cluing in on?
participants (6)
-
Ken Garland
-
Luigi Augello
-
Peter Nahas
-
Robert.Becker＠motoristsgroup.com
-
sawall
-
Valdis.Kletnieks＠vt.edu