Hello, i have a few questions about the message parser, basically i want to parse/split up the MESSAGE field itself and write the splitted up message in a mysql database, i cant find any documents about how this can be done, do i need an external parser (perl or whatever) or can this be done within syslog-ng.conf? right now i just can write the whole message to the mysqldb syslog-ng.conf template("INSERT INTO logs (host, facility, priority, level, tag, date,time, program, msg) VALUES ( '$HOST', '$FACILITY','$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); so it gets written to the database in this way: INSERT INTO logs (host, facility, priority, level, tag, date,time, program, msg) VALUES ( '10.44.10.253', 'local4','notice', 'notice', 'a5', '2006-06-29', '14:39:46', 'NS25', 'NS25: NetScreen device_id=NS25 [Root]system-notification-00257(traffic): start_time=\"2006-06-29 14:38:38\" duration=0 policy_id=95 service=http proto=6 src zone=Untrust dst zone=Untrust action=Deny sent=0 rcvd=0 src=10.10.10.225 dst=208.174.52.61 src_port=2042 dst_port=80 session_id=0' ); now i want to split up the message part itself system-notification traffic, insert the start_time/duration/policy_id/service/proto/src-zone etc etc in a different table this perl script http://www.optekconsulting.com/tools/nstf.pl has every field i need, Any Help is really welcome Cheers Hubert -- Die e-Mail-Boxes von Brennercom sind Virus-gesichert und Spam-gefiltert. Le caselle e-Mail di Brennercom sono protette da sistemi antivirus e antispam. http://www.brennercom.it
I have an idea I would like to run by everyone, but first I will provide a brief background of my situation. I am using syslog-ng on an embedded Linux device with a limited amount of resources. To this end, I must maintain very small log files. I have looked into log rotation, but do not wish to take the crond / logrotate approach. Ideally, it would be nice if Linux natively supported a cyclic file type so that I may limit the file size externally from syslog-ng. I have explored the idea of developing an external sever which will manage a circular file for me, but I feel it would be much more efficient to add this functionality to syslog-ng directly. Let me know if this sounds feasible.
On Thu, 29 Jun 2006 16:46:01 EDT, Kenneth Kassing said:
I am using syslog-ng on an embedded Linux device with a limited amount of resources. To this end, I must maintain very small log files. I have looked into log rotation, but do not wish to take the crond / logrotate approach.
Logrotate is a 43K binary on my Fedora Core box. I'm not sure you're going to have an easy time keeping the bloat in adding it natively to syslog-ng to be much smaller (since you'll have to add in all the support for config files that logrotate has into syslog-ng's parser, etc). Or even cheaper - have syslog-ng log to /var/log/$MON/$DAY or similar (doesn't matter what, as long as it's different per-day), and then just have cron run a 'find /var/log -type f -mtime +7 -exec rm {};'). Hard to get much more lightweight than that if you have 'find' onboard already (though it's 156K, so logrotate is a better idea if you have to add a binary). I'm thinking it's also a non-starter on the maintenance end, as it will be a big patch you'll be dragging from version to version, if Balazs isn't interested in putting it in the base release....
On 6/29/06, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote:
On Thu, 29 Jun 2006 16:46:01 EDT, Kenneth Kassing said:
To this end, I must maintain very small log files. I have looked into log rotation, but do not wish to take the crond / logrotate approach.
You might want to look at how DJB solved this in daemontools, via the "multilog" program, http://cr.yp.to/daemontools/multilog.html See the section on "Automatically rotated logs". If you want to experiment with multilog, you can call it from syslog-ng as a 'program' destination. Kevin
On Thu, 2006-06-29 at 14:48 +0200, Hubert Lang wrote:
Hello,
i have a few questions about the message parser, basically i want to parse/split up the MESSAGE field itself and write the splitted up message in a mysql database, i cant find any documents about how this can be done, do i need an external parser (perl or whatever) or can this be done within syslog-ng.conf?
right now i just can write the whole message to the mysqldb
In 1.9.x you can match patterns in the message and then reference the matches in templates via $N (where N is the sequence number of the match), similar to what perl has. -- Bazsi
Hey Balazs, i still have this problem, even with 1.9 i can catch the message in a better way but im not able to split the message up, do you have some parsers even external ones for syslog-ng ? im prety sure some users wrote one. Cheers and thenks for your help On Sat, 01 Jul 2006 17:40:47 +0200 Balazs Scheidler <bazsi@balabit.hu> wrote:
On Thu, 2006-06-29 at 14:48 +0200, Hubert Lang wrote:
Hello,
i have a few questions about the message parser, basically i want to parse/split up the MESSAGE field itself and write the splitted up message in a mysql database, i cant find any documents about how this can be done, do i need an external parser (perl or whatever) or can this be done within syslog-ng.conf?
right now i just can write the whole message to the mysqldb
In 1.9.x you can match patterns in the message and then reference the matches in templates via $N (where N is the sequence number of the match), similar to what perl has.
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- Die e-Mail-Boxes von Brennercom sind Virus-gesichert und Spam-gefiltert. Le caselle e-Mail di Brennercom sono protette da sistemi antivirus e antispam. http://www.brennercom.it
participants (5)
-
Balazs Scheidler
-
Hubert Lang
-
Kenneth Kassing
-
Kevin
-
Valdis.Kletnieks＠vt.edu