Hi All I am using syslog-ng which pipes the log to mysql database in this format host, facility, priority, level, tag, date, time, program, msg I installed the php plugin and now I can browse through the web real fast. Is there a way I can push all the previous logs in the database in the same format ? That way users can also look at the previous datam Thanks -- Asif Iqbal http://pgpkeys.mit.edu:11371/pks/lookup?op=get&search=0x8B686E08 There's no place like 127.0.0.1
On Tue, 2003-12-23 at 03:45, Asif Iqbal wrote:
Hi All
I am using syslog-ng which pipes the log to mysql database in this format
host, facility, priority, level, tag, date, time, program, msg
I installed the php plugin and now I can browse through the web real fast.
Is there a way I can push all the previous logs in the database in the same format ? That way users can also look at the previous datam
Thanks
Were you running syslogd or syslog-ng (and log to a file) before? By default neither of these formats contain facility and priority settings, but apart from that there shouldn't be any problems importing the old logs into a database with some perl scripting. For speed reasons I would cut up the message with a unique delimeter and use mysqlimport instead of hundreds if not thousands of INSERT commands, but of course the later also works. Best regards Michael Boman -- Michael Boman Security Architect, SecureCiRT Pte Ltd http://www.securecirt.com
On Tue, 23 Dec 2003, Michael Boman wrote:
Date: Tue, 23 Dec 2003 09:44:09 +0800 From: Michael Boman <michael.boman@securecirt.com> Reply-To: syslog-ng@lists.balabit.hu To: Syslog-NG ML <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng]Old Syslog Data
On Tue, 2003-12-23 at 03:45, Asif Iqbal wrote:
Hi All
I am using syslog-ng which pipes the log to mysql database in this format
host, facility, priority, level, tag, date, time, program, msg
I installed the php plugin and now I can browse through the web real fast.
Is there a way I can push all the previous logs in the database in the same format ? That way users can also look at the previous datam
Thanks
Were you running syslogd or syslog-ng (and log to a file) before? By
I was using syslogd in the past until I found this awesome tool
default neither of these formats contain facility and priority settings, but apart from that there shouldn't be any problems importing the old logs into a database with some perl scripting.
Any hint or example would be greatly appreciated
For speed reasons I would cut up the message with a unique delimeter and use mysqlimport instead of hundreds if not thousands of INSERT commands, but of course the later also works.
Best regards Michael Boman
Thanks a lot -- Asif Iqbal http://pgpkeys.mit.edu:11371/pks/lookup?op=get&search=0x8B686E08 There's no place like 127.0.0.1
Asif Iqbal (Monday 22 December 2003 21:38)
default neither of these formats contain facility and priority settings, but apart from that there shouldn't be any problems importing the old logs into a database with some perl scripting.
Any hint or example would be greatly appreciated
It should be fairly easy (this is how I've done such things in the past) to modify your log file into a file of SQL statements, then just pipe that into psql (I assume it works the same with mysql). Let's say your syslog file contains lines looking like this: Oct 6 00:32:09 eirny kernel: hdc: LITE-ON LTR-12101B, ATAPI CD/DVD-ROM drive ...and your database table contains columns like this: date, time, host, message, facility, priority Then you'd just want to use your favorite editor (which ought to be vim ;-P) and use a few simple commands to change all of the lines to read: insert into db_table (timestamp, host, message) values ('6-10-2003', '00:32:09', 'eirny', 'kernel: hdc: LITE-ON LTR-12101B, ATAPI CD/DVD-ROM drive'); If you don't have defaults specified for the missing columns, then you could do something like: insert into db_table (timestamp, host, message, facility, priority) values ('6-10-2003', '00:32:09', 'eirny', 'kernel: hdc: LITE-ON LTR-12101B, ATAPI CD/DVD-ROM drive', '', ''); You might need to adjust the date format depending on what DBMS you're using...since the year wasn't specified you can just get it from 'date +%Y' if everything is from this year. If your log data spans multiple years, the easiest way to deal with it would probably be to split the files at the year changes, then just globally insert the different year in each file. -- Sigþór Björn Jarðarson (Casey Allen Shobe) cshobe@softhome.net Jabber: sigthor@jabber.org; ICQ: 1494523; AIM/Yahoo: SomeLinuxGuy -- Free development contributor of:
KDE toolbar icons Kopete user interface, usability, and testing X11 Icelandic Dvorak keymaps http://rivyn.livejournal.com
participants (3)
-
Asif Iqbal
-
Casey Allen Shobe
-
Michael Boman