[syslog-ng] How to convert syslog-ng logs into xml format

[Alexander Clouter]

Hi,

Padmanabhan, Rajeesh (GE Healthcare) <Rajeesh.Padmanabhan at ge.com> [20070211 15:40:27 +0530]:
>
> Hi All,
>  
> I need to convert syslog-ng logs into xml format. Could you please help
> me to do the same. If someone can send me a sample format, would be
> great.
>  
Oh dear...sounds like someone caught 'xml fever', it happens to all of us at 
some stage and we usually recover from the illness when its too late and the 
project is already using it...

XML is something thats abused more often than its used properly.  Its useful 
if you have two rather different systems that have no common data exchange 
mechanism between them and so XML is used as a 'common' format.  Bear in mind 
that this means is at both ends you have to get programming tech-monkeys to 
write scripts to convert the data to XML and then back from XML into another 
format.  In situations like this you do not care that it could take two weeks 
of solid CPU crunching to import the data, just as long as it gets there...

Now as you have not given much detail about why you need the files in an XML 
format or what you plan on doing with the files the follow format is probably 
'suitable':

<?xml version="1.0" encoding="UTF-8"?>
<syslog>
  <message>first syslog message</message>
  <message>second syslog message</message>
  ...
</syslog>

I'm finding it hard to think of a situation where XML and syslog data would 
be a good idea given that its rather easy to parse and store syslog data 
already.  People want to archive and easily search (usually with nothing more 
than grep) their syslog logs and in an XML format searching your logs *will* 
slow down by at least a factor of 1000.

A far more suitable format, if flat text files is not good enough, is a SQL 
database; probably the only real useful application of an SQL database[1].

I'm unsure how to help you recover from your XML fever, it was difficult for 
me to grow out of.  I learn from my mistakes and unfortunately for me when 
you base a project around XML the mistakes (for me at least) do not have an 
effect till much later on.  There is nothing on the Internet as suggested 
reading but I am happy to discuss this off list if you want to.

I did some digging around to try to see if there was anything commercial 
doing this and found[2] something and was shocked at how they were doing 
it...its a bad schema and seems to be a solution to something that is not 
actually a problem.

*Everything* that deals with logging data should support syslog built 
logfiles (you might need to use syslog-ng with its template() command to 
structure the logfile to be in a format that the import engine expects 
though), if it does not you should look for some other software.  My gut 
feelings tell me this is an internal homebrew project though..

I hope the 'fever' breaks soon :)

Cheers

Alex

[1] thats another rant of mine though :)
[2] http://www.kiwisyslog.com/kb/idx/4/125/article/