[syslog-ng] Multi-line Logs

[Enigma]

Balazs Scheidler wrote:
> On Thu, 2008-07-03 at 11:17 +0200, Balazs Scheidler wrote:
>   
>> On Wed, 2008-07-02 at 18:31 -0400, Enigma wrote:
>>     
>>> I have a host that sends mult-line messages (embedded newlines, not 
>>> separate syslog messages) from what I can tell syslog-ng strips out all 
>>> the newlines and replaces them with a space.
>>>
>>> Is there anyway to disable or modify (replace them with something else 
>>> that can be easily s/// in post-processing) this functionality without 
>>> changing the code and compiling from source?
>>>
>>> I have been through the syslog-ng manual and asked Mr. Google and I 
>>> cannot find anything on this topic.
>>>       
>> Newlines and stuff are incompatible with TCP transport. If you are using
>> UDP, it could work, however as you point out syslog-ng removes all NLs
>> from log messages in order not to ruin your logfiles.
>>
>> With my development snapshot the new syslog-protocol drafts are
>> implemented, that too allows embedded NLs.
>>
>> This crude patch deletes the part that removes NLs from messages:
>>
>> diff --git a/src/logmsg.c b/src/logmsg.c
>> index 139fb3a..adb9f2d 100644
>> --- a/src/logmsg.c
>> +++ b/src/logmsg.c
>> @@ -522,11 +522,6 @@ log_msg_parse(LogMessage *self, gchar *data, gint length, guint flags, regex_t *
>>        self->stamp.time.tv_sec = now;
>>      }
>>  
>> -  for (oldsrc = src, oldleft = left; oldleft >= 0; oldleft--, oldsrc++)
>> -    {
>> -      if (*oldsrc == '\n' || *oldsrc == '\r')
>> -       *oldsrc = ' ';
>> -    }
>>    g_string_assign_len(&self->msg, src, left);
>>  }
>>  
>>
>> I might add something more sophisticated if you or anyone else can help me with finding out a 
>> good idea how to handle NLs when they are written to logfiles.
>>
>> E.g. you have a template like this:
>>
>> template("$DATE $HOST $MSG\n");
>>
>> If there's an NL in $MSG it'd probably break a lot of log parsers. If syslog-ng would repeat the syslog header
>>
>>     
>
> sorry, sent too early. Would it be enough if syslog-ng would be capable
> of repeating the $DATE $HOST part for each line produced because of NLs
> in MSG?
>
> How do you want to use multi-line messages?
>
>   
Basically I want to preserve the logs as they come from the source.  
They are delivered to syslog-ng (via UDP so the TCP issue isn't a 
factor) with embedded newlines and we need them to be forwarded (again 
via UDP) and written to the log files with those embedded newlines.  
This is kind of a corner case because we know it will not affect any log 
parsing.  Unfortunately patching the code will not work since this 
syslog-ng server resides on client prem and will not patch/compile from 
source on a production machine.

The only other option would be to replace the newlines with some other 
char (with a regex) that we can so a substitute with a newline on the 
backend before writing to the log file or forwarding the message.  I 
haven't been able to find this functionality either (quite possibly a 
PEBCAK situation).  Is this the rewrite stuff you were talking about in 
your 'migrate over to PCRE' thread?