[syslog-ng] Multi-line Logs

[Balazs Scheidler]

On Thu, 2008-07-03 at 05:56 -0400, Enigma wrote:
> Balazs Scheidler wrote:
> > On Thu, 2008-07-03 at 11:17 +0200, Balazs Scheidler wrote:
> >   
> >> On Wed, 2008-07-02 at 18:31 -0400, Enigma wrote:
> >>     
> >>> I have a host that sends mult-line messages (embedded newlines, not 
> >>> separate syslog messages) from what I can tell syslog-ng strips out all 
> >>> the newlines and replaces them with a space.
> >>>
> >>> Is there anyway to disable or modify (replace them with something else 
> >>> that can be easily s/// in post-processing) this functionality without 
> >>> changing the code and compiling from source?
> >>>
> >>> I have been through the syslog-ng manual and asked Mr. Google and I 
> >>> cannot find anything on this topic.
> >>>       
> >> Newlines and stuff are incompatible with TCP transport. If you are using
> >> UDP, it could work, however as you point out syslog-ng removes all NLs
> >> from log messages in order not to ruin your logfiles.
> >>
> >> With my development snapshot the new syslog-protocol drafts are
> >> implemented, that too allows embedded NLs.
> >>
> >> This crude patch deletes the part that removes NLs from messages:
> >>
> >> diff --git a/src/logmsg.c b/src/logmsg.c
> >> index 139fb3a..adb9f2d 100644
> >> --- a/src/logmsg.c
> >> +++ b/src/logmsg.c
> >> @@ -522,11 +522,6 @@ log_msg_parse(LogMessage *self, gchar *data, gint length, guint flags, regex_t *
> >>        self->stamp.time.tv_sec = now;
> >>      }
> >>  
> >> -  for (oldsrc = src, oldleft = left; oldleft >= 0; oldleft--, oldsrc++)
> >> -    {
> >> -      if (*oldsrc == '\n' || *oldsrc == '\r')
> >> -       *oldsrc = ' ';
> >> -    }
> >>    g_string_assign_len(&self->msg, src, left);
> >>  }
> >>  
> >>
> >> I might add something more sophisticated if you or anyone else can help me with finding out a 
> >> good idea how to handle NLs when they are written to logfiles.
> >>
> >> E.g. you have a template like this:
> >>
> >> template("$DATE $HOST $MSG\n");
> >>
> >> If there's an NL in $MSG it'd probably break a lot of log parsers. If syslog-ng would repeat the syslog header
> >>
> >>     
> >
> > sorry, sent too early. Would it be enough if syslog-ng would be capable
> > of repeating the $DATE $HOST part for each line produced because of NLs
> > in MSG?
> >
> > How do you want to use multi-line messages?
> >
> >   
> Basically I want to preserve the logs as they come from the source.  
> They are delivered to syslog-ng (via UDP so the TCP issue isn't a 
> factor) with embedded newlines and we need them to be forwarded (again 
> via UDP) and written to the log files with those embedded newlines.  
> This is kind of a corner case because we know it will not affect any log 
> parsing.  Unfortunately patching the code will not work since this 
> syslog-ng server resides on client prem and will not patch/compile from 
> source on a production machine.
> 
> The only other option would be to replace the newlines with some other 
> char (with a regex) that we can so a substitute with a newline on the 
> backend before writing to the log file or forwarding the message.  I 
> haven't been able to find this functionality either (quite possibly a 
> PEBCAK situation).  Is this the rewrite stuff you were talking about in 
> your 'migrate over to PCRE' thread?

The first problem that I'd have to disable the "substitute every NL with
a space" code in the input path, and probably put that into the
destination, controlled by a flag or something.

This way you can either have NL in your destination if you choose to, or
you can also rewrite it using regexps.

-- 
Bazsi