On Thu, 2008-07-03 at 05:56 -0400, Enigma wrote:
Balazs Scheidler wrote:
On Thu, 2008-07-03 at 11:17 +0200, Balazs Scheidler wrote:
On Wed, 2008-07-02 at 18:31 -0400, Enigma wrote:
I have a host that sends mult-line messages (embedded newlines, not separate syslog messages) from what I can tell syslog-ng strips out all the newlines and replaces them with a space.
Is there anyway to disable or modify (replace them with something else that can be easily s/// in post-processing) this functionality without changing the code and compiling from source?
I have been through the syslog-ng manual and asked Mr. Google and I cannot find anything on this topic.
Newlines and stuff are incompatible with TCP transport. If you are using UDP, it could work, however as you point out syslog-ng removes all NLs from log messages in order not to ruin your logfiles.
With my development snapshot the new syslog-protocol drafts are implemented, that too allows embedded NLs.
This crude patch deletes the part that removes NLs from messages:
diff --git a/src/logmsg.c b/src/logmsg.c index 139fb3a..adb9f2d 100644 --- a/src/logmsg.c +++ b/src/logmsg.c @@ -522,11 +522,6 @@ log_msg_parse(LogMessage *self, gchar *data, gint length, guint flags, regex_t * self->stamp.time.tv_sec = now; }
- for (oldsrc = src, oldleft = left; oldleft >= 0; oldleft--, oldsrc++) - { - if (*oldsrc == '\n' || *oldsrc == '\r') - *oldsrc = ' '; - } g_string_assign_len(&self->msg, src, left); }
I might add something more sophisticated if you or anyone else can help me with finding out a good idea how to handle NLs when they are written to logfiles.
E.g. you have a template like this:
template("$DATE $HOST $MSG\n");
If there's an NL in $MSG it'd probably break a lot of log parsers. If syslog-ng would repeat the syslog header
sorry, sent too early. Would it be enough if syslog-ng would be capable of repeating the $DATE $HOST part for each line produced because of NLs in MSG?
How do you want to use multi-line messages?
Basically I want to preserve the logs as they come from the source. They are delivered to syslog-ng (via UDP so the TCP issue isn't a factor) with embedded newlines and we need them to be forwarded (again via UDP) and written to the log files with those embedded newlines. This is kind of a corner case because we know it will not affect any log parsing. Unfortunately patching the code will not work since this syslog-ng server resides on client prem and will not patch/compile from source on a production machine.
The only other option would be to replace the newlines with some other char (with a regex) that we can so a substitute with a newline on the backend before writing to the log file or forwarding the message. I haven't been able to find this functionality either (quite possibly a PEBCAK situation). Is this the rewrite stuff you were talking about in your 'migrate over to PCRE' thread?
The first problem that I'd have to disable the "substitute every NL with a space" code in the input path, and probably put that into the destination, controlled by a flag or something. This way you can either have NL in your destination if you choose to, or you can also rewrite it using regexps. -- Bazsi