[syslog-ng] are nested quotes possible in parser

13 Jan 2009

      Hi there,

my Avocent console servers are sending console port output via syslog  
to my syslog server running syslog-ng 3.0.1. Console output looks  
like this:

Jan 13 00:18:39 sysl@cyc2 Buffering: S39.gwa [Jan 13 00:14:04.379: % 
EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM threshold exceeded, TCAM  
Utilization [97%]]
Jan 13 00:18:53 sysl@cyc2 Buffering: S42.sw2f [2009 Jan 13 00:18:53  
EST -05:00 %ETHC-5-PORTTOSTP:Port 3/27 joined bridge port 3/27]
Jan 13 00:18:53 sysl@cyc2 Buffering: S42.sw2f [2009 Jan 13 00:18:53  
EST -05:00 %DTP-7-PORTLINKDOWN:Port 3/27 Link down]
Jan 13 00:18:53 sysl@cyc2 Buffering: S42.sw2f [2009 Jan 13 00:18:53  
EST -05:00 %ETHC-5-PORTFROMSTP:Port 3/27 left bridge port 3/27]

The goal is to store the console output within square brackets into  
separate files named after the server that created this output. The  
first line of the example above should go into the file "gwa" while  
the others go into "sw2f". This is what I have so far:
...
source s_udp { udp (); };
# --- parse console server output
# separate port description from message
parser p_console_output {
       csv-parser (columns ("CONSOLE.SOURCE", "CONSOLE.MSG")
                   delimiters (" ")
                   quote-pairs ("[]")
                   template ("${MSGONLY}"));
};
# extract port label from port description
parser p_console_source {
       csv-parser (columns ("CONSOLE.PORT", "CONSOLE.LABEL")
                   delimiters (".")
                   template ("${CONSOLE.SOURCE}"));
};
# --- destination of console output
destination d_console_output {
        file ("/usr/local/var/log/remote/${HOST_FROM}/console/$ 
{CONSOLE.LABEL}"
              template ("${CONSOLE.MSG}\n"));
};
# --- filter console output
filter f_console_output {
       facility (local7) and host ("^sysl@cyc.*");
};
# --- log console output
log {
       source (s_udp);
       filter (f_console_output);
       parser (p_console_output);
       parser (p_console_source);
       destination (d_console_output);
};
This works just fine with the last three lines of my example data  
above. The problem I am having is that if the console output (the  
text between square brackets) contains its own square brackets the  
message will cut off right after the first occurrence of the closing  
bracket. The first line of my example data will look like this:

Jan 13 00:14:04.379: %EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM  
threshold exceeded, TCAM Utilization [97%

instead of

Jan 13 00:14:04.379: %EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM  
threshold exceeded, TCAM Utilization [97%]

I could probably get around this by using a rewrite rule using PCRE  
but considering the amount of data that needs to be looked at this  
solution is going to be very expensive.

Is there a way to make syslog-ng aware of nested quotes? If not, is  
there something in the pipeline to support this in future releases?  
If not, I will be willing to take a shot and implement this.

Any pointers or suggestions are welcome.

Thanks.

- Michael

[syslog-ng] are nested quotes possible in parser

Michael Hocke