On Fri, Nov 05, 2010 at 03:49:05PM -0600, Bill Anderson wrote:

...

I'm using a tab separated format from apache for access logs. My last two fields are referrer and user-agent. Obviously sometimes there is no referrer. Unfortunately when there isn't one apache only logs an empty string instead of the more common "-". This isn't a problem in scripts that parse the resulting logfile as they see the resulting empty field when I log $MSG.

However, I just started a new log file that uses the csv-parser w/tab as delimiter and when the referrer field is empty, APACHE.USERAGENT (the last field) gets rolled into APACHE.REFERRER, the second to last field. As a result the template for this page (which uses APACHE.REFERRER) isn't reliable. When REFERRER is empty I want it to be empty (or something I can specify, like a default) not he next field in the parser definition.

I've look at the manual and don't see anything about handling empty fields. How do I get syslog-ng/csv-parser to log the empty field instead of moving to the next one?

... On Nov 5, 2010, at 4:55 PM, Matthew Hall wrote:

I think you need to configure some of your flags to the parser.

Did you try something like these directions here:

https://www.icts.uiowa.edu/confluence/display/ICTSit/Using+syslog-ng+to+coll...

Matthew.

Thanks for your reply, Mathew. Perhaps I wasn't clear enough. The syslog-ng produced logfile that logs $MSG is *just fine*. The tabs are there, and anything that parses it and expects the fields gets them just fine. The problem arises when the template only needs to log fields from the csv parser and a preceding field is empty. If there are any flags on that page that affect how the csv-parser handles empty fields, I'd appreciate them being pointed out, as I didn't see any. Cheers, Bill

On Saturday, November 06, 2010 00:40 CET, Matthew Hall <mhall@mhcomputing.net> wrote:

I'm surprised it doesn't handle doubled delimiters right.

However I am likewise stumped. I can't find any explanation anywhere in the Balabit docs which explain what escape-* options do specifically. You´re right, that is missing. I´ll hunt down what they do, and add it to the docs.

@Bill: I can think of two possibilities that might do what you´re looking for: 1. In your template, try to specify a default value for your macros, for example, to specify a dash: {$APACHE.REFERRER:-} The problem with this is that it might not work - default values probably work only during parsing, and not when the macro is used is a template, but I think it is worth a try. If they were implemented in a universal way, it might work. 2. In syslog-ng OSE 3.2, you can use template functions and other operators to check and compare macro values. You would need to use the "if" template function to return a dash or other suitable value if your macros are empty (""). So instead of $APACHE.REFERRER, your template would look something like: $(if ("${APACHE.REFERRER}" == "") "-" $APACHE.REFERRER) See http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid... and http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid... for details. Regards, Robert

I guess reading the code is the only option for now. :/

On Fri, Nov 05, 2010 at 05:26:38PM -0600, Bill Anderson wrote:

...

Thanks for your reply, Mathew. Perhaps I wasn't clear enough. The syslog-ng produced logfile that logs $MSG is *just fine*. The tabs are there, and anything that parses it and expects the fields gets them just fine. The problem arises when the template only needs to log fields from the csv parser and a preceding field is empty. If there are any flags on that page that affect how the csv-parser handles empty fields, I'd appreciate them being pointed out, as I didn't see any.

Cheers, Bill

---

Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

On Nov 6, 2010, at 1:13 PM, Fekete Róbert wrote:

On Saturday, November 06, 2010 00:40 CET, Matthew Hall <mhall@mhcomputing.net> wrote:

...

I'm surprised it doesn't handle doubled delimiters right.

However I am likewise stumped. I can't find any explanation anywhere in the Balabit docs which explain what escape-* options do specifically. You´re right, that is missing. I´ll hunt down what they do, and add it to the docs.

@Bill: I can think of two possibilities that might do what you´re looking for: 1. In your template, try to specify a default value for your macros, for example, to specify a dash: {$APACHE.REFERRER:-} The problem with this is that it might not work - default values probably work only during parsing, and not when the macro is used is a template, but I think it is worth a try. If they were implemented in a universal way, it might work.

I'm with you in the suspicion it won't work but I'll give it a shot. If it doesn't, I'll go with below for now.

2. In syslog-ng OSE 3.2, you can use template functions and other operators to check and compare macro values. You would need to use the "if" template function to return a dash or other suitable value if your macros are empty (""). So instead of $APACHE.REFERRER, your template would look something like: $(if ("${APACHE.REFERRER}" == "") "-" $APACHE.REFERRER)

3.2 isn't an option just yet here, so this won't work. However, it gives me an idea. Perhaps prior to calling the parser I could to a rewrite on "\t\t" to "\t-\t". It feels hackish, but might work as long as the performance impact isn't too bad. Ultimately though, the parser needs to not "skip" empty-value fields. Cheers, Bill

On Sat, 2010-11-06 at 20:13 +0100, Fekete Róbert wrote:

On Saturday, November 06, 2010 00:40 CET, Matthew Hall <mhall@mhcomputing.net> wrote:

...

I'm surprised it doesn't handle doubled delimiters right.

However I am likewise stumped. I can't find any explanation anywhere in the Balabit docs which explain what escape-* options do specifically. You´re right, that is missing. I´ll hunt down what they do, and add it to the docs.

@Bill: I can think of two possibilities that might do what you´re looking for: 1. In your template, try to specify a default value for your macros, for example, to specify a dash: {$APACHE.REFERRER:-} The problem with this is that it might not work - default values probably work only during parsing, and not when the macro is used is a template, but I think it is worth a try. If they were implemented in a universal way, it might work.

Robert, I don't exactly understand what refer to here. The default expansion work everywhere a template is used. E.g. if you have an empty value, you can assign a default to it within the template. But when expanding a template an undefined name-value pair becomes an empty string. E.g. ${APACHE.REFERER} and ${APACHE.REFERER:-} is the same, since if APACHE.REFERER is not defined both become the empty string. And if it is defined, both become the value stored in APACHE.REFERER.

2. In syslog-ng OSE 3.2, you can use template functions and other operators to check and compare macro values. You would need to use the "if" template function to return a dash or other suitable value if your macros are empty (""). So instead of $APACHE.REFERRER, your template would look something like: $(if ("${APACHE.REFERRER}" == "") "-" $APACHE.REFERRER)

See http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid... and http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-guid... for details.

-- Bazsi