Re: [syslog-ng] need help with a match filter
Someone else on the list may correct me but I think you'll find that the destination is evaluated at the time syslog is started (or restarted) not when a log message is processed. I'm pretty sure there was a discussion on the list about something similar a while back (although I couldn't find it in my emails) So short answer is that you can't use a file desination. You may however get some mileage out of writing a custom program destination and do the last bit of processing in your program.
Rory Toma <rory@ooma.com> 01/25/10 5:12 PM >>> Looking more at the docs, it looks likes I also need a value for 3.0 syntax. So I am trying something like below. However, the value of $foo_hostdir is not being set correctly. It is being set to "foo_hostdir". How do I get the value to be the value of the match?
filter foo_filter { match('host("foo_[0-9A-F]\{9\}$")' value("foo_hostdir") type("string") ); }; # foo destinations destination foo_dest { file("/logs/$foo_hostdir/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); }; On 1/25/10 12:02 PM, Rory Toma wrote:
I'm trying to dynamically create lognames and directories form the host name that is sent. My hosts' names are of the form
foo_<mac address>
I want to create a destination that would be
/logs/foo_<first 9 digits of mac address>//foo_.log
Here is what I tried so far, and it does not work, but is close. Can someone suggest a config that might work? Thx.
I can do this individually, but having 100+ filters and 100+ log lines in the config really slows things down.
#filter for foo filter foo_filter { match('host("foo_[0-9A-F]\{9\}$")' flags("store-matches")); };
# foo destinations destination foo_dest {
file("/logs/$0/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Hmmm... Example 3.40 in the 3.0 syslog-ng documentation would seem to suggest otherwise, or am I just misunderstanding the syntax? On 1/25/10 2:24 PM, chris packham wrote:
Someone else on the list may correct me but I think you'll find that the destination is evaluated at the time syslog is started (or restarted) not when a log message is processed. I'm pretty sure there was a discussion on the list about something similar a while back (although I couldn't find it in my emails)
So short answer is that you can't use a file desination. You may however get some mileage out of writing a custom program destination and do the last bit of processing in your program.
Rory Toma<rory@ooma.com> 01/25/10 5:12 PM>>>
Looking more at the docs, it looks likes I also need a value for 3.0 syntax. So I am trying something like below. However, the value of $foo_hostdir is not being set correctly. It is being set to "foo_hostdir". How do I get the value to be the value of the match?
filter foo_filter { match('host("foo_[0-9A-F]\{9\}$")' value("foo_hostdir") type("string") ); };
# foo destinations destination foo_dest {
file("/logs/$foo_hostdir/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); };
On 1/25/10 12:02 PM, Rory Toma wrote:
I'm trying to dynamically create lognames and directories form the host name that is sent. My hosts' names are of the form
foo_<mac address>
I want to create a destination that would be
/logs/foo_<first 9 digits of mac address>//foo_.log
Here is what I tried so far, and it does not work, but is close. Can someone suggest a config that might work? Thx.
I can do this individually, but having 100+ filters and 100+ log lines in the config really slows things down.
#filter for foo filter foo_filter { match('host("foo_[0-9A-F]\{9\}$")' flags("store-matches")); };
# foo destinations destination foo_dest {
file("/logs/$0/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On 1/25/10 2:38 PM, Rory Toma wrote:
Hmmm... Example 3.40 in the 3.0 syslog-ng documentation would seem to suggest otherwise, or am I just misunderstanding the syntax?
OK, I've tried another approach, and it almost works... I use the parser command to break up my hostname (which is always 16 chars) into 16 parts, and then I want to use the first 13 columns in the dest... However, I don't know how to construct a delimiters statement to break this up, delimiters("") doesn't work. Any suggestions?
In case it is not obvious, the problem I am trying to solve here is to not have tens of thousands of files in the same directory, which destroys performance. Is there another, built-in way that I don't know about to have syslog-ng do a hash on the destination filename and generate a directory automagically based on that? thx On 1/25/10 3:07 PM, Rory Toma wrote:
On 1/25/10 2:38 PM, Rory Toma wrote:
Hmmm... Example 3.40 in the 3.0 syslog-ng documentation would seem to suggest otherwise, or am I just misunderstanding the syntax?
OK, I've tried another approach, and it almost works...
I use the parser command to break up my hostname (which is always 16 chars) into 16 parts, and then I want to use the first 13 columns in the dest... However, I don't know how to construct a delimiters statement to break this up, delimiters("") doesn't work. Any suggestions? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
OK, after looking at about a hundred web pages, and having my office mate do the same, here's what works: filter foo_filter { match("\(foo_[0-9A-F]\{9\}\)[0-9A-F]\{3\}" value("HOST") flags("store-matches")); }; # foo destinations destination foo_dest { file("/logs/${1}/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); }; log {source(telo); filter(foo_filter); destination(foo_dest); };
On Mon, 2010-01-25 at 16:37 -0800, Rory Toma wrote:
OK, after looking at about a hundred web pages, and having my office mate do the same, here's what works:
filter foo_filter { match("\(foo_[0-9A-F]\{9\}\)[0-9A-F]\{3\}" value("HOST") flags("store-matches")); };
# foo destinations destination foo_dest {
file("/logs/${1}/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); };
log {source(telo); filter(foo_filter); destination(foo_dest); };
Great work. Can you perhaps have a suggestion on the documentation whether we could clear up something to make this easier? I know it is difficult to tell after the fact what was missing from the docs, but perhaps we could improve it this way. Thanks. -- Bazsi
Honestly, I no longer remember. However, having something like an "examples" section with several working examples and explanations would be helpful, and if there are subtle differences, pointing out the difference and why it makes a difference. On 2/6/10 7:54 AM, Balazs Scheidler wrote:
On Mon, 2010-01-25 at 16:37 -0800, Rory Toma wrote:
OK, after looking at about a hundred web pages, and having my office mate do the same, here's what works:
filter foo_filter { match("\(foo_[0-9A-F]\{9\}\)[0-9A-F]\{3\}" value("HOST") flags("store-matches")); };
# foo destinations destination foo_dest {
file("/logs/${1}/$R_YEAR$R_MONTH$R_DAY/$HOST-$R_YEAR$R_MONTH$R_DAY.log" owner(root) group(root) perm(0444) template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC $MSG\n") template_escape(no)); };
log {source(telo); filter(foo_filter); destination(foo_dest); };
Great work. Can you perhaps have a suggestion on the documentation whether we could clear up something to make this easier?
I know it is difficult to tell after the fact what was missing from the docs, but perhaps we could improve it this way.
Thanks.
On Mon, 2010-02-08 at 11:20 -0800, Rory Toma wrote:
Honestly, I no longer remember. However, having something like an "examples" section with several working examples and explanations would be helpful, and if there are subtle differences, pointing out the difference and why it makes a difference.
we're planning a wiki on our website, where these examples could be stored in the future. -- Bazsi
participants (3)
-
Balazs Scheidler
-
chris packham
-
Rory Toma