Using macro substring as another macro
Hi, I'd like to implement simple hashing in file() destination, mainly to avoid ext3 32k directories limit, but also for convenience. Is it possible to do something simliar what this shell snippet does: $ NAME=value $ echo ${NAME[*]:0:1} v $ but for macros (especially those coming from parsers)? What I want is logs written to /var/log/h/ho/hos/hostname/service.log (with configurable level of subdirs) using a single destination(). Regards, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
Jakub Jankowski <shasta@toxcorp.com> writes:
I'd like to implement simple hashing in file() destination, mainly to avoid ext3 32k directories limit, but also for convenience. Is it possible to do something simliar what this shell snippet does:
$ NAME=value $ echo ${NAME[*]:0:1} v $
but for macros (especially those coming from parsers)?
Off the top of my head: no, it's not currently possible with stock syslog-ng (except with some dirty hacks I'd rather not share). On the other hand, writing a template function that would do just what you want should be very very easy.
What I want is logs written to /var/log/h/ho/hos/hostname/service.log (with configurable level of subdirs) using a single destination().
I can throw together a patch that would introduce a $(substr START END STR) template function, which you could use like this: destination d_file { file ("/var/log/$(substr 1 1 ${HOST})/$(substr 1 2 ${HOST})/$(substr 1 3 ${HOST})/${HOST}/service.log"); }; It's a bit more verbose than ${NAME[*]:0:1}, but does pretty much the same thing. I'll knock something up either tonight or sometime tomorrow, both for syslog-ng 3.2 and for 3.3 aswell. On the other hand, if you have basic C knowledge, you can do this too, and it's actually quite fun to do so! I've written a short blog post a few months ago about writing trivial template functions, that should be enough to get you started: http://algernon.blogs.balabit.com/2011/01/howto-write-a-simple-template-func... If anyone beats me to it: you're my guest for your (reasonable) choice of beverage if you ever come to Hungary, or wherever we might meet. -- |8]
On Thu, 2011-04-07 at 15:20 +0200, Gergely Nagy wrote:
Jakub Jankowski <shasta@toxcorp.com> writes:
I'd like to implement simple hashing in file() destination, mainly to avoid ext3 32k directories limit, but also for convenience. Is it possible to do something simliar what this shell snippet does:
$ NAME=value $ echo ${NAME[*]:0:1} v $
but for macros (especially those coming from parsers)?
Off the top of my head: no, it's not currently possible with stock syslog-ng (except with some dirty hacks I'd rather not share). On the other hand, writing a template function that would do just what you want should be very very easy.
What I want is logs written to /var/log/h/ho/hos/hostname/service.log (with configurable level of subdirs) using a single destination().
I can throw together a patch that would introduce a $(substr START END STR) template function, which you could use like this:
destination d_file { file ("/var/log/$(substr 1 1 ${HOST})/$(substr 1 2 ${HOST})/$(substr 1 3 ${HOST})/${HOST}/service.log"); };
It's a bit more verbose than ${NAME[*]:0:1}, but does pretty much the same thing.
I'll knock something up either tonight or sometime tomorrow, both for syslog-ng 3.2 and for 3.3 aswell. On the other hand, if you have basic C knowledge, you can do this too, and it's actually quite fun to do so!
I've written a short blog post a few months ago about writing trivial template functions, that should be enough to get you started:
http://algernon.blogs.balabit.com/2011/01/howto-write-a-simple-template-func...
If anyone beats me to it: you're my guest for your (reasonable) choice of beverage if you ever come to Hungary, or wherever we might meet.
Ah, I didn't know that article, but it's definitely worth a read, who has a seemingly complex issue with generating filenames or content. PS: I'm thinking about how to invent a complete functional language in syslog-ng template functions, yammm :) No, not really :) -- Bazsi
Balazs Scheidler <bazsi@balabit.hu> writes:
PS: I'm thinking about how to invent a complete functional language in syslog-ng template functions, yammm :) No, not really :)
That sounds like a challenge! $(lisp), here I come! (And I was thinking I'll 'soon' run out of syslog-ng projects...) -- |8]
On Thu, 2011-04-07 at 15:01 +0200, Jakub Jankowski wrote:
Hi,
I'd like to implement simple hashing in file() destination, mainly to avoid ext3 32k directories limit, but also for convenience. Is it possible to do something simliar what this shell snippet does:
$ NAME=value $ echo ${NAME[*]:0:1} v $
but for macros (especially those coming from parsers)?
What I want is logs written to /var/log/h/ho/hos/hostname/service.log (with configurable level of subdirs) using a single destination().
There was a similar patch for IP addresses recently, which I haven't had the time to integrate properly, which created directories over IP address bytes (e.g. 10.0.0.1 would become 10/0/0/1) although that is more difficult to do with substring like functionality. hmm.... thinking out loud. there are basically two syntactic options: 1) to use the new template function syntax 2) to continue extending our bash-like expansion syntax. The 1) option would be something like: $(cut -c 1-10 $NAME) or $(substr $NAME 1 10) The alternative would be to add really simple functions as an addition to the basic template syntax. We currently have the "default" values like this way: ${NAME:-default} this results in "default" if $NAME is not-defined or empty. The template function stuff is easier to write (as you can write plugins for that, and there's a whole framework to support it). The bash syntax-like stuff is more difficult to write as the template parser is manually coded, and each expansion mode has to be open-coded, although it has better performance (since we can avoid a copy of $NAME into a temporary buffer). I'm somewhat undecided. What do others think? Can you also propose a syntax for 1) above? I wouldn't copy the bash case exactly, as that is an array operator if I'm right, and syslog-ng doesn't (yet) support arrays in name-value pairs. -- Bazsi
Friday 08 of April 2011 00:51:23 Balazs Scheidler napisał(a):
On Thu, 2011-04-07 at 15:01 +0200, Jakub Jankowski wrote:
I'd like to implement simple hashing in file() destination, mainly to avoid ext3 32k directories limit, but also for convenience. Is it possible to do something simliar what this shell snippet does: $ NAME=value $ echo ${NAME[*]:0:1} v $ but for macros (especially those coming from parsers)? [...] hmm.... thinking out loud. there are basically two syntactic options:
1) to use the new template function syntax
I had a look at the code yesterday, using Gergely's pointers from the other mail (thanks for that!), but one thing stopped me from doing some pointer arithmetics there myself - I wasn't sure how it'd work for unicode (i.e. multibyte) strings. Also pity that glib doesn't have a g_substr() :)
2) to continue extending our bash-like expansion syntax.
The 1) option would be something like:
$(cut -c 1-10 $NAME)
or
$(substr $NAME 1 10)
As for the syntax - I'd stick with substr <string> <offset> <length>, just like Perl, PHP, or MySQL do.
The alternative would be to add really simple functions as an addition to the basic template syntax. We currently have the "default" values like this way:
${NAME:-default}
this results in "default" if $NAME is not-defined or empty.
The template function stuff is easier to write (as you can write plugins for that, and there's a whole framework to support it). The bash syntax-like stuff is more difficult to write as the template parser is manually coded, and each expansion mode has to be open-coded, although it has better performance (since we can avoid a copy of $NAME into a temporary buffer).
Efficiency is one of the factors I need to consider, so I'd love to see that ${macro:offset:len} working eventually, but for a starter, having it even as a (slower) template function would be neat too. Do you think syslog-ng could have both implemented $someday?
I'm somewhat undecided. What do others think? Can you also propose a syntax for 1) above? I wouldn't copy the bash case exactly, as that is an array operator if I'm right, and syslog-ng doesn't (yet) support arrays in name-value pairs.
It works with simple syntax as well ${foo:0:1}; in fact - ${foo[*]:x:y} doesn't work in bash < 3.1. I wouldn't mind syslog-ng using the simpler form :) Regards, -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
Jakub Jankowski <shasta@toxcorp.com> writes:
Friday 08 of April 2011 00:51:23 Balazs Scheidler napisał(a):
On Thu, 2011-04-07 at 15:01 +0200, Jakub Jankowski wrote:
I'd like to implement simple hashing in file() destination, mainly to avoid ext3 32k directories limit, but also for convenience. Is it possible to do something simliar what this shell snippet does: $ NAME=value $ echo ${NAME[*]:0:1} v $ but for macros (especially those coming from parsers)? [...] hmm.... thinking out loud. there are basically two syntactic options:
1) to use the new template function syntax
I had a look at the code yesterday, using Gergely's pointers from the other mail (thanks for that!), but one thing stopped me from doing some pointer arithmetics there myself - I wasn't sure how it'd work for unicode (i.e. multibyte) strings. Also pity that glib doesn't have a g_substr() :)
I wouldn't worry about unicode for now, we could call it $(cut), as Bazsi suggested, and document that it workes on bytes treams, and has no knowledge of unicode. For the vast majority of cases, that's good enough in my opinion. I couldn't think of a case where I would want to split unicode strings in template functions. (Also, there's g_strstr_len() in glib, which is almost like substr)
2) to continue extending our bash-like expansion syntax.
The 1) option would be something like:
$(cut -c 1-10 $NAME)
or
$(substr $NAME 1 10)
As for the syntax - I'd stick with substr <string> <offset> <length>, just like Perl, PHP, or MySQL do.
$(substr $NAME 1 10) is also easier to parse (and syslog-ng does that for us to begin with) than -c 1-10 (which would require a second parse), so I'd vote for the latter too.
The alternative would be to add really simple functions as an addition to the basic template syntax. We currently have the "default" values like this way:
${NAME:-default}
this results in "default" if $NAME is not-defined or empty.
The template function stuff is easier to write (as you can write plugins for that, and there's a whole framework to support it). The bash syntax-like stuff is more difficult to write as the template parser is manually coded, and each expansion mode has to be open-coded, although it has better performance (since we can avoid a copy of $NAME into a temporary buffer).
Efficiency is one of the factors I need to consider, so I'd love to see that ${macro:offset:len} working eventually, but for a starter, having it even as a (slower) template function would be neat too. Do you think syslog-ng could have both implemented $someday?
${macro:offset:len} sounds like a good candidate for a HOWTO! But, I'll knock up $(substr) first (though, due to work commitments, that'll be done later tonight, later than I've planned). I'll post the patch in this thread once it's ready. My offer still stands, nevertheless! -- |8]
participants (3)
-
Balazs Scheidler
-
Gergely Nagy
-
Jakub Jankowski