patterndb and context - access fields from initial message
Hi guys, I did have a conversation today on IRC with algernon (again many thanks to him) about the problem I am faced with and we agreed that I should send my question to the list. My aim is to consolidate a bunch of related syslog messages into a single message once the final message has come in (or a timeout has occured for whatever reason). To do this I use patterndb together with a defined context linking related messages to the context. Unfortunately the length of the context (i.e. the number of messages within the context) is unknown from the outset. Basically it (mainly) depends on how long the context is active for - which can be any time and is completely unforseeable. Now my problem is as follows: From the initial message I parse (using patterndb) certain fields into a number of named MACROS. This is all solved and works very well. When the final message comes in (or in case a timeout occurs) I would like to trigger an action (triggered by match or timeout respectively) to generate and subsequently inject a message about the full context using certain fields, mainly from the final message, but also including a number of fields from the inital message of the context. Generally a field is available using the syntax ${MACRO}@# where @# denotes a reference to a previous message within the same context - so ${MACRO}@5 would use the value of MACRO from a message with a distance of 5 from the curren message (where the distance of the current message to itself is 1 - so @5 references the 4th previous message). That also works very well but obviously requires that one knows which message to refer to using the distance value. The problem comes when the context-length is unknown and thus the distance of the initial message to the final message is not a fixed, pre-determined number but rather varies. Although there seems to be a variable/macro indicating/storing the context-length which (according to the manual) can be accessed by $(context-length) [NOTE: the paranthesis instead of the curly braces] the combination of the two does not seem to work out: I was expecting that ${MACRO}@$(context-length) would provide me with the required information - that is: the value of the parsed field from the initial message - but my construct unfortunately fails. Is there anything I am doing wrong here (i.e. is there any quoting/esacping required for certain characters), is my thought process flawed or does this not work/is it a bug. In case it does not work, how would I be able to achieve my goal. Many thanks and best regards, Atom2 P.S. The version of syslog-ng I am using is 3.4.7, compiled from sources, using the latest stable gentoo ebuild.
Hi, I'm AFK for a while but did you check out the `grep` template function? Cheers
Hi! I think, the negative notation could solve this situation eg.: $MACRO@-N would mean the first Nth message in the context and not the last Nth. I checked the code and it is not terrible hard to implement. I can make a PoC for you in the next week, if you would like to test it. Regards, Viktor On Tue, Jul 8, 2014 at 11:54 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
I'm AFK for a while but did you check out the `grep` template function?
Cheers
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Am 08.07.14 18:02, schrieb Tusa Viktor:
Hi!
I think, the negative notation could solve this situation eg.: $MACRO@-N would mean the first Nth message in the context and not the last Nth. I checked the code and it is not terrible hard to implement. I can make a PoC for you in the next week, if you would like to test it.
Regards, Viktor Thanks Victor, that sounds like a very promising solution and I am more than happy to test it. Auto-applying a patch to my 3.4.7 version is not really hard as I use gentoo and its emerge system to re-install ebuilds is very flexible given that gentoo is a source based distribution. The only thing is that I would prefer to stick to version 3.4.7 for my testst as this is the latest stable version available for gentoo. I hope that also works out for you.
I am looking forward to your patch whenever you have a chance to make it happen. Many thanks Atom2
On Tue, Jul 8, 2014 at 11:54 AM, Fabien Wernli <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote:
Hi,
I'm AFK for a while but did you check out the `grep` template function?
Thanks Fabien, I haven't checked your suggested grep template function yet and to be honest, up to your kind suggestion, I was not even aware of any such tenplate. Having looked at the documentation and assuming that I have fully understood it, I am however not sure whether it would be the best solution for at least the following two reasons: 1) I guess it might not be very efficient for a largish contexts as grep needs to search through all messages within the context which might be processing intensive and 2) If the same named macro is available in more than one message within the context (which is the case in my patterndb), I would get a comma-seperated list of (identical) values which would require further processing to extract only one of the values delivered by the tempate. I am sure that could somehow be sorted, but Victor's extension described above seems to be easier on the outset and it might also be of interest to others. Having said that, your approach is really creative and I have learned something new. Many thanks for that. Atom2
Cheers ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I think we should rather use a template function that operates on the entire context. I wouldn't use indexing any further than we have now, unless there's a very specific usecase. In the current one, the issue is that we don't know how many messages there are, If I understand correctly. On Tue, Jul 8, 2014 at 6:02 PM, Tusa Viktor <tusavik@gmail.com> wrote:
Hi!
I think, the negative notation could solve this situation eg.: $MACRO@-N would mean the first Nth message in the context and not the last Nth. I checked the code and it is not terrible hard to implement. I can make a PoC for you in the next week, if you would like to test it.
Regards, Viktor
On Tue, Jul 8, 2014 at 11:54 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
I'm AFK for a while but did you check out the `grep` template function?
Cheers
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Am 09.07.14 00:42, schrieb Balazs Scheidler:
I think we should rather use a template function that operates on the entire context. I wouldn't use indexing any further than we have now, unless there's a very specific usecase. In the current one, the issue is that we don't know how many messages there are, If I understand correctly. My (in terms of syslog-ng) uneducated view is slightly different, probably it boils down to a specific use-case:
I'd rather have the ability to access a specific message directly than use a template that goes through all messages in the context for a number of obvious reasons - probably there are more: Firstly I am convinced that accessing a message directly by using an index is more efficient than searching the full context for a specific message - although I have to admit that this clearly depends on the implementation details and that I don't know. So my assumption might well be incorrect. Secondly whilst it is true that the number of messages is unknown, I do, however, know in which message (measured from the initial message) the required information is stored. So the case for grep that states "The grep template function can be used to filter the messages of the same context when the index of the particular message is not known" is not really given. Giving this some more thought, it probably is a common scenario that initial messages of a context are predictable and you know what you require later on in a consolidated message (e.g. start time, ports, etc.). So having an option to access messages indexed from the initial message would make this very easy and thus would be a nice feature. Obviously this also holds for the final messages of a context - but that functionality is already there. Intermediary messages (which determine the context-length) may not really be interesting as those might just regularily indicate that the "context" is still alive (e.g Key exchange [Kex] messages in an ssh session context). For an ssh session it is the few inital messages and the few final messages that are required to do useful message consolidation. The intermediary messages serve the sole purpose of indicating "session still alive" and would ease error handling in case of unexpected connection drops (action "timeout" instead of "match"). So in essence having indexed access to both the inital and the final messages would in my view go a long way of easy and efficient message consolidation. Finally grep requires something to grep (i.e. search) for and that might not be easily available or it might be so wide that numerous matches will be returned requiring to extract the one message that is right from a comma separated list of values - another step that could easily be avoided by allowing indexed access to just the relevant messages. I hope this makes sense, Regards Atom2
On Tue, Jul 8, 2014 at 6:02 PM, Tusa Viktor <tusavik@gmail.com <mailto:tusavik@gmail.com>> wrote:
Hi!
I think, the negative notation could solve this situation eg.: $MACRO@-N would mean the first Nth message in the context and not the last Nth. I checked the code and it is not terrible hard to implement. I can make a PoC for you in the next week, if you would like to test it.
Regards, Viktor
On Tue, Jul 8, 2014 at 11:54 AM, Fabien Wernli <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote:
Hi,
I'm AFK for a while but did you check out the `grep` template function?
Cheers ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Yeah, this is what I wanted to mention, $(grep) iterates over all the messages in the context and evaluates its arguments for all messages. $(grep <syslog-ng filter> $MSG) returns a comma delimited list of $MSG fields where the filter matches. This is the only function that actually iterates over the context, but we could do more of those. On Tue, Jul 8, 2014 at 11:54 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
I'm AFK for a while but did you check out the `grep` template function?
Cheers
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
participants (4)
-
Atom2
-
Balazs Scheidler
-
Fabien Wernli
-
Tusa Viktor