[syslog-ng] patterndb and context - access fields from initial message

[Atom2]

Am 09.07.14 00:42, schrieb Balazs Scheidler:
> I think we should rather use a template function that operates on the
> entire context. I wouldn't use indexing any further than we have now,
> unless there's a very specific usecase. In the current one, the issue is
> that we don't know how many messages there are, If I understand correctly.
My (in terms of syslog-ng) uneducated view is slightly different, 
probably it boils down to a specific use-case:

I'd rather have the ability to access a specific message directly than 
use a template that goes through all messages in the context for a 
number of obvious reasons - probably there are more:

Firstly I am convinced that accessing a message directly by using an 
index is more efficient than searching the full context for a specific 
message - although I have to admit that this clearly depends on the 
implementation details and that I don't know. So my assumption might 
well be incorrect.

Secondly whilst it is true that the number of messages is unknown, I do, 
however, know in which message (measured from the initial message) the 
required information is stored. So the case for grep that states "The 
grep template function can be used to filter the messages of the same 
context when the index of the particular message is not known" is not 
really given.

Giving this some more thought, it probably is a common scenario that 
initial messages of a context are predictable and you know what you 
require later on in a consolidated message (e.g. start time, ports, 
etc.). So having an option to access messages indexed from the initial 
message would make this very easy and thus would be a nice feature. 
Obviously this also holds for the final messages of a context - but that 
functionality is already there.

Intermediary messages (which determine the context-length) may not 
really be interesting as those might just regularily indicate that the 
"context" is still alive (e.g Key exchange [Kex] messages in an ssh 
session context). For an ssh session it is the few inital messages and 
the few final messages that are required to do useful message 
consolidation. The intermediary messages serve the sole purpose of 
indicating "session still alive" and would ease error handling in case 
of unexpected connection drops (action "timeout" instead of "match").

So in essence having indexed access to both the inital and the final 
messages would in my view go a long way of easy and efficient message 
consolidation.

Finally grep requires something to grep (i.e. search) for and that might 
not be easily available or it might be so wide that numerous matches 
will be returned requiring to extract the one message that is right from 
a comma separated list of values - another step that could easily be 
avoided by allowing indexed access to just the relevant messages.

I hope this makes sense,

Regards Atom2

>
>
> On Tue, Jul 8, 2014 at 6:02 PM, Tusa Viktor <tusavik at gmail.com
> <mailto:tusavik at gmail.com>> wrote:
>
>     Hi!
>
>     I think, the negative notation could solve this situation eg.:
>     $MACRO at -N would mean the first Nth message in the context and not
>     the last Nth. I checked the code and it is not terrible hard to
>     implement. I can make a PoC for you in the next week, if you would
>     like to test it.
>
>     Regards,
>     Viktor
>
>
>     On Tue, Jul 8, 2014 at 11:54 AM, Fabien Wernli <wernli at in2p3.fr
>     <mailto:wernli at in2p3.fr>> wrote:
>
>         Hi,
>
>         I'm AFK for a while but did you check out the `grep` template
>         function?
>
>         Cheers
>         ______________________________________________________________________________
>         Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>         Documentation:
>         http://www.balabit.com/support/documentation/?product=syslog-ng
>         FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Documentation:
>     http://www.balabit.com/support/documentation/?product=syslog-ng
>     FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>