Syslog-ng & Distributed File Systems??
Is there a version of syslog-ng that can properly use distributed filesystems like GlusterFS? For example, I have two nodes each running syslog-ng but also sharing a GlusterFS filesystem. Can those independent syslog-ng processes running on different servers write to the same file on the GlusterFS?
I don't really know honestly, but I doubt that. This doesn't work properly on NFS either. ----- Original message -----
Is there a version of syslog-ng that can properly use distributed filesystems like GlusterFS? For example, I have two nodes each running syslog-ng but also sharing a GlusterFS filesystem. Can those independent syslog-ng processes running on different servers write to the same file on the GlusterFS? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I have a situation where syslog-ng processes a syslog line, users paserdb and does lots of work and finally sends the complete object via json to an external application. This application does some thinking and based on some other data sources needs to send the log message back into syslog-ng with a different set of TAGS so that it gets routed through syslog-ng to a different destination program. The problem I am having is that syslog-ng does not use the TAGS in the incoming json object. The TAGS get replaced with the TAGS on the "source" of the syslog-ng that reads the json object, and augmented with any patterndb processing. Can anyone think of a way to get some arbitrary set of TAGS (possibly in a different custom macro) placed into the TAGS macro so that all of the filters on tags can be used. For example, I could make a patterndb for each individual tag value, and invoke each patterndb on the MyTags value. If there is a match then tag the message with the TAG. I would need to know all of the TAGS in advance and would probably not perform all that well, but it would work. Thanks in advance for any other suggestions. -- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
Hi, the simplest way right now is the set-tag() rewrite operation, that together with condition() might do the trick. I know there should be an easier way, but I'm afraid there isn't. I'm thinking about how this should work in the long term, but right now I don't have a clear idea. ----- Original message -----
I have a situation where syslog-ng processes a syslog line, users paserdb and does lots of work and finally sends the complete object via json to an external application. This application does some thinking and based on some other data sources needs to send the log message back into syslog-ng with a different set of TAGS so that it gets routed through syslog-ng to a different destination program.
The problem I am having is that syslog-ng does not use the TAGS in the incoming json object. The TAGS get replaced with the TAGS on the "source" of the syslog-ng that reads the json object, and augmented with any patterndb processing.
Can anyone think of a way to get some arbitrary set of TAGS (possibly in a different custom macro) placed into the TAGS macro so that all of the filters on tags can be used.
For example, I could make a patterndb for each individual tag value, and invoke each patterndb on the MyTags value. If there is a match then tag the message with the TAG. I would need to know all of the TAGS in advance and would probably not perform all that well, but it would work.
Thanks in advance for any other suggestions.
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Did the ability to trigger on matches of the number of messages in the context every get merged into 3.4? I'm at a point where I could really use this functionality :-) https://lists.balabit.hu/pipermail/syslog-ng/2012-November/019691.html
as far as I remember, yes it did. It is a template function and not a macro. it is called $(context-length) ----- Original message -----
Did the ability to trigger on matches of the number of messages in the context every get merged into 3.4?
I'm at a point where I could really use this functionality :-)
https://lists.balabit.hu/pipermail/syslog-ng/2012-November/019691.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
so the syntax would be <action condition="$(context-length) == $num"> wher $num is some macro from the pattern used to match a line. Is that correct? Sent from my ASUS Pad Balazs Scheidler <bazsi77@gmail.com> wrote: as far as I remember, yes it did. It is a template function and not a macro. it is called $(context-length) ----- Original message -----
Did the ability to trigger on matches of the number of messages in the context every get merged into 3.4?
I'm at a point where I could really use this functionality :-)
https://lists.balabit.hu/pipermail/syslog-ng/2012-November/019691.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Evan Rempel <erempel@uvic.ca> writes:
so the syntax would be
<action condition="$(context-length) == $num">
wher $num is some macro from the pattern used to match a line.
Is that correct?
$num can be pretty much anything: a number, a macro, another template function - it is entirely up to you. It does not need to be extracted from the pattern, but that should work too. -- |8]
As of 2 days ago a new syslog-ng guide was published that now documents this :-) Slightly different syntax <action condition='"$(context-length)" >= "$max"'> Works like a charm. Also, it isn't specified that <tag>xxx</tag> can be in the <message> part of an action. syslog-ng never stops amazing me. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] on behalf of Gergely Nagy [algernon@balabit.hu] Sent: Saturday, April 13, 2013 5:32 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] min and max message count condition in correlation actions Evan Rempel <erempel@uvic.ca> writes:
so the syntax would be
<action condition="$(context-length) == $num">
wher $num is some macro from the pattern used to match a line.
Is that correct?
$num can be pretty much anything: a number, a macro, another template function - it is entirely up to you. It does not need to be extracted from the pattern, but that should work too. -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Could you please give an example of using 'context-length' condition? I wonder if I can use it for sending an alert to monitoring system when there are more than 'N' exceptions per 'T' second are sent by my app hosts. On Sun, Apr 14, 2013 at 5:30 AM, Evan Rempel <erempel@uvic.ca> wrote:
As of 2 days ago a new syslog-ng guide was published that now documents this :-)
Slightly different syntax
<action condition='"$(context-length)" >= "$max"'>
Works like a charm.
Also, it isn't specified that <tag>xxx</tag> can be in the <message> part of an action.
syslog-ng never stops amazing me.
Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [ syslog-ng-bounces@lists.balabit.hu] on behalf of Gergely Nagy [ algernon@balabit.hu] Sent: Saturday, April 13, 2013 5:32 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] min and max message count condition in correlation actions
Evan Rempel <erempel@uvic.ca> writes:
so the syntax would be
<action condition="$(context-length) == $num">
wher $num is some macro from the pattern used to match a line.
Is that correct?
$num can be pretty much anything: a number, a macro, another template function - it is entirely up to you. It does not need to be extracted from the pattern, but that should work too.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Best regards, Koldaev Anton
Well, what you are asking isn't achievable with syslog-ng itself. We do this at our site, but we have built an inftrastructure around syslog-ng that passes classified events (at first it is a syslog message) to programs which create other events that get passes via syslog-ng to other programs that finally create e-mail, tickets, jabber, SMS, twitter and IP phone alerts. What you want to do is a great idea, you just need more than syslog-ng to accomplish it. Evan Anton Koldaev <koldaevav@gmail.com> wrote: Could you please give an example of using 'context-length' condition? I wonder if I can use it for sending an alert to monitoring system when there are more than 'N' exceptions per 'T' second are sent by my app hosts. On Sun, Apr 14, 2013 at 5:30 AM, Evan Rempel <erempel@uvic.ca<mailto:erempel@uvic.ca>> wrote: As of 2 days ago a new syslog-ng guide was published that now documents this :-) Slightly different syntax <action condition='"$(context-length)" >= "$max"'> Works like a charm. Also, it isn't specified that <tag>xxx</tag> can be in the <message> part of an action. syslog-ng never stops amazing me. Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>] on behalf of Gergely Nagy [algernon@balabit.hu<mailto:algernon@balabit.hu>] Sent: Saturday, April 13, 2013 5:32 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] min and max message count condition in correlation actions Evan Rempel <erempel@uvic.ca<mailto:erempel@uvic.ca>> writes:
so the syntax would be
<action condition="$(context-length) == $num">
wher $num is some macro from the pattern used to match a line.
Is that correct?
$num can be pretty much anything: a number, a macro, another template function - it is entirely up to you. It does not need to be extracted from the pattern, but that should work too. -- |8] ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq -- Best regards, Koldaev Anton
Hmm... It feels like syslog-ng should to be able to do it on it's own: *> rate*: Specifies maximum how many messages should be generated in the specified time period in the following format: * <number-of-messages>/<period-in-seconds>*.
... then maximum one message is generated per minute for every host that sends a log message matching the rule. Excess messages are dropped.
Balabit guys: any way to force it not to drop messages by rate, but execute an action instead? (For example: execute log() if rate >= 10/60s) On Mon, Apr 15, 2013 at 8:35 PM, Evan Rempel <erempel@uvic.ca> wrote:
Well, what you are asking isn't achievable with syslog-ng itself. We do this at our site, but we have built an inftrastructure around syslog-ng that passes classified events (at first it is a syslog message) to programs which create other events that get passes via syslog-ng to other programs that finally create e-mail, tickets, jabber, SMS, twitter and IP phone alerts.
What you want to do is a great idea, you just need more than syslog-ng to accomplish it.
Evan
Anton Koldaev <koldaevav@gmail.com> wrote:
Could you please give an example of using 'context-length' condition? I wonder if I can use it for sending an alert to monitoring system when there are more than 'N' exceptions per 'T' second are sent by my app hosts.
On Sun, Apr 14, 2013 at 5:30 AM, Evan Rempel <erempel@uvic.ca> wrote:
As of 2 days ago a new syslog-ng guide was published that now documents this :-)
Slightly different syntax
<action condition='"$(context-length)" >= "$max"'>
Works like a charm.
Also, it isn't specified that <tag>xxx</tag> can be in the <message> part of an action.
syslog-ng never stops amazing me.
Evan. ________________________________________ From: syslog-ng-bounces@lists.balabit.hu [ syslog-ng-bounces@lists.balabit.hu] on behalf of Gergely Nagy [ algernon@balabit.hu] Sent: Saturday, April 13, 2013 5:32 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] min and max message count condition in correlation actions
Evan Rempel <erempel@uvic.ca> writes:
so the syntax would be
<action condition="$(context-length) == $num">
wher $num is some macro from the pattern used to match a line.
Is that correct?
$num can be pretty much anything: a number, a macro, another template function - it is entirely up to you. It does not need to be extracted from the pattern, but that should work too.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Best regards, Koldaev Anton
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Best regards, Koldaev Anton
Anton Koldaev <koldaevav@gmail.com> writes:
Balabit guys: any way to force it not to drop messages by rate, but execute an action instead? (For example: execute log() if rate >= 10/60s)
I'd love to teach syslog-ng to do that, but that needs quite a bit of work, and isn't likely to happen anytime soon, I'm afraid. (Unless I'm missing something, but I find that unlikely.) -- |8]
On 2013-04-16, Gergely Nagy wrote:
Anton Koldaev <koldaevav@gmail.com> writes:
Balabit guys: any way to force it not to drop messages by rate, but execute an action instead? (For example: execute log() if rate >= 10/60s)
I'd love to teach syslog-ng to do that, but that needs quite a bit of work, and isn't likely to happen anytime soon, I'm afraid. (Unless I'm missing something, but I find that unlikely.)
In the meantime - this is trivial to do with SEC[1] (Simple Event Correlator). The man page[2] is quite a good source of documentation, but I highly recommend reading this great two[3] part[4] article "Working with SEC", where you'll see examples of how to make SEC do exactly what OP wantts. Namely: type=SingleWithThreshold ptype=RegExp pattern=foo bar baz=(\S+) desc=$0 action=shellcmd /bin/echo -e "foo baz=$1 above 3/10s" | mail admin@example window=10 thresh=3 or, if you want "high" and "low" thresholds: type=SingleWith2Thresholds ptype=RegExp pattern=foo bar baz=(\S+) desc=$0 high action=shellcmd /bin/echo -e "foo baz=$1 above 3/10s" | mail admin@example window=10 thresh=3 desc2=$0 low action2=shellcmd /bin/echo -e "foo baz=$1 stopped" | mail admin@example window2=120 thresh2=0 Hope this helps. Cheers, Jakub. [1] http://simple-evcorr.sourceforge.net/ [2] http://simple-evcorr.sourceforge.net/man.html [3] http://simple-evcorr.sourceforge.net/SEC-tutorial/article.html [4] http://simple-evcorr.sourceforge.net/SEC-tutorial/article-part2.html -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
A while back I was asking about the json parser and how to get the TAGS from the json object into the TAGS of the syslog-ng internal structure so that I could make filters on them. Now I am trying again, following Balazs suggestion of using set-tag(), however, I find that the rewrite rewrite r_set_sometag { set-tag("sometag" condition(filter(f_should_set_sometag)) ); }; is not supported. I thought that I would be able to use the filter in the log statement such as log { source(s_some); log { filter(f_should_set_sometag); rewrite(r_set_sometag); destination(d_dest); }; }; This works for one tag, but I have multiple tags that I want to set which gives be a factorial(N) branching sequence which is not feasible. Using junction/channels ends up producing multiple output lines when multiple TAGS are set :-( Can anyone think of a way to set an arbitrary set of TAGS on a syslog message given an arbitrary set of "set-tag-flags" presented in other macro values? Should the set-tag rewrite support conditionals? Thanks in advance ... again. Evan. On 03/28/2013 09:56 PM, Balazs Scheidler wrote:
Hi,
the simplest way right now is the set-tag() rewrite operation, that together with condition() might do the trick.
I know there should be an easier way, but I'm afraid there isn't.
I'm thinking about how this should work in the long term, but right now I don't have a clear idea.
----- Original message -----
I have a situation where syslog-ng processes a syslog line, users paserdb and does lots of work and finally sends the complete object via json to an external application. This application does some thinking and based on some other data sources needs to send the log message back into syslog-ng with a different set of TAGS so that it gets routed through syslog-ng to a different destination program.
The problem I am having is that syslog-ng does not use the TAGS in the incoming json object. The TAGS get replaced with the TAGS on the "source" of the syslog-ng that reads the json object, and augmented with any patterndb processing.
Can anyone think of a way to get some arbitrary set of TAGS (possibly in a different custom macro) placed into the TAGS macro so that all of the filters on tags can be used.
For example, I could make a patterndb for each individual tag value, and invoke each patterndb on the MyTags value. If there is a match then tag the message with the TAG. I would need to know all of the TAGS in advance and would probably not perform all that well, but it would work.
The requirement of pythong >= 2.5 make compiling syslog 3.4 a problem for Redhat 5.x which only comes iwth python 2.4 Is there any way you could relax this requirement to live with python 2.4? Redhat 5.x is still a fully supported OS and it would be nice not to limit these hosts to syslog 3.3 Thanks, -- Evan Rempel
do you happen to have the error message that happens with 2.4? that way I could judge it a bit better w/o having to install rhel5 myself. also, you might be able to compile syslog-ng on a different host, but I might consider adding compatibility fixes to make things simpler. ----- Original message -----
The requirement of pythong >= 2.5 make compiling syslog 3.4 a problem for Redhat 5.x which only comes iwth python 2.4
Is there any way you could relax this requirement to live with python 2.4?
Redhat 5.x is still a fully supported OS and it would be nice not to limit these hosts to syslog 3.3
Thanks, -- Evan Rempel ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Ahhh I should have looked at the build log a little better. ... === configuring in modules/afamqp/rabbitmq-c (/usr/local/uvredhat/BUILD/syslog-ng-3.4.1/modules/afamqp/rabbitmq-c) configure: running /bin/sh ./configure.gnu --disable-option-checking '--prefix=/usr/local' '--libdir=/usr/local/lib64' '--with-module-dir=/usr/local/lib64/syslog-ng' '--mandir=/usr/local/share/man' '--infodir=/usr/local/share/info' '--sysconfdir=/usr/local/etc/syslog-ng' '--localstatedir=/var/lib/syslog-ng' '--enable-pcre' 'PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/share/pkgconfig' '--disable-tcp-wrapper' --cache-file=/dev/null --srcdir=. ./configure.gnu Running: ./configure --disable-option-checking --prefix=/usr/local --libdir=/usr/local/lib64 --with-module-dir=/usr/local/lib64/syslog-ng --mandir=/usr/local/share/man --infodir=/usr/local/share/info --sysconfdir=/usr/local/etc/syslog-ng --localstatedir=/var/lib/syslog-ng --enable-pcre PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/share/pkgconfig --disable-tcp-wrapper --cache-file=/dev/null --srcdir=. --disable-shared --enable-static --with-pic ... checking for a Python interpreter with version >= 2.5... none configure: error: no suitable Python interpreter found configure: error: ./configure.gnu failed for modules/afamqp/rabbitmq-c so it is a .configure limitation in the rabbitmq-c Thanks. On 04/03/2013 10:23 AM, Balazs Scheidler wrote:
do you happen to have the error message that happens with 2.4? that way I could judge it a bit better w/o having to install rhel5 myself.
also, you might be able to compile syslog-ng on a different host, but I might consider adding compatibility fixes to make things simpler.
----- Original message -----
The requirement of pythong >= 2.5 make compiling syslog 3.4 a problem for Redhat 5.x which only comes iwth python 2.4
Is there any way you could relax this requirement to live with python 2.4?
Redhat 5.x is still a fully supported OS and it would be nice not to limit these hosts to syslog 3.3
Thanks, -- Evan Rempel ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
----- Original message -----
Ahhh I should have looked at the build log a little better.
do you use amqp on these hosts? if not, I would simply disable in-tree building of rabbitmq in case the python version is too old. I would not want to change rabbitmq from upstream to support older python versions. btw, if you disable amqp you should be fine.
Yup, that is just what I am doing. and now I have a 3.4.1 rpm for my redhat 4 systems. Thanks for making me read :-) Evan. On 04/03/2013 12:03 PM, Balazs Scheidler wrote:
----- Original message -----
Ahhh I should have looked at the build log a little better.
do you use amqp on these hosts? if not, I would simply disable in-tree building of rabbitmq in case the python version is too old.
I would not want to change rabbitmq from upstream to support older python versions.
btw, if you disable amqp you should be fine.
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
It depends on what os you are using. There are some cluster softwares available for configuring this. For example, on redhat you can use its cluster software + gfs and syslog to accomplish this. 发自我的 iPad 在 2013-3-28，13:46，Jiann-Ming Su <su_js1@yahoo.com> 写道：
Is there a version of syslog-ng that can properly use distributed filesystems like GlusterFS? For example, I have two nodes each running syslog-ng but also sharing a GlusterFS filesystem. Can those independent syslog-ng processes running on different servers write to the same file on the GlusterFS? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (7)
-
Anton Koldaev
-
Balazs Scheidler
-
Evan Rempel
-
Gergely Nagy
-
Jakub Jankowski
-
Jian
-
Jiann-Ming Su