I appreciate the help Bazsi, I receive around 600 of this messages daily, do you know when this messages are generated? I there any way to change that time to lets say one message/hour? TIA!<br><br>On Sun, 2007-04-29 at 10:25 -0600, Fabian Pucciarelli wrote:
<br>> I have syslog ng writing to a mysql pipe and then a little script<br>> reading the pipe and writing to the db. I currently receive many<br>> messages like the following, I wonder if somenone can help me figure
<br>> out what this message is saying..... thanks.<br>><br>><br>> | mailux | syslog | notice | notice | 2d | 2007-04-13 | 01:16:40<br>> | syslog-ng | syslog-ng[30548]: Log statistics;<br>> dropped='pipe(/tmp/mysql.pipe)
<div style="direction: ltr;">=0', processed='center(queued)=40295',<br>> processed='center(received)=13206',<br>> processed='destination(d_mysql)=40295', processed='source(net_tcp)=0',
<br>> processed='source(src)=8962', processed='source(net_udp)=4244' |<br>> 46674 |<br><br>This contains the various message counters that syslog-ng maintains<br>internally.<br><br>'processed' counters just count how much messages were processed at the
<br>given point, 'dropped' counters indicate how much log messages were<br>dropped by syslog-ng itself.<br><br>The string in the parenthesis tell were the given counter is counting<br>inside syslog-ng.<br><br>'center' is the main log message dispatching mechanism, it received
<br>13206 messages from various sources and then sent 40295 messages out,<br>e.g. you are probably sending individual messages to multiple<br>destinations.<br><br>You have two sources, one named 'src' generated 8962, the other called
<br>'net_udp' 4244 messages.<br><br>--<br>Bazsi</div><br><br><div><span class="gmail_quote">On 4/30/07, <b class="gmail_sendername"><a href="mailto:syslog-ng-request@lists.balabit.hu">syslog-ng-request@lists.balabit.hu
</a></b> <<a href="mailto:syslog-ng-request@lists.balabit.hu">syslog-ng-request@lists.balabit.hu</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Send syslog-ng mailing list submissions to<br> <a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>or, via email, send a message with subject or body 'help' to<br> <a href="mailto:syslog-ng-request@lists.balabit.hu">syslog-ng-request@lists.balabit.hu
</a><br><br>You can reach the person managing the list at<br> <a href="mailto:syslog-ng-owner@lists.balabit.hu">syslog-ng-owner@lists.balabit.hu</a><br><br>When replying, please edit your Subject line so it is more specific
<br>than "Re: Contents of syslog-ng digest..."<br><br><br>Today's Topics:<br><br> 1. Re: syslog-ng Digest, Vol 24, Issue 32 (Fabian Pucciarelli)<br> 2. Re: turn off case sensitivity for match regex filter
<br> (Balazs Scheidler)<br> 3. Re: Re: syslog-ng Digest, Vol 24, Issue 32 (Balazs Scheidler)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Sun, 29 Apr 2007 10:25:18 -0600
<br>From: "Fabian Pucciarelli" <<a href="mailto:fabiangp@gmail.com">fabiangp@gmail.com</a>><br>Subject: [syslog-ng] Re: syslog-ng Digest, Vol 24, Issue 32<br>To: <a href="mailto:syslog-ng@lists.balabit.hu">
syslog-ng@lists.balabit.hu</a><br>Message-ID:<br> <<a href="mailto:1e6757090704290925n541deb73t13eace5731aced3b@mail.gmail.com">1e6757090704290925n541deb73t13eace5731aced3b@mail.gmail.com</a>><br>Content-Type: text/plain; charset="iso-8859-1"
<br><br>I have syslog ng writing to a mysql pipe and then a little script reading<br>the pipe and writing to the db. I currently receive many messages like the<br>following, I wonder if somenone can help me figure out what this message is
<br>saying..... thanks.<br><br><br>| mailux | syslog | notice | notice | 2d | 2007-04-13 | 01:16:40 |<br>syslog-ng | syslog-ng[30548]: Log statistics;<br>dropped='pipe(/tmp/mysql.pipe)=0', processed='center(queued)=40295',
<br>processed='center(received)=13206', processed='destination(d_mysql)=40295',<br>processed='source(net_tcp)=0', processed='source(src)=8962',<br>processed='source(net_udp)=4244' | 46674 |
<br><br>On 4/29/07, <a href="mailto:syslog-ng-request@lists.balabit.hu">syslog-ng-request@lists.balabit.hu</a> <<br><a href="mailto:syslog-ng-request@lists.balabit.hu">syslog-ng-request@lists.balabit.hu</a>> wrote:<br>
><br>> Send syslog-ng mailing list submissions to<br>> <a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><br>><br>> To subscribe or unsubscribe via the World Wide Web, visit<br>
> <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>> or, via email, send a message with subject or body 'help' to<br>>
<a href="mailto:syslog-ng-request@lists.balabit.hu">syslog-ng-request@lists.balabit.hu</a><br>><br>> You can reach the person managing the list at<br>> <a href="mailto:syslog-ng-owner@lists.balabit.hu">syslog-ng-owner@lists.balabit.hu
</a><br>><br>> When replying, please edit your Subject line so it is more specific<br>> than "Re: Contents of syslog-ng digest..."<br>><br>><br>> Today's Topics:<br>><br>> 1. Re: turn off case sensitivity for match regex filter
<br>> (Balazs Scheidler)<br>> 2. Re: turn off case sensitivity for match regex filter (stucky)<br>><br>><br>> ----------------------------------------------------------------------<br>><br>> Message: 1
<br>> Date: Sat, 28 Apr 2007 12:42:06 +0200<br>> From: Balazs Scheidler <<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu</a>><br>> Subject: Re: [syslog-ng] turn off case sensitivity for match regex<br>> filter
<br>> To: Syslog-ng users' and developers' mailing list<br>> <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>> Message-ID: <<a href="mailto:1177756926.14925.17.camel@bzorp.balabit">
1177756926.14925.17.camel@bzorp.balabit</a>><br>> Content-Type: text/plain<br>><br>> On Sat, 2007-04-28 at 01:52 -0700, stucky wrote:<br>> > Guys<br>> ><br>> > Playing around with ng 2 and I started looking at the match filter
<br>> > again.<br>> > Simple question that I cannot find an answer to anywhere on the net.<br>> > How do I turn off case sensitivity for the match target ?<br>> > I'd like the following line to match "error' or 'ERROR' or 'Error'
<br>> ><br>> > filter logparse { match("error"); };<br>> ><br>> > but of course it only matches 'error' since by default regex is case<br>> > sensitive.<br>> > Basically I'm trying to emulate 'grep -i'
<br>> > I guess I could do this :<br>> ><br>> > filter logparse { match("[Ee][Rr][Rr][Oo][Rr]"); }; but it'd be soo<br>> > much simpler to turn off case sensitivity.<br>><br>> Yes, you are right. But it's not currently possible. It should be
<br>> however, I'll try to add it in the nearfuture.<br>><br>> ><br>> > And while we're talking regex. Shouldn't the above line actually read<br>> > like this :<br>> ><br>> > filter logparse { match(".+error.+"); }; ?
<br>> ><br>> > meaning "anything followed by 'error' followed by anything"<br>> > Both appear to work so I assume the first line is interpreted by<br>> > syslog-ng like the second line correct ?
<br>><br>> syslog-ng interprets "match" the same as grep, e.g. it does not care<br>> where the pattern is found. if you want to match the beginning or the<br>> end of line, you need to use explicit ^ and $ characters.
<br>><br>> --<br>> Bazsi<br>><br>><br>><br>> ------------------------------<br>><br>> Message: 2<br>> Date: Sat, 28 Apr 2007 12:44:04 -0700<br>> From: stucky <<a href="mailto:stucky101@gmail.com">
stucky101@gmail.com</a>><br>> Subject: Re: [syslog-ng] turn off case sensitivity for match regex<br>> filter<br>> To: "Syslog-ng users' and developers' mailing list"<br>> <
<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>> Message-ID:<br>> <<a href="mailto:30997e260704281244g6f1225bfwc7da2b4e05fdfd9f@mail.gmail.com">30997e260704281244g6f1225bfwc7da2b4e05fdfd9f@mail.gmail.com
</a>><br>> Content-Type: text/plain; charset="utf-8"<br>><br>> Baszi<br>><br>> Cool. I'm in the middle of building a new infrastructure and would like to<br>> use this feature. I'm not a programmer but I assume adding this feature
<br>> shouldn't be very hard at all right ?<br>> If you had a rough ETA that'd help me.<br>><br>> thx<br>><br>> On 4/28/07, Balazs Scheidler <<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu
</a>> wrote:<br>> ><br>> > On Sat, 2007-04-28 at 01:52 -0700, stucky wrote:<br>> > > Guys<br>> > ><br>> > > Playing around with ng 2 and I started looking at the match filter<br>> > > again.
<br>> > > Simple question that I cannot find an answer to anywhere on the net.<br>> > > How do I turn off case sensitivity for the match target ?<br>> > > I'd like the following line to match "error' or 'ERROR' or 'Error'
<br>> > ><br>> > > filter logparse { match("error"); };<br>> > ><br>> > > but of course it only matches 'error' since by default regex is case<br>> > > sensitive.
<br>> > > Basically I'm trying to emulate 'grep -i'<br>> > > I guess I could do this :<br>> > ><br>> > > filter logparse { match("[Ee][Rr][Rr][Oo][Rr]"); }; but it'd be soo
<br>> > > much simpler to turn off case sensitivity.<br>> ><br>> > Yes, you are right. But it's not currently possible. It should be<br>> > however, I'll try to add it in the nearfuture.
<br>> ><br>> > ><br>> > > And while we're talking regex. Shouldn't the above line actually read<br>> > > like this :<br>> > ><br>> > > filter logparse { match(".+error.+"); }; ?
<br>> > ><br>> > > meaning "anything followed by 'error' followed by anything"<br>> > > Both appear to work so I assume the first line is interpreted by<br>> > > syslog-ng like the second line correct ?
<br>> ><br>> > syslog-ng interprets "match" the same as grep, e.g. it does not care<br>> > where the pattern is found. if you want to match the beginning or the<br>> > end of line, you need to use explicit ^ and $ characters.
<br>> ><br>> > --<br>> > Bazsi<br>> ><br>> > _______________________________________________<br>> > syslog-ng maillist - <a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu
</a><br>> > <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>> > Frequently asked questions at <a href="http://www.campin.net/syslog-ng/faq.html">
http://www.campin.net/syslog-ng/faq.html</a><br>> ><br>> ><br>><br>><br>> --<br>> stucky<br>> -------------- next part --------------<br>> An HTML attachment was scrubbed...<br>> URL:<br>>
<a href="http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070428/c3bdb0f9/attachment.html">http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070428/c3bdb0f9/attachment.html</a><br>><br>> ------------------------------
<br>><br>> _______________________________________________<br>> syslog-ng maillist - <a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><br>> <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>><br>><br>> End of syslog-ng Digest, Vol 24, Issue 32<br>> *****************************************<br>><br><br><br><br>--<br>Regards,<br><br>Fabian Pucciarelli
<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <a href="http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070429/92be486d/attachment-0001.html">http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070429/92be486d/attachment-0001.html
</a><br><br>------------------------------<br><br>Message: 2<br>Date: Sun, 29 Apr 2007 19:21:11 +0200<br>From: Balazs Scheidler <<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu</a>><br>Subject: Re: [syslog-ng] turn off case sensitivity for match regex
<br> filter<br>To: Syslog-ng users' and developers' mailing list<br> <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>Message-ID: <<a href="mailto:1177867271.9878.11.camel@bzorp.balabit">
1177867271.9878.11.camel@bzorp.balabit</a>><br>Content-Type: text/plain<br><br>On Sat, 2007-04-28 at 12:44 -0700, stucky wrote:<br>> Baszi<br>><br>> Cool. I'm in the middle of building a new infrastructure and would
<br>> like to use this feature. I'm not a programmer but I assume adding<br>> this feature shouldn't be very hard at all right ?<br>> If you had a rough ETA that'd help me.<br><br>attached patch implements it using perl-like syntax,
e.g.:<br><br>filter f_case { match('(?i)regexp'); };<br><br>It works in all filters that use regexps (e.g. match, host, program,<br>etc) The regexp must begin with '(?' or otherwise the flag will not be<br>
recognized.<br><br>Tomorrow's snapshot should contain it.<br><br>>From 47f53555268efb72ab8db2d620d7669b8e5dc7a4 Mon Sep 17 00:00:00 2001<br>From: Balazs Scheidler <<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu
</a>><br>Date: Sun, 29 Apr 2007 18:53:46 +0200<br>Subject: [PATCH] added support for Perl-like regexp flags to support case-ignoring matches<br><br>2007-04-28 Balazs Scheidler <<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu
</a>><br><br> * src/filter.c (filter_re_compile): parse '(?i)' at the beginning of<br> regexps as an ignore-case flag, just like Perl does<br>---<br> src/filter.c | 34 +++++++++++++++++++++++++++++++---
<br> tests/unit/test_filters.c | 5 +++++<br> 2 files changed, 36 insertions(+), 3 deletions(-)<br><br>diff --git a/src/filter.c b/src/filter.c<br>index 8e19440..cf7e3aa 100644<br>--- a/src/filter.c<br>+++ b/src/filter.c
<br>@@ -230,11 +230,39 @@ typedef struct _FilterRE<br> } FilterRE;<br><br> static gboolean<br>-filter_re_compile(const char *re, regex_t *regex)<br>+filter_re_compile(const gchar *re, regex_t *regex)<br> {<br>- int rc;<br>
+ gint rc;<br>+ const gchar *re_comp = re;<br>+ gint flags = REG_EXTENDED;<br><br>- rc = regcomp(regex, re, REG_EXTENDED);<br>+ if (re[0] == '(' && re[1] == '?')<br>+ {<br>+ gint i;<br>
+<br>+ for (i = 2; re[i] && re[i] != ')'; i++)<br>+ {<br>+ switch (re[i])<br>+ {<br>+ case 'i':<br>+ flags |= REG_ICASE;<br>+ break;
<br>+ }<br>+ }<br>+ if (re[i])<br>+ {<br>+ re_comp = &re[i + 1];<br>+ }<br>+ else<br>+ {<br>+ msg_error("Invalid regexp flags",<br>+ evt_tag_str("re", re),
<br>+ NULL);<br>+ return FALSE;<br>+ }<br>+ }<br>+<br>+ rc = regcomp(regex, re_comp, flags);<br> if (rc)<br> {<br> gchar buf[256];<br>diff --git a/tests/unit/test_filters.c b/tests/unit/test_filters.c
<br>index 58236d7..187b572 100644<br>--- a/tests/unit/test_filters.c<br>+++ b/tests/unit/test_filters.c<br>@@ -137,11 +137,16 @@ main(int argc G_GNUC_UNUSED, char *argv[] G_GNUC_UNUSED)<br><br> testcase("<15>Oct 15 16:17:01 host openvpn[2499]: PTHREAD support initialized", 0, filter_host_new("^host$"), 1);
<br> testcase("<15>Oct 15 16:17:01 host openvpn[2499]: PTHREAD support initialized", 0, filter_host_new("^hos$"), 0);<br>+ testcase("<15>Oct 15 16:17:01 host openvpn[2499]: PTHREAD support initialized", 0, filter_host_new("pthread"), 0);
<br> fprintf(stderr, "One \"invalid regular expressions\" message is to be expected\n");<br> TEST_ASSERT(filter_host_new("((") == NULL);<br><br>+ fprintf(stderr, "One \"invalid regular expressions\" message is to be expected\n");
<br>+ TEST_ASSERT(filter_host_new("(?iana") == NULL);<br>+<br> testcase("<15>Oct 15 16:17:01 host openvpn[2499]: PTHREAD support initialized", 0, filter_match_new(" PTHREAD "), 1);<br>
testcase("<15>Oct 15 16:17:01 host openvpn[2499]: PTHREAD support initialized", 0, filter_match_new("^PTHREAD$"), 0);<br>+ testcase("<15>Oct 15 16:17:01 host openvpn[2499]: PTHREAD support initialized", 0, filter_match_new("(?i)pthread"), 1);
<br> fprintf(stderr, "One \"invalid regular expression\" message is to be expected\n");<br> TEST_ASSERT(filter_match_new("((") == NULL);<br><br><br><br>--<br>Bazsi<br><br><br><br>------------------------------
<br><br>Message: 3<br>Date: Sun, 29 Apr 2007 19:24:51 +0200<br>From: Balazs Scheidler <<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu</a>><br>Subject: Re: [syslog-ng] Re: syslog-ng Digest, Vol 24, Issue 32<br>To: Syslog-ng users' and developers' mailing list
<br> <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>Message-ID: <<a href="mailto:1177867491.9878.16.camel@bzorp.balabit">1177867491.9878.16.camel@bzorp.balabit</a>><br>Content-Type: text/plain
<br><br>On Sun, 2007-04-29 at 10:25 -0600, Fabian Pucciarelli wrote:<br>> I have syslog ng writing to a mysql pipe and then a little script<br>> reading the pipe and writing to the db. I currently receive many<br>> messages like the following, I wonder if somenone can help me figure
<br>> out what this message is saying..... thanks.<br>><br>><br>> | mailux | syslog | notice | notice | 2d | 2007-04-13 | 01:16:40<br>> | syslog-ng | syslog-ng[30548]: Log statistics;<br>> dropped='pipe(/tmp/mysql.pipe)=0', processed='center(queued)=40295',
<br>> processed='center(received)=13206',<br>> processed='destination(d_mysql)=40295', processed='source(net_tcp)=0',<br>> processed='source(src)=8962', processed='source(net_udp)=4244' |
<br>> 46674 |<br><br>This contains the various message counters that syslog-ng maintains<br>internally.<br><br>'processed' counters just count how much messages were processed at the<br>given point, 'dropped' counters indicate how much log messages were
<br>dropped by syslog-ng itself.<br><br>The string in the parenthesis tell were the given counter is counting<br>inside syslog-ng.<br><br>'center' is the main log message dispatching mechanism, it received<br>13206 messages from various sources and then sent 40295 messages out,
<br>e.g. you are probably sending individual messages to multiple<br>destinations.<br><br>You have two sources, one named 'src' generated 8962, the other called<br>'net_udp' 4244 messages.<br><br>--<br>Bazsi
<br><br><br><br>------------------------------<br><br>_______________________________________________<br>syslog-ng maillist - <a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><br><a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br><br><br>End of syslog-ng Digest, Vol 24, Issue 33<br>*****************************************<br></blockquote></div><br><br clear="all"><br>-- <br>Regards,<br><br>
Fabian Pucciarelli