- syslog-ng - lists.balabit.hu

Re: [syslog-ng] db-parser reuse for multiple logs?
by Jim Hendrick 14 Apr '15

14 Apr '15

I think it would depend on the filter. If it uses something that is readily available then maybe filter first. If is a match within the message, then possibly not. The patterndb parser is very fast. I have a single system pulling 24 fields out of a 7000 event per second load and it shows no sign of stress. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Mikkel Leth Carlsen <mlca(a)tdc.dk> Date: 04/10/2015 2:11 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] db-parser reuse for multiple logs? Hi Evan Thanks! I'm not too familiar with how syslog-ng actually implements the configuration and processes syslog messages, but I would think that parsing before filtering could have an impact on performance? I.e. I would want to exclude unwanted data before applying the parser - which I assume is a more costly operation even if it does not match? Perhaps by inserting: filter(filter_host1_or_host2); before the parser below? /Mikkel > -----Oprindelig meddelelse----- > Fra: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng- > bounces(a)lists.balabit.hu] På vegne af Evan Rempel > Sendt: 8. april 2015 17:38 > Til: syslog-ng(a)lists.balabit.hu > Emne: Re: [syslog-ng] db-parser reuse for multiple logs? > > I don't have an answer to your actual question, however, you can work > around it by > > log { > source(src_udp); > parser(myparser); > log { > filter(filter_host1); > destination(dst_host1); > flags(final); > }; > log { > filter(filter_host2); > destination(dst_host2); > flags(final); > }; > }; > > Hope that helps. > > Evan. > > On 04/08/2015 04:31 AM, Mikkel Leth Carlsen wrote: > > Hi > > > > Are db-parsers defined in syslog-ng configurations not reusable for > multiple logs? A simplified example (syslog 3.6.2): > > > > parser myparser { > > db_parser( > > file("/usr/local/etc/patterndb.d/myparser.xml") > > ); > > }; > > > > template mytemplate { > > template("${A};${B};${C}\n"); > > } > > > > filter filter_host1 { > > netmask(10.0.0.1/255.255.255.255); > > }; > > > > filter filter_host2 { > > netmask(10.0.0.2/255.255.255.255); > > }; > > > > destination dst_host1 { > > file("host1.log" perm(0644) template(mytemplate)); > > }; > > > > > > destination dst_host2 { > > file("host2.log" perm(0644) template(mytemplate)); > > }; > > > > log { > > source(src_udp); > > filter(filter_host1); > > parser(myparser); > > destination(dst_host1); > > flags(final); > > }; > > > > log { > > source(src_udp); > > filter(filter_host2); > > parser(myparser); > > destination(dst_host2); > > flags(final); > > }; > > > > This seems to work as expected and 'syslog-ng -s' does not report any > problems, but I see the following in the syslog-ng internal log: > > > > Internal error, duplicate configuration elements refer to the same > persistent config; name='db- > parser(/usr/local/etc/patterndb.d/myparser.xml)' > > Internal error, duplicate configuration elements refer to the same > persistent config; name='db- > parser(/usr/local/etc/patterndb.d/myparser.xml)' > > > > /Mikkel > > > _______________________________________________________________________ > _______ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > _______________________________________________________________________ > _______ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

5 7

Re: [syslog-ng] Understanding Sources & Destinations
by Jim Hendrick 10 Apr '15

10 Apr '15

This could be used for example to process historical log data as part of an investigation. Say you have a search tool (like ELK or something that needs preprocessing to json, sql, etc). Being able to send those logs through a command line version that does the exact same processing as the real time one would be extremely valuable. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: "Scheidler, Balázs" <balazs.scheidler(a)balabit.com> Date: 04/10/2015 12:39 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Subject: Re: [syslog-ng] Understanding Sources & Destinations Hi, The command line tool should work primarily between stdin/stdout just like for instance awk. We may make it possible to fire up tcp listeners but the primary use-case is batch processing already received data. Another use case is to debug configuration snippets in a faster way than edit/reload/test cycle that nromal syslog-ng offers. On Apr 10, 2015 12:55 AM, "Asadullah Hussain" <asadxflow(a)gmail.com> wrote: Hi Guys, I am working to understand the task of developing syslog-ng as a command line tool (GSoC2015) and I am trying to figure what "possible outputs" the command line tool will take. As per my understanding the input & outputs to syslog are defined as Sources & Destinations (which are places where applications output their logs from respectively). I have read the documentation about sources & destinations [Table 6.2 ](http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#chapter-sources) and my understanding is that syslog has following 3 types of sources: files, streams on tcp/ip sockets and stdout from a program. and 3 types of destinations: files or remote hosts (TCP/IP socket) and applications (mentioned in section 7). So are these all possible types of inputs/outputs that syslog-ng can have? which have to be supported by the command line tool. Or am I missing some points? -- Cheers, Asadullah Hussain ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

2 1

db-parser reuse for multiple logs?
by Mikkel Leth Carlsen 10 Apr '15

10 Apr '15

Hi Are db-parsers defined in syslog-ng configurations not reusable for multiple logs? A simplified example (syslog 3.6.2): parser myparser { db_parser( file("/usr/local/etc/patterndb.d/myparser.xml") ); }; template mytemplate { template("${A};${B};${C}\n"); } filter filter_host1 { netmask(10.0.0.1/255.255.255.255); }; filter filter_host2 { netmask(10.0.0.2/255.255.255.255); }; destination dst_host1 { file("host1.log" perm(0644) template(mytemplate)); }; destination dst_host2 { file("host2.log" perm(0644) template(mytemplate)); }; log { source(src_udp); filter(filter_host1); parser(myparser); destination(dst_host1); flags(final); }; log { source(src_udp); filter(filter_host2); parser(myparser); destination(dst_host2); flags(final); }; This seems to work as expected and 'syslog-ng -s' does not report any problems, but I see the following in the syslog-ng internal log: Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)' Internal error, duplicate configuration elements refer to the same persistent config; name='db-parser(/usr/local/etc/patterndb.d/myparser.xml)' /Mikkel

4 4

Understanding Sources & Destinations
by Asadullah Hussain 10 Apr '15

10 Apr '15

Hi Guys, I am working to understand the task of developing syslog-ng as a command line tool (GSoC2015) and I am trying to figure what "possible outputs" the command line tool will take. As per my understanding the input & outputs to syslog are defined as Sources & Destinations (which are places where applications output their logs from respectively). I have read the documentation about sources & destinations [Table 6.2 ]( http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guid…) and my understanding is that syslog has following 3 types of sources: files, streams on tcp/ip sockets and stdout from a program. and 3 types of destinations: files or remote hosts (TCP/IP socket) and applications (mentioned in section 7). So are these all possible types of inputs/outputs that syslog-ng can have? which have to be supported by the command line tool. Or am I missing some points? -- Cheers, Asadullah Hussain

2 1

UDP errors and lost UDP messages
by Matt Zagrabelny 07 Apr '15

07 Apr '15

Greetings list! Using syslog-ng 3.1 with Debian Squeeze, 2.6.32-5-amd64. The system has 8GB of RAM. I'm losing some UDP logs. I know to not use UDP - we use TLS for our Debian systems, but our Cisco gear leaves us with few options. According to netstat, the rate is anywhere from 600 to 3000 UDP errors per second. Using a tcpdump query of "dst port 514" show about the same rate of UDP traffic coming to the system. I've bumped the buffer size according to various docs: $ head -n -0 /proc/sys/net/core/[rw]mem_* ==> /proc/sys/net/core/rmem_default <== 16777216 ==> /proc/sys/net/core/rmem_max <== 16777216 ==> /proc/sys/net/core/wmem_default <== 16777216 ==> /proc/sys/net/core/wmem_max <== 16777216 And the udp specific memory limits: $ head -n -0 /proc/sys/net/ipv4/*udp* ==> /proc/sys/net/ipv4/udp_mem <== 768384 1024512 1536768 ==> /proc/sys/net/ipv4/udp_rmem_min <== 16777216 ==> /proc/sys/net/ipv4/udp_wmem_min <== 16777216 My UDP source for syslog-ng is also using a larger buffer: $ grep -A4 -B1 'udp(' /etc/syslog-ng/syslog-ng.conf source s_udp { udp( keep_hostname(yes) so_rcvbuf(16777216) ); }; According to syslog-ng-ctl stats the system is processing ~270 UDP messages per second. This hasn't really changed since I've made the kernel variable tweaks, nor after changing the so_rcvbuf size either. Any ideas of what to look for next? Thanks! -m

3 4

so_rcvbuf default value of 0
by Matt Zagrabelny 07 Apr '15

07 Apr '15

Greetings, Just looking for confirmation about the so_rcvbuf parameter for the udp() source driver. The default is 0. Does that mean that syslog-ng uses the default value from the kernel? ie. cat /proc/sys/net/core/rmem_default 124928 If that is true, then perhaps the syslog-ng admin docs could mention that. I've scanned "man 7 socket" and don't seen anything that suggests that a value of 0 for so_rcvbuf means that the socket will inherit the value. Thanks for the help! -m

2 2

Compiling Github Source (not Balabit source)
by Asadullah Hussain 03 Apr '15

03 Apr '15

Hello, I have recently started exploring the syslog source with plan of contributing in future. I am able to run syslog using the source from Balabit site. But is there a way to compile the Github source? I have tried using Autoconf/Autoscan to generate the configure files but without success. I just have future pull requests in mind as Github source and Balabit source differ in a few files). -- Cheers, Asad

3 4

Re: [syslog-ng] Compiling Github Source (not Balabit source)
by Laszlo Budai 03 Apr '15

03 Apr '15

Hi Asad, I've created a docker image which can helps you: https://registry.hub.docker.com/u/lbudai/docker-syslog-ng-build-env/ L. Sent from my HTC ----- Reply message ----- From: "Asadullah Hussain" <asadxflow(a)gmail.com> To: <syslog-ng(a)lists.balabit.hu> Subject: [syslog-ng] Compiling Github Source (not Balabit source) Date: Thu, Apr 2, 2015 11:33 Hello, I have recently started exploring the syslog source with plan of contributing in future. I am able to run syslog using the source from Balabit site. But is there a way to compile the Github source? I have tried using Autoconf/Autoscan to generate the configure files but without success. I just have future pull requests in mind as Github source and Balabit source differ in a few files). -- Cheers, Asad ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

Re: [syslog-ng] Compiling Github Source (not Balabit source) (Asadullah Hussain)
by Anshul Kushwaha 02 Apr '15

02 Apr '15

Hi Asad, You can try running autoconf ./autogen.sh ./configure make make install for compiling from the github source. This worked well for me. If there are errors while running autoconf, try posting them here or on irc. Regards, Anshul Kushwaha

1 0

Stats missing from remote sources in syslog-ng 3.5?
by Pradeep Sanders 31 Mar '15

31 Mar '15

Hello, I'm seeing a similar problem to what was reported here: http://communities.balabit.com/balabit/topics/stats-from-syslog-remote-sour… <http://communities.balabit.com/balabit/topics/stats-from-syslog-remote-sour…> I'm trying to upgrade from the RHEL 6 default 3.2 version to 3.5 or 3.6, in order to enable threaded support. I've tried a 3.6 build from the following location: https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng36epel6/epe… <https://copr-be.cloud.fedoraproject.org/results/czanik/syslog-ng36epel6/epe…> However, this build consumes all my host memory (48GB) in a matter of hours before starting on swap space. I've tried a 3.5.4 and a 3.5.6 package but both of them exhibit the same issue, with remote sources not reported in "syslog-ng-ctl stats". Does anyone have any advice on how I might fix the stats for remote sources in 3.5? Thanks, -Pradeep

2 1