- syslog-ng - lists.balabit.hu

Issue regarding the configurations of syslog-ng application on Centos
by Christian Hangi 09 May '14

09 May '14

Hello Guys, I am new in Syslog-ng application and also Linux and I need your help regarding the functionality of syslog-ng on Linux. I am building a syslog-ng server capable of receiving syslog messages from syslog-ng clients. In doing so, I would like to those syslog messages to be received and stored inside the Syslog MySQL database of the server inside the syslog_incoming table so that I can be able view those messages. I am using the Centos 6.5 of Redhat distributions and the Syslog-ng application running on that PC acts as a Syslog-ng server. This is my first attempt to allow the syslog messages storage on the database: @version:3.2 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; #source s_sys { #file ("/proc/kmsg" program_override("kernel: ")); # unix-stream ("/dev/log"); # internal(); # udp(ip(0.0.0.0) port(514)); #}; #A change was made here! source network { syslog(ip(0.0.0.0) port(514) transport(udp));}; destination db_mysql{sql(type(mysql) host("localhost") username("cacti") password("P@ssw0rd") database("syslog") table("syslog_incoming") columns("host", "facility", "priority", "date", "time", "message") values("$HOST","$FACILITY","$PRIORITY","$YEAR-$MONTH-$DAY","$HOUR:$MIN:$SEC","$MSG") indexes("host", "facility", "priority", "date", "time", "message") );}; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #A change was made here! log { source(s_sys);source(network); destination(d_mysql);}; #log { source(network); destination(d_mysql);}; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; #log { source(s_sys); filter(f_kernel); destination(d_kern); }; #log { source(s_sys); filter(f_default); destination(d_mesg); }; #log { source(s_sys); filter(f_auth); destination(d_auth); }; #log { source(s_sys); filter(f_mail); destination(d_mail); }; #log { source(s_sys); filter(f_emergency); destination(d_mlal); }; #log { source(s_sys); filter(f_news); destination(d_spol); }; #log { source(s_sys); filter(f_boot); destination(d_boot); }; #log { source(s_sys); filter(f_cron); destination(d_cron); }; # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #A change was made here! log { source(network); destination(d_mysql);}; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; #log { source(s_sys); filter(f_kernel); destination(d_kern); }; #log { source(s_sys); filter(f_default); destination(d_mesg); }; #log { source(s_sys); filter(f_auth); destination(d_auth); }; #log { source(s_sys); filter(f_mail); destination(d_mail); }; #log { source(s_sys); filter(f_emergency); destination(d_mlal); }; #log { source(s_sys); filter(f_news); destination(d_spol); }; #log { source(s_sys); filter(f_boot); destination(d_boot); }; #log { source(s_sys); filter(f_cron); destination(d_cron); }; # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: And this is the message I got when I tried to restart the application : Stopping syslog-ng: [FAILED] Starting syslog-ng: Error in configuration, unresolved destination reference; destination='d_mysql' [FAILED] Then I tried to use a named pipe to accept and send thoses messages to the syslog_incoming table of the Syslog MySQL database through a template: @version:3.2 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; #source s_sys { # file ("/proc/kmsg" program_override("kernel: ")); # unix-stream ("/dev/log"); # internal(); # udp(ip(0.0.0.0) port(514)); #}; destination d_mysql{ pipe("/var/log/mysql.pipe" template("INSERT INTO syslog_incoming(host, facility, priority, date, time, message) VALUES('$HOST', '$FACILITY', '$PRIORITY', '$ YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$MSG');\n") template-escape(yes)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #A change was made here! log { source(network); destination(d_mysql);}; #log { source(network); destination(d_mysql);}; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; #log { source(s_sys); filter(f_kernel); destination(d_kern); }; #log { source(s_sys); filter(f_default); destination(d_mesg); }; #log { source(s_sys); filter(f_auth); destination(d_auth); }; #log { source(s_sys); filter(f_mail); destination(d_mail); }; #log { source(s_sys); filter(f_emergency); destination(d_mlal); }; #log { source(s_sys); filter(f_news); destination(d_spol); }; #log { source(s_sys); filter(f_boot); destination(d_boot); }; #log { source(s_sys); filter(f_cron); destination(d_cron); }; # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: And I got this message assuming that the syslog-ng server is working Stopping syslog-ng: [ OK ] Starting syslog-ng: [ OK ] But when I checked to the table of the database it was empty. So, I would like to know, do you miss something on the first attempt to allow the syslog-server to work perfectly or should I just not to uncomment the source system (s_sys) to accept the syslog messages from the Syslog-clients? On the 2nd attempt, what needs to be done to acctually see those messages? Please I need some advice on those two attempts. Kindly regards, Chris Hangi

2 1

[ANN]: syslog-ng Incubator maintenance
by Gergely Nagy 07 May '14

07 May '14

Hi! While it has effectively happened about a month ago, I didn't get around to announce the good news yet: syslog-ng Incubator maintenance has been handed over to Viktor Tusa <tusa(a)balabit.hu>, author of (in no particular order) the Lua and RSS destinations, graphite-output template function, and more. He has been doing a terrific job so far, and I'm very happy with the maintainer switch. The Incubator is in the best hands possible. Thank you, Viktor! -- |8]

1 0

AMQP source
by MÓZES Ádám István 05 May '14

05 May '14

Hi All, My name is MÓZES Ádám István, and I'll develop the AMQP source this summer. A short introduction about me and the progress of the development can be found at this blog: http://dnsjts.blogspot.hu/search/label/gsoc2014 . Best regards, Ádám

1 0

Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages
by Chaman Chakalaka 05 May '14

05 May '14

Just to close the circle on this, I found the problem. After having increased the rmem_max I was no longer seeing any UDP packet drops. I was still not seeing some log messages, and at this point it was a configuration issue. I noticed I had filters such as: filter f_info { level(info);} Obviously this was only catching info level messages. Changed to: filter f_info {level(info...emerg);} Sorry for the noise, but I still wanted to bring some closure in case someone runs into this. The initial packet drop blinded me into looking into some basic configs, thanks all for the suggestions! > From: syslog-ng-request(a)lists.balabit.hu > Subject: syslog-ng Digest, Vol 108, Issue 27 > To: syslog-ng(a)lists.balabit.hu > Date: Wed, 30 Apr 2014 12:00:02 +0200 > > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. Re: Ubuntu Precise -ng filling out buffer, dropping messages > (Evan Rempel) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 29 Apr 2014 12:38:52 -0700 > From: Evan Rempel <erempel(a)uvic.ca> > Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, > dropping messages > To: syslog-ng(a)lists.balabit.hu > Message-ID: <535FFFCC.8000203(a)uvic.ca> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 04/29/2014 12:20 PM, Chaman Chakalaka wrote: > > I fixed the issue with udp being dropped at the system level by changing the linux core files but this time restarting the system, now I know they are missing somewhere between getting to the system and syslog-ng :( > > > > I'm lost once again... > > > > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > trysource s_network_udp { udp(so_rcvbuf(33554432) log_fetch_limit(20000) log_iw_size(1000000) ); }; > > > This assumes that you have a large net.core.rmem_max > > net.core.rmem_max = 52428800 > > See how that goes. > > > ------------------------------ > > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > End of syslog-ng Digest, Vol 108, Issue 27 > ******************************************

1 0

ZMQ transport, source and destination
by Laci Mészáros 04 May '14

04 May '14

Hy all, I am László Mészáros a student at Szechenyi Istvan University learning software engineering. I've made a proposal for Google Summer of Code to implement the zeromq transport, source and destination. The organization accepted it, so during this summer I will work on it. You can find the proposal here: https://github.com/balabit/syslog-ng/wiki/GSoC2014-Proposal:-ZMQ-transport,… I made a blog where you can follow the status, it can be found here: http://lmesz.tumblr.com/post/84705072064/introduction-aims I think one part it will be fun and on another it will be difficult, so I would like to ask all of you, if you have some use cases on your shelf about how you would/will use zmq for logging via network, please share it with us. You can send it to the list or to me personally it would be very useful during the implementation. Br, Laci

1 0

MongoDB destination TLS support
by György Demarcsek Ifj. 01 May '14

01 May '14

Hi All! My name is György Demarcsek and I'm a Computer Science BSc student at Budapest University of Technology and Economics. I will graduate in 2015, but planning to go for the MSc as well. I participate in the Google Summer of Code program this year, and my proposal "TLS Support for the MongoDB destination" has been accepted by the syslog-ng project. You may verify it right here: https://algernon.blogs.balabit.com/2014/04/gsoc2014-syslog-ng-accepted-proj… So this summer, I'll be coding TLS support for the MongoDB client ( https://github.com/algernon/libmongo-client) library used by syslog-ng ose. I'm really enthusiastic, I really want to do a great job. I will regularly report about the development process on my blog - created mostly for this - at http://demarcsek92.blogspot.hu/search/label/gsoc2014 so you can all follow my process and comment on anything. Suggestions, tips, critics are all welcome from anyone, especially developers and fellow GSoC participants contributing to syslog-ng ose. Thanks folks, happy hacking to all of you! :) Cheers, Gyuri

1 0

Pattern DB Parser "Default Values"
by David Hauck 01 May '14

01 May '14

Hi, I was wondering if there was a way to specify default values for pattern DB parsers that include a value, but where the parsed value is <null>[/empty]? In particular if I have something like the following: <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> I'd like to be able to do something like either, 1: <pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern> Or 2: <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> <values> <value name="field1.default">foo</value> <value name="field2.default">bar</value> <value name="field4.default">beef</value> Just curious... Thanks, -David

4 5

Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, dropping messages
by Chaman Chakalaka 29 Apr '14

29 Apr '14

Thanks for responding! The exact version is of syslog-ng is 3.3.4. Here's something you may find of interest. 1K Messages root@badbox:/proc/net# nc -lku 514 > /tmp/testing123.txt root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 1000 --interval 30 badbox.cbf 514 average rate = 996.43 msg/sec, count=29893 <--------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 24081 <------------ received/processed Lower the rate to 150 messages per second: root@goodbox:/proc/sys/net# loggen --inet --dgram --size 500 --rate 150 --interval 30 badbox.cbf 514 average rate = 149.03 msg/sec, count=4471 < -------------------- sent root@badbox:/proc/net# cat /tmp/testing123.txt | wc -l 4471 < -------------------------- received/processed At this point, given the test above, I don't know if this is a system issue or a syslog-ng issue. It seems to be system, but I really can't tell what. > From: syslog-ng-request(a)lists.balabit.hu > Subject: syslog-ng Digest, Vol 108, Issue 24 > To: syslog-ng(a)lists.balabit.hu > Date: Tue, 29 Apr 2014 13:25:01 +0200 > > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > > > Today's Topics: > > 1. Re: Ubuntu Precise -ng filling out buffer, dropping messages > (Gergely Nagy) > 2. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 3. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 4. Re: Pattern DB Parser "Default Values" (Gergely Nagy) > 5. Re: syslog-ng does not start if destination host not found > (Gergely Nagy) > 6. [Bug 275] lib/filter/filter-in-list.c does not compile under > Solaris 10 (bugzilla(a)bugzilla.balabit.com) > 7. [Bug 279] Syslog-ng central loging server seg fault gentoo > (bugzilla(a)bugzilla.balabit.com) > 8. Re: Basic (?) multi line question (Jim Hendrick) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 29 Apr 2014 12:24:09 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Ubuntu Precise -ng filling out buffer, > dropping messages > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87mwf4xwl2.fsf(a)balabit.hu> > Content-Type: text/plain > > Hi! > > Chaman Chakalaka <chebannedmeagain(a)hotmail.com> writes: > > > I'm trying to process ~800 UDP messages second, which I don't think is > > much. The current setup worked fine in Ubuntu 10.04 (Lucid) and > > syslog-ng 2.6 (I believe). I'm running into what I believe is receive > > buffer problems on Ubuntu Server 12.04 (Precise) w/ ng 3.XX > > First of all, what's the exact version of your syslog-ng? Precise has a > fairly old version, one that's... not exactly the best release. I'd > suggest you give a try to the packages at: > http://asylum.madhouse-project.org/projects/debian/ > > I'd suggest the syslog-ng 3.5 branch from there, and see if the problem > persists with an upgraded syslog-ng. If it persists, let us know, and > we'll help debug the issue further. > > -- > |8] > > > > ------------------------------ > > Message: 2 > Date: Tue, 29 Apr 2014 12:25:38 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429102538.E2BAC39DC88(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > Gergely Nagy <algernon(a)balabit.hu> changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |algernon(a)balabit.hu > AssignedTo|bazsi(a)balabit.hu |algernon(a)balabit.hu > > > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 3 > Date: Tue, 29 Apr 2014 12:29:08 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429102908.849C539DC78(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > > > > --- Comment #1 from Gergely Nagy <algernon(a)balabit.hu> 2014-04-29 12:29:08 --- > Without debug symbols, the backtrace is fairly useless for debugging purposes, unfortunately. It would help tremendously, if you could reproduce the problem > with a non-stripped binary, so we see the functions in the backtrace. > > Meanwhile, can I ask what config you use on the host where the segmentation fault happened? Maybe we can figure something out from that... > > Thanks! > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 4 > Date: Tue, 29 Apr 2014 12:41:44 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Pattern DB Parser "Default Values" > To: syslog-ng(a)lists.balabit.hu > Cc: Balazs Scheidler <bazsi(a)balabit.com> > Message-ID: <87iopsxvrr.fsf(a)balabit.hu> > Content-Type: text/plain > > David Hauck <davidh(a)netacquire.com> writes: > > > I was wondering if there was a way to specify default values for > > pattern DB parsers that include a value, but where the parsed value is > > <null>[/empty]? > > > > In particular if I have something like the following: > > > > <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> > > > > I'd like to be able to do something like either, 1: > > > > <pattern>test message; field1=@ESTRING:field1<foo>: @field2=@ESTRING:field2<bar>: @field3=@ESTRING:: @field4=@ESTRING:field4<beef>: @</pattern> > > > > Or 2: > > > > <pattern>test message; field1=@ESTRING:field1: @field2=@ESTRING:field2: @field3=@ESTRING:: @field4=@ESTRING:field4: @</pattern> > > <values> > > <value name="field1.default">foo</value> > > <value name="field2.default">bar</value> > > <value name="field4.default">beef</value> > > > > Just curious... > > You can use ${field1:-foo} in templates, to set a default if none is > set. It doesn't work for empty fields, though, but that can be worked > around with an $(if $(length $field1) eq 0 "default" $field1) template, > possibly in a rewrite rule. > > Though, maybe ${field1:-foo} should work for empty values too, not just > unset ones (to mimic shell better, which does just that). I can make it > do so, if that'd be desired, would make it unnecessary to use the $(if) > hack. > > @Bazsi: What do you think? > > -- > |8] > > > > ------------------------------ > > Message: 5 > Date: Tue, 29 Apr 2014 12:59:56 +0200 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] syslog-ng does not start if destination host > not found > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87eh0gxuxf.fsf(a)balabit.hu> > Content-Type: text/plain > > "Bendler, Ehren" <ebendler(a)ciena.com> writes: > > [...] > > If this is the intended behavior, that's fine too. We can deploy our > > own patch to the afsocket module if it isn't going to be changed in a > > release. > > No, this is definitely not the intended behaviour. Some change between > 3.3.5 and 3.5.7 broke the fix, I'll go ahead and restore the intended > behaviour. Thanks for reporting the issue! > > Unfortunately, I can't help with the other issue at the moment, but I'll > try to revisit it later. > > -- > |8] > > > > ------------------------------ > > Message: 6 > Date: Tue, 29 Apr 2014 13:09:22 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 275] lib/filter/filter-in-list.c does not > compile under Solaris 10 > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429110922.7AD9939DC99(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=275 > > > > > > --- Comment #2 from Gergely Nagy <algernon(a)balabit.hu> 2014-04-29 13:09:22 --- > I think we can change the code to use find_cr_or_lf(), instead of using getline(), or reimplement something like getline() in terms of find_cr_or_lf() + fgets > (or mmap or something). That would solve the problem without having to add much to misc.c. I'll see what I can do. > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 7 > Date: Tue, 29 Apr 2014 13:21:01 +0200 (CEST) > From: bugzilla(a)bugzilla.balabit.com > Subject: [syslog-ng] [Bug 279] Syslog-ng central loging server seg > fault gentoo > To: syslog-ng(a)lists.balabit.hu > Message-ID: <20140429112101.BD6F339DCA1(a)lists.balabit.hu> > Content-Type: text/plain; charset="UTF-8" > > https://bugzilla.balabit.com/show_bug.cgi?id=279 > > > > > > --- Comment #2 from Martin <hlavacek(a)gmx.com> 2014-04-29 13:21:02 --- > I thought that I have recompiled syslog with debug symbols because I have added --enable-debug to my ebuild: > > syslog1 ~ # syslog-ng -V > syslog-ng 3.4.7 > Installer-Version: 3.4.7 > Revision: ssh+git://algernon@git.balabit/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#detached_from_v3.4.3#999a7a6102d40da44b75a2acf78e54244164771f > Compile-Date: Apr 29 2014 13:06:40 > Available-Modules: > affile,afprog,afsocket-notls,afsocket-tls,afuser,basicfuncs,confgen,csvparser,dbparser,syslogformat,cryptofuncs,system-source,afamqp,afsocket > Enable-Debug: on > Enable-GProf: off > Enable-Memtrace: off > Enable-IPv6: on > Enable-Spoof-Source: off > Enable-TCP-Wrapper: on > Enable-Linux-Caps: off > Enable-Pcre: on > > You can see that opt "Enable-Debug:" is ON. It is not enought? If not can you please give me any advice how should I recompile this binary in proper way in > gentoo? > > Size of binary is: > syslog1 ~ # ls -lah /usr/sbin/syslog-ng > 16K -rwxr-xr-x 1 root root 15K Apr 29 13:18 /usr/sbin/syslog-ng* > > Configure options by emerge: > ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share > --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --disable-silent-rules --disable-dependency-tracking --with-ivykis=internal > --with-libmongo-client=internal --sysconfdir=/etc/syslog-ng --localstatedir=/var/lib/syslog-ng --with-pidfile-dir=/var/run > --with-module-dir=/usr/lib64/syslog-ng --enable-debug --with-systemdsystemunitdir=/usr/lib/systemd/system --disable-systemd --disable-linux-caps > --disable-geoip --enable-ipv6 --disable-json --disable-mongodb --enable-pcre --disable-smtp --disable-spoof-source --disable-sql --enable-ssl > --enable-tcp-wrapper > > > -- > Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are watching all bug changes. > > > ------------------------------ > > Message: 8 > Date: Tue, 29 Apr 2014 07:24:58 -0400 > From: Jim Hendrick <jrhendri(a)roadrunner.com> > Subject: Re: [syslog-ng] Basic (?) multi line question > To: Syslog-ng users' and developers' mailing list > <syslog-ng(a)lists.balabit.hu> > Message-ID: <535F8C0A.5060104(a)roadrunner.com> > Content-Type: text/plain; charset="iso-8859-1" > > Thanks all for the thoughts - > > I will try to write up some of the patterns and correlations, starting > with the most simple. > > This would (I think) be a valuable addition to track different logs that > have some dynamic id as a key. > > (ultimately I am hoping to parse specific data out of these multi-line > beasties and be able to populate a database directly from syslog-ng) > > I will work on writing this up this week. > > Thanks again! > Jim > > > On 04/29/2014 04:53 AM, Tusa Viktor wrote: > > Hi! > > > > If you know the format of all the messages which possibly contains a > > MID, you can write patterns for them and then you can use correlation > > to extract information from these messages. But it only works with > > special conditions, I think it wouldn't work in your case. But it > > wouldn't be so hard to create such functionality in syslog-ng, so if > > you open a github issue in http://github.com/balabit/syslog-ng, some > > of us will try to make it work. > > > > Best Regards, > > Viktor > > > > > > On Tue, Apr 29, 2014 at 8:14 AM, C. L. Martinez <carlopmart(a)gmail.com > > <mailto:carlopmart@gmail.com>> wrote: > > > > Hi Jim, > > > > Some time ago, I have tried the same: correlate logs for Ironport > > devices. And my conclusion was: impossible. I loose a lot info and > > some correlated logs are wrong ... > > > > The only approach that maybe should work with opensource tools, IMO, > > is rsyslog+sec.pl <http://sec.pl>. But, as a Orangepeel says, > > logstash can be an > > option. > > > > Bye. > > > > On Mon, Apr 28, 2014 at 2:44 PM, <jrhendri(a)roadrunner.com > > <mailto:jrhendri@roadrunner.com>> wrote: > > > Hmmm - crickets :-) > > > > > > I have some examples like this: > > > <date> <host> <program>: Info: New SMTP ICID [0-9]{9} <rest of > > message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: Start MID [0-9]{9} ICID [0-9]{9} > > <rest of message> > > > <date> <host> <program>: Info: New SMTP DCID [0-9]{9} <rest of > > message> > > > <date> <host> <program>: Info: Message done DCID [0-9]{9} MID > > [0-9]{9} <rest of message> > > > <date> <host> <program>: Info: ICID [0-9]{9} close > > > > > > this is only an example to illustrate the different message > > elements that contain different kinds of IDs. > > > > > > The issue is there will be interleaving with *different* ICID > > (inbound connections from different SMTP servers) each sending > > multiple MIDs (message IDs) and also different DCID (destination > > connections *to* different mail relays). > > > > > > I have been looking at multi-line-mode(regexp) but that seems to > > imply all consecutive lines until the next regex match are assumed > > to be part of the same message. > > > > > > I hope I can do something where all matching ICIDs are treated > > as part of one line, that can be parsed separately. > > > > > > Not sure if this is possible with multi-line-mode *or* with some > > patterndb wizardry. > > > > > > Has anyone addressed this? > > > > > > Thanks for any working-examples/guidance/sympathy (in roughly > > that order :-) > > > > > > Jim > > > > > > > > > > > > > > > ---- jrhendri(a)roadrunner.com <mailto:jrhendri@roadrunner.com> wrote: > > >> Hi, > > >> > > >> I am trying to parse data elements out of a variable number > > of log lines that all are associated by a single unique key. > > >> > > >> Specifically - they are Cisco IronPort email logs that have > > various "ID" fields (MID - message ID is the most common) > > >> > > >> > > >> Essentially I want to pull the MID out of the line marked marked: > > >> > > >> "Start MID (\d+) <other stuff>" > > >> > > >> and then process every line that matches that specific MID > > value as part of the message. > > >> > > >> Note: they all have this string included somewhere: > > >> > > >> "MID (\d+) " > > >> > > >> Up to a reasonable timeout - or ended by: > > >> > > >> "Message finished mid (\d+) done" with the matching ID. > > >> > > >> Is this possible with syslog-ng? (OSE or PE?) > > >> > > >> I thought I had seen something using patterndb but I cannot > > seem to find the reference > > >> > > >> Clearly there will be interleaved lines with *different* MIDs > > that need to be processed independently. > > >> > > >> Thanks in advance! > > >> Jim > > > > > > > > ______________________________________________________________________________ > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > > > > > ______________________________________________________________________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > >

2 2

Correct Usage of Multiple 'pattern' Databases
by David Hauck 29 Apr '14

29 Apr '14

Hello, I've only recently dug into some more intricate 'syslog-ng' configurations and had a question regarding 'log' construct blocks where multiple 'parser' references exist. I've been trying to do something like the following (testing with the supplied example pattern databases): log { filter(f_auth); parser("login"); parser("sshd"); parser("su"); parser("sudo"); log { filter(f_class_system); ... }; }; The problem I'm having is that extracted values from matched rules appear to be lost when the matched rule exists in a pattern db *other than the last referenced parser() db*. Specifically, if a rule is matched in the 'sshd' db above the following 'f_class_system' filter (which attempts to match '.classifier.class') *does not* match; however, if a rule is matched in the 'sudo' db above the 'f_class_system' filter *does* match. I'm sure this is perfectly explainable, but I can't find any documentation/Google references specifically outlining this behaviour. Given the above and in order to work around this I assume I would have to, either: 1) combine all of the rules into a single db file, or 2) break out each 'parser' reference into a separate embedded 'log' construct (not ideal since the filtering et mechanics in each would be identical and for maintenance reasons I'd like to consolidate these into a single 'log' construct). Both options are less than ideal. Is there a better way? Really appreciate any help you might be able to offer. Thanks, -David

9 17

syslog-ng does not start if destination host not found
by Bendler, Ehren 29 Apr '14

29 Apr '14

In syslog-ng 3.5.4.1, this seems to be a recurrence of an issue that has appeared several times before over the years, most recently said to be fixed/changed in v3.3.5 here (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=660897) We had previously been using syslog-ng 3.0.4, which would fail this test gracefully and retry the connection based on the time_reopen() parameter. We don't use DNS, but we generate a hard-coded /etc/hosts file after startup that associates internal names with a customer chosen subnet address so that we can collect logs without editing the syslog-ng.conf file while we are running. As in, we know that the primary controller will always be present with the name "CTXP", but we can't promise that it will have a particular address because the customer can change it, so we have to write it out to /etc/hosts at a point in the startup process *after* syslog-ng is started. * Starting syslog-ng Error resolving hostname; host='CTXP' Error initializing message pipeline; ..[FAIL(2)] If this is the intended behavior, that's fine too. We can deploy our own patch to the afsocket module if it isn't going to be changed in a release. Another thing I need help with (when I use a hard-coded IP in syslog-ng.conf for testing) is how to get this message to go away: * Starting syslog-ng WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='12000' [OK] Here are the settings we have for these parameters: source remote_lms { syslog(ip(0.0.0.0) port(5149) max-connections(120) log_iw_size(1200) so_keepalive(yes)); }; With log_fifo_size(12000) set in the global options. The documentation says I did the math correctly here, with log_fifo_size = 10 * log_iw_size(), and log_iw_size() = 10 * max-connections()? This is the only remote source in our configuration. Thanks, Ehren Bendler

2 1