November 2013 - syslog-ng - lists.balabit.hu

syslog-ng 3.5.1 - question about flags(final)...
by Johnson, Chris (HP TippingPoint Roseville) 10 Dec '13

10 Dec '13

Hello all, I'm in the process of upgrading from version 3.3.9 to 3.5.1 and have a question about how the 'flags(final);' is working in 3.5.1. In 3.3.9, I use the following structure of imbedded log statements: ################################################################################ # Service ipsec # filter f_ipsec_pgm{program("IPSEC-*" type("glob")) or program("IKE-*" type("glob")) or program("CHARON-*" type("glob")) or program("charon-*" type("glob")); }; filter f_ipsec_lvl_01{level(warning..emerg)}; filter f_ipsec_lvl_02{level(info..emerg)}; log { source(s_local); filter(f_ipsec_pgm); log { filter(f_ipsec_lvl_01); destination(d_logID_11); }; log { filter(f_ipsec_lvl_02); rewrite(r_quote_newlines); destination(d_logID_13); }; flags(final); }; In this case log messages of the 'correct' program would further be filtered on their severity level. * Info level messages would only be sent to 'd_logID_13' * Warning level messages (and above) would be sent to BOTH 'd_logID_11' and 'd_logID_13'. * Debug level messages would be discarded. Under 3.5.1, the 'filtering ' stops after it matches once: * Warning messages (and above) are only sent to 'd_logID_11' and NOT 'd_logID_13'. * Info messages are still (correctly) being sent only to 'd_logID_13'. If I remove (or comment out) the 'flags(final);' statement, messages are filtered correctly (i.e. the way I *want* them to be filtered :)) except that they also are being processed by all the following log statements and are being caught in my final filter of 'program("*" type("glob"))'. NOTE: yes, I know that I could use 'flags(fallback)' in my final filter, but that would still have every message processed by every log filter, and I would like to avoid that. So, what would be the correct way to set up my log statement to re-create the 3.3 behavior? Thanks, Chris ---------------------------------------- Christopher Johnson chris.johnson3(a)hp.com<mailto:chris.johnson3@hp.com> HP Software - Security Product Group (916) 785-2817 ----------------------------------------

3 7

[review request] 3.6queue/f/stats-reset
by Balazs Scheidler 10 Dec '13

10 Dec '13

Hi, I'm about to integrate a stats-counter related change and prior to doing that I'd like to solicit feedback. There are both user-visible changes and developer visible changes, feedback on both is sought. Code related comments are preferred on github, functionality comments are best sent via email in response to this one. Here's the branch: https://github.com/balabit/syslog-ng/tree/3.6queue/f/stats-reset Thanks in advance. User-visible changes: ===================== syslog-ng is keeping track on a number of statistics counters that can be used to track what is happening. The detail level of these counters is controlled by the stats-level() global option, which defaults to zero. syslog-ng reports these counters periodically every stats-freq() seconds, or preferably on-demand by invoking "syslog-ng-ctl stats" on the command line. At stats-level(3) a number of "dynamic" counters are enabled, which track the number of messages on a per $HOST, per $PROGRAM and per $HOST_FROM basis. This means that you can follow how many messages a given $PROGRAM is generating in a period. The issue with dynamic counters is that they can consume a lot of memory if the rate of new $PRORGRAM/$HOST values is high. Previously the cleanup happened in a limited way at configuration reload (e.g. SIGHUP) time, which is not really useful for controlling memory usage by this feature. The current patch series introduces a stats-lifetime() option and syslog-ng starts inspecting counters every once in a while and prune those that are updated earlier than stats-lifetime(). Not-so visible changes ====================== I have fixed a few bugs and renamed a function to make it easier to understand. Details are in the git history. Balazs Scheidler (7): logsource: use stats_instant_inc_dynamic_counter() for HOST counters logsource: track time for PROGRAM counters mainloop: remove old orphanage based stats cleanup stats: add stats_prune_old_counters() stats-timer: implement stats_lifetime option cfg: add stats-lifetime() global option stats: rename stats_instance_inc_dynamic_counter() to stats_register_and_increment_dynamic_counter() lib/cfg-grammar.y | 16 ++++++++------- lib/cfg-parser.c | 1 + lib/cfg.c | 3 ++- lib/cfg.h | 1 + lib/logsource.c | 15 +++----------- lib/mainloop.c | 1 - lib/stats-timer.c | 61 +++++++++++++++++++++++++++++++++++++++---------------- lib/stats-timer.h | 2 +- lib/stats.c | 52 +++++++++++++++++++++++++++++++++++++++-------- lib/stats.h | 7 +++---- 10 files changed, 108 insertions(+), 51 deletions(-) -- Bazsi

6 12

syslog - sql error
by vijay 03 Dec '13

03 Dec '13

HI All, Iam using syslog-ng3.2.5 rpm in centos x86_64 environment.Past days it is working well.Suddenly iam seeing a problem in writing to mysql Error running SQL query; type='mysql', host='192.168.9.68', port='3306', user='cygnet', database='CygNetDB', error='1072: Key column \'date\' doesn\'t exist in table', query='CREATE INDEX SYSLOGS_20131130_date_idx ON SYSLOGS_20131130 (date)' Error adding missing index; table='SYSLOGS_20131130', column='date' Error running SQL query; type='mysql', host='192.168.9.68', port='3306', user='cygnet', database='CygNetDB', error='1170: BLOB/TEXT column \'facility\' used in key specification without a key length', query='CREATE INDEX SYSLOGS_20131130_facility_idx ON SYSLOGS_20131130 (facility)' Error adding missing index; table='SYSLOGS_20131130', column='facility' vijay

2 1

[Bug 264] New: docbug: all-hard-macros does not exist in value-pairs
by bugzilla＠bugzilla.balabit.com 02 Dec '13

02 Dec '13

https://bugzilla.balabit.com/show_bug.cgi?id=264 Summary: docbug: all-hard-macros does not exist in value-pairs Product: syslog-ng Version: 3.5.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.balabit(a)faxm0dem.org Type of the Report: --- Estimated Hours: 0.0 using this yields "Error parsing config, Error compiling template (Error parsing value-pairs: unknown scope all-hard-macros)" Indeed, the key "all-hard-macros" is not being referenced in 'lib/value-pairs.c' -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Syslog-ng OSE Question.
by Malkin, Vlad 02 Dec '13

02 Dec '13

Hi, Is it possible to have syslog-ng client to invoke an external (to it) software to acknowledge when each individual log statement is sent to the server? Background: logs should not be lost. The system needs to store logs internally until they are transmitted to the server (use of TCP transport provides a level of reliable delivery). After each log statement is transmitted by syslog-ng client, it needs to be removed from internal store to preserve allocated space. Thank you for your help! --Vlad

5 6

syslog-ng 3.5.2 has been released
by devel＠balabit.hu 29 Nov '13

29 Nov '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.5.2 SUMMARY : new stable release DATE : Nov 29, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.5.2) has been released. For latest fixes in the 3.5.x feature branch you are recommended to upgrade to this version. CHANGES: 3.5.2 Fri, 29 Nov 2013 17:09:40 +0100 This is the first bug-fix release for the 3.5.x series, with a number of significant fixes over the original 3.5.1 release. Upgrading to this release is highly recommended. Bugfixes ======== * The system() source has been fixed on Solaris and HP-UX: it correctly recognises the system on Solaris, and uses /dev/log (and not /dev/pipe) on HP-UX. * The system() source has much stricter requirements to select /dev/kmsg on Linux, so it won't choose it over /proc/kmsg unless the kernel really does support it. * The syslog-format() module has been fixed to only set the PROGRAM to "kernel" if all of the previous conditions apply, and the message is a local message. * The linux-kmsg-format parser (and system() on Linux 3.5+ as a consequence) will correctly calculate the timestamp of kernel messages on 32-bit systems now. * The sun-streams() source has been fixed, it compiles and works again. * A very rare and short race condition has been fixed in the pipe() source and destination modules, which could leave the pipe available with elevated permissions for a short time. * The SQL, MongoDB and SMTP modules were fixed to not cause a segmentation fault when loaded before the socket module. * A crash was fixed in the no-tls build of the socket destination driver. * It is now possible to specify the host() and port() parameters for the redis() destination. * Various stability fixes, such as: - A use-after-free condition was corrected in the queue backlog acknowledgement code. - Always use setsid() when detaching, even if we're not detaching from a TTY. - Memory leaks were fixed in the persist state handling code. - Changing stats-level() and reloading syslog-ng produced a crash, which has been fixed. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Alexandre Lecuyer Balazs Scheidler <bazsi(a)balabit.hu> Daniel Gados <gdani(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Howard Tsai <htsai(a)skytap.com> Imre Lazar <imre(a)balabit.hu> Jakub Wilk <jwilk(a)debian.org> Laszlo Budai <lbudai(a)balabit.hu> Martin Bagge / brother <brother(a)bsnet.se> Matyas Koszik <koszik(a)atw.hu> Oscar Muñoz Otto Berger <otto(a)bergerdata.de> Viktor Juhasz <jviktor(a)balabit.hu> Viktor Tusa <tusa(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

syslog-ng 3.4.6 has been released
by devel＠balabit.hu 27 Nov '13

27 Nov '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.6 SUMMARY : new stable release DATE : Nov 27, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.6) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.6 Wed, 27 Nov 2013 16:21:47 +0100 This is the fifth bug-fix release for the 3.4.x series, with a number of significant fixes over the 3.4.5 release. Upgrading to this release is highly recommended. Bugfixes ======== * The system() source has been fixed on Solaris and HP-UX: it correctly recognises the system on Solaris, and uses /dev/log (and not /dev/pipe) on HP-UX. * The sun-streams() source has been fixed, it compiles and works again. * A very rare and short race condition has been fixed in the pipe() source and destination modules, which could leave the pipe available with elevated permissions for a short time. * The SQL, MongoDB and SMTP modules were fixed to not cause a segmentation fault when loaded before the socket module. * The syslog-format() module has been fixed to only set the PROGRAM to "kernel" if all of the previous conditions apply, and the message is a local message. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Alexandre Lecuyer Balazs Scheidler <bazsi(a)balabit.hu> Gergely Nagy <algernon(a)balabit.hu> Imre Lazar <imre(a)balabit.hu> Laszlo Budai <lbudai(a)balabit.hu> Matyas Koszik <koszik(a)atw.hu> Oscar Muñoz DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Mongodb
by Nilo Cabeza Diaz 27 Nov '13

27 Nov '13

how send log to mongodb??? this is what I do in syslog-ng.conf and does nothing for me destination d_dw_v3_mongo { mongodb( host("localhost") port(27017) database("syslog") collection("messages") value-pairs( key("ISODATE") key("PRIORITY") pair("svc_id" "${.SDATA.syslog(a)0.svcid}") pair("tid" "${.SDATA.syslog(a)0.tid}") pair("act_id" "${.SDATA.syslog(a)0.id}") pair("act_eid" "${.SDATA.syslog(a)0.eid}") key("MESSAGE") key("SDATA") key("HOST") ); };

2 1

Error of parser
by Nilo Cabeza Diaz 26 Nov '13

26 Nov '13

hello: look this error Failed to start Syslog-NG server : * Starting system logging syslog-ng syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted' Error parsing config, syntax error, unexpected KW_DESTINATION, expecting ';' in /etc/syslog-ng/syslog-ng.conf at line 94, column 1: destination d_mongodb { ^^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng ...fail!

2 1

syslog-ng 3.6 rebase
by Balazs Scheidler 25 Nov '13

25 Nov '13

Hi, After integrating everything I could, I would like to rebase the current master to merge fixups into the base patches. This is the last such rebase in 3.6. You can see the rebased series in the "next" branch, I intend to reset the master branch to that history tomorrow evening. the HEAD of both branches are identical, only the history differs. In the meantime, you can submit patches either the current "master" or "next" branches, I'll integrate stuff to "next" myself. Thanks. -- Bazsi

1 1