- syslog-ng - lists.balabit.hu

[Bug 61] udp6 source declaration causes syslog forwarding to assume IPv6
by bugzilla＠bugzilla.balabit.com 20 Dec '12

20 Dec '12

https://bugzilla.balabit.com/show_bug.cgi?id=61 Gergely Nagy <algernon(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|3.0.6 |3.4.1 --- Comment #5 from Gergely Nagy <algernon(a)balabit.hu> 2012-12-20 13:12:03 --- This is partially fixed in 3.4 already with the network() statement, we only need to force syslog() to set last_addr_family to IPv4. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

syslog-ng Insider - December 2012
by Peter Czanik 20 Dec '12

20 Dec '12

Dear syslog-ng users, This is the 19th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng 3.4 beta1 is released ------------------------------- Version 3.4 beta1was released thisweek, right before the Christmasholidays. It has many new features and bug fixes even since the last alpha release, most notably a new AMQP destination a JSON parser and a reworked syslog parser and network configuration, which makes configuring syslog-ng even more simple and flexible. For a complete list of changes, check the announcement at https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-December/000150.… For binary package availability check our 3^rd party binary page at http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… balabit.logstore 0.1.0 is released ---------------------------------- The second version of the balabit.logstore project was announced last week. It is a library written in Clojure, that tries to provide a convenient API to read syslog-ng PE LogStore files. Development is still in its early phases, but it can already read unencrypted logstore files, search in them and print many useful information about them. Compared to the previous version, this has a Java API. For more details, check the announcement at https://lists.balabit.hu/pipermail/syslog-ng/2012-December/019788.html NEW RELEASES: * syslog-ng OSE 3.4 beta1: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-December/000150.… * balabit.logstore 0.1.0: https://lists.balabit.hu/pipermail/syslog-ng/2012-December/019788.html ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[ANN]: balabit.logstore 0.1.0 - Open Source LogStore reader API
by Gergely Nagy 19 Dec '12

19 Dec '12

--------------------------------------------------------------- PACKAGE : com.balabit/logstore VERSION : 0.1.0 SUMMARY : Rewritten from scratch, now includes simple Java API DATE : 2012 December 15 HOMEPAGE : http://algernon.github.com/balabit.logstore --------------------------------------------------------------- DESCRIPTION: The balabit.logstore project is a library written in Clojure, that tries to provide a convenient API to read syslog-ng PE LogStore files. The reason behind the implementation is to have an independent, open source reader for the LogStore file format, so that one is not tied to syslog-ng to read one's logs stored in this format. An open implementation makes it possible to read these logs on systems where syslog-ng is not installed, or where the lgstool program is not available. LogStore itself is a storage format for log messages, supporting encryption, compression, secure timestamps, serialized messages (think name-value pairs as opposed to just a plain, formatted string) and a lot more. This format is being opened, and this library is meant to provide an implenetation for a LogStore reader, that will eventually support reading all kinds of LogStore files. This release is a complete rewrite of an earlier attempt, and this time, it comes with a simple Java API. EXAMPLE: Clojure: (ns example (:require [balabit.logstore.sweet :as logstore])) (prn (logstore/messages (logstore/from-file "/path/to/logstore/messages.store"))) Java: import BalaBit.LogStore; import clojure.lang.LazySeq; import clojure.lang.Keyword; import java.util.Map; public class LGSCat { public static void main(String[] args) { Keyword k = BalaBit.LogStore.keyword("MESSAGE"); Object o = BalaBit.LogStore.fromFile (args[0]); LazySeq s = (LazySeq) BalaBit.LogStore.messages (o); for (Object m : s.toArray()) { Map msg = (Map) m; System.out.println(msg.get(k)); } } } More examples can be found bundled with the sources, and as part of the documentation: http://algernon.github.com/balabit.logstore/#balabit.logstore.cli FEATURES: The library - while still experimental - can read most unencrypted LogStores, created by syslog-ng PE 4.0 or any later version, and the messages stored within can be explored. It comes with a test suite and quite a few examples. FUTURE DIRECTION: The library is very simple right now, the error handling is pretty much non-existent, and it does not perform any kind of validation. The Java API is very thin, and exposes far more Clojure-isms than it ideally should, this will be improved upon in the future. Once things stabilized a bit, we plan to publish ready to use JAR files that can be used from any Java or Clojure project. DOWNLOADS: The source is available from the git repository at github: git://github.com/algernon/balabit.logstore.git Documentation and more information about the library is available on its homepage at http://algernon.github.com/balabit.logstore/. -- |8]

1 1

newbie question
by Groups 14 Dec '12

14 Dec '12

Newbie question My Goals (2) To replace the current CentOS 5.x log server with a SL 6.x syslog-ng server. *************************************** *************************************** Goal 1 To have all Linux servers log by the following: All Linux servers are running either syslog or rsyslog - Accept logs from Linux servers and separate them to messages.$YEAR$MONTH$DAY secure.$YEAR$MONTH$DAY maillog.$YEAR$MONTH$DAY messages can be a 'catchall' ================== My success on Linux servers... The mods in the syslog-ng.conf I have for Linux servers does work but catches everything. *************************************** *************************************** Goal 2 To have all (Cisco) switches / routers log according to their current syslog settings Current syslog server uses /etc/hosts to log the following. Receive logging messages from network devices (Router/Switch/VPN) to 3 files #based on priority local4.err /var/log/NetLog/NetDeverr local4.notice,local4.!err /var/log/NetLog/NetDevnote local4.=debug,local4.=info /var/log/NetLog/NetDevdebug #Receive logging messages from all other devices to 3 files based on priority syslog.err /var/log/OtherLog/OthDeverr syslog.notice,syslog.!err /var/log/OtherLog/OthDevnote syslog.=debug,syslog.=info /var/log/OtherLog/OthDevdebug #Receive logging messages from all wireless devices to 3 files based on priority local3.err /var/log/NetLog/Wirelesserr local3.notice,local3.!err /var/log/NetLog/Wirelessnote local3.=debug,local3.=info /var/log/NetLog/Wirelessdebug #Receive logging messages from all UPS's to 3 files based on priority local5.err /var/log/UPSLog/UPSerr local5.notice,local5.!err /var/log/UPSLog/UPSnote local5.=debug,local5.=info /var/log/UPSLog/UPSdebug *************************************** Thx for any pointers. tk --------------------- rsyslog is turned off -- OS: Scientific Linux 6.2 (RHEL 6 clone) -- #syslog-ng -V syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 15 2012 19:52:28 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off -- syslog-ng.conf (out of the box) with one change create_dirs (yes); --------------------- @version:3.2 # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; ## Begin Mods source s_udp { udp( flags(no-parse) ); };# An UDP source source s_tcp { tcp( flags(no-parse) ); };# An TCP source ###### Filter statements ####### ### Linux logs filter f_mailhost {host(172.25.85.41); }; filter f_merlin {host(172.25.45.10); }; ###### Destination statements ####### destination d_linux_servers {file("/var/log/devices/linux_servers/$HOST/log.$YEAR$MONTH$DAY"); }; ###### Log statements ####### ### Linux servers log {source(s_udp); filter(f_merlin); destination(d_linux_servers); }; log {source(s_udp); filter(f_mailhost); destination(d_linux_servers); }; --------------------- Thx

1 0

FreeBSD 3.4 pre beta testing
by Peter Czanik 14 Dec '12

14 Dec '12

Hello, Attached is a diff against the current sysutils/syslog-nd-devel port. Please test it, so problems could be discovered before the beta release. To use the attached diff, change to the /usr/ports/sysutils/syslog-ng-devel directory and run: patch -p1 < /path/to/prebeta.diff Changes since alpha3: - latest sources - redundant patches removed - glib json support removed (json parser needs json-c) - added patch to fix libnet (will be part of the beta release) - lot of pkg-list changes Once 3.4 beta is released I plan to update the port too. Happy testing! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

3.4 head on FreeBSD
by Peter Czanik 12 Dec '12

12 Dec '12

Hello, While trying to compile on FreeBSD 9.1, I ran into this problem: /bin/sh ../../libtool --tag=CC --mode=compile cc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -D_REENTRANT -I/usr/local/include/glib-2.0 -I/usr/local/include/eventlog -I/usr/local/include -I/usr/local/include -I../../lib/ivykis/lib/include -I../../lib/ivykis/lib/include -I../../lib/ivykis/modules/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -pipe -I/usr/local/include/libnet116 -I/usr/local/include -fno-strict-aliasing -Wall -g -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c -o libafsocket_notls_la-afinet-dest.lo `test -f 'afinet-dest.c' || echo './'`afinet-dest.c libtool: compile: cc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -D_REENTRANT -I/usr/local/include/glib-2.0 -I/usr/local/include/eventlog -I/usr/local/include -I/usr/local/include -I../../lib/ivykis/lib/include -I../../lib/ivykis/lib/include -I../../lib/ivykis/modules/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -pipe -I/usr/local/include/libnet116 -I/usr/local/include -fno-strict-aliasing -Wall -g -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c afinet-dest.c -fPIC -DPIC -o .libs/libafsocket_notls_la-afinet-dest.o In file included from /usr/local/include/libnet116/libnet.h:120, from afinet-dest.c:43: /usr/local/include/libnet116/./libnet/libnet-structures.h:235: error: redefinition of typedef 'libnet_t' afinet-dest.h:35: error: previous declaration of 'libnet_t' was here gmake[4]: *** [libafsocket_notls_la-afinet-dest.lo] Error 1 gmake[4]: Leaving directory `/root/syslog-ng-devel/work/syslog-ng-3.4.0alpha3/modules/afsocket' gmake[3]: *** [all] Error 2 gmake[3]: Leaving directory `/root/syslog-ng-devel/work/syslog-ng-3.4.0alpha3/modules/afsocket' gmake[2]: *** [all-recursive] Error 1 gmake[2]: Leaving directory `/root/syslog-ng-devel/work/syslog-ng-3.4.0alpha3/modules' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/root/syslog-ng-devel/work/syslog-ng-3.4.0alpha3' gmake: *** [all] Error 2 *** [do-build] Error code 1 Disabling libnet support results in a successful build, but it can't be started: root@fb91:/root/syslog-ng-devel # /usr/local/etc/rc.d/syslog-ng onerestart syslog_ng not running? (check /var/run/syslog.pid). Starting syslog_ng. WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode Please update it to use the syslog-ng 3.4 format at your time of convinience, compatibility mode can operate less efficiently in some cases. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.; Unable to determine how to monitor this fd, follow_freq() not set and it is not possible to poll it with the current ivykis polling method, try changing IV_EXCLUDE_POLL_METHOD environment variable; fd='11' Error initializing log_reader, closing fd; fd='11' Error initializing message pipeline; /usr/local/etc/rc.d/syslog-ng: WARNING: failed to start syslog_ng FreeBSD installs syslog-ng-3.4.0alpha3/scl/syslog-ng.conf by default, so it needs a version bump. It can easily be fixed by me. But system() is now in C instead of shell scripts... Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 3

WARNING: window sizing for tcp sources
by Anton Koldaev 11 Dec '12

11 Dec '12

Hello, I have two questions: 1. % sudo /etc/init.d/syslog-ng restart * Stopping system logging syslog-ng [ OK ] * Starting system logging syslog-ng WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='80', new_log_iw_size='100', min_log_fifo_size='25000' I have a lot of tcp sources, destination and options. Where should I look for this warning? I don't have "log_iw_size=80" anywhere. 2. `syslog-ng --preprocess-into=/tmp/pre.conf` ^Does it work for anybody? It writes the config file but it hangs after it for me. Is it supposed to be used when syslog-ng daemon is down? -- Best regards, Koldaev Anton

2 4

syslog-ng 3.4beta
by Balazs Scheidler 10 Dec '12

10 Dec '12

Hi, Having finally been able to finish the network() source and destination drivers I declare 3.4 as feature complete, and plan to publish 3.4beta1 fairly soon. Let's say, before the winter holiday. In hindsight, this took much longer than it should have, 3.4 has grown a lot of interesting functionality: - junctions and more complicated log processing paths - amqp destination - smtp destination - json output and parser plugins - key rewriting in destination drivers (add, remove prefixes) - a number of new template functions (uuid, hash functions, etc) - a number of new parsers in db-parser (set, pcre, ...) - allow 32 bit message and nvpair sizes - improved systemd support - demand loaded plugins - plugin support for log transport protocols and the network driver to be able to use them easily And other things I've probably forgot. I have just merged Algernon's accumulated merge queue, and will probably cherry-pick stuff before the final release and merge in the fixes from 3.3 before the beta release. If you have stuff that you want to see in the beta, please speak up now. Testing _before_ the beta would also be appreciated, so that the beta is actually usable by laymen. Thanks :) As always, the latest code is in git, Algernon builds daily snapshots. Thanks -- Bazsi

5 9

[PATCH 1/2] pdbtool: match with supplied message did not perform processing
by balint.kovacs＠balabit.com 10 Dec '12

10 Dec '12

From: Balint Kovacs <blint(a)blint.hu> Under some circumstances pdbtool match did not process the message if it was provided on the command line due to a regression introduced as part of a LogProto refactoring (2d0fb042) Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/pdbtool/pdbtool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/dbparser/pdbtool/pdbtool.c b/modules/dbparser/pdbtool/pdbtool.c index deab84d..25014fe 100644 --- a/modules/dbparser/pdbtool/pdbtool.c +++ b/modules/dbparser/pdbtool/pdbtool.c @@ -503,7 +503,7 @@ pdbtool_match(int argc, char *argv[]) { dbg_list = g_array_new(FALSE, FALSE, sizeof(RDebugInfo)); } - while (!eof && buf) + while (!eof && (buf || match_message)) { invalidate_cached_time(); if (G_LIKELY(proto)) -- 1.7.9.5

1 1

[PATCH] afsocket: Add an ip-protocol() option for the network driver
by Gergely Nagy 08 Dec '12

08 Dec '12

To make it easy to set up IPv6 sources or destinations, the network() driver should have an option to select the IP protocol. This patch does just that, in the form of an ip-protocol() option, which allows one to select either IPv4 or IPv6. In the end, this allows us to have a source as follows: source s_v6_syslog { network(port(12345) transport(framed) ip-protocol(6)); }; The option is available for both the source and destination version of network(). Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket-grammar.ym | 29 ++++++++++++++++++++++++++++- modules/afsocket/afsocket-parser.c | 1 + 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/modules/afsocket/afsocket-grammar.ym b/modules/afsocket/afsocket-grammar.ym index 7beb6e8..464d5d8 100644 --- a/modules/afsocket/afsocket-grammar.ym +++ b/modules/afsocket/afsocket-grammar.ym @@ -79,6 +79,7 @@ TLSContext *last_tls_context; %token KW_UDP6 %token KW_NETWORK %token KW_TRANSPORT +%token KW_IP_PROTOCOL %token KW_IP_TTL %token KW_SO_BROADCAST @@ -136,6 +137,8 @@ TLSContext *last_tls_context; %type <ptr> dest_afnetwork %type <ptr> dest_afnetwork_params +%type <num> afinet_ip_protocol_option + %% start @@ -149,6 +152,25 @@ start | LL_CONTEXT_DESTINATION dest_afnetwork { YYACCEPT; } ; +afinet_ip_protocol_option + : LL_NUMBER + { + CHECK_ERROR($1 == 4 || $1 == 6, @1, "ip-protocol option can only be 4 or 6!"); + if ($1 == 4) + { + $$ = AF_INET; + } + else + { + $$ = AF_INET6; + } + } + ; + +source_afinet_ip_protocol + : KW_IP_PROTOCOL '(' afinet_ip_protocol_option ')' { ((AFSocketSourceDriver *)last_driver)->address_family = $3; } + ; + source_afunix : KW_UNIX_DGRAM '(' source_afunix_dgram_params ')' { $$ = $3; } | KW_UNIX_STREAM '(' source_afunix_stream_params ')' { $$ = $3; } @@ -314,6 +336,7 @@ source_afnetwork_option : source_afinet_option | source_afsocket_transport | source_afsocket_stream_params {} + | source_afinet_ip_protocol ; source_afsocket_transport @@ -335,7 +358,10 @@ source_afsocket_transport } ; - +dest_afinet_ip_protocol + : KW_IP_PROTOCOL '(' afinet_ip_protocol_option ')' { ((AFSocketDestDriver *)last_driver)->address_family = $3; } + ; + dest_afunix : KW_UNIX_DGRAM '(' dest_afunix_dgram_params ')' { $$ = $3; } | KW_UNIX_STREAM '(' dest_afunix_stream_params ')' { $$ = $3; } @@ -500,6 +526,7 @@ dest_afnetwork_options dest_afnetwork_option : dest_afinet_option | dest_afsocket_transport + | dest_afinet_ip_protocol ; dest_afsocket_transport diff --git a/modules/afsocket/afsocket-parser.c b/modules/afsocket/afsocket-parser.c index 4065d31..f9e14a4 100644 --- a/modules/afsocket/afsocket-parser.c +++ b/modules/afsocket/afsocket-parser.c @@ -72,6 +72,7 @@ static CfgLexerKeyword afsocket_keywords[] = { { "tcp_keepalive_intvl", KW_TCP_KEEPALIVE_INTVL, 0x0304 }, { "spoof_source", KW_SPOOF_SOURCE }, { "transport", KW_TRANSPORT }, + { "ip_protocol", KW_IP_PROTOCOL }, { "max_connections", KW_MAX_CONNECTIONS }, { "keep_alive", KW_KEEP_ALIVE }, { NULL } -- 1.7.10.4

2 1