- syslog-ng - lists.balabit.hu

max recursion depth on filters?
by Patrick Hemmer 11 Mar '12

11 Mar '12

Is there a maximum depth on recursive filter statements? I seem to have an issue which would indicate so. For example, in the config below, i have f_discard, which calls f_iptables_discard, which calls f_iptables. I have it like this so that its easy to read, configure, and add other filters. Its supposed to result in lines matching this to not be logged, but its not working. However if I take the contents of `f_iptables_discard` and put them directly into `f_discard` it works fine. filter f_iptables { program('^kernel') and message('^\s*\[\s*[\d\.]+\] iptables/' type(pcre)) }; filter f_iptables_discard { # ignore iptables broadcast messages filter(f_iptables) and message('MAC=ff:ff:ff:ff:ff:ff') ; }; filter f_discard { not ( filter(f_iptables_discard) ); }; rewrite r_tag { set('iptables', value('DBTAG') condition(filter(f_iptables))); }; log { source(s_local); source(s_net); rewrite(r_tag); filter(f_discard); destination(d_sqlite); };

2 2

Parser not parsing :-)
by T. A. Smooth 11 Mar '12

11 Mar '12

I can only assume I am not implementing this correctly. :-) But I have a parser I am trying to use so I can take a subset of the information of a message and send that subset to another receiver. This is the whole message: <13>Feb 4 18:40:17 myhost syslogng: 2012-02-04T18:40:17-08:00 > myhostserver-http /tmp/logs/access_log Hi Mom > What I want to do is send out the message as : <13>Feb 4 18:40:17 myhost syslogng: Hi Mom Notice how I dropped the middle part out. >From what I have read, the parser acts on the message body alone. Is this correct? So I set it up to look for four(4) columns of data and to be "greedy" on the last column. I have played around with the number of columns and even used a rewrite function instead. But the Parser continues to produce empty variables. And my template just echos out my default value. Any thoughts? > > > parser p_et_logmessage { > csv-parser( > #columns("ETMSG") > #columns("ETMSG.ISODATE") > columns("ETMSG.ISODATE", "ETMSG.EASI", "ETMSG.SOURCE", > "ETMSG.BODY") > delimiters(" ") > #template("${MSG}") > flags(greedy) > ); > }; > > rewrite r_rewrite_set{set('${ETMSG.BODY:-nothing}', value("MESSAGE"));}; > > template t_et_basic_logmessage { > template("${ETMSG.BODY:-nothing}\n"); template_escape(no); }; > > > destination destination_info { > tcp("host2" port(8080) > template(t_et_basic_logmessage) > log_disk_fifo_size(32212254720) > ); > }; > > log { > source(INTAKE); > parser(p_et_logmessage); > destination(destination_info); > }; > My latest Post: Givenchy Fall/Winter 2012 Collection – Runway | Highsnobiety.com<http://www.webeclubbin.com/blog/2012/01/givenchy-fallwinter-2012-collection…> Get a signature like this. <http://r1.wisestamp.com/r/landing?promo=19&dest=http%3A%2F%2Fwww.wisestamp.…> CLICK HERE.<http://r1.wisestamp.com/r/landing?promo=19&dest=http%3A%2F%2Fwww.wisestamp.…>

4 9

syslog-ng 3.4.0alpha1 has been released
by devel＠balabit.hu 11 Mar '12

11 Mar '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0alpha1 SUMMARY : new stable release DATE : Mar 11, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.0alpha1) has been released. This is an alpha release aiming at collecting feedback and testing on the upcoming 3.4 version. Use in production environments is not recommended. CHANGES: 3.4.0alpha1 Features: ========= * Support for junctions & channels were added, which improve the flexibility of the syslog-ng configuration language. This allows combining sources with their closely tied processing functionality (like parser, rewrite and filter statements). Read this blog post for more information: http://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/ In the final form of the functionality the "log" keyword as described in the blog post above was replaced with "channel". * The functionality to query and manipulate sets of name-value pairs (often referenced as value-pairs and used in the mongodb() destination driver and the $(format-json) template function). got significantly improved. It is now possible to change the name of the keys when creating the output. See this commit for more information: https://github.com/bazsi/syslog-ng-3.4/commit/ddc7c2539bd66fa35e8df441e4baf… * Plugins & modules are now demand-loaded automatically if the "autoload-compiled-modules" global variable is set to 1, which is the default. Any shared libraries found on the module search path is considered for loading if the configuration file contains a reference to a functionality it provides. To disable this functionality simply set the referenced variable to 0 with a "@define" statement and load modules explicitly via "@module" statements. To list the available plugins & modules, use the --module-registry command line option for syslog-ng, which results in a detailed listing. * Added a new parser named json-parser() to parse incoming JSON formatted messages. See this commit for more information: https://github.com/bazsi/syslog-ng-3.4/commit/e5569687bba2551c89a78faee55bc… * Added a number of template functions: $(length ARG) - length of a template expression $(substr ARG START [LEN]) - substring of a string $(strip ARG) - remove white space from the start and end $(sanitize ARG1 ARG2) - join args to form a filename while removing special characters like '/' $(+), $(-), $(*), $(/), $(%) - perform numeric operations * Added support for replicasets to the mongodb() destination driver. https://github.com/bazsi/syslog-ng-3.4/commit/a980b9d268efa54879d3deeb4a53f… * Added support for safe-mode() to the mongodb() destination driver that ensures inserts were properly executed at the cost of some performance penalties. https://github.com/bazsi/syslog-ng-3.4/commit/768f0c6ec8eba2ad51531f2331fb5… * Added support for SMTP destination to send out emails triggered by log messages. https://github.com/bazsi/syslog-ng-3.4/commit/404ceb959efe9715ce7437d7dcdc2… * Added support for generating UUIDs via the $(uuid) template function. https://github.com/bazsi/syslog-ng-3.4/commit/a16798c653c057239236945034114… * Added the @SET@ parser to db-parser(). https://github.com/bazsi/syslog-ng-3.4/commit/ab08c84abbeda5a200b8150a6af7b… * Added support for dbd-option() in the sql() destination driver that makes it possible to supply driver specific options to the DBI driver. https://github.com/bazsi/syslog-ng-3.4/commit/9254fe9e6236746bf1f955f0cac12… * Reload of the configuration can now be triggered using "syslog-ng-ctl reload". * A new macro named $LOGHOST was added, which expands to the local hostname running syslog-ng. * A set of time macros were added prefixed with "C_" that use the current time instead of the reception time (prefixed R_) and and the time that was included in the message (prefixed S_). This means that C_DATE expands to the current date, whereas R_DATE would expand to the date the current message was received at. https://github.com/bazsi/syslog-ng-3.4/commit/c2d17009e2ce14960acb519750fe2… * Improved error reporting by including the configuration-file location of the object associated with the error. This makes it easier to diagnose errors even in the case of otherwise unnamed objects. * This release also includes all fixes of the 3.3 branch, which are not listed here for brevity's sake. The merged commit ID is: bf742b0, which is a couple of patches ahead of "3.3.4". Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andreas Piesk Balazs Scheidler (BalaBit) Balint Kovacs (BalaBit) Evan Rempel (University of Victoria) Gergely Nagy (BalaBit) Heiko Gerstung Hendrik Völker (Verizon) Jakub Jankowski (superhost.pl) Martin Grauel (BalaBit) Matthias Runge (Fedora) Patrick Hemmer Russ Milne (Seccuris) DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Strong Cipher for syslogng
by Pramod Pillai 07 Mar '12

07 Mar '12

Hi We are using Syslog NG version is 3.2.4 We haave a requirement to use strong cipher for syslogng listening ports. Is it possible in syslogng version 3.2.4. Regards Pramod

2 1

[Bug 167] New: "configure" error on Solaris 10 SPARC: Glib headers inconsistent....
by bugzilla＠bugzilla.balabit.com 06 Mar '12

06 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=167 Summary: "configure" error on Solaris 10 SPARC: Glib headers inconsistent.... Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: david.j.fenske(a)usps.gov Type of the Report: bug Estimated Hours: 0.0 1. Downloaded syslog-ng-3.3.4 source 2. gunzip/de-tar'ed source 3. Updated eventlog, glib, gcc packages from Sunfreeware.com 4. set PKG_CONFIG_PATH=/usr/local/lib/pkgconfig 5. Ran: ./configure 6. Receive error: configure: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH 7. Have tried setting CFLAGS to -m64 and -m32 resulting in same error. I know this problem was reported in 2008 or so but no "solution" was ever given. (other than to download from sunfreeware.com, which I would do if the latest version was was available). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

start problems
by Jenei Gábor 02 Mar '12

02 Mar '12

Hello, I've already adopted the recommended init script for debian, but still my syslog-ng 3.4 doesn't seem to work. At the moment actually I can run my compiled program just by entering ./syslog-ng, however it does not seem to have access to files, because I don't see any trace in my log files of starting. If I try to start the program by its init script it waits for about a minute, and then says: failed. However after init script there are still running instances. Can you tell me what my problem is? Gabor

1 0

Bazsi's blog: Project Lumberjack to improve Linux logging
by Balazs Scheidler 01 Mar '12

01 Mar '12

Project Lumberjack to improve Linux logging In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Gibbs (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team. We discussed a couple of pain points for logging, logging is usually an afterthought during development, computer based processing, correllation of application logs is nearly impossible. We roughly agreed that the key to improve the situation is to involve the community at large, initiate a momentum and try to get application developers on board and have them create structured logs. We also agreed that this will not happen overnight and we need to take a gradual approach. To move into that direction, the benefits of good logging needs to be communicated and delivered to both application developers and their users. We also talked about what kind of building blocks are needed to deliver a solution fast, and concluded that we basically have everything available, and even better they are open source. The key is to tie these components together, document best practices and perhaps provide better integration. Thus project Lumberjack was born, hosted as a Fedora project at https://fedorahosted.org/lumberjack/. The building blocks that need some care are: * some applications already produce logs in structured format, those should be integrated (auditd for instance) * we need to define a mechanism to submit structured logs to local logging services for further processing (ELAPI and some enhanced syslog) * we need to make sure that local logging services cope with structured data (already available for a long time now) * we need to define a mechanism to store messages in a structured form and a way query them * last, but not least we need to define a naming scheme for event data which CEE can bring to the table Most of these is already possible by using a combination of tools and proper configuration, however learning how to do this is not a trivial undertaking for those who only want to develop or use applications. Changing that is the primary aim of Project Lumberjack. If you are interested in logging, make sure to check that out.

1 0

Re: [syslog-ng] [libivykis-discuss] Cross compiling syslog-ng - problem with pthread_spin_lock
by Balazs Scheidler 29 Feb '12

29 Feb '12

----- Original message ----- > Hey guys, > > > On Wed, Feb 29, 2012 at 09:24:01AM +0100, Balazs Scheidler wrote: > > > > > I am trying to cross compile syslog-ng for my embedded platform. > > > > When I run ./configure pointing it to the proper tool chain, it > > > > indicates "checking for pthread_spin_lock... no". And when I > > > > compile, it complains - "iv_thr.h:97: error: conflicting type > > > > qualifiers for 'pthread_spinlock_t'" > > > > > > I suspect ivykis (lib/ivykis/) is a bit buggy when > > > cross-compiling. Since it fails to detect pthread_spin_lock, it will > > > have HAVE_PTHREAD_SPIN_LOCK disabled, which in turn means that it > > > will try to typedef pthread_spinlock_t to an int. But! Your embedded > > > platform DOES have pthread_spin_lock_t, so this will fail. > > > > > > I would suggest editing lib/ivykis/config.h after configure, and > > > enabling HAVE_PTHREAD_SPIN_LOCK. > > > > > > Hopefully this'll help. > > > > > > (Why the detection fails, though, is not entirely clear to me..) > > > > It'd make sense to forward this email to > > libivykis-discuss(a)lists.sourceforge.net so the ivykis authors become > > aware of the issue. > > Upstream ivykis just assumes that pthread_spin_lock() is always > there, i.e. it doesn't even do a configure test for it, so I think > that this issue is due to a patch unique to syslog-ng's branch/fork > of ivykis. > oops. my bad. probably messed up the configure test. sorry for bothering you Lennart.

1 0

[Bug 164] New: syslog-ng sql destination starts dropping all messages
by bugzilla＠bugzilla.balabit.com 29 Feb '12

29 Feb '12

https://bugzilla.balabit.com/show_bug.cgi?id=164 Summary: syslog-ng sql destination starts dropping all messages Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 syslog-ng 3.3.4 with afsql explicit commit deadlock patch applied libdbi 0.8.3 I have a box writing to an SQLite3 database, and it seems that every few days, syslog-ng gets into a state where it starts dropping all messages supposed to be written to the database. There are no errors written to the syslog-ng log. # syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number ... dst.sql;d_sqlite#0;sqlite3,,,/var/log/logs.sql3,logs;a;dropped;61372 dst.sql;d_sqlite#0;sqlite3,,,/var/log/logs.sql3,logs;a;stored;30 destination;d_sqlite;;a;processed;122885 destination d_sqlite { sql( type('sqlite3') database("/var/log/logs.sql3") table("logs") columns("time", "time_r", "host", "facility", "priority", "program", "pid", "tag", "message") values("$S_UNIXTIME", "$R_UNIXTIME", "$FULLHOST", "$FACILITY_NUM", "$LEVEL_NUM", "$PROGRAM", "$PID", "$DBTAG", "$MSG") null("") flags(explicit-commits) flush_lines(10) flush_timeout(200) log_fifo_size(30) #debug ); }; (gdb) info threads Id Target Id Frame 2 Thread 0x3b1963e2700 (LWP 606) "syslog-ng" 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 * 1 Thread 0x3b1963e5b00 (LWP 605) "syslog-ng" 0x000003b19572fd73 in epoll_wait () from /lib64/libc.so.6 (gdb) bt #0 0x000003b19572fd73 in epoll_wait () from /lib64/libc.so.6 #1 0x000003b195fa176d in iv_epoll_poll (numfds=36, active=0x3c985736130, msec=3600000) at iv_method_epoll.c:73 #2 0x000003b195fa0d3b in iv_main () at iv_main.c:265 #3 0x000003b195f764d1 in main_loop_run () at mainloop.c:731 #4 0x0000004ff3a61d08 in main (argc=1, argv=0x3c9857362b8) at main.c:260 (gdb) thread 2 [Switching to thread 2 (Thread 0x3b1963e2700 (LWP 606))] #0 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt full #0 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000003b192b16170 in afsql_dd_database_thread (arg=0x4ff3cb5e00) at afsql.c:900 self = 0x4ff3cb5e00 #2 0x000003b195f784d5 in worker_thread_func (st=0x4ff3c82570) at misc.c:623 p = 0x4ff3c82570 res = 0x0 mask = {__val = {81923, 0 <repeats 15 times>}} #3 0x000003b195c77dc6 in ?? () from /usr/lib64/libglib-2.0.so.0 No symbol table info available. #4 0x000003b1959e3b2a in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x000003b19572f71d in clone () from /lib64/libc.so.6 No symbol table info available. (gdb) frame 1 #1 0x000003b192b16170 in afsql_dd_database_thread (arg=0x4ff3cb5e00) at afsql.c:900 900 afsql.c: No such file or directory. in afsql.c (gdb) print *self $1 = {super = {super = {super = {ref_cnt = {counter = 1}, flags = 1, cfg = 0x4ff3c7c4c0, pipe_next = 0x0, queue_data = 0x0, queue = 0x3b192b16f02 <afsql_dd_queue>, init = 0x3b192b165b5 <afsql_dd_init>, deinit = 0x3b192b16d42 <afsql_dd_deinit>, free_fn = 0x3b192b1703d <afsql_dd_free>, notify = 0}, optional = 0, group = 0x4ff3cb1420 "d_sqlite", id = 0x4ff3cb14b0 "d_sqlite#0", plugins = 0x0, drv_next = 0x0}, acquire_queue_data = 0x0, acquire_queue = 0x3b195f5715c <log_dest_driver_acquire_queue_method>, release_queue_data = 0x0, release_queue = 0x3b195f5724f <log_dest_driver_release_queue_method>, queues = 0x4ff3c724e0, log_fifo_size = 30, throttle = 0}, type = 0x4ff3cb5da0 "sqlite3", host = 0x4ff3cb5de0 "", port = 0x4ff3cb5fe0 "", user = 0x4ff3cb6000 "syslog-ng", password = 0x4ff3cb6020 "", database = 0x4ff3cb6040 "/var/log/logs.sql3", encoding = 0x4ff3cb6060 "UTF-8", columns = 0x4ff3cb38c0, values = 0x4ff3c72580, indexes = 0x0, table = 0x4ff3cb6080, fields_len = 9, fields = 0x4ff3cb17a0, null_value = 0x4ff3cb63a0 "", time_reopen = 60, num_retries = 3, flush_lines = 10, flush_timeout = 200, flush_lines_queued = 0, flags = 1, session_statements = 0x0, template_options = {ts_format = 0, time_zone = {0x0, 0x0}, time_zone_info = {0x4ff3cb2850, 0x4ff3cb2870}, frac_digits = 0}, dropped_messages = 0x4ff3cb14d0, stored_messages = 0x4ff3cb14d8, db_thread = 0x4ff3c82590, db_thread_mutex = 0x4ff3c82540, db_thread_wakeup_cond = 0x4ff3c82500, db_thread_terminate = 0, db_thread_suspended = 0, db_thread_suspend_target = {tv_sec = 0, tv_usec = 0}, queue = 0x4ff3cb1560, seq_num = 61484, pending_msg = 0x0, pending_msg_ack_needed = 0, dbi_ctx = 0x4ff3c88ad0, validated_tables = 0x4ff3c77d80, failed_message_counter = 0} For some reason gdb is unable to work with a gcore file I've taken, only with the live process, so I can get further information, but I may have to wait a day or two for the issue to recur. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

Cross compiling syslog-ng - problem with pthread_spin_lock
by Radha Krishna Srimanthula 29 Feb '12

29 Feb '12

Hi, I am trying to cross compile syslog-ng for my embedded platform. When I run ./configure pointing it to the proper tool chain, it indicates "checking for pthread_spin_lock... no". And when I compile, it complains - "iv_thr.h:97: error: conflicting type qualifiers for 'pthread_spinlock_t'" Can someone please let me know how to go past this error? Am I missing a prerequisite library? -- Regards, Radha <http://www.sradhakrishna.com>

4 4

[PATCH (3.4) 0/2] value-pairs: Efficient sorting
by Gergely Nagy 28 Feb '12

28 Feb '12

In the two patches that will follow, a subtle, yet very important change will be made to the core of value-pairs: instead of building a GHashTable during a value_pairs_foreach(), we now build a binary tree, that's sorted at insert time, and traversing through it is much faster. This also buys us sorted keys almost free of charge! And once we have that, the second patch exports a value_pairs_foreach_sorted() function, so that modules who want to override the default alphabetical sort will be able to do so. -- |8]

2 8

Rewrite backreference oddity
by Ti Leggett 27 Feb '12

27 Feb '12

I have a rewrite rule that is used to replace private IPs that have no DNS name to an internally used name. It looks like: rewrite r_ddn { subst("172.30.60.(\d*)", "d9550-$1.example.com", value("HOST"), type("pcre")); }; For IP 172.30.60.1 I should get a HOST that is d9550-1.example.com, but what I actually get is d9550-.example.com1. Any ideas what I'm doing incorrectly? I've tried encapsulating the back reference in {}. Here's info about my syslog-ng: [root@admin 02]# syslog-ng -V syslog-ng 3.3.4 Installer-Version: 3.3.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#5e44eb46b0d7b86b62f17698e2b6de875ac8d7c6 Compile-Date: Feb 17 2012 10:55:40 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat Available-Modules: affile,afsocket,dummy,afsocket-tls,afprog,syslogformat,csvparser,afmongodb,confgen,afuser,basicfuncs,dbparser,convertfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on

3 7

RFC: value-pairs() improvements, and the way forward
by Gergely Nagy 25 Feb '12

25 Feb '12

Hi! As is my custom when I get stuck thinking about something probably trivial for those more versed in algorithmical thinking, I turn to the list, and dump my ideas, in the hopes to borrow all the good ideas you lot usually have. And unlike most times, I'll try to be short and to the point for a change. Some of you might have heard about this value-pairs thing that found its way into syslog-ng 3.3, and received some neat new features in 3.4, but as a reminder, let me tell you a few words about it: it's a reasonably convenient, simple, yet flexible way to build a set of key-value pairs from all the data syslog-ng managed to collect about and parse out of a message. In 3.4, one can even apply various transformations to the keys, to massage them into a format suitable for applications outside of syslog-ng. As of this writing, the mongodb destination driver and the format-json template function are the two users of this feature, and a few more users will likely see the light of day in the not too distant future. However, there are multiple issues with the current implementation: - It's far slower than it needs to be. - It's hard to sort the keys alphabetically (which would be useful for format-welf, for example, or anywhere else where we want the order of keys to remain constant between messages) - It would be very, very inconvenient to enhance the implementation to be able to handle nested structures. And we do need nested structures: the json parser (in 3.4) already transforms nested structures into an internal format, that could - in theory - be expanded back to the original, or something very close, but value-pairs does not support that. So, I want to change the *implementation* of value-pairs, to fix all of the problems above. The syntax in the config file *will* remain the same (though, both mongodb and format-json will probably grow an extra parameter, that controls whether we want a flat structure, or a nested object structure). As a first step, I will replace the GHashTables (which is a key-value store, optimised for random access, basically) with GTree (which is a sorted key-value store, optimised for ordered traversal, pretty much what we need). This has the advantage of being a bit more efficient than GHashTables, and gaining us an ordered key list for free. The hard part is writing the flat format => tree conversion, and making it efficient. I have some crude code to do it, which iterates through a sorted key list, splits the key up by dots, finds the appropriate node in the tree, if any, and adds a new child node. This means we iterate through the original key list only once (this is good), but we have to perform a lot of lookups in another tree for each key (this is bad). Now, I can optimise this a bit, and use the ordered property of the original list, and store the path to the last inserted node, and try to use that for speedier backtracking. But that's something I'll probably do in a second version of the update.. I believe this method would be reasonably efficient, but if anyone has a better suggestion, please let me know. -- |8]

2 1

[PATCH] afsql: Fix a deadlock when using explicit commits.
by Gergely Nagy 25 Feb '12

25 Feb '12

When using explicit commits, in case the COMMIT results in an error, syslog-ng forgot to release a mutex, resulting in a deadlock the next time the same error happened. The fix is to do release the mutex in this error path, too. Signed-off-by: Fried Zoltan <deirf(a)balabit.hu> Signed-off-by: Patrick Hemmer <syslogng(a)stormcloud9.net> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsql/afsql.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/modules/afsql/afsql.c b/modules/afsql/afsql.c index b3fec86..d3c72c9 100644 --- a/modules/afsql/afsql.c +++ b/modules/afsql/afsql.c @@ -890,6 +890,7 @@ afsql_dd_database_thread(gpointer arg) { afsql_dd_disconnect(self); afsql_dd_suspend(self); + g_mutex_unlock(self->db_thread_mutex); continue; } } -- 1.7.9

2 1

help with SDATA
by Chris Johnson 25 Feb '12

25 Feb '12

Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log. 2012-02-05T16:24:45.368 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos(a)2636.1.1.1.2.36 reason=\”unset\” ] Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information destination d_mongodb { mongodb( value-pairs( scope("everything") key(".SDATA.junos(a)2636.1.1.1.2.36.reason") ) ); }; Thoughts? Any help is greatly appreciated. Chris

3 4

How to use parser
by 大町靖 25 Feb '12

25 Feb '12

Hello Robert, Thanks for your effort. I rewrite the configurtion below. parser p1{ csv-parser(columns("Msg1","Msg2") delimiters("#AA#") # template ("$Msg1\n$Msg2"); }; destination d_mesge { file("/var/log/messages" template("$MSG1\nMSG2\n")); }; log { source(s_sys); parser(p1); destination(d_mesg); }; But the result isn't unexpected. I want to divide by "#AA#". ("#AA#" is a string.) It seems that the result is divided by the '#' or 'A' . (character). For example, the messages is "12A456 #AA# 789". The result is MSG1 = 12 MSG2=456. I expect that MSG1=12A456 MSG2=789. What should I do ? Thanks, Yasu === Hi, remove the template() option from the parser definition and add it to the definition of the destination. Regards, Robert On 02/07/2012 03:50 PM, 大町靖 wrote: > Hello Experts, > > Please let me know how to use parser setting. > > This is the syslog message. > Msg1 #AA# Msg2 > > I want to divide this message. > I expect the below. > Msg1 > Msg2 > > But It doesn't work well , though I wrote the configuration below. > parser p1{ > csv-parser(columns("Msg1","Msg2") > delimiters("#AA#") > template ("$Msg1\n$Msg2"); > }; > log { source(s_sys); parser(p1); destination(d_mesg); }; > > I check the output. The result wasn't unexpected. > Msg1 #AA# Msg2 > > If my configuration is wrong, please let me know. > > Thanks, > Yasu > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

2 1

Re: [syslog-ng] Installing syslog-ng 3.3.4 OSS on multiple Centos servers
by Srinivasan Subramanian 25 Feb '12

25 Feb '12

What about Mageia ? I see the latest binaries and source RPMS in the Mageia repository? Looks like the latest will always be available there. Srini > Date: Fri, 24 Feb 2012 14:22:46 +0100 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Installing syslog-ng 3.3.4 OSS on multiple > Centos servers > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87pqd49xi1.fsf(a)luthien.mhp> > Content-Type: text/plain > > Srinivasan Subramanian <ssrini_vasan(a)hotmail.com> writes: > > > I have a need to install syslog-ng 3.3.4 on multiple Centos servers. > > Unfortunately i am unable to find RPMs for the newer versions and only > > find the source downloads. > > I don't know of any Centos RPMs, but you might try the sRPMs from > SuSE[1]. At worst, they'll be a reasonable starting point. > > [1]: http://software.opensuse.org/search/download?base=openSUSE%3AFactory&file=B… > > --

1 0

3.3.4 anyone else notice a memory leak
by Evan Rempel 25 Feb '12

25 Feb '12

I run a number of syslog-ng processes, and I am noticing that the processes that read from a pipe, and write to a program destination all seem to have a memory leak in them. The rate of leak seems to be dependent on the volume of log lines (not surprising). Has anyone else noticed this. My config is; @version: 3.3 # # UVic syslog-ng configuration options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; source local { pipe("/var/log/syslog.pipes/syslogstats" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); }; destination syslog_stats { program("/home/sysprog/syslog-apps/syslog_stats/syslogstats" template("$MESSAGE\n") template_escape(no) ); }; log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); };

6 12

syslog-ng Insider - February 2012
by Peter Czanik 23 Feb '12

23 Feb '12

Dear syslog-ng users, This is the 11th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS Brno: Fedora, CEE, journal and syslog-ng ---------------------------------------- Last week the Brno Red Hat office hosted two conferences, one small about logging and the Fedora Developer conference. The logging miniconf covered some very hot topics: CEE, journal, auditd and some lesser known projects, like ELAPI. After the formal program, we had some very good discussions about the future of logging. You can check the diagram drawn up as conclusion here: https://czanik.blogs.balabit.com/2012/02/brno-fedora-cee-journal-and-syslog… And read more about how syslog-ng supports CEE: http://algernon.blogs.balabit.com/2012/02/cee-handling-with-syslog-ng/ BalaBit has just released the latest version of its leading log management tool, syslog-ng 4 F2 ------------------------------- Adding to the existing, rich feature set which includes high-performance multi-thread processing, encrypted and timestamped log files, disk-based buffering, direct database access, native TLS support, the syslog-ng 4 F2 now supports Application-levelAcknowledgement via Reliable Log Transport Protocol (RLTP)^TM , a new transport protocol that prevents message loss during connection breaks. In addition, the latest version of syslog-ng can now natively collect and process log messages from SQL databases enabling users to easily manage log messages from a wide variety of enterprise software and custom applications. syslog-ng 3.3.4 is released --------------------------- It is a bugfix release, which fixes all previously known problems in the 3.3 series. There is only one change in 3.3 sources since the last release: manual pages were put under the GPL, and XML sources are now also available, so that the entire source code of syslog-ng is free from this point onwards. Sources are available at http://www.balabit.com/network- <http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…>security/syslog-ng/opensource-logging-system/downloads/download <http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…> Packages for some distributions are available from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… Detailed changelog is available at http://www.balabit.com/files/syslog-ng/open-source-edition/3.3.4/changelog-… Documentation was also updated: http://www.balabit.com/support/documentation/documents/syslog-ng-ose-3.3-gu… EU Data Protection Directive – How a single regulation could boost the transparency in IT security? -------------------------------- Personal opinion from Balázs Scheidler, CEO of BalaBit Overall, the *EU Data Protection Directive can be a milestone in boosting the transparency of IT security at* organizations – similarly to the regulatory compliance regulations after the Enron case. If adopted, the new directive could bring about a change in *the implementation of IT security policies so that the current focus on audits could shift to the deeper integration of IT security processes into business processes. *As logging and log management are the base of every monitoring method, *technologies with high-speed and zero message loss capabilities, like syslog-ng, will come to the front.* Encrypting log files, in which companies store user names, passwords and other sensitive company data is also key to prevent data loss. http://bscheidler.blogs.balabit.com/2012/02/eu-data-protection-directive-ho… OTHER SHORT NEWS * What makes the upcoming v3.4 even more flexible? It's summarized by Bazsi, lead developer of syslog-ng at http://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/ Watch out for an upcoming Alpha version! * There are still many syslog-ng v2.0 users. For those, here is a teaser, what changed since that release: http://czanik.blogs.balabit.com/2012/02/whats-new-in-syslog-ng-3-x/ * A syslog-ng presentation for FOSDEM: http://czanik.blogs.balabit.com/2012/02/fosdem-syslog-ng-as-upstream/ * ELSA, the high performance web GUI for syslog-ng, received a very useful new feature, post processing of search results: http://ossectools.blogspot.com/2012/01/hunting-with-elsa-transforms.html NEW RELEASES: * syslog-ng OSE 3.3.4: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-January/000133.h… * syslog-ng PE 4F2 ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Extract portion of SDATA file name?
by Andika Daud 21 Feb '12

21 Feb '12

Hi, I'm a noobie and just about beginning to try out syslog-ng at my company. I have the following configuration on the client: source s_apache { file("/apps/log/apache/store-dev03/*" program_override("apache") flags(no-parse) ); }; destination d_remote { syslog("10.5.76.125" transport("tcp") port(514) # template("${MSGONLY}") ); }; log { source(s_apache); destination(d_remote); }; And something like this on the server side: source s_network { syslog(ip(0.0.0.0) port(514) transport("tcp") ); }; destination d_local { file("/apps/log/syslong-ng/$PROGRAM/$HOST/$YEAR$MONTH$DAY.log" # ignore the use of template below, I just wanted to prove the server side could # see the value of the filepath from the client template("${.SDATA.file(a)18372.4.name} - ${MSGONLY}\n") ); }; log { source(s_network); destination(d_local); }; My ("${.SDATA.file(a)18372.4.name} value can be something like this: '/apps/log/apache/store-dev03/access.log'. What I want to be able to do is to parse that path. That is to discard the begining of the path, /apps/log/apache. Get the rest of the value, especially the file name (access.log) to be used to construct destination path. Is this possible? Thank you for your kind help.

2 2

[PATCH (3.3)] doc: Fix EXTRA_DIST
by Gergely Nagy 21 Feb '12

21 Feb '12

There is a \ missing from the end of the second line of EXTRA_DIST, which results in a couple of files being left out of make dist, which in turn makes distcheck and building from dist tarballs fail. This patch adds the missing \. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- doc/Makefile.am | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/doc/Makefile.am b/doc/Makefile.am index 33b113c..6c6c1c2 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -1,7 +1,7 @@ man_MANS = man/syslog-ng.8 man/syslog-ng.conf.5 man/pdbtool.1 man/loggen.1 man/syslog-ng-ctl.1 EXTRA_DIST = man/syslog-ng.8.xml man/syslog-ng.conf.5.xml man/pdbtool.1.xml man/loggen.1.xml man/syslog-ng-ctl.1.xml \ - $(man_MANS) + $(man_MANS) \ security/bof-2002-09-27.txt \ security/dos-2000-11-22.txt \ xsd/patterndb-1.xsd \ -- 1.7.9

1 0

[Bug 152] New: Vers 3.3.3 pipe source does not parse host from input line
by bugzilla＠bugzilla.balabit.com 14 Feb '12

14 Feb '12

https://bugzilla.balabit.com/show_bug.cgi?id=152 Summary: Vers 3.3.3 pipe source does not parse host from input line Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Try as I might, I can not get a pipe source to maintain the hostname. Syslog-ng includes the hostname as the $MSGHDR (or the $PROGRAM). my config # ----------------------------------- source patterndb { pipe("/var/log/syslog.pipes/patterndb" log_fetch_limit(500) log_iw_size(100000) ); }; source int { internal(); }; template t_standardfile { template("$S_ISODATE $FULLHOST $FACILITY.$LEVEL $MSGHDR][$PROGRAM][$MESSAGE\n"); template_escape(no); }; destination d_var_patterndb { file("/var/log/patterndb.$R_YEAR$R_MONTH$R_DAY.000000" owner("root") group("syslogs") perm(0640) template(t_standardfile)); }; log { source(patterndb); destination(d_var_patterndb); }; # ----------------------------------- data I write tothe pipe # ----------------------------------- <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> <12>2011-11-25T00:00:30-08:00 somehost.uvic.ca mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- The output file # ----------------------------------- 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> 2011-11-25T00:00:30-08:00 patterndb(a)catamount.comp.uvic.ca user.warning somehost.uvic.ca ][somehost.uvic.ca][mmfs: Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- I would expect the output file to read # ----------------------------------- 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.618 2011: Accepted and connected to 172.20.102.38 hermes0080 <c0n350> 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.620 2011: Connecting to 172.20.107.23 nestor0167 <c0n200> 2011-11-25T00:00:30-08:00 somehost.uvic.ca user.warning mmfs: ][mmfs][Fri Nov 25 00:00:29.621 2011: Connected to 172.20.107.23 nestor0167 <c0n200> # ----------------------------------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Network compression
by Rory Toma 14 Feb '12

14 Feb '12

I notice that network compression is listed as a feature for syslog-ng. I can't find any documentation on how to enable this anywhere, however. How do I tell a syslog-ng receiver to accept compressed traffic?

2 3

Documentation questions
by Chris Johnson 13 Feb '12

13 Feb '12

I was reading through the documentation, and noticed that in some examples the log statements are named: log demo_filteredlog { source(s1); filter(demo_filter); destination(d1);}; When I tried using this method in my config file, syslog-ng fails to start. Is this method supported or is this just an oversight with the documentation? Also, the links to the sample pattern database (http://www.balabit.com/downloads/files/patterndb-snapshot/?utm_source=syslo…) apprears to be incorrect. Is this the same pattern database at /downloads/files?path=/patterndb-snapshot? or is there a more recent one available to download? Thanks Chris

2 1

Cant get pattern matching to work
by Chris Johnson 12 Feb '12

12 Feb '12

Sorry for bombarding the mailing list with questions. I have been working through the documentation but just seem to keep hitting obstacles. I have compared this to the examples in the manual and some samples I was able to track down, but Im not seeing where I am going wrong. For this question, I am working with the following data: "MESSAGE" : "RT_FLOW_SESSION_CLOSE: session closed unset: 192.168.199.253/55189->8.8.8.8/53 junos-dns-udp X.X.X.X/60836->8.8.8.8/53 source-nat-rule None 17 trust-to-untrust trust untrust 30455 1(83) 1(83) 2 N/A N/A N/A(N/A) vlan.0", "PROGRAM" : "RT_FLOW" Based on this , I have generated the following pattern database and config just to see if I am heading down the right path. Unfortunately, what I thought would generate a pattern match on the logs and generate the appropriate macros, are not retuning anything. I have also tried taking out the $PROGRAM patter, but that did not seem to help. I have also validated this with pdbtool and it succeeds. Any help is greatly appreciated Chris patterndb.xml <patterndb version='3' pub_date='2011-02-11'> <ruleset name='session_close' id='123456678'> <rules> <rule provider='cj' id='182437592347598' class='session'> <patterns> <pattern> @ESTRING:TEST1:::@ @ANYSTRING:TEST2@ </pattern> </patterns> </rule> </rules> </ruleset> </patterndb> syslog-ng.conf #####Destinations##### destination d_mongodb { mongodb( value-pairs( key("TEST1") key("TEST2") scope("base") ) ); }; #####Parser##### parser pattern_db { db_parser( file("/usr/local/etc/patterndb.xml") ); }; #####Log##### log { source(s_network); parser(pattern_db); destination(d_mongodb); }; Unfortunately, what I thought would pattern match and generate a macro for

2 6