- syslog-ng - lists.balabit.hu

Bazsi's blog: Project Lumberjack to improve Linux logging
by Balazs Scheidler 01 Mar '12

01 Mar '12

Project Lumberjack to improve Linux logging In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Gibbs (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team. We discussed a couple of pain points for logging, logging is usually an afterthought during development, computer based processing, correllation of application logs is nearly impossible. We roughly agreed that the key to improve the situation is to involve the community at large, initiate a momentum and try to get application developers on board and have them create structured logs. We also agreed that this will not happen overnight and we need to take a gradual approach. To move into that direction, the benefits of good logging needs to be communicated and delivered to both application developers and their users. We also talked about what kind of building blocks are needed to deliver a solution fast, and concluded that we basically have everything available, and even better they are open source. The key is to tie these components together, document best practices and perhaps provide better integration. Thus project Lumberjack was born, hosted as a Fedora project at https://fedorahosted.org/lumberjack/. The building blocks that need some care are: * some applications already produce logs in structured format, those should be integrated (auditd for instance) * we need to define a mechanism to submit structured logs to local logging services for further processing (ELAPI and some enhanced syslog) * we need to make sure that local logging services cope with structured data (already available for a long time now) * we need to define a mechanism to store messages in a structured form and a way query them * last, but not least we need to define a naming scheme for event data which CEE can bring to the table Most of these is already possible by using a combination of tools and proper configuration, however learning how to do this is not a trivial undertaking for those who only want to develop or use applications. Changing that is the primary aim of Project Lumberjack. If you are interested in logging, make sure to check that out.

1 0

Re: [syslog-ng] [libivykis-discuss] Cross compiling syslog-ng - problem with pthread_spin_lock
by Balazs Scheidler 29 Feb '12

29 Feb '12

----- Original message ----- > Hey guys, > > > On Wed, Feb 29, 2012 at 09:24:01AM +0100, Balazs Scheidler wrote: > > > > > I am trying to cross compile syslog-ng for my embedded platform. > > > > When I run ./configure pointing it to the proper tool chain, it > > > > indicates "checking for pthread_spin_lock... no". And when I > > > > compile, it complains - "iv_thr.h:97: error: conflicting type > > > > qualifiers for 'pthread_spinlock_t'" > > > > > > I suspect ivykis (lib/ivykis/) is a bit buggy when > > > cross-compiling. Since it fails to detect pthread_spin_lock, it will > > > have HAVE_PTHREAD_SPIN_LOCK disabled, which in turn means that it > > > will try to typedef pthread_spinlock_t to an int. But! Your embedded > > > platform DOES have pthread_spin_lock_t, so this will fail. > > > > > > I would suggest editing lib/ivykis/config.h after configure, and > > > enabling HAVE_PTHREAD_SPIN_LOCK. > > > > > > Hopefully this'll help. > > > > > > (Why the detection fails, though, is not entirely clear to me..) > > > > It'd make sense to forward this email to > > libivykis-discuss(a)lists.sourceforge.net so the ivykis authors become > > aware of the issue. > > Upstream ivykis just assumes that pthread_spin_lock() is always > there, i.e. it doesn't even do a configure test for it, so I think > that this issue is due to a patch unique to syslog-ng's branch/fork > of ivykis. > oops. my bad. probably messed up the configure test. sorry for bothering you Lennart.

1 0

[Bug 164] New: syslog-ng sql destination starts dropping all messages
by bugzilla＠bugzilla.balabit.com 29 Feb '12

29 Feb '12

https://bugzilla.balabit.com/show_bug.cgi?id=164 Summary: syslog-ng sql destination starts dropping all messages Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 syslog-ng 3.3.4 with afsql explicit commit deadlock patch applied libdbi 0.8.3 I have a box writing to an SQLite3 database, and it seems that every few days, syslog-ng gets into a state where it starts dropping all messages supposed to be written to the database. There are no errors written to the syslog-ng log. # syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number ... dst.sql;d_sqlite#0;sqlite3,,,/var/log/logs.sql3,logs;a;dropped;61372 dst.sql;d_sqlite#0;sqlite3,,,/var/log/logs.sql3,logs;a;stored;30 destination;d_sqlite;;a;processed;122885 destination d_sqlite { sql( type('sqlite3') database("/var/log/logs.sql3") table("logs") columns("time", "time_r", "host", "facility", "priority", "program", "pid", "tag", "message") values("$S_UNIXTIME", "$R_UNIXTIME", "$FULLHOST", "$FACILITY_NUM", "$LEVEL_NUM", "$PROGRAM", "$PID", "$DBTAG", "$MSG") null("") flags(explicit-commits) flush_lines(10) flush_timeout(200) log_fifo_size(30) #debug ); }; (gdb) info threads Id Target Id Frame 2 Thread 0x3b1963e2700 (LWP 606) "syslog-ng" 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 * 1 Thread 0x3b1963e5b00 (LWP 605) "syslog-ng" 0x000003b19572fd73 in epoll_wait () from /lib64/libc.so.6 (gdb) bt #0 0x000003b19572fd73 in epoll_wait () from /lib64/libc.so.6 #1 0x000003b195fa176d in iv_epoll_poll (numfds=36, active=0x3c985736130, msec=3600000) at iv_method_epoll.c:73 #2 0x000003b195fa0d3b in iv_main () at iv_main.c:265 #3 0x000003b195f764d1 in main_loop_run () at mainloop.c:731 #4 0x0000004ff3a61d08 in main (argc=1, argv=0x3c9857362b8) at main.c:260 (gdb) thread 2 [Switching to thread 2 (Thread 0x3b1963e2700 (LWP 606))] #0 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt full #0 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000003b192b16170 in afsql_dd_database_thread (arg=0x4ff3cb5e00) at afsql.c:900 self = 0x4ff3cb5e00 #2 0x000003b195f784d5 in worker_thread_func (st=0x4ff3c82570) at misc.c:623 p = 0x4ff3c82570 res = 0x0 mask = {__val = {81923, 0 <repeats 15 times>}} #3 0x000003b195c77dc6 in ?? () from /usr/lib64/libglib-2.0.so.0 No symbol table info available. #4 0x000003b1959e3b2a in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x000003b19572f71d in clone () from /lib64/libc.so.6 No symbol table info available. (gdb) frame 1 #1 0x000003b192b16170 in afsql_dd_database_thread (arg=0x4ff3cb5e00) at afsql.c:900 900 afsql.c: No such file or directory. in afsql.c (gdb) print *self $1 = {super = {super = {super = {ref_cnt = {counter = 1}, flags = 1, cfg = 0x4ff3c7c4c0, pipe_next = 0x0, queue_data = 0x0, queue = 0x3b192b16f02 <afsql_dd_queue>, init = 0x3b192b165b5 <afsql_dd_init>, deinit = 0x3b192b16d42 <afsql_dd_deinit>, free_fn = 0x3b192b1703d <afsql_dd_free>, notify = 0}, optional = 0, group = 0x4ff3cb1420 "d_sqlite", id = 0x4ff3cb14b0 "d_sqlite#0", plugins = 0x0, drv_next = 0x0}, acquire_queue_data = 0x0, acquire_queue = 0x3b195f5715c <log_dest_driver_acquire_queue_method>, release_queue_data = 0x0, release_queue = 0x3b195f5724f <log_dest_driver_release_queue_method>, queues = 0x4ff3c724e0, log_fifo_size = 30, throttle = 0}, type = 0x4ff3cb5da0 "sqlite3", host = 0x4ff3cb5de0 "", port = 0x4ff3cb5fe0 "", user = 0x4ff3cb6000 "syslog-ng", password = 0x4ff3cb6020 "", database = 0x4ff3cb6040 "/var/log/logs.sql3", encoding = 0x4ff3cb6060 "UTF-8", columns = 0x4ff3cb38c0, values = 0x4ff3c72580, indexes = 0x0, table = 0x4ff3cb6080, fields_len = 9, fields = 0x4ff3cb17a0, null_value = 0x4ff3cb63a0 "", time_reopen = 60, num_retries = 3, flush_lines = 10, flush_timeout = 200, flush_lines_queued = 0, flags = 1, session_statements = 0x0, template_options = {ts_format = 0, time_zone = {0x0, 0x0}, time_zone_info = {0x4ff3cb2850, 0x4ff3cb2870}, frac_digits = 0}, dropped_messages = 0x4ff3cb14d0, stored_messages = 0x4ff3cb14d8, db_thread = 0x4ff3c82590, db_thread_mutex = 0x4ff3c82540, db_thread_wakeup_cond = 0x4ff3c82500, db_thread_terminate = 0, db_thread_suspended = 0, db_thread_suspend_target = {tv_sec = 0, tv_usec = 0}, queue = 0x4ff3cb1560, seq_num = 61484, pending_msg = 0x0, pending_msg_ack_needed = 0, dbi_ctx = 0x4ff3c88ad0, validated_tables = 0x4ff3c77d80, failed_message_counter = 0} For some reason gdb is unable to work with a gcore file I've taken, only with the live process, so I can get further information, but I may have to wait a day or two for the issue to recur. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

Cross compiling syslog-ng - problem with pthread_spin_lock
by Radha Krishna Srimanthula 29 Feb '12

29 Feb '12

Hi, I am trying to cross compile syslog-ng for my embedded platform. When I run ./configure pointing it to the proper tool chain, it indicates "checking for pthread_spin_lock... no". And when I compile, it complains - "iv_thr.h:97: error: conflicting type qualifiers for 'pthread_spinlock_t'" Can someone please let me know how to go past this error? Am I missing a prerequisite library? -- Regards, Radha <http://www.sradhakrishna.com>

4 4

[PATCH (3.4) 0/2] value-pairs: Efficient sorting
by Gergely Nagy 28 Feb '12

28 Feb '12

In the two patches that will follow, a subtle, yet very important change will be made to the core of value-pairs: instead of building a GHashTable during a value_pairs_foreach(), we now build a binary tree, that's sorted at insert time, and traversing through it is much faster. This also buys us sorted keys almost free of charge! And once we have that, the second patch exports a value_pairs_foreach_sorted() function, so that modules who want to override the default alphabetical sort will be able to do so. -- |8]

2 8

Rewrite backreference oddity
by Ti Leggett 27 Feb '12

27 Feb '12

I have a rewrite rule that is used to replace private IPs that have no DNS name to an internally used name. It looks like: rewrite r_ddn { subst("172.30.60.(\d*)", "d9550-$1.example.com", value("HOST"), type("pcre")); }; For IP 172.30.60.1 I should get a HOST that is d9550-1.example.com, but what I actually get is d9550-.example.com1. Any ideas what I'm doing incorrectly? I've tried encapsulating the back reference in {}. Here's info about my syslog-ng: [root@admin 02]# syslog-ng -V syslog-ng 3.3.4 Installer-Version: 3.3.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#5e44eb46b0d7b86b62f17698e2b6de875ac8d7c6 Compile-Date: Feb 17 2012 10:55:40 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat Available-Modules: affile,afsocket,dummy,afsocket-tls,afprog,syslogformat,csvparser,afmongodb,confgen,afuser,basicfuncs,dbparser,convertfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on

3 7

RFC: value-pairs() improvements, and the way forward
by Gergely Nagy 25 Feb '12

25 Feb '12

Hi! As is my custom when I get stuck thinking about something probably trivial for those more versed in algorithmical thinking, I turn to the list, and dump my ideas, in the hopes to borrow all the good ideas you lot usually have. And unlike most times, I'll try to be short and to the point for a change. Some of you might have heard about this value-pairs thing that found its way into syslog-ng 3.3, and received some neat new features in 3.4, but as a reminder, let me tell you a few words about it: it's a reasonably convenient, simple, yet flexible way to build a set of key-value pairs from all the data syslog-ng managed to collect about and parse out of a message. In 3.4, one can even apply various transformations to the keys, to massage them into a format suitable for applications outside of syslog-ng. As of this writing, the mongodb destination driver and the format-json template function are the two users of this feature, and a few more users will likely see the light of day in the not too distant future. However, there are multiple issues with the current implementation: - It's far slower than it needs to be. - It's hard to sort the keys alphabetically (which would be useful for format-welf, for example, or anywhere else where we want the order of keys to remain constant between messages) - It would be very, very inconvenient to enhance the implementation to be able to handle nested structures. And we do need nested structures: the json parser (in 3.4) already transforms nested structures into an internal format, that could - in theory - be expanded back to the original, or something very close, but value-pairs does not support that. So, I want to change the *implementation* of value-pairs, to fix all of the problems above. The syntax in the config file *will* remain the same (though, both mongodb and format-json will probably grow an extra parameter, that controls whether we want a flat structure, or a nested object structure). As a first step, I will replace the GHashTables (which is a key-value store, optimised for random access, basically) with GTree (which is a sorted key-value store, optimised for ordered traversal, pretty much what we need). This has the advantage of being a bit more efficient than GHashTables, and gaining us an ordered key list for free. The hard part is writing the flat format => tree conversion, and making it efficient. I have some crude code to do it, which iterates through a sorted key list, splits the key up by dots, finds the appropriate node in the tree, if any, and adds a new child node. This means we iterate through the original key list only once (this is good), but we have to perform a lot of lookups in another tree for each key (this is bad). Now, I can optimise this a bit, and use the ordered property of the original list, and store the path to the last inserted node, and try to use that for speedier backtracking. But that's something I'll probably do in a second version of the update.. I believe this method would be reasonably efficient, but if anyone has a better suggestion, please let me know. -- |8]

2 1

[PATCH] afsql: Fix a deadlock when using explicit commits.
by Gergely Nagy 25 Feb '12

25 Feb '12

When using explicit commits, in case the COMMIT results in an error, syslog-ng forgot to release a mutex, resulting in a deadlock the next time the same error happened. The fix is to do release the mutex in this error path, too. Signed-off-by: Fried Zoltan <deirf(a)balabit.hu> Signed-off-by: Patrick Hemmer <syslogng(a)stormcloud9.net> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsql/afsql.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/modules/afsql/afsql.c b/modules/afsql/afsql.c index b3fec86..d3c72c9 100644 --- a/modules/afsql/afsql.c +++ b/modules/afsql/afsql.c @@ -890,6 +890,7 @@ afsql_dd_database_thread(gpointer arg) { afsql_dd_disconnect(self); afsql_dd_suspend(self); + g_mutex_unlock(self->db_thread_mutex); continue; } } -- 1.7.9

2 1

help with SDATA
by Chris Johnson 25 Feb '12

25 Feb '12

Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log. 2012-02-05T16:24:45.368 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos(a)2636.1.1.1.2.36 reason=\”unset\” ] Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information destination d_mongodb { mongodb( value-pairs( scope("everything") key(".SDATA.junos(a)2636.1.1.1.2.36.reason") ) ); }; Thoughts? Any help is greatly appreciated. Chris

3 4

How to use parser
by 大町靖 25 Feb '12

25 Feb '12

Hello Robert, Thanks for your effort. I rewrite the configurtion below. parser p1{ csv-parser(columns("Msg1","Msg2") delimiters("#AA#") # template ("$Msg1\n$Msg2"); }; destination d_mesge { file("/var/log/messages" template("$MSG1\nMSG2\n")); }; log { source(s_sys); parser(p1); destination(d_mesg); }; But the result isn't unexpected. I want to divide by "#AA#". ("#AA#" is a string.) It seems that the result is divided by the '#' or 'A' . (character). For example, the messages is "12A456 #AA# 789". The result is MSG1 = 12 MSG2=456. I expect that MSG1=12A456 MSG2=789. What should I do ? Thanks, Yasu === Hi, remove the template() option from the parser definition and add it to the definition of the destination. Regards, Robert On 02/07/2012 03:50 PM, 大町靖 wrote: > Hello Experts, > > Please let me know how to use parser setting. > > This is the syslog message. > Msg1 #AA# Msg2 > > I want to divide this message. > I expect the below. > Msg1 > Msg2 > > But It doesn't work well , though I wrote the configuration below. > parser p1{ > csv-parser(columns("Msg1","Msg2") > delimiters("#AA#") > template ("$Msg1\n$Msg2"); > }; > log { source(s_sys); parser(p1); destination(d_mesg); }; > > I check the output. The result wasn't unexpected. > Msg1 #AA# Msg2 > > If my configuration is wrong, please let me know. > > Thanks, > Yasu > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

2 1