- syslog-ng - lists.balabit.hu

max recursion depth on filters?
by Patrick Hemmer 11 Mar '12

11 Mar '12

Is there a maximum depth on recursive filter statements? I seem to have an issue which would indicate so. For example, in the config below, i have f_discard, which calls f_iptables_discard, which calls f_iptables. I have it like this so that its easy to read, configure, and add other filters. Its supposed to result in lines matching this to not be logged, but its not working. However if I take the contents of `f_iptables_discard` and put them directly into `f_discard` it works fine. filter f_iptables { program('^kernel') and message('^\s*\[\s*[\d\.]+\] iptables/' type(pcre)) }; filter f_iptables_discard { # ignore iptables broadcast messages filter(f_iptables) and message('MAC=ff:ff:ff:ff:ff:ff') ; }; filter f_discard { not ( filter(f_iptables_discard) ); }; rewrite r_tag { set('iptables', value('DBTAG') condition(filter(f_iptables))); }; log { source(s_local); source(s_net); rewrite(r_tag); filter(f_discard); destination(d_sqlite); };

2 2

Parser not parsing :-)
by T. A. Smooth 11 Mar '12

11 Mar '12

I can only assume I am not implementing this correctly. :-) But I have a parser I am trying to use so I can take a subset of the information of a message and send that subset to another receiver. This is the whole message: <13>Feb 4 18:40:17 myhost syslogng: 2012-02-04T18:40:17-08:00 > myhostserver-http /tmp/logs/access_log Hi Mom > What I want to do is send out the message as : <13>Feb 4 18:40:17 myhost syslogng: Hi Mom Notice how I dropped the middle part out. >From what I have read, the parser acts on the message body alone. Is this correct? So I set it up to look for four(4) columns of data and to be "greedy" on the last column. I have played around with the number of columns and even used a rewrite function instead. But the Parser continues to produce empty variables. And my template just echos out my default value. Any thoughts? > > > parser p_et_logmessage { > csv-parser( > #columns("ETMSG") > #columns("ETMSG.ISODATE") > columns("ETMSG.ISODATE", "ETMSG.EASI", "ETMSG.SOURCE", > "ETMSG.BODY") > delimiters(" ") > #template("${MSG}") > flags(greedy) > ); > }; > > rewrite r_rewrite_set{set('${ETMSG.BODY:-nothing}', value("MESSAGE"));}; > > template t_et_basic_logmessage { > template("${ETMSG.BODY:-nothing}\n"); template_escape(no); }; > > > destination destination_info { > tcp("host2" port(8080) > template(t_et_basic_logmessage) > log_disk_fifo_size(32212254720) > ); > }; > > log { > source(INTAKE); > parser(p_et_logmessage); > destination(destination_info); > }; > My latest Post: Givenchy Fall/Winter 2012 Collection – Runway | Highsnobiety.com<http://www.webeclubbin.com/blog/2012/01/givenchy-fallwinter-2012-collection…> Get a signature like this. <http://r1.wisestamp.com/r/landing?promo=19&dest=http%3A%2F%2Fwww.wisestamp.…> CLICK HERE.<http://r1.wisestamp.com/r/landing?promo=19&dest=http%3A%2F%2Fwww.wisestamp.…>

4 9

syslog-ng 3.4.0alpha1 has been released
by devel＠balabit.hu 11 Mar '12

11 Mar '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0alpha1 SUMMARY : new stable release DATE : Mar 11, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.4.0alpha1) has been released. This is an alpha release aiming at collecting feedback and testing on the upcoming 3.4 version. Use in production environments is not recommended. CHANGES: 3.4.0alpha1 Features: ========= * Support for junctions & channels were added, which improve the flexibility of the syslog-ng configuration language. This allows combining sources with their closely tied processing functionality (like parser, rewrite and filter statements). Read this blog post for more information: http://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/ In the final form of the functionality the "log" keyword as described in the blog post above was replaced with "channel". * The functionality to query and manipulate sets of name-value pairs (often referenced as value-pairs and used in the mongodb() destination driver and the $(format-json) template function). got significantly improved. It is now possible to change the name of the keys when creating the output. See this commit for more information: https://github.com/bazsi/syslog-ng-3.4/commit/ddc7c2539bd66fa35e8df441e4baf… * Plugins & modules are now demand-loaded automatically if the "autoload-compiled-modules" global variable is set to 1, which is the default. Any shared libraries found on the module search path is considered for loading if the configuration file contains a reference to a functionality it provides. To disable this functionality simply set the referenced variable to 0 with a "@define" statement and load modules explicitly via "@module" statements. To list the available plugins & modules, use the --module-registry command line option for syslog-ng, which results in a detailed listing. * Added a new parser named json-parser() to parse incoming JSON formatted messages. See this commit for more information: https://github.com/bazsi/syslog-ng-3.4/commit/e5569687bba2551c89a78faee55bc… * Added a number of template functions: $(length ARG) - length of a template expression $(substr ARG START [LEN]) - substring of a string $(strip ARG) - remove white space from the start and end $(sanitize ARG1 ARG2) - join args to form a filename while removing special characters like '/' $(+), $(-), $(*), $(/), $(%) - perform numeric operations * Added support for replicasets to the mongodb() destination driver. https://github.com/bazsi/syslog-ng-3.4/commit/a980b9d268efa54879d3deeb4a53f… * Added support for safe-mode() to the mongodb() destination driver that ensures inserts were properly executed at the cost of some performance penalties. https://github.com/bazsi/syslog-ng-3.4/commit/768f0c6ec8eba2ad51531f2331fb5… * Added support for SMTP destination to send out emails triggered by log messages. https://github.com/bazsi/syslog-ng-3.4/commit/404ceb959efe9715ce7437d7dcdc2… * Added support for generating UUIDs via the $(uuid) template function. https://github.com/bazsi/syslog-ng-3.4/commit/a16798c653c057239236945034114… * Added the @SET@ parser to db-parser(). https://github.com/bazsi/syslog-ng-3.4/commit/ab08c84abbeda5a200b8150a6af7b… * Added support for dbd-option() in the sql() destination driver that makes it possible to supply driver specific options to the DBI driver. https://github.com/bazsi/syslog-ng-3.4/commit/9254fe9e6236746bf1f955f0cac12… * Reload of the configuration can now be triggered using "syslog-ng-ctl reload". * A new macro named $LOGHOST was added, which expands to the local hostname running syslog-ng. * A set of time macros were added prefixed with "C_" that use the current time instead of the reception time (prefixed R_) and and the time that was included in the message (prefixed S_). This means that C_DATE expands to the current date, whereas R_DATE would expand to the date the current message was received at. https://github.com/bazsi/syslog-ng-3.4/commit/c2d17009e2ce14960acb519750fe2… * Improved error reporting by including the configuration-file location of the object associated with the error. This makes it easier to diagnose errors even in the case of otherwise unnamed objects. * This release also includes all fixes of the 3.3 branch, which are not listed here for brevity's sake. The merged commit ID is: bf742b0, which is a couple of patches ahead of "3.3.4". Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Andreas Piesk Balazs Scheidler (BalaBit) Balint Kovacs (BalaBit) Evan Rempel (University of Victoria) Gergely Nagy (BalaBit) Heiko Gerstung Hendrik Völker (Verizon) Jakub Jankowski (superhost.pl) Martin Grauel (BalaBit) Matthias Runge (Fedora) Patrick Hemmer Russ Milne (Seccuris) DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Strong Cipher for syslogng
by Pramod Pillai 07 Mar '12

07 Mar '12

Hi We are using Syslog NG version is 3.2.4 We haave a requirement to use strong cipher for syslogng listening ports. Is it possible in syslogng version 3.2.4. Regards Pramod

2 1

[Bug 167] New: "configure" error on Solaris 10 SPARC: Glib headers inconsistent....
by bugzilla＠bugzilla.balabit.com 06 Mar '12

06 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=167 Summary: "configure" error on Solaris 10 SPARC: Glib headers inconsistent.... Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: david.j.fenske(a)usps.gov Type of the Report: bug Estimated Hours: 0.0 1. Downloaded syslog-ng-3.3.4 source 2. gunzip/de-tar'ed source 3. Updated eventlog, glib, gcc packages from Sunfreeware.com 4. set PKG_CONFIG_PATH=/usr/local/lib/pkgconfig 5. Ran: ./configure 6. Receive error: configure: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH 7. Have tried setting CFLAGS to -m64 and -m32 resulting in same error. I know this problem was reported in 2008 or so but no "solution" was ever given. (other than to download from sunfreeware.com, which I would do if the latest version was was available). -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

start problems
by Jenei Gábor 02 Mar '12

02 Mar '12

Hello, I've already adopted the recommended init script for debian, but still my syslog-ng 3.4 doesn't seem to work. At the moment actually I can run my compiled program just by entering ./syslog-ng, however it does not seem to have access to files, because I don't see any trace in my log files of starting. If I try to start the program by its init script it waits for about a minute, and then says: failed. However after init script there are still running instances. Can you tell me what my problem is? Gabor

1 0

Bazsi's blog: Project Lumberjack to improve Linux logging
by Balazs Scheidler 01 Mar '12

01 Mar '12

Project Lumberjack to improve Linux logging In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Gibbs (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team. We discussed a couple of pain points for logging, logging is usually an afterthought during development, computer based processing, correllation of application logs is nearly impossible. We roughly agreed that the key to improve the situation is to involve the community at large, initiate a momentum and try to get application developers on board and have them create structured logs. We also agreed that this will not happen overnight and we need to take a gradual approach. To move into that direction, the benefits of good logging needs to be communicated and delivered to both application developers and their users. We also talked about what kind of building blocks are needed to deliver a solution fast, and concluded that we basically have everything available, and even better they are open source. The key is to tie these components together, document best practices and perhaps provide better integration. Thus project Lumberjack was born, hosted as a Fedora project at https://fedorahosted.org/lumberjack/. The building blocks that need some care are: * some applications already produce logs in structured format, those should be integrated (auditd for instance) * we need to define a mechanism to submit structured logs to local logging services for further processing (ELAPI and some enhanced syslog) * we need to make sure that local logging services cope with structured data (already available for a long time now) * we need to define a mechanism to store messages in a structured form and a way query them * last, but not least we need to define a naming scheme for event data which CEE can bring to the table Most of these is already possible by using a combination of tools and proper configuration, however learning how to do this is not a trivial undertaking for those who only want to develop or use applications. Changing that is the primary aim of Project Lumberjack. If you are interested in logging, make sure to check that out.

1 0

Re: [syslog-ng] [libivykis-discuss] Cross compiling syslog-ng - problem with pthread_spin_lock
by Balazs Scheidler 29 Feb '12

29 Feb '12

----- Original message ----- > Hey guys, > > > On Wed, Feb 29, 2012 at 09:24:01AM +0100, Balazs Scheidler wrote: > > > > > I am trying to cross compile syslog-ng for my embedded platform. > > > > When I run ./configure pointing it to the proper tool chain, it > > > > indicates "checking for pthread_spin_lock... no". And when I > > > > compile, it complains - "iv_thr.h:97: error: conflicting type > > > > qualifiers for 'pthread_spinlock_t'" > > > > > > I suspect ivykis (lib/ivykis/) is a bit buggy when > > > cross-compiling. Since it fails to detect pthread_spin_lock, it will > > > have HAVE_PTHREAD_SPIN_LOCK disabled, which in turn means that it > > > will try to typedef pthread_spinlock_t to an int. But! Your embedded > > > platform DOES have pthread_spin_lock_t, so this will fail. > > > > > > I would suggest editing lib/ivykis/config.h after configure, and > > > enabling HAVE_PTHREAD_SPIN_LOCK. > > > > > > Hopefully this'll help. > > > > > > (Why the detection fails, though, is not entirely clear to me..) > > > > It'd make sense to forward this email to > > libivykis-discuss(a)lists.sourceforge.net so the ivykis authors become > > aware of the issue. > > Upstream ivykis just assumes that pthread_spin_lock() is always > there, i.e. it doesn't even do a configure test for it, so I think > that this issue is due to a patch unique to syslog-ng's branch/fork > of ivykis. > oops. my bad. probably messed up the configure test. sorry for bothering you Lennart.

1 0

[Bug 164] New: syslog-ng sql destination starts dropping all messages
by bugzilla＠bugzilla.balabit.com 29 Feb '12

29 Feb '12

https://bugzilla.balabit.com/show_bug.cgi?id=164 Summary: syslog-ng sql destination starts dropping all messages Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 syslog-ng 3.3.4 with afsql explicit commit deadlock patch applied libdbi 0.8.3 I have a box writing to an SQLite3 database, and it seems that every few days, syslog-ng gets into a state where it starts dropping all messages supposed to be written to the database. There are no errors written to the syslog-ng log. # syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number ... dst.sql;d_sqlite#0;sqlite3,,,/var/log/logs.sql3,logs;a;dropped;61372 dst.sql;d_sqlite#0;sqlite3,,,/var/log/logs.sql3,logs;a;stored;30 destination;d_sqlite;;a;processed;122885 destination d_sqlite { sql( type('sqlite3') database("/var/log/logs.sql3") table("logs") columns("time", "time_r", "host", "facility", "priority", "program", "pid", "tag", "message") values("$S_UNIXTIME", "$R_UNIXTIME", "$FULLHOST", "$FACILITY_NUM", "$LEVEL_NUM", "$PROGRAM", "$PID", "$DBTAG", "$MSG") null("") flags(explicit-commits) flush_lines(10) flush_timeout(200) log_fifo_size(30) #debug ); }; (gdb) info threads Id Target Id Frame 2 Thread 0x3b1963e2700 (LWP 606) "syslog-ng" 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 * 1 Thread 0x3b1963e5b00 (LWP 605) "syslog-ng" 0x000003b19572fd73 in epoll_wait () from /lib64/libc.so.6 (gdb) bt #0 0x000003b19572fd73 in epoll_wait () from /lib64/libc.so.6 #1 0x000003b195fa176d in iv_epoll_poll (numfds=36, active=0x3c985736130, msec=3600000) at iv_method_epoll.c:73 #2 0x000003b195fa0d3b in iv_main () at iv_main.c:265 #3 0x000003b195f764d1 in main_loop_run () at mainloop.c:731 #4 0x0000004ff3a61d08 in main (argc=1, argv=0x3c9857362b8) at main.c:260 (gdb) thread 2 [Switching to thread 2 (Thread 0x3b1963e2700 (LWP 606))] #0 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt full #0 0x000003b1959e84bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000003b192b16170 in afsql_dd_database_thread (arg=0x4ff3cb5e00) at afsql.c:900 self = 0x4ff3cb5e00 #2 0x000003b195f784d5 in worker_thread_func (st=0x4ff3c82570) at misc.c:623 p = 0x4ff3c82570 res = 0x0 mask = {__val = {81923, 0 <repeats 15 times>}} #3 0x000003b195c77dc6 in ?? () from /usr/lib64/libglib-2.0.so.0 No symbol table info available. #4 0x000003b1959e3b2a in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x000003b19572f71d in clone () from /lib64/libc.so.6 No symbol table info available. (gdb) frame 1 #1 0x000003b192b16170 in afsql_dd_database_thread (arg=0x4ff3cb5e00) at afsql.c:900 900 afsql.c: No such file or directory. in afsql.c (gdb) print *self $1 = {super = {super = {super = {ref_cnt = {counter = 1}, flags = 1, cfg = 0x4ff3c7c4c0, pipe_next = 0x0, queue_data = 0x0, queue = 0x3b192b16f02 <afsql_dd_queue>, init = 0x3b192b165b5 <afsql_dd_init>, deinit = 0x3b192b16d42 <afsql_dd_deinit>, free_fn = 0x3b192b1703d <afsql_dd_free>, notify = 0}, optional = 0, group = 0x4ff3cb1420 "d_sqlite", id = 0x4ff3cb14b0 "d_sqlite#0", plugins = 0x0, drv_next = 0x0}, acquire_queue_data = 0x0, acquire_queue = 0x3b195f5715c <log_dest_driver_acquire_queue_method>, release_queue_data = 0x0, release_queue = 0x3b195f5724f <log_dest_driver_release_queue_method>, queues = 0x4ff3c724e0, log_fifo_size = 30, throttle = 0}, type = 0x4ff3cb5da0 "sqlite3", host = 0x4ff3cb5de0 "", port = 0x4ff3cb5fe0 "", user = 0x4ff3cb6000 "syslog-ng", password = 0x4ff3cb6020 "", database = 0x4ff3cb6040 "/var/log/logs.sql3", encoding = 0x4ff3cb6060 "UTF-8", columns = 0x4ff3cb38c0, values = 0x4ff3c72580, indexes = 0x0, table = 0x4ff3cb6080, fields_len = 9, fields = 0x4ff3cb17a0, null_value = 0x4ff3cb63a0 "", time_reopen = 60, num_retries = 3, flush_lines = 10, flush_timeout = 200, flush_lines_queued = 0, flags = 1, session_statements = 0x0, template_options = {ts_format = 0, time_zone = {0x0, 0x0}, time_zone_info = {0x4ff3cb2850, 0x4ff3cb2870}, frac_digits = 0}, dropped_messages = 0x4ff3cb14d0, stored_messages = 0x4ff3cb14d8, db_thread = 0x4ff3c82590, db_thread_mutex = 0x4ff3c82540, db_thread_wakeup_cond = 0x4ff3c82500, db_thread_terminate = 0, db_thread_suspended = 0, db_thread_suspend_target = {tv_sec = 0, tv_usec = 0}, queue = 0x4ff3cb1560, seq_num = 61484, pending_msg = 0x0, pending_msg_ack_needed = 0, dbi_ctx = 0x4ff3c88ad0, validated_tables = 0x4ff3c77d80, failed_message_counter = 0} For some reason gdb is unable to work with a gcore file I've taken, only with the live process, so I can get further information, but I may have to wait a day or two for the issue to recur. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

Cross compiling syslog-ng - problem with pthread_spin_lock
by Radha Krishna Srimanthula 29 Feb '12

29 Feb '12

Hi, I am trying to cross compile syslog-ng for my embedded platform. When I run ./configure pointing it to the proper tool chain, it indicates "checking for pthread_spin_lock... no". And when I compile, it complains - "iv_thr.h:97: error: conflicting type qualifiers for 'pthread_spinlock_t'" Can someone please let me know how to go past this error? Am I missing a prerequisite library? -- Regards, Radha <http://www.sradhakrishna.com>

4 4