- syslog-ng - lists.balabit.hu

Log Directories created with IP and not hostnames
by James McDonald 10 May '12

10 May '12

Hi, I am new to syslog-ng, as well as this board. I posted a question before my account was verified, so I'm worried it didn't get posted, or I missed the responses. The Windows clients in my environment are creating their log directories named after their IP's and not their hostnames. The RHEL clients are using their hostnames. I would like to get the Windows to use their hostnames too. All the config files look to be configured correctly. Does anyone have any suggestions on what I can check or try to get the Windows clients to create their log directories using their hostnames? Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

1 0

[Bug 179] New: SDATA from Loggen Not Recorded
by bugzilla＠bugzilla.balabit.com 09 May '12

09 May '12

https://bugzilla.balabit.com/show_bug.cgi?id=179 Summary: SDATA from Loggen Not Recorded Product: syslog-ng Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bmehne(a)gmail.com Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=60) --> (https://bugzilla.balabit.com/attachment.cgi?id=60) Syslog-ng conf I have been trying to find some API to write SDATA to mongodb, but it seems that syslog-ng is not parsing the sdata correctly or loggen is not being called correctly. I am calling loggen with: loggen -r 5 -P -p "[syslog@0 uid=\"system\" id=\"4937903063198901248\" sid=\"-\" svchome=\"/run/cronsvc/01/1\" svcid=\"dummy\" name=\"JobExecutionStateManagementJob\" tid=\"__system__\" type=\"SYSTEM\" cid=\"-\" eid=\"-\"]" -S -s 1024 -i -I 2000 localhost 1000 My syslog-ng.conf is attached. I have compiled and run with syslog-ng 3.3.1 (from tar), 3.3.4 (as in debian repos), and 3.4 (from github). I have also tried with the mongodb driver patched to insert rather than upsert (from algernon github repo, flat-insert branch). My testing is taken from http://www.syslog.org/forum/index.php?topic=1233.0 My /var/log/messages.json spits out empty braces (e.g. { } ) for each message, and mongodb does record "SDATA" : "[meta sequenceId=\"70\"]",but that is not the intended sdata. Any help would be appreciated. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

if no host; do use_dns
by Lance Laursen 09 May '12

09 May '12

Hi list, With keep_hostname(yes), when the syslog-ng box receives a message which does not have a host field, you end up with the last hop IP as the hostname. Is there already a way to tell sysng to use_dns() when this situation is encountered? I'd like to continue using keep_hostname(yes) for all log messages which do have a hostname included, but in the instance that hostname is not included, have syslog-ng do a reverse dns lookup in order to resolve hostname. Thanks,

2 1

Windows clients using IP instead of hostname to create directories
by James McDonald 09 May '12

09 May '12

I have a number of Windows clients sending logs back to a central log server running RHEL 5.5. The Linux clients each create a directory for their logs using their respective hostnames as the name of the directory. The Windows clients, however, are using their IP addresses to name their respective directories. The hostnames and IP's are resolving correctly (tested via nslookup and ping) and, from what I can tell, the config files seem to be correct. Is there anything I can double-check or test that might help resolve this issue? I am using version 4.0.1 on the Windows clients and they are running Server 2008R2. Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

1 0

How do I get pdbtool to create version 4 patterndb.xml?
by Kurt Yoder 08 May '12

08 May '12

All, I'm trying to create patterndb version 4 XML so I can make use of its additional features over version 3. However pdbtool seems only to output version 3 XML. I verified that all of my component XML files are version 4, and used pdbtool from syslog-ng 3.3.5: $ /opt/syslog-ng/bin/pdbtool merge --pdb=/tmp/patterndb.xml --recursive --glob='*.pdb' --directory=/etc/syslog-ng/patterndb.d/ The generated file is version 3: $ head -2 /tmp/patterndb.xml <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2012-05-07'> How do I generate version 4 XML?

2 1

[Bug 177] New: crashes in syslog-ng-2.0.10-1 in Fedora i386
by bugzilla＠bugzilla.balabit.com 07 May '12

07 May '12

https://bugzilla.balabit.com/show_bug.cgi?id=177 Summary: crashes in syslog-ng-2.0.10-1 in Fedora i386 Product: syslog-ng Version: 2.0.x Platform: Other OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: rcp(a)sentientmeat.ca Type of the Report: bug Estimated Hours: 0.0 Let me start by apologizing for the age of the software and the platform. Our situation is that we need to support millisecond timestamps on an older branch of an embedded platform. This platform was previously happily running the Fedora Core 4 package syslog-ng-1.6.11-1.fc4.i386.rpm. We took the syslog-ng-2.0.10-1.fc10.src.rpm and eventlog-0.2.7-1.fc10.src.rpm packages from Fedora 10, and have recompiled them for Fedora 4. Unfortunately we've ended up with stability issues, in that the daemon crashes. We haven't yet found a recipe for reproducing the problem, but it happens daily in our lab. We'd like to know if these issues have been encountered before and if there are fixes available. We've attempted upgrading past the issue, by recompiling Fedora 12's syslog-ng-2.1.4-6.fc12.src.rpm and eventlog-0.2.7-4.fc12.src.rpm, and when that also suffered crashes, we added glib2-2.22.5-1.fc12.src.rpm. We've looked at syslog-ng 3, but the long list of dependancies that need to be updated for this platform is rather daunting. I'm including below the backtraces from our corefiles. We've seen the following many times. The environment included: export MALLOC_CHECK_=3 export LIBC_FATAL_STDERR_=1 export MALLOC_PERTURB_=171 core.2196.b50 warning: Can't read pathname for load map: Input/output error. Failed to read a valid object file image from memory. Core was generated by `/sbin/syslog-ng -p /var/run/syslogd.pid'. Program terminated with signal 6, Aborted. #0 0x002f0410 in ?? () (gdb) bt #0 0x002f0410 in ?? () #1 0xbff5de20 in ?? () #2 0x00000006 in ?? () #3 0x00000894 in ?? () #4 0x4ddda1f8 in raise () from /lib/tls/libc.so.6 #5 0x4dddb948 in abort () from /lib/tls/libc.so.6 #6 0x4de0f52a in __libc_message () from /lib/tls/libc.so.6 #7 0x4de15424 in _int_free () from /lib/tls/libc.so.6 #8 0x4de1595f in free () from /lib/tls/libc.so.6 #9 0x4dfb8b44 in g_free () from /usr/lib/libglib-2.0.so.0 #10 0x0806000a in log_reader_free (s=0x8ece698) at logreader.c:457 #11 0x08054274 in log_pipe_unref (self=0x8ece698) at logpipe.c:51 #12 0x0805f582 in log_reader_fd_finalize (source=0x8ecf560) at logreader.c:167 #13 0x4dfb0e7a in ?? () from /usr/lib/libglib-2.0.so.0 #14 0x4dfb250e in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 #15 0x4dfb53f6 in ?? () from /usr/lib/libglib-2.0.so.0 #16 0x4dfb58d8 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 #17 0x0804b73e in main_loop_run (cfg=0xbff5e940) at main.c:167 #18 0x0804c0e4 in main (argc=3, argv=0xbff5e9e4) at main.c:483 (gdb) frame 6 #6 0x4de0f52a in __libc_message () from /lib/tls/libc.so.6 (gdb) info registers eax 0x0 0 ecx 0x894 2196 edx 0x6 6 ebx 0x4ded7ff4 1307410420 esp 0xbff5df54 0xbff5df54 ebp 0xbff5e5a4 0xbff5e5a4 esi 0x0 0 edi 0xc 12 eip 0x4de0f52a 0x4de0f52a <__libc_fatal> eflags 0x202 [ IF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0xc010007b -1072693125 fs 0x0 0 gs 0x33 51 (gdb) x/32x 0xbff5df54 0xbff5df54: 0x0000000c 0xbff5e090 0x00000400 0x4decd3b4 0xbff5df64: 0x00000017 0xbff5eeab 0x0000000f 0x4decd3cd 0xbff5df74: 0x00000002 0x4decd46c 0x0000001f 0x4decd3d1 0xbff5df84: 0x00000004 0xbff5e607 0x00000008 0x4decd3d7 0xbff5df94: 0x00000005 0x00000000 0x4e01317c 0x08eca1b0 0xbff5dfa4: 0x4e01317c 0x00000000 0x00000000 0x4decd3d7 0xbff5dfb4: 0x00000005 0xbff5dfd0 0x00000025 0x00000000 0xbff5dfc4: 0x00000000 0x00000000 0xffffffff 0xbff5e607 (gdb) x/s 0x4decd3b4 0x4decd3b4: "*** glibc detected *** %s: %s: 0x%s ***\n" (gdb) x/s 0xbff5eeab 0xbff5eeab: "/sbin/syslog-ng" (gdb) x/s 0x4decd3cd 0x4decd3cd: ": %s: 0x%s ***\n" (gdb) x/s 0x4decd46c 0x4decd46c: "double free or corruption (out)" (gdb) x/s 0x4decd3d1 0x4decd3d1: ": 0x%s ***\n" (gdb) x/s 0xbff5e607 0xbff5e607: "08ee8bd0" # Same error, but without the Heap Consistency Checking environment variables: core.2229.a34 warning: Can't read pathname for load map: Input/output error. Failed to read a valid object file image from memory. Core was generated by `/sbin/syslog-ng -p /var/run/syslogd.pid'. Program terminated with signal 11, Segmentation fault. #0 0x4de15982 in free () from /lib/tls/libc.so.6 #0 0x4de15982 in free () from /lib/tls/libc.so.6 #1 0x4dfb8b44 in g_free () from /usr/lib/libglib-2.0.so.0 #2 0x0806000a in log_reader_free (s=0x9532ae8) at logreader.c:457 #3 0x08054274 in log_pipe_unref (self=0x9532ae8) at logpipe.c:51 #4 0x0805f582 in log_reader_fd_finalize (source=0x9538190) at logreader.c:167 #5 0x4dfb0e7a in ?? () from /usr/lib/libglib-2.0.so.0 #6 0x4dfb250e in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 #7 0x4dfb53f6 in ?? () from /usr/lib/libglib-2.0.so.0 #8 0x4dfb58d8 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 #9 0x0804b73e in main_loop_run (cfg=0xbfb8af40) at main.c:167 #10 0x0804c0e4 in main (argc=3, argv=0xbfb8afe4) at main.c:483 # This traceback was a one-time error: core.2351.b79 warning: Can't read pathname for load map: Input/output error. Failed to read a valid object file image from memory. Core was generated by `/sbin/syslog-ng -p /var/run/syslogd.pid'. Program terminated with signal 11, Segmentation fault. #0 0x080555c2 in log_center_queue (s=0x88c50d0, msg=0x88bb918, path_flags=0) at center.c:354 354 center.c: No such file or directory. in center.c #0 0x080555c2 in log_center_queue (s=0x88c50d0, msg=0x88bb918, path_flags=0) at center.c:354 #1 0x080547e2 in log_pipe_queue (s=0x88c50d0, msg=0x88bb918, path_flags=0) at logpipe.h:72 #2 0x080547a9 in log_source_group_queue (s=0x88c4568, msg=0x88bb918, path_flags=0) at sgroup.c:140 #3 0x080542c9 in log_pipe_queue (s=0x88c4568, msg=0x88bb918, path_flags=0) at logpipe.h:72 #4 0x080542a5 in log_pipe_forward_msg (self=0x88c5658, msg=0x88bb918, path_flags=0) at logpipe.c:60 #5 0x080599e6 in log_pipe_queue (s=0x88c5658, msg=0x88bb918, path_flags=0) at logpipe.h:72 #6 0x080599c2 in afsocket_sc_queue (s=0x88bd2c8, msg=0x88bb918, path_flags=0) at afsocket.c:261 #7 0x0805f0fc in log_pipe_queue (s=0x88bd2c8, msg=0x88bb918, path_flags=0) at logpipe.h:72 #8 0x0805f0d8 in log_source_queue (s=0x88bd2f8, msg=0x88bb918, path_flags=0) at logsource.c:49 #9 0x0805f72f in log_pipe_queue (s=0x88bd2f8, msg=0x88bb918, path_flags=0) at logpipe.h:72 #10 0x0805f700 in log_reader_handle_line (self=0x88bd2f8, line=0x88d9bd0 "<131>Apr 18 23:14:51 root[9796]: /usr/sw", ' ' <repeats 24 times>, "ipcMsg.cpp:297", ' ' <repeats 24 times>, "(BASE_IPC - 0x00000000) MplThread(4)@mgmtplane(9)", ' ' <repeats 21 times>, "ERROR /usr/sw/loa"..., length=307, saddr=0x0, parse_flags=4) at logreader.c:210 #11 0x0805f9c8 in log_reader_iterate_buf (self=0x88bd2f8, saddr=0x0, flush=0, msg_count=0xbfa7c230) at logreader.c:295 #12 0x0805fe58 in log_reader_fetch_log (self=0x88bd2f8, fd=0x88c46b0) at logreader.c:400 #13 0x0805f541 in log_reader_fd_dispatch (source=0x88b50c8, callback=0, user_data=0x0) at logreader.c:153 #14 0x4dfb23ee in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 #15 0x4dfb53f6 in ?? () from /usr/lib/libglib-2.0.so.0 #16 0x4dfb58d8 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 #17 0x0804b73e in main_loop_run (cfg=0xbfa7c4d0) at main.c:167 #18 0x0804c0e4 in main (argc=3, argv=0xbfa7c574) at main.c:483 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 170] New: feature request: include only load *.conf files
by bugzilla＠bugzilla.balabit.com 07 May '12

07 May '12

https://bugzilla.balabit.com/show_bug.cgi?id=170 Summary: feature request: include only load *.conf files Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: enhancement Estimated Hours: 0.0 When an rpm package is used to place a configuration file in an include directory and the package is later updated, a file of the name *.conf.rpmnew or *.conf.rpmsave but the contents of this file will be a conflict with the *.conf file and syslog-ng will not start. Syslog-ng should only load the *.conf files so that it can work seamlessly with the rpm method. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

A question about filtering messages
by C. L. Martinez 06 May '12

06 May '12

Hi all, I need to filter different message types with some special words in it. Is this a correct filter to do this?? filter f_test { netmask("192.168.1.15") and ( message("FIRST_OPTION:") or message("SECOND_OPTION:") or message("THIRD_OPTION:")); }; Thanks.

2 1

(no subject)
by pedro castro 05 May '12

05 May '12

Learn How To Earn Money 0nline Now http://san-francisco-tips.com/perduol.php?gjgpageID=159 __________________ " De cow up n died on myhans." (c) annette albern Sat, 5 May 2012 8:55:20

1 0

[PATCH] Don't use deprecated glib symbols on newer glibs
by Gergely Nagy 04 May '12

04 May '12

A couple of symbols have been deprecated by glib in versions newer than the minimum version syslog-ng supports, and with 2.30 which deprecated g_atomic_int_exchange_and_add(), these warnings are way over the normal threshold, and turn build logs into not much more than noise. This patch solves this issue by checking the glib version at compile time, and deciding then which functions to use. This makes it possible to remain compatible with ancient glib, yet, still compile without deprecation warnings on systems with recent glibs. No functionality is lost or changed, just different interfaces used to accomplish the same thing. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/atomic.h | 4 ++++ lib/timeutils.c | 8 ++++++++ syslog-ng/syslog-ng-ctl.c | 12 ++++++++---- 3 files changed, 20 insertions(+), 4 deletions(-) diff --git a/lib/atomic.h b/lib/atomic.h index 05cfb88..32d421e 100644 --- a/lib/atomic.h +++ b/lib/atomic.h @@ -52,7 +52,11 @@ g_atomic_counter_get(GAtomicCounter *c) static inline gint g_atomic_counter_exchange_and_add(GAtomicCounter *c, gint val) { +#if GLIB_CHECK_VERSION(2, 30, 0) + return g_atomic_int_add(&c->counter, val); +#else return g_atomic_int_exchange_and_add(&c->counter, val); +#endif } static inline gint diff --git a/lib/timeutils.c b/lib/timeutils.c index 9e7822b..313d61c 100644 --- a/lib/timeutils.c +++ b/lib/timeutils.c @@ -844,7 +844,11 @@ zone_info_read(const gchar *zonename, ZoneInfo **zone, ZoneInfo **zone64) if (byte_read == -1) { msg_error("Failed to read the time zone file", evt_tag_str("filename", filename), NULL); +#if GLIB_CHECK_VERSION(2, 22, 0) + g_mapped_file_unref(file_map); +#else g_mapped_file_free(file_map); +#endif g_free(filename); return FALSE; } @@ -857,7 +861,11 @@ zone_info_read(const gchar *zonename, ZoneInfo **zone, ZoneInfo **zone64) *zone64 = zone_info_parser(&buff, TRUE, &version); } +#if GLIB_CHECK_VERSION(2, 22, 0) + g_mapped_file_unref(file_map); +#else g_mapped_file_free(file_map); +#endif g_free(filename); return TRUE; } diff --git a/syslog-ng/syslog-ng-ctl.c b/syslog-ng/syslog-ng-ctl.c index 25948b0..16348f1 100644 --- a/syslog-ng/syslog-ng-ctl.c +++ b/syslog-ng/syslog-ng-ctl.c @@ -131,7 +131,7 @@ slng_verbose(int argc, char *argv[], const gchar *mode) { gint ret = 0; GString *rsp = NULL; - gchar buff[256]; + gchar buff[256], *ubuff; if (!verbose_set) snprintf(buff, 255, "LOG %s\n", mode); @@ -139,10 +139,14 @@ slng_verbose(int argc, char *argv[], const gchar *mode) snprintf(buff, 255, "LOG %s %s\n", mode, strncasecmp(verbose_set, "on", 2) == 0 || verbose_set[0] == '1' ? "ON" : "OFF"); - g_strup(buff); + ubuff = g_ascii_strup(buff, -1); - if (!(slng_send_cmd(buff) && ((rsp = slng_read_response()) != NULL))) - return 1; + if (!(slng_send_cmd(ubuff) && ((rsp = slng_read_response()) != NULL))) + { + g_free(ubuff); + return 1; + } + g_free(ubuff); if (!verbose_set) printf("%s\n", rsp->str); -- 1.7.10

1 0

[ANN]: balabit.logstore - Open Source LogStore reader API
by Gergely Nagy 04 May '12

04 May '12

--------------------------------------------------------------- PACKAGE : com.balabit/logstore VERSION : 0.0.1 SUMMARY : First alpha release DATE : 2012 April 24 HOMEPAGE : http://algernon.github.com/balabit.logstore --------------------------------------------------------------- DESCRIPTION: The balabit.logstore project is a library written in Clojure, that tries to provide a convenient API to read syslog-ng PE LogStore files. The reason behind the implementation is to have an independent, open source reader for the LogStore file format, so that one is not tied to syslog-ng to read one's logs stored in this format. An open implementation makes it possible to read these logs on systems where syslog-ng is not installed, or where the lgstool program is not available. LogStore itself is a storage format for log messages, supporting encryption, compression, secure timestamps, serialized messages (think name-value pairs as opposed to just a plain, formatted string) and a lot more. This format is being opened, and this library is meant to provide an implenetation for a LogStore reader, that will eventually support reading all kinds of LogStore files. EXAMPLE: (ns example (:require [balabit.logstore :as logstore])) (defn lgstool-cat ([] (lgstool-cat "resources/logstores/short.compressed.store")) ([filename] (logstore/with-file filename (let [indexed-records (zipmap (range (count (logstore/records))) (logstore/records)) chunk-records (keys (filter #(= :chunk (:type (val %))) indexed-records)) print-msgs (fn [index] (dorun (map println (:messages (logstore/nth-record index)))))] (dorun (map print-msgs chunk-records)))))) (lgstool-cat "/path/to/logstore/messages.store") More examples can be found bundled with the sources, and as part of the documentation: http://algernon.github.com/balabit.logstore/#balabit.logstore.examples FEATURES: The library - being in very early alpha stage - can already read some LogStore files, and transparently decompress them, collect meta-data about both the logstore, and the information stored within. It comes with an extensive test suite and a lot of examples. FUTURE DIRECTION: We are working on making the library able to deserialize name-value pairs, to make it support decrypting logstores, and paralell to that, we're working on a Java API, so that it can be used from a more familiar language, and still provide a convenient, easy to use API. In the future, we also plan to publish ready to use JAR files that one can easily use from any Java or Clojure project. DOWNLOADS: The source is available from the git repository at github: git://github.com/algernon/balabit.logstore.git Documentation and more information about the library is available on its homepage at http://algernon.github.com/balabit.logstore/. People interested in the development of the library can also look at the developer docs at: http://algernon.github.com/balabit.logstore/developer-api.html -- |8]

2 2

[PATCH] afsocket: Support a smooth upgrade path to recent systemd
by Gergely Nagy 04 May '12

04 May '12

When the journal was introduced in systemd, it started to pass /run/systemd/journal/syslog to syslogds (it was passing down /dev/log before). Since we verify that the FD to open matches the expected filename, we didn't pick up the fd due to the filename mismatch. This can be fixed on the system() source level, where it can decide which to generate into the config. However, for setups that do not use system() for one reason or the other, this does not work. For these systems that do not use the system() source, the best solution is to try and replace /dev/log with /run/systemd/journal/syslog at run-time. This patch does just that, and ties it to config versions 3.4 or earlier. This means that post-3.4, the automatic replacement will stop working, and users will get the generic failed to acquire socket error message instead. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afunix.c | 56 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 13 deletions(-) diff --git a/modules/afsocket/afunix.c b/modules/afsocket/afunix.c index 8145f1a..a5644ac 100644 --- a/modules/afsocket/afunix.c +++ b/modules/afsocket/afunix.c @@ -72,7 +72,8 @@ afunix_sd_set_perm(LogDriver *s, gint perm) #if ENABLE_SYSTEMD static gboolean -afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) +afunix_sd_acquire_named_socket(AFSocketSourceDriver *s, gint *result_fd, + const gchar *filename) { AFUnixSourceDriver *self = (AFUnixSourceDriver *) s; gint fd, fds; @@ -101,12 +102,12 @@ afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) for (fd = SD_LISTEN_FDS_START; fd < SD_LISTEN_FDS_START + fds; fd++) { /* check if any type is available */ - if (sd_is_socket_unix(fd, 0, -1, self->filename, 0)) + if (sd_is_socket_unix(fd, 0, -1, filename, 0)) { int type = (self->super.flags & AFSOCKET_STREAM) ? SOCK_STREAM : SOCK_DGRAM; /* check if it matches our idea of the socket type */ - if (sd_is_socket_unix(fd, type, -1, self->filename, 0)) + if (sd_is_socket_unix(fd, type, -1, filename, 0)) { *result_fd = fd; break; @@ -114,7 +115,7 @@ afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) else { msg_error("The systemd supplied UNIX domain socket is of a different type, check the configured driver and the matching systemd unit file", - evt_tag_str("filename", self->filename), + evt_tag_str("filename", filename), evt_tag_int("systemd-sock-fd", fd), evt_tag_str("expecting", type == SOCK_STREAM ? "unix-stream()" : "unix-dgram()"), NULL); @@ -130,31 +131,60 @@ afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) */ msg_debug("Ignoring systemd supplied fd as it is not a UNIX domain socket", - evt_tag_str("filename", self->filename), + evt_tag_str("filename", filename), evt_tag_int("systemd-sock-fd", fd), NULL); } } } - else - return TRUE; if (*result_fd != -1) { g_fd_set_nonblock(*result_fd, TRUE); g_fd_set_cloexec(*result_fd, TRUE); msg_verbose("Acquired systemd socket", - evt_tag_str("filename", self->filename), + evt_tag_str("filename", filename), evt_tag_int("systemd-sock-fd", *result_fd), NULL); + return TRUE; } - else + return TRUE; +} + +static gboolean +afunix_sd_acquire_socket(AFSocketSourceDriver *s, gint *result_fd) +{ + AFUnixSourceDriver *self = (AFUnixSourceDriver *) s; + gboolean fd_ok; + GlobalConfig *cfg = log_pipe_get_config(&s->super.super.super); + + fd_ok = afunix_sd_acquire_named_socket(s, result_fd, self->filename); + + if (fd_ok && (*result_fd == -1) && (strcmp(self->filename, "/dev/log") == 0)) { - msg_debug("Failed to acquire systemd socket, trying to open ourselves", - evt_tag_str("filename", self->filename), - NULL); + fd_ok = afunix_sd_acquire_named_socket(s, result_fd, "/run/systemd/journal/syslog"); + + if (fd_ok && *result_fd > -1) + { + if (cfg->version <= 0x0304) + { + msg_warning("WARNING: systemd detected while using /dev/log; migrating automatically to /run/systemd/journal/syslog. Please update your configuration to use the system() source.", + evt_tag_str("id", self->super.super.super.id), + NULL); + + g_free(self->filename); + self->filename = g_strdup("/run/systemd/journal/syslog"); + return TRUE; + } + } } - return TRUE; + + if (!fd_ok) + msg_debug("Failed to acquire systemd socket, trying to open ourselves", + evt_tag_str("filename", self->filename), + NULL); + + return fd_ok; } #else -- 1.7.10

1 0

[PATCH 0/2] systemd enhancements
by Gergely Nagy 04 May '12

04 May '12

The two patches that follow improve syslog-ng's support for systemd-enabled platforms. The first patch simply teaches the system() source to prefer /run/systemd/journal/syslog over /dev/log if running under systemd. That alone makes configs that use system() work under both sysvinit and systemd without any change at all to the config. However, for configs that use /dev/log hard-coded, the upgrade path isn't that smooth. In that case, we still check whether running under systemd, and if so, but we couldn't accept any of the offered fds, we check again with /run/systemd/journal/syslog as the filename, and replace the hard-coded one if the other filename worked. The second patch implements the above - I'm not entirely sold on the idea, and haven't explored the possible side-effects yet, but I believe it's reasonably safe for the vast majority of cases. For now, the second patch is mostly meant as a proposal, and will need a review before it can be considered for a merge.

2 5

[PATCH] afsocket: Fix unix-dgram initialisation
by Gergely Nagy 04 May '12

04 May '12

When initializing an unix-dgram socket, set self->fd to the acquired socket FD, so that fds inherited from systemd actually get used and polled on. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/modules/afsocket/afsocket.c b/modules/afsocket/afsocket.c index 5b192f4..c1bee5c 100644 --- a/modules/afsocket/afsocket.c +++ b/modules/afsocket/afsocket.c @@ -679,7 +679,7 @@ afsocket_sd_init(LogPipe *s) if (sock == -1 && !afsocket_open_socket(self->bind_addr, !!(self->flags & AFSOCKET_STREAM), &sock)) return self->super.super.optional; } - self->fd = -1; + self->fd = sock; if (!self->setup_socket(self, sock)) { -- 1.7.9.1

2 3

[PATCH] afsmtp: Fix afsmtp linking when configured with --with-libesmtp
by Gergely Nagy 04 May '12

04 May '12

In case configure received a --with-libesmtp=/path option, we forgot to add -lesmtp to LIBESMTP_LIBS, which resulted in afsmtp getting compiled, but not linked to libesmtp, which is obviously not desirable. This patch fixes that, and adds the -lesmtp. Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure.in b/configure.in index 2d56bbd..03cc76c 100644 --- a/configure.in +++ b/configure.in @@ -815,7 +815,7 @@ if test "x$enable_smtp" != "xno" && test "x$with_libesmtp" != "no"; then CPPFLAGS="$CPPFLAGS -I$with_libesmtp/include" LDFLAGS="$LDFLAGS -L$with_libesmtp/lib" AC_CHECK_HEADER(libesmtp.h, [LIBESMTP_CFLAGS="-I$with_libesmtp/include" - LIBESMTP_LIBS="-L$with_libesmtp/lib"], [libesmtp=no]) + LIBESMTP_LIBS="-L$with_libesmtp/lib -lesmtp"], [libesmtp=no]) CPPFLAGS="$CPPFLAGS_SAVE" LDFLAGS="$LDFLAGS_SAVE" else -- 1.7.10

1 0

[PATCH] lib/gprocess.c: Ignore SIGHUP inside the supervisor.
by Gergely Nagy 04 May '12

04 May '12

Reloading syslog-ng's config is often done via a killall -HUP syslog-ng command, which also sends a signal to the supervisor. However, the supervisor did not handle said signal, thus it exited as is the default for it. To work around this issue, lets ignore SIGHUP inside the supervisor. Reported-by: Patrick Hemmer Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/gprocess.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/gprocess.c b/lib/gprocess.c index aaefbab..d68521d 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -1055,10 +1055,15 @@ g_process_perform_supervise(void) pid_t pid; gboolean first = TRUE, exited = FALSE; gchar proc_title[PROC_TITLE_SPACE]; + struct sigaction sa; g_snprintf(proc_title, PROC_TITLE_SPACE, "supervising %s", process_opts.name); g_process_setproctitle(proc_title); + memset(&sa, 0, sizeof(sa)); + sa.sa_handler = SIG_IGN; + sigaction(SIGHUP, &sa, NULL); + while (1) { if (pipe(init_result_pipe) != 0) -- 1.7.10

1 0

syslog-ng Insider - May 2012
by Peter Czanik 03 May '12

03 May '12

Dear syslog-ng users, This is the 13th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS Have you tried to add custom information to log messages, fix mis-formatted logs or anonymize logs? --------------------------------- The next long-time-supported release of SSB version 3 LTS is about to be released. This release includes a switch to 64-bit architecture, a huge performance improvement in the indexing/searching feature for a large number of message and search patterns and a couple of new features, too. The following post plans to introduce those new features to you. The updated User’s Manual will contain a detailed description of them — this post is written more to serve as teaser and to highlight some of the use cases we had in mind when we’d planned the features, and, of course, to ask for your feedback about them. http://gyp.blogs.balabit.com/2012/03/new-features-in-ssb-3-lts-rewriting-pa… Admin guide: http://www.balabit.com/sites/default/files/documents/ssb-3.0-guides/ssb-v3.… syslog-ng participates in GSoC ------------------------------ This year syslog-ng participates in GSoC under the umbrella of openSUSE. We have one student accepted, who will work on syslog-ng’s mongodb destination. The story of BalaBit's GSoC participation: http://czanik.blogs.balabit.com/2012/04/syslog-ng-on-gsoc-thanks-opensuse/ What the student will do: http://algernon.blogs.balabit.com/2012/05/in-the-land-of-open-source-the-fr… MongoDB howto ------------- One of the major reasons to update to 3.3 other than threading is MongoDB. It allows great flexibility when using patterns and storing parsed data from logs. The syslog-ng documentation covers most information necessary to use MongoDB, but this HowTo compiles all these into a single document and extends it with features from the upcoming syslog-ng 3.4 version. http://asylum.madhouse-project.org/blog/2012/04/26/mongodb-howto/ OTHER SHORT NEWS * There is now an open source LogStore reader work in progress: https://github.com/algernon/balabit.logstore * syslog-ng 3.4 alpha packages are now available in the FreeBSD ports and also for openSUSE: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… * syslog-ng and ELSA were presented at LOADays: http://czanik.blogs.balabit.com/2012/04/syslog-ng-at-linux-open-administrat… NEW RELEASES: * syslog-ng OSE 3.3.5: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-May/000142.html * syslog-ng OSE 3.4 alpha2: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-May/000143.html * libumberlog 0.2.1: http://algernon.github.com/libumberlog/ ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

smtp on FreeBSD
by Peter Czanik 02 May '12

02 May '12

Hello, I tried to compile syslog-ng 3.4 alpha git on FreeBSD 8.1 with smtp support, but it does not start: fb81# /usr/local/etc/rc.d/syslog-ng start Starting syslog_ng. Error opening plugin module; module='afsmtp', error='/usr/local/lib/syslog-ng/libafsmtp.so: Undefined symbol "_smtp_message_str_cb"' smtp support can only be disabled by removing libesmtp from the system... There is also a segfault now for each incoming log (nothing appears in log files), so I'm checking if it's related to any of the enabled modules. At first glance it seems to be something FreeBSD specific, I have all modules enabled on openSUSE, and logging seems to work (at least the basics, I checked). Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

4 18

3.4 packages
by Peter Czanik 02 May '12

02 May '12

Hello, I created packages from current 3.4 git for openSUSE and helped to update the FreeBSD syslog-ng-devel port. They had some light testing and seem to work. Please refer to http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… for download information! The openSUSE packages have smtp enabled (in a separate sub package, untested), on FreeBSD the module can't be loaded, so marked as broken. Please let me know, if you run into any problems! Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

[RFC]: LogThreadedDestDriver - making threaded destinations better
by Gergely Nagy 01 May '12

01 May '12

Hi! There have been a number of fixes going into afsql or afmongodb (or both) recently, fixing bugs that affected both, for the simple reason that when I made afmongodb, I copied parts of the architecture from afsql. Now there is the SMTP destination aswell, suffering from the same issues, and a Redis destination is in the works aswell. Therefore, it is about time to rethink how these drivers that all look very similar under the hood could be rebased onto a common foundation, something that abstracts away all the common stuff, such as handling the worker thread, and the queue method - so all the affected drivers need to do, is implement their own tasks, and skip all the copy-paste-ish boilerplate they have now. To this end, we had a short brainstorming with Bazsi, and I'll try to summarize what I remember (with a few later ideas added). The goal is to create a new class, based on LogDestDriver, for all those destination drivers that use a worker thread to do their blocking operations. This new class, which for the time being shall be named LogThreadedDestDriver shall implement the following things: * It must handle the setup, starting and stopping of the worker thread, with a module-supplied worker function, and that function shall be assisted by class functions to do the dirty work. * The queue handling must be safe and free of races (which pretty much means we have to find a way to wrap log_queue_pop_head into a synchronous call) * The class must be prepared to work not only with single messages, but in bulk aswell (think SQL's transactions, or the upcoming MongoDB bulk insert) * The class must implement its own queue method, downstream modules should not need to care about that. * We must be able to wake up the worker thread, even if there is nothing to handle: this is needed for a clean shutdown. And that's pretty much it. The class would have callbacks for the following things: * A pair of callbacks to call during the driver's _init() and _deinit() functions. These can either be implemented as callbacks, and the driver will then use it's own init/deinit and set it in its parent, and call the module-set callbacks from those. Or, the class can provide _init/_deinit methods, and leave it up to the module to implement its own methods, and call the class methods itself. The former makes drivers that do not require extra init/deinit simpler, the latter has less callbacks and looks neater. When implementing, I'll probably go with the latter. * An 'do-the-work' callback, which is called whenever work needs to be performed, and an error callback to fire whenever the do-the-work failed. I'm not entirely sure how best to handle the transaction case here, but that's something that needs to be done around here. One possibility is to allow two kinds of do-the-work callbacks: one for the simple case where no transactions are done (afsmtp and current afmongodb) and one that supports transactions/bulk inserts (afsql and future afmongodb and redis). Off the top of my head, this should be enough to move the boring parts of the current threaded destination drivers into a common class. Once the transaction case is figured out, all of our drivers can be reimplemented on top of this class, reducing copied and slightly altered code by a HUGE margin, and finally making it so that we have a single place to fix possible bugs in, instead of 3 slightly different copies of the same thing. As a side effect, time_reopen and stored/dropped counter handling would get moved from individual drivers into this base class, along with flush_timeout and flush_lines (if those still exist in 3.4) and a couple of other things. I plan to have a Proof of Concept patch ready by friday night, we'll see how it goes! (And shortly after, a redis destination will follow) -- |8]

2 2

Multi-core use in threaded mode
by Martin Holste 01 May '12

01 May '12

I'm seeing that on a very busy syslog-ng 3.3.4 box, CPU never rises above 103% and log loss occurs. Does threaded mode not allow separate threads to utilize multiple cores? I need to scale above a single CPU.

6 8

[PATCH (3.4) 0/3] afsql code cleanups
by Gergely Nagy 01 May '12

01 May '12

The patches following this mail are my attempts at cleaning up the afsql driver here and there, mostly afsql_dd_insert_db(), in preparation for moving towards LogThreadedDestDriver (which will be posted separately a bit later). The migration is possible without these patches, but these are worth applying even without the LogThreadedDestDriver patches, as these present no real architectural change, just a little refactoring. I think the code could be cleaned up further, but for the time being, I only did what I thought necessary to prepare for migration to LogThreadedDestDriver.

2 5

[PATCH (3.4)] afsmtp: Add SCS_SMTP to lib/stats.h
by Gergely Nagy 01 May '12

01 May '12

Add SCS_SMTP to lib/stats.h, so we get proper smtp stats too. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/stats.h | 1 + modules/afsmtp/afsmtp.c | 4 ---- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/stats.h b/lib/stats.h index 99936da..9174408 100644 --- a/lib/stats.h +++ b/lib/stats.h @@ -70,6 +70,7 @@ enum SCS_SEVERITY = 24, SCS_FACILITY = 25, SCS_SENDER = 26, + SCS_SMTP = 27, SCS_MAX, SCS_SOURCE_MASK = 0xff }; diff --git a/modules/afsmtp/afsmtp.c b/modules/afsmtp/afsmtp.c index bd8adfe..841100d 100644 --- a/modules/afsmtp/afsmtp.c +++ b/modules/afsmtp/afsmtp.c @@ -33,10 +33,6 @@ #include <libesmtp.h> -#ifndef SCS_SMTP -#define SCS_SMTP 0 -#endif - typedef struct { gchar *name; -- 1.7.10

2 1

syslog-ng 3.3.5 and openssl
by Alexander Komyagin 01 May '12

01 May '12

Hi! I noticed that when I'm compiling syslog-ng 3.3.5 with openssl enabled (at least in my environment), libsyslog-ng-crypto.so somehow is not linked with libssl (checked with ldd), thus resulting with unresolved symbols error on syslog-ng startup for afsocket module. Attached patch fixes the issue. --- syslog-ng-3.3.5.orig/lib/Makefile.am 2012-03-31 22:23:43.000000000 +0400 +++ syslog-ng-3.3.5/lib/Makefile.am 2012-04-26 17:32:58.000000000 +0400 @@ -12,7 +12,7 @@ libsyslog_ng_la_LDFLAGS = -no-undefined -release @VERSION@ module_LTLIBRARIES = libsyslog-ng-crypto.la -libsyslog_ng_crypto_la_LIBADD = @CORE_DEPS_LIBS@ libsyslog-ng.la +libsyslog_ng_crypto_la_LIBADD = @CORE_DEPS_LIBS@ @OPENSSL_LIBS@ libsyslog-ng.la libsyslog_ng_crypto_la_LDFLAGS = -no-undefined -avoid-version # this is intentionally formatted so conflicts are less likely to arise. one name in every line. -- Best wishes, Alexander Komyagin

2 1

[PATCH] lib/gprocess.c: Ignore SIGHUP inside the supervisor.
by Gergely Nagy 01 May '12

01 May '12

Reloading syslog-ng's config is often done via a killall -HUP syslog-ng command, which also sends a signal to the supervisor. However, the supervisor did not handle said signal, thus it exited as is the default for it. To work around this issue, lets ignore SIGHUP inside the supervisor. Reported-by: Patrick Hemmer Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/gprocess.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/lib/gprocess.c b/lib/gprocess.c index aaefbab..9246d88 100644 --- a/lib/gprocess.c +++ b/lib/gprocess.c @@ -1110,7 +1110,9 @@ g_process_perform_supervise(void) g_process_message("Initialization failed but the daemon did not exit, even when forced to, trying to recover; pid='%d'", pid); continue; } - + + sigignore(SIGHUP); + if (process_opts.check_fn && (process_opts.check_period >= 0)) { gint i = 1; -- 1.7.9.1

2 1