- syslog-ng - lists.balabit.hu

6 identical servers, only 4 sending logs
by Brian Johnson 11 Sep '12

11 Sep '12

Hello all, I hope what I'm asking hasn't been covered previously, I tried some searches with no luck. If I'm duplicating something else, I apologize. My problem is, I have 6 DHCP servers with identical syslog-ng.conf and syslog.conf files on them. The set up is as so: dhcp-a-01 and dhcp-b-01 are a DHCP failover pair dhcp-a-02 and dhcp-b-02 are a DHCP failover pair dhcp-a-03 and dhcp-b-03 are a DHCP failover pair The 'dhcp-a' servers are in the A data center. 'dhcp-b' servers are in the B data center. Again, the syslog-ng.conf files on all of them are identical, checked with sha1sum. It is confirmed that all of them are using syslog-ng for logging. I have them all set to log to the same remote logging server. Logs from dhcp-[a,b]-01 and dhcp-[a,b]-03 are making it to the remote server with no issues. I can see it on the remote server and I can see it when doing a 'tcpdump port 514' on the servers themselves. For some reason, I'm not seeing any logs from dhcp-[a,b]-02 on the remote server and when I do 'tcpdump port 514' for a length of time, I get: dhcp-b-02:~# tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel when the other servers, done at the same time, show packets captured. I just did a "tail -f /var/log/syslog > /tmp/test" all of the servers between 11:43:26 and 11:45:38 (2m12s). In that time: dhcp-[a,b]-01 had roughly 2700 lines dhcp-[a-b]-02 had roughly 11000 lines dhcp-[a-b]-03 had roughly 1100 lines So to me it seems like there's some sort of throttling on the data that's able to be sent. There's ~5x more traffic on pair 2 than 1 (which will be rebalanced, just trying to get this working first) so that would make sense. The only thing that I could find that looks like it would help is the log_fifo_size option, but that doesn't seem to help -- I've made several adjustments to it, but it doesn't seem to make any difference. Can someone please let me know what I'm missing? Thanks! Brian

2 2

[RFC]: Pattern matching & corellation ideas
by Gergely Nagy 11 Sep '12

11 Sep '12

In the past few months, I talked a lot about patterndb and related things with colleagues - during coffee break, over a beer, etc -, and last night, I got as far as drafting a proof of concept tool that realizes some of the ideas we've had. Some of these might have been discussed on this list too, in the past, I really can't remember all the influences I'm afraid. Anyway! I like the concept of patterndb, but I absolutely hate XML. It's not a natural format for describing how to match patterns, and what to do with them. Not for me, anyway. I'd like something that is much closer to how I normally think, something that feels more like a programming language, a domain specific one, engineered for this task only, but still somewhat familiar. I also want it to be fast. For a long time, I also wanted to play with developing both a Domain Specific Language (DSL for short) and a compiler, but never had the opportunity. Pattern matching is one now! How about a small language, that we'd compile down to C, automatically add some boilerplate, and we'd get a syslog-ng parser plugin in return? That would mean the language is fairly easy to extend, it produces native code, which will hopefully run as fast - if not faster - than patterndb, and we skip the entire XML pile too! To demonstrate, this is what I've been thinking of: ,---- | (cond :message | (match "foo (bar) ([:number:])") | (do | (set! :bar "$1") | (set! :stuff "$2") | (conj! :tags "test stuff"))) | | (deftest message-match | "this is a foo bar 1234 message!" | | (== :bar "bar") | (== :stuff "1234") | (contains? :tags "test stuff")) `---- This would compile down to roughly the following C code (with the test excluded, for now): ,---- | gboolean | m_something(LogMsg *message, GString *subject) | { | GString *m1; /* ([:number:]) */ | | /* The subject must be at least as long as the static strings in the | pattern, if it's shorter, we don't match */ | if (subject->len < 8) | return FALSE; | | /* If we don't find part of the pattern, bail out. */ | if (strncmp(subject->str, "foo bar ") != 0) | return FALSE; | | if (!find_number(subject->str + 8, &m1)) | return FALSE; | | /* The whole stuff matched, yay! Lets fill in the fields. */ | | /* "bar" is a static string, fill it in as-is, no need to extract it | from the subject. */ | log_msg_set_value(msg, "bar", "bar", 3); | | /* :stuff is a number, so that needs to come from the | subject. Thankfully find_number() already did the extraction for | us. */ | log_msg_set_value(msg, "stuff", m1->str, m1->len); | | log_msg_set_tag_by_name(msg, "test stuff"); | | return TRUE; | } `---- I believe this is pretty efficient, the code generator can comment the the generated source nicely too. If we add named capture-groups, then the variables used can have meaningful names too! For example: (match "this (?<object>[:qstring:]) is good") In this case, the variable would be called m_object. However, the pattern-string is a bit awkward, when the rest is lispy, it also complicates the generator, so I was thinking of turning the pattern into a lispy syntax too: ,---- | (match "foo " (capture "bar") " " (capture :number)) `---- Or, with named capture groups added: ,---- | (match "this " (capture :qstring :as "object") " is good") `---- Of course, the action to take on a match can also contain another cond+match pair, so it can be nested as deep as one wishes to, the compiler will compile each cond into a separate function, and just call the appropriate one. Or perhaps inline them - that's an implementation detail, and doesn't really matter. The big advantage I see, is a DSL that is much closer to how I think, one that has the potential to produce a compact parser, one that is also easy to debug with conventional tools (gdb ;) because it compiles down to C. There is less run-time overhead too. Also, if implemented correctly, the generator would have the parser and the code generator entirely independent, so adding a different syntax would be as easy as writing a parser that produces the same abstract syntax tree the generator works with. This way, for those who're more familiar with C-like languages, the above matcher could be rewritten like this: ,---- | switch ($message) | { | case match("foo ", capture("bar"), " ", capture(:number:)) | { | set("bar", "$1"); | set("stuff", "$2"); | append($tags, "test stuff"); | } | } `---- And it would compile down to the exact same C code, accompanied by an appropriate autotools-based build system, so all you'd have to do in the end is to write the matcher, and issue the following commands: ,---- | $ matcher-generate test-patterns.pm | $ cd test-patterns | $ autoreconf -i && ./configure && make && make install `---- And finally, modify your syslog-ng.conf: ,---- | @module test-patterns | parser p_test { parser(test-patterns); }; `---- It does have downsides, though, namely that you need to regenerate & recompile the module and restart syslog-ng each time you modify the source, which is less convenient than just restarting syslog-ng itself. One also needs to learn a 'new' language to write pattern matchers in (but one has to learn patterndb too, anyway, so this isn't that big a disadvantage, especially since a more language-like thing is, in my opinion, easier to learn :). However, I believe that the advantages are worth it. For me, they certainly do, so I already started to hash out a proof of concept. So far, my PoC code can generate C functions, and supports a small subset of the DSL explained below. Do note that all this does not include corellation, because I believe that corellation should be separate from parsing, and a similar technique could be used to write advanced corellation setups - I will go into detail once I have a working proof of concept compiler for the parser. As a start, the DSL would support the following constructs: Top-level constructs: --------------------- * (cond :field condition action ...) Where :field can be any field, condition is a single condition function (see below) and action is a single action too (see even further below). Any number of condition-action paris can be specified, the first one matching will win, and the rest won't be tried. These two must always be paired together. * (deftest test-name "source string" test-conditions) Right now, lets ignore this. But in the long run, I want to be able to write down reasonably complex tests too. Not entirely sure yet what I need it to do though. Condition functions: -------------------- * (match pattern-spec) Matches a pattern-spec, simple as that. See below for the definition of the pattern-spec! * (not-match pattern-spec) The opposite of (match): action is triggered if the pattern does not match. * (exists?) Triggers the action if the field in the condition exists. * (not-exists?) Opposite of (exists?). * :default or (always) Always triggers, so one can do catch-all actions. Action functions: ----------------- * (set! :field value) * (clear! :field) These two should speak for themselves, I believe. * (conj! :field values) Conjoin (append) values to the specified field. Some fields (:tags) that need to be, will be treated specially, otherwise it just appends a separator (",") and the values. * (do actions...) Does all the specified actions. Pattern spec: ------------- The pattern can be built up from the sequence of the following things, in any order: * A plain string * (capture pattern-spec [:as name]) This just marks the pattern-spec as something to capture. While the implementation may produce code that captures things anyway, the only guarantee that something will be available for the actions, is to wrap it in (capture). The pattern-spec can be anything, it can contain nested captures. If :as name is specified, the capture will be named, and actions can refer to the captured thing by name. Otherwise, they need to refer it by a number. Each capture - named or anonymous - has a different number, starting from one, increasing with each occurrence of (capture). * Any of the following special keywords: * :number * :string * :qstring * :ipv4-address * :ipv6-address * :mac-address Future ideas ------------ Later, once the basics are ready and work, it would make sense to introduce a way to share common blocks of code: functions and perhaps variables. Conclusion ========== XML sucks, DSL rocks. Feedback appreciated, be that on the syntax, or the initially proposed features/functions/etc, or anything else. -- |8], who perhaps spent a little too much time near LISP-y stuff

4 8

Syslog-NG/Log Analyzer
by James McDonald 11 Sep '12

11 Sep '12

I finally got my syslog-ng servers to pass logs to the MySQL db, but was having some problems getting LogAnalzyer to access the db. I re-installed Log Analyzer but now I am getting an access denied error: Connection to the database server failed. Access denied for user 'syslog'@'FQDN_for_mysql_server.com' (using password: YES). Here is the output of the grants for the 'syslog' user: CODE: <http://kb.monitorware.com/> SELECT ALL mysql> show grants for 'syslog'@'FQDN_for_mysql_server.com'; +--------------------------------------------------------------------------- ---------+ | Grants for syslog(a)FQDN_for_mysql_server.com | +--------------------------------------------------------------------------- ---------+ | GRANT USAGE ON *.* TO 'syslog'@'FQDN_for_mysql_server.com' | | GRANT ALL PRIVILEGES ON `Syslog`.* TO 'syslog'@'FQDN_for_mysql_server.com' | +--------------------------------------------------------------------------- ---------+ 2 rows in set (0.00 sec) And here is what my config.php looks like: $CFG['DefaultSourceID'] = 'Source1'; $CFG['Sources']['Source1']['ID'] = 'Source1'; $CFG['Sources']['Source1']['Name'] = 'Syslog'; $CFG['Sources']['Source1']['ViewID'] = 'SYSLOG'; $CFG['Sources']['Source1']['SourceType'] = SOURCE_DB; $CFG['Sources']['Source1']['DBTableType'] = 'syslogng'; $CFG['Sources']['Source1']['DBType'] = DB_MYSQL; $CFG['Sources']['Source1']['DBServer'] = 'ustc-mysql'; $CFG['Sources']['Source1']['DBName'] = 'Syslog'; $CFG['Sources']['Source1']['DBUser'] = 'syslog'; $CFG['Sources']['Source1']['DBPassword'] = 'pTRl819ELI0viRmXaEUO'; $CFG['Sources']['Source1']['DBTableName'] = 'SystemEvents'; $CFG['Sources']['Source1']['DBEnableRowCounting'] = false; Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

2 3

[Bug 161] New: Syslog-ng stops processing after reload
by bugzilla＠bugzilla.balabit.com 10 Sep '12

10 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=161 Summary: Syslog-ng stops processing after reload Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 syslog-ng version: 3.3.4 OS: RHEL 6.2 Whenever I issue a reload to our syslog-ng instance syslog-ng stops processing messages and sits there doing nothing. The issue only seems to occur when the server is processing a high log volume. This server is part of a 2-node active/passive HA group. So we have 2 servers with the exact same configuration. Whenever I issue the reload on the server that is not currently active, it works fine. I can issue a failover and the server that was able to reload fine now hangs. The active server in this HA group processes around 35,000 messages per second. I havent posted the config because it is fairly large, however I can if desired. The most notable thing in it is that threading is enabled. Below is the backtrace of syslog-ng stuck in this state. I took a gcore, so I can gather any additional information that is needed. Also, if I try to do a full backtrace ('where full') in gdb on any thread other than thread 5, gdb segfaults. Not sure if that has any relevance or not. (gdb) info threads 5 Thread 0x7fb9c654ab00 (LWP 11040) 0x000000353cedd967 in writev () from /lib64/libc.so.6 4 Thread 0x7fb9c25fb700 (LWP 11049) 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 3 Thread 0x7fb9c2ffc700 (LWP 11048) 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 2 Thread 0x7fb9c39fd700 (LWP 11046) 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 * 1 Thread 0x7fb9c43fe700 (LWP 11045) 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 (gdb) where #0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c43fb9a0, msec=10000) at iv_method_epoll.c:73 #2 0x00007fb9c65cf1bb in iv_main () at iv_main.c:265 #3 0x00007fb9c65d3c36 in iv_work_thread (_thr=0x14fc610) at iv_work.c:196 #4 0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc6e0) at iv_thread.c:100 #5 0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000353cee570d in clone () from /lib64/libc.so.6 (gdb) thread 2 [Switching to thread 2 (Thread 0x7fb9c39fd700 (LWP 11046))]#0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 (gdb) where #0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c39fa9a0, msec=10000) at iv_method_epoll.c:73 #2 0x00007fb9c65cf1bb in iv_main () at iv_main.c:265 #3 0x00007fb9c65d3c36 in iv_work_thread (_thr=0x14fc8e0) at iv_work.c:196 #4 0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc9b0) at iv_thread.c:100 #5 0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000353cee570d in clone () from /lib64/libc.so.6 (gdb) thread 3 [Switching to thread 3 (Thread 0x7fb9c2ffc700 (LWP 11048))]#0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 (gdb) where #0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c2ff99a0, msec=10000) at iv_method_epoll.c:73 #2 0x00007fb9c65cf1bb in iv_main () at iv_main.c:265 #3 0x00007fb9c65d3c36 in iv_work_thread (_thr=0x15635c0) at iv_work.c:196 #4 0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc380) at iv_thread.c:100 #5 0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000353cee570d in clone () from /lib64/libc.so.6 (gdb) thread 4 [Switching to thread 4 (Thread 0x7fb9c25fb700 (LWP 11049))]#0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 (gdb) where #0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c25f89a0, msec=10000) at iv_method_epoll.c:73 #2 0x00007fb9c65cf1bb in iv_main () at iv_main.c:265 #3 0x00007fb9c65d3c36 in iv_work_thread (_thr=0x1566d20) at iv_work.c:196 #4 0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x1566a70) at iv_thread.c:100 #5 0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000353cee570d in clone () from /lib64/libc.so.6 (gdb) thread 5 [Switching to thread 5 (Thread 0x7fb9c654ab00 (LWP 11040))]#0 0x000000353cedd967 in writev () from /lib64/libc.so.6 (gdb) where #0 0x000000353cedd967 in writev () from /lib64/libc.so.6 #1 0x00007fb9c65820f8 in log_proto_file_writer_flush (s=0x171b5a0) at logproto.c:270 #2 0x00007fb9c658c403 in log_proto_flush (s=0x171b5a0) at logproto.h:86 #3 0x00007fb9c658ec3e in log_writer_flush (self=0x171adb0, flush_mode=LW_FLUSH_QUEUE) at logwriter.c:1032 #4 0x00007fb9c658f006 in log_writer_deinit (s=0x171adb0) at logwriter.c:1106 #5 0x00007fb9c63399d6 in log_pipe_deinit (s=0x171adb0) at ../../lib/logpipe.h:254 #6 0x00007fb9c633b3cd in affile_dw_deinit (s=0x171ac20) at affile.c:627 #7 0x00007fb9c63399d6 in log_pipe_deinit (s=0x171ac20) at ../../lib/logpipe.h:254 #8 0x00007fb9c633c211 in affile_dd_deinit_writer (key=0x171ad60, value=0x171ac20, user_data=0x0) at affile.c:974 #9 0x000000353e22a7a3 in g_hash_table_foreach () from /lib64/libglib-2.0.so.0 #10 0x00007fb9c633c349 in affile_dd_deinit (s=0x1539990) at affile.c:996 #11 0x00007fb9c6575d86 in log_pipe_deinit (s=0x1539990) at logpipe.h:254 #12 0x00007fb9c65761b4 in log_dest_group_deinit (s=0x1539ef0) at dgroup.c:76 #13 0x00007fb9c656f54c in log_pipe_deinit (s=0x1539ef0) at logpipe.h:254 #14 0x00007fb9c657078c in log_center_deinit (self=0x14f35c0) at center.c:576 #15 0x00007fb9c6571296 in cfg_deinit (cfg=0x14f1ca0) at cfg.c:261 #16 0x00007fb9c6590cac in main_loop_reload_config_apply () at mainloop.c:494 #17 0x00007fb9c6590aab in main_loop_io_worker_job_complete (self=0x156d160) at mainloop.c:417 #18 0x00007fb9c65d3ccc in iv_work_event (_pool=0x14f1bf0) at iv_work.c:221 #19 0x00007fb9c65d2862 in iv_event_run_pending_events (_dummy=0x0) at iv_event.c:67 #20 0x00007fb9c65d2cb5 in iv_event_raw_got_event (_this=0x7fb9c654aa18) at iv_event_raw.c:82 #21 0x00007fb9c65cefec in iv_run_active_list (active=0x7fff33ed17b0) at iv_main.c:219 #22 0x00007fb9c65cf1cc in iv_main () at iv_main.c:269 #23 0x00007fb9c65913e9 in main_loop_run () at mainloop.c:731 #24 0x00000000004016d0 in main () -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

[Bug 195] New: syslog 3.3.6.90 stats for src. unix-dgram do not show filesystem name
by bugzilla＠bugzilla.balabit.com 10 Sep '12

10 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=195 Summary: syslog 3.3.6.90 stats for src.unix-dgram do not show filesystem name Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When using the syslog-ng-ctl to get statistics from syslog-ng the source stats look like; src.unix-dgram;local#1;;a;processed;2025 src.internal;local#0;;a;processed;54 src.file;local#2;/proc/kmsg;a;processed;273 for a configuration of source local { internal( tags("syslog.internal") ); unix-dgram("/dev/log" max_connections(10000) log_iw_size(1000000) ); file("/proc/kmsg" program_override("kernel") ); }; The file name of /proc/kmsg shows in the stats line for it, but the file name of /dev/log does not show in the stats line related to it. I think this is a bug and the /dev/log should show in the stats line. Evan. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 5

[Bug 144] New: incremental statistics and on reload and stop
by bugzilla＠bugzilla.balabit.com 07 Sep '12

07 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=144 Summary: incremental statistics and on reload and stop Product: syslog-ng Version: unspecified Platform: All OS/Version: All Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: mdione.ext(a)orange.com Type of the Report: --- Estimated Hours: 0.0 syslog-ng log statistics in /var/log/messages (or some other defined destination), but it does it accumulatively. this makes it harder to make graphs, because the data collector must detect reloads and stop/starts. this most probably should not be done if the statistics are asked via syslog-ng-ctl. also, it does not print statistics on reload or stop, so the last values before the reload or stop are lost. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 61] udp6 source declaration causes syslog forwarding to assume IPv6
by bugzilla＠bugzilla.balabit.com 07 Sep '12

07 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=61 --- Comment #4 from Gergely Nagy <algernon(a)balabit.hu> 2012-09-07 15:45:20 --- For what its worth, the quick workaround mentioned above is available here: https://github.com/algernon/syslog-ng/commit/edd84a5fcedb998949beccc18794b9… -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 61] udp6 source declaration causes syslog forwarding to assume IPv6
by bugzilla＠bugzilla.balabit.com 07 Sep '12

07 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=61 Gergely Nagy <algernon(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #3 from Gergely Nagy <algernon(a)balabit.hu> 2012-09-07 15:39:23 --- The straightforward fix is to add a { last_addr_family = AF_INET; } to dest_afsyslog, similarly how it is done for the source part. But then we need syslog6() to use the protocol over IPv6, and we might need syslog(transport(udp6)) too (although, syslog6(transport(udp)) should be the same). I could send a patch - against 3.4 - that turns syslog() ipv4-only, and introduces syslog6(). On the other hand, the best would be if we didn't need the ipv4/ipv6 distinction at all, and whichever resolves would be used. I'd rather see that done. I'll try to see if the PE core has this, the team did some refactoring in the area. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 61] udp6 source declaration causes syslog forwarding to assume IPv6
by bugzilla＠bugzilla.balabit.com 07 Sep '12

07 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=61 Gergely Nagy <algernon(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |algernon(a)balabit.hu AssignedTo|bazsi(a)balabit.hu |algernon(a)balabit.hu --- Comment #2 from Gergely Nagy <algernon(a)balabit.hu> 2012-09-07 15:14:09 --- This is still present in 3.3, and that is unfortunate since it just bit me. I'll see what I can do about it. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

Syslog-NG/MySQL question
by James McDonald 07 Sep '12

07 Sep '12

Here is the output of /opt/syslog-ng/sbin/syslog-ng -V: [root@ustc-syslog2 ~]# /opt/syslog-ng/sbin/syslog-ng -V syslog-ng-premium-edition 4 F2 (4.2.3) Installer-Version: 4.2.3 Revision: ssh+git://folti@git.balabit//var/scm/git/syslog-ng/syslog-ng-core--mainline- -4.2#master#725fff94b398b564b71f9de88808148b0dd2fc8d Compile-Date: Jul 4 2012 14:01:24 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afl ogstore,disk-buffer,confighash,convertfuncs,basic-proto,rltp-proto,afsql,afs qlsource Available-Modules: afsql,license,afsocket-tls,afuser,tfjson,csvparser,affile,dbparser,rltp-prot o,afsocket-notls,dummy,confgen,aflogstore,basicfuncs,basic-proto,afsqlsource ,snmp,afsocket,syslogformat,disk-buffer,afprog,convertfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: on Enable-Pcre: on Does anyone know what needs to be edited/installed to have the line "Enable-SQL: on" be included in that output? I installed the following MySQL rpms on this server but I'm not sure what else needs to be done to have that line appear. MySQL-client-5.5.14-1.rhel5.x86_64.rpm MySQL-server-5.5.14-1.rhel5.x86_64.rpm MySQL-shared-compat-5.5.14-1.rhel5.x86_64.rpm MySQL-devel-5.5.14-1.rhel5.x86_64.rpm MySQL-shared-5.5.14-1.rhel5.x86_64.rpm MySQL-test-5.5.14-1.rhel5.x86_64.rpm Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

2 3

[Bug 188] New: Syslog-ng 3.3.5.90-20120810 problems: undefined symbol and ld.so library path
by bugzilla＠bugzilla.balabit.com 06 Sep '12

06 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=188 Summary: Syslog-ng 3.3.5.90-20120810 problems: undefined symbol and ld.so library path Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jpo(a)di.uminho.pt Type of the Report: --- Estimated Hours: 0.0 Linux distro: Fedora 17 x86_64 Syslog-ng source tarball: syslog-ng-3.3.5.90-20120810-v3.3.5-43-g19f6c.tar.gz we have encountered a couple of problems when running syslog-ng-3.3.5.90-20120810 under Fedora 17 x86_64: 1) ld.so library path 2) undefined symbol: tls_context_setup_session Steps to reproduce the problem: 1) just run "/usr/sbin/syslog-ng --version" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Error opening plugin module; module='afsocket-tls', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Available-Modules: dummy,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- The library path problem can be solved by creating the file "/etc/ld.so.conf.d/syslog-ng.conf" containing the syslog-ng module directory ( /usr/lib64/syslog-ng in this particular case ). Running syslog-ng under strace without touching the ld.so library paths produces: ---------- # # strace -e trace=file /usr/sbin/syslog-ng --version ... access("//usr/lib64/syslog-ng/libafsocket.so", F_OK) = 0 stat("//usr/lib64/syslog-ng/libafsocket.so", {st_mode=S_IFREG|0755, st_size=103560, ...}) = 0 open("//usr/lib64/syslog-ng/libafsocket.so", O_RDONLY|O_CLOEXEC) = 4 open("/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 open("/usr/lib64/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=49152, ...}) = 0 Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' ... ----------- Note that all syslog-ng modules with the exception of libsyslog-ng-crypto.so are loaded from the syslog-ng modules directory. Output of "syslog-ng --version" after creating the file "/etc/ld.so.conf.d/syslog-ng.conf" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Available-Modules: dummy,afsocket,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,afsocket-tls,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- /jpo -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 13

DNS-based port knocking demo
by Endre Szabo 06 Sep '12

06 Sep '12

Hi list, Here's a proof of concept of DNS based port knocking. Firewall opens up port of SSHd for 10 seconds after nameserver answering to a predefined DNS query. It uses pattern db and netfilter only. http://vimeo.com/endreszabo/dns-portknocking Endre

5 5

The "3.3.7 RC"
by Marvin Nipper 05 Sep '12

05 Sep '12

Sorry for what will be "stupid questions". I know that you were wanting us to "try out" the 3.3.7 RC, and that (last week) you were expecting to produce that as earlier as last Friday. However... there was all of that weekend traffic, with a whole new set of ivykis-related problems, so I'm not sure what the state of the "current" packaging is at this point, and also not 100% sure what packaging I should be pulling down. My _guess_ is maybe the HEAD file from here: http://packages.madhouse-project.org/syslog-ng/3.3/3.3.6/, and I see that a new package was built last night. Bottom line... can you just let us know when you have all of the latest fixes rolled up into a potential RC package, and just verify where we should pull that from. I'm more than happy to run/test the RC package, but I just need to know when/where I should grab it. Thanks. Marvin Nipper Director of Security Stream Global Services This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

2 1

[Bug 193] New: syslog-ng 3.3. 6 spins at 100% on FreeBSD with spoof-source compiled in
by bugzilla＠bugzilla.balabit.com 04 Sep '12

04 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=193 Summary: syslog-ng 3.3.6 spins at 100% on FreeBSD with spoof- source compiled in Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: FreeBSD Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: regression Estimated Hours: 0.0 With the default syslog-ng.conf on FreeBSD, syslog-ng 3.3.6 will spin at 100% CPU when spoof source is enabled. 3.3.5 did not have this problem, neither does it happen with --disable-spoof-source, and by all accounts, this is a problem in syslog-ng, so a lib/ivykis upgrade will not help. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 12

[Bug 192] New: 3.3.6 fails to compile with --disable-spoof-source
by bugzilla＠bugzilla.balabit.com 03 Sep '12

03 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=192 Summary: 3.3.6 fails to compile with --disable-spoof-source Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: dreisner(a)archlinux.org Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=65) --> (https://bugzilla.balabit.com/attachment.cgi?id=65) fix build with --disable-spoof-source modules/afsocket/afinet.c misses some #ifdef protection of AFInetDestDriver.spoof_source, which leads to compilation failures with --disable-spoof-source. afinet.c: In function 'afinet_dd_init': afinet.c:545:11: error: 'AFInetDestDriver' has no member named 'spoof_source' mv -f .deps/libafsocket_notls_la-afsocket-plugin.Tpo .deps/libafsocket_notls_la-afsocket-plugin.Plo mv -f .deps/libafsocket_notls_la-afsocket-parser.Tpo .deps/libafsocket_notls_la-afsocket-parser.Plo make[4]: *** [libafsocket_notls_la-afinet.lo] Error 1 make[4]: *** Waiting for unfinished jobs.... mv -f .deps/libafsocket_tls_la-afsocket-parser.Tpo .deps/libafsocket_tls_la-afsocket-parser.Plo afinet.c: In function 'afinet_dd_init': afinet.c:545:11: error: 'AFInetDestDriver' has no member named 'spoof_source' It looks like your public facing git repo isn't updated yet for the 3.3.x branch, so the attached patch is against the 3.3.6 tarball. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[Bug 190] New: syslog-ng with TCP source, fails to shutdown properly, and generates core dump
by bugzilla＠bugzilla.balabit.com 01 Sep '12

01 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=190 Summary: syslog-ng with TCP source, fails to shutdown properly, and generates core dump Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Solaris Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: marvin.nipper(a)stream.com Type of the Report: bug Estimated Hours: 0.0 OK. I decided to jump to 3.3.6, so that I could get rid of the source patching that I was doing to fix the address spoofing bug in 3.3.5. It compiled cleanly, and I was hoping that I was finally going to be on an "unpatched" version of 3.3.x!! However, there is now a completely new bug that has apparently been introduced into 3.3.6, related to TCP sources. My environment is Solaris 10 U10 x86. What I am seeing: The daemon starts fine, and starts collecting log data just fine; When I execute the standard /etc/init.d script to stop the daemon (using the kill of the PID): I now instantly get a core dump; I find that both of the syslog-ng processes are still running; I find that syslog-ng no longer appears to be processing any of the UDP input (as the files that should grow, because of UDP input, are no longer growing); and I find that the one file that I generate from TCP input, is still continuing to grow. Put simply, it appears that the "kill" does not clearly shutdown the TCP source, and so syslog-ng just continues to run, and continues to "eat" TCP packets, and feed them to their designated target file. I end up having to do a kill -9 on both of the syslog-ng daemons, in order to get them to stop. The relevant statements are fairly trivial, and they work flawlessly in 3.3.5 (as well as all previous releases): source any_tcp { tcp(port(601) max-connections(40) flags("store-legacy-msghdr", "threaded") use_dns(no) log_fetch_limit(100) log_iw_size(250)); }; destination workstation_log { file("/var/adm/log/workstation.log" create_dirs(yes) flags("threaded")); }; log { source(any_tcp); destination(workstation_log); flags(final); }; Anyway, I'm hoping that you will possibly have some clue as to a particular bit of code that might have been changed between 3.3.5 and 3.3.6, that might be causing this behavior(??). Let me know what all you might want/need from me, to assist in sorting this out. SORRY for the bad news (really). I was really hoping that 3.3.6 would be "painless", and I'm sure that you were hoping for the same thing. As with the spoofing bug in 3.3.5, I am ALWAYS more than happy to try out a patch, to see if that fixes the problem. I know that it may not be easy to try out Solaris-related changes (and maybe this issue is just something in Solaris). As always, THANKS for your time and help. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

4 24

syslog-ng 3.3.7 roadmap
by Gergely Nagy 01 Sep '12

01 Sep '12

Hi! Since there were a couple of pending issues reported around the time 3.3.6 was released, a few little bugs remained in the release. The nastiest one only affects Solaris, and can be solved by updating the embedded ivykis. The others are - in my opinion - fairly rare corner cases. Nevertheless, they are all fixed in git, and I will release an RC sometime during the weekend. Any testing and feedback would be most appreciated, and if you have any issues you'd like to get fixed or implemented before the next release, do let me know! As of this writing, the following bugs were fixed in git: [#188]: afsocket-notls mistakenly being compiled as if TLS was enabled [#189]: syslog-ng --debug --version segfaulting at the module list [#190]: Solaris TCP source shutdown issue [#191]: glob-based @include working differently than the dir-based one I will have the RC going on for a little while, mostly to allow people time to install 3.3.6 (which, by and large, is the best 3.3 release so far, if I may says so myself, even with the Solaris issue), and find any bugs I may have missed. The plan is to release syslog-ng 3.3.7 on October 31st, because that's almost like a christmas present, and is far enough to allow people to test 3.3.6 and the upcoming RC, but soon enough to not let the already fixed bugs cause more damage. -- |8]

1 0

[Bug 189] New: Syslog-ng 3.3.5. 90 segfaults with cmd line options --version and --debug
by bugzilla＠bugzilla.balabit.com 01 Sep '12

01 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=189 Summary: Syslog-ng 3.3.5.90 segfaults with cmd line options -- version and --debug Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: jpo(a)di.uminho.pt Type of the Report: --- Estimated Hours: 0.0 Linux distro: Fedora 17 x86_64 Syslog-ng source tarball: syslog-ng-3.3.5.90-20120810-v3.3.5-43-g19f6c.tar.gz Summary: Using the command line options --version and --debug simultaneously causes a segmentation fault. ---------- # /usr/sbin/syslog-ng --version --debug syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 20 2012 05:49:44 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Segmentation fault (core dumped) ---------- # gdb /usr/sbin/syslog-ng core.1065 GNU gdb (GDB) Fedora (7.4.50.20120120-49.fc17) Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/syslog-ng...Reading symbols from /usr/lib/debug/usr/sbin/syslog-ng.debug...done. done. [New LWP 1065] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/syslog-ng --version --debug'. Program terminated with signal 11, Segmentation fault. #0 nv_registry_get_handle_name (length=0x7ffff9f5a1c8, handle=4, self=<optimized out>) at nvtable.h:86 86 stored = &g_array_index(self->names, NVHandleDesc, handle - 1); (gdb) backtrace #0 nv_registry_get_handle_name (length=0x7ffff9f5a1c8, handle=4, self=<optimized out>) at nvtable.h:86 #1 log_msg_get_value_name (handle=handle@entry=4, name_len=name_len@entry=0x7ffff9f5a1c8) at logmsg.c:305 #2 0x00007f52258d8bea in log_msg_set_value (self=self@entry=0xe7da00, handle=handle@entry=4, value=value@entry=0x7f522590069b "syslog-ng", value_len=value_len@entry=9) at logmsg.c:458 #3 0x00007f52258da937 in log_msg_new_internal (prio=47, msg=0xe7cfa0 "Trying to open module; module='dummy', filename='//usr/lib64/syslog-ng/libdummy.so'") at logmsg.c:1128 #4 0x00007f52258e7703 in msg_send_internal_message (prio=<optimized out>, msg=<optimized out>) at messages.c:141 #5 0x00007f52258e78d0 in msg_event_send (e=0xe7cf30) at messages.c:179 #6 0x00007f52258eb825 in plugin_dlopen_module (module_name=module_name@entry=0xe72560 "dummy", module_path=0x7f522590068f "//usr/lib64/syslog-ng") at plugin.c:188 #7 0x00007f52258ebf73 in plugin_list_modules (out=0x7f52240d3260, verbose=verbose@entry=0) at plugin.c:293 #8 0x00000000004018a4 in version () at main.c:134 #9 0x000000000040163c in main (argc=1, argv=0x7ffff9f5a488) at main.c:217 (gdb) ---------- -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[Bug 191] New: glob-based includes aren't evaluated in proper order
by bugzilla＠bugzilla.balabit.com 01 Sep '12

01 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=191 Summary: glob-based includes aren't evaluated in proper order Product: syslog-ng Version: 3.3.x Platform: All OS/Version: All Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: bug Estimated Hours: 0.0 While upgrading one of my trickier configs, I noticed that syslog-ng suddenly refused to start, saying it can't find the mongodb destination - this was clearly bogus. A little research revealed the problem: I had a setup where /etc/syslog-ng/syslog-ng.conf had an @include "/etc/syslog-ng/conf.d/" line prior to the upgrade, but when upgraded, that changed to @include "/etc/syslog-ng/conf.d/*.conf". It turns out, that while syslog-ng loads the files in the correct order in both cases, when using globs, the include stack gets botched up: Starting to read include file; filename='/home/algernon/install/sng/3.3/etc/conf.d/00-mongodb.conf', depth='1' Starting to read include file; filename='/home/algernon/install/sng/3.3/etc/conf.d/99-test.conf', depth='2' Error parsing destination, destination plugin mongodb not found in /home/algernon/install/sng/3.3/etc/conf.d/99-test.conf at line 2, column 2: included from /home/algernon/install/sng/3.3/etc/conf.d/00-mongodb.conf line 1, column 1 included from /home/algernon/install/sng/3.3/etc/syslog-ng.conf line 26, column 1 mongodb(); 00-mongodb.conf does NOT include 99-test.conf. syslog-ng.conf includes both with a glob include. But since it thinks 99-test.conf was included by 00-mongodb.conf, it did not evaluate the contents of the latter, and as such, the mongodb module did not get loaded, and then, of course, the 99-test.conf which would use it, will fail with a parse error. With a directory include, this does not happen. A similar issue affects 3.4 too, but there plugin loading happens automatically, so this particular config would work. However, if a later config relies on a former (eg, the former defines a block), then that would break too. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

recovering corrupt disk buffer?
by Jiann-Ming Su 31 Aug '12

31 Aug '12

Is it possible to recover a corrupted disk buffer? If so, how? I don't think I saw it in the admin guide. Thanks for any tips.

2 1

Error: Number of licensed hostnames exceeded.
by sagar naravane 30 Aug '12

30 Aug '12

Hi, I am using syslog-ng client rpm to work as a relay. I keep getting this message. Aug 30 07:22:07 sj1glm190 syslog-ng[9009]: Number of licensed hostnames exceeded; error='Licensed limit exceeded, rejecting request (license_limit=3, tag=sj1slm275)' This is my relay config @version: 4.2 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # file:///opt/syslog-ng/share/doc/admin-guide/index.html # or # http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-v4.1-guid… # @include "scl.conf" options {log_msg_size(5000000);}; ###### # sources source s_local { # message generated by Syslog-NG internal(); system(); }; ## define source for incoming log messages from client. Use it in custom configs source s_client { syslog(ip(0.0.0.0) port(514) transport("tcp") max-connections(100)); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_master { syslog("x.x.x.x" transport ("tcp") port(514)); }; ###### # output log { source(s_local); destination(d_messages); }; log { source(s_client); destination(d_master); }; @include "include/" -- Regards, Sagar Naravane

3 2

Syslog-NG / Log Analyzer
by James McDonald 29 Aug '12

29 Aug '12

I have asked about this problem before, but I have been t/sing with some tips from fellow message board members. I have been able to get MySQL and Log Analyzer installed but I get an error within Log Analyzer, which reads: No syslog records found. I have attached the config.php and the syslog-ng.conf files for your viewing pleasure. The MySQL additions are at the very bottom of the syslog-ng.conf file. Let me know if you need any more info, to help me get this thing working. Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

1 0

syslog-ng 3.3.6 has been released
by devel＠balabit.hu 28 Aug '12

28 Aug '12

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.3.6 SUMMARY : new stable release DATE : Aug 27, 2012 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.3.6) has been released. For latest fixes in the 3.3.x feature branch you are recommended to upgrade to this version. CHANGES: 3.3.6 Fri, 10 Aug 2012 11:00:00 +0200 Highlights ========== This release is a bug-fix release mostly, with a few minor - yet useful - feature enhancements. The most important one is that the patched ivykis syslog-ng 3.3 shipped with until now is no more. We build against upstream ivykis now (still included for convenience, though). Other highlights include much improved systemd support, and an enhancement to the @include feature, and many bugfixes. See below for more information! Features ======== * The system() source automatically detects systemd, and defines the proper log socket accordingly. If the system() source is not used, a workaround to use the systemd supplied log socket instead of /dev/log was applied. This kicks in if syslog-ng is running under systemd, and its config version is 3.4 or less. * The @include statement now accepts globs, allowing one to limit what gets included, and limit it to files matching "*.conf", for example. Bugfixes ======== * When using spoof-source, the destination address is now correctly resolved again after a SIGHUP. * When the supervisor receives a SIGHUP, it will not terminate anymore, but ignore it. * The fsync(yes) functionality was restored, and will now do what it is supposed to do. * A memory corruption was fixed in the DNS cache when used in threaded mode and dns-cache-hosts() is used. * The program() destination now correctly closes the standard input of the running program when syslog-ng is being reloaded (as opposed to only sending a signal). * Incorrect encoding of data caused UDP and unix-dgram() receives to be erroneously closed, so no further data would be received. This has been fixed. Incorrect encoding of incoming data causes the connection for connection oriented transports (like tcp() and unix-stream()) to be closed, but the same condition is ignored for datagram oriented transports (like udp() or unix-dgram()). * Fixed a possible NULL dereference in patterndb, when an action was used without a context-scope. * The max-connections() setting now works correctly accross reloads, and does not reset the counter. Other changes ============= * On systems that have utmpx(5), syslog-ng will prefer and use that over the older utmp(5). This means that there is no more patching needed for systems like FreeBSD 9, which only support utmpx. Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Alex Zimnitsky <avz(a)dagotel.ru> Alexander Komyagin <komyagin(a)altell.ru> Balazs Scheidler <bazsi(a)balabit.hu> Cary Taylor <ctaylor(a)ptgi.com> EgonB <egon(a)local.ee> Evan Rempel <erempel(a)uvic.ca> Gergely Nagy <algernon(a)balabit.hu> Imre Lazar <imre(a)balabit.hu> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Lennert Buytenhek <buytenh(a)wantstofly.org> Mads <mads(a)ab3.no> Mark Ulmer <mark.ulmer(a)apollogrp.edu> Mark Wooding <mdw(a)distorted.org.uk> Marvin Nipper <marvin.nipper(a)stream.com> Patrick Hemmer Peter Czanik <czanik(a)balabit.hu> Sandor Geller <wildy(a)muhelybt.hu> Viktor Juhasz <jviktor(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

2 1

[PATCH (3.4)] system-source: Introduce a new system() generator.
by Gergely Nagy 27 Aug '12

27 Aug '12

This patch replaces the old confgen-based system() generator, that relied on a script (generate-system-source.sh) to generate the block system() expands into. While that was flexible and easy to modify, it has had the disadvantage of running an external script at startup, which is undesirable in many situations, including when using a strict AppArmor profile. However, a generated system() is useful in a lot of situations, ranging from creating portable, system-agnostic configuration files to supporting sysvinit and systemd with the same config. For these reasons, this patch implements a built-in system() generator, that does not rely on the generate-system-source.sh script at all, but does everything itself, from C. It is loaded by the default scl.conf, but can be loaded separately with: @module system-source The system() generator this patch implements results in the exact same expansion the old script generated. The old script-based solution is also deprecated (and thus, removed), in order to not duplicate code: we only wish to maintain one implementation. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- configure.in | 1 + modules/Makefile.am | 2 +- modules/system-source/Makefile.am | 9 ++ modules/system-source/system-source.c | 196 +++++++++++++++++++++++++++++++++ scl/system/generate-system-source.sh | 110 ------------------ scl/system/plugin.conf | 27 +---- 6 files changed, 211 insertions(+), 134 deletions(-) create mode 100644 modules/system-source/Makefile.am create mode 100644 modules/system-source/system-source.c delete mode 100755 scl/system/generate-system-source.sh diff --git a/configure.in b/configure.in index 294066a..5c00e0f 100644 --- a/configure.in +++ b/configure.in @@ -1184,6 +1184,7 @@ AC_OUTPUT(dist.conf modules/csvparser/Makefile modules/csvparser/tests/Makefile modules/confgen/Makefile + modules/system-source/Makefile modules/syslogformat/Makefile modules/pacctformat/Makefile modules/basicfuncs/Makefile diff --git a/modules/Makefile.am b/modules/Makefile.am index fd557e1..eb9548d 100644 --- a/modules/Makefile.am +++ b/modules/Makefile.am @@ -1 +1 @@ -SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb afsmtp csvparser confgen syslogformat pacctformat basicfuncs cryptofuncs dbparser tfjson jsonparser +SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb afsmtp csvparser confgen system-source syslogformat pacctformat basicfuncs cryptofuncs dbparser tfjson jsonparser diff --git a/modules/system-source/Makefile.am b/modules/system-source/Makefile.am new file mode 100644 index 0000000..3bf0bf4 --- /dev/null +++ b/modules/system-source/Makefile.am @@ -0,0 +1,9 @@ +moduledir = @moduledir@ +AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib +export top_srcdir + +module_LTLIBRARIES := libsystem-source.la +libsystem_source_la_SOURCES = system-source.c +libsystem_source_la_CPPFLAGS = $(AM_CPPFLAGS) +libsystem_source_la_LIBADD = $(MODULE_DEPS_LIBS) +libsystem_source_la_LDFLAGS = $(MODULE_LDFLAGS) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c new file mode 100644 index 0000000..9ba0480 --- /dev/null +++ b/modules/system-source/system-source.c @@ -0,0 +1,196 @@ +/* + * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012 Gergely Nagy <algernon(a)balabit.hu> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + * As an additional exemption you are allowed to compile & link against the + * OpenSSL libraries as published by the OpenSSL project. See the file + * COPYING for details. + * + */ + +#include "cfg.h" +#include "cfg-lexer.h" +#include "cfg-grammar.h" +#include "messages.h" +#include "plugin.h" + +#include <sys/utsname.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <unistd.h> + +static void +system_sysblock_add_unix_dgram(GString *sysblock, const gchar *path, + const gchar *perms) +{ + g_string_append_printf(sysblock, "unix-dgram(\"%s\"", path); + if (perms) + g_string_append_printf(sysblock, " perms(%s)", perms); + g_string_append(sysblock, ");\n"); +} + +static void +system_sysblock_add_file(GString *sysblock, const gchar *path, + gint follow_freq, const gchar *prg_override, + const gchar *flags) +{ + g_string_append_printf(sysblock, "file(\"%s\"", path); + if (follow_freq >= 0) + g_string_append_printf(sysblock, " follow-freq(%d)", follow_freq); + if (prg_override) + g_string_append_printf(sysblock, " program-override(\"%s\")", prg_override); + if (flags) + g_string_append_printf(sysblock, " flags(%s)", flags); + g_string_append(sysblock, ");\n"); +} + +static void +system_sysblock_add_module(GString *sysblock, const gchar *mod) +{ + g_string_append_printf(sysblock, "@module %s\n", mod); +} + +static void +system_sysblock_add_sun_streams(GString *sysblock, const gchar *path, + const gchar *door) +{ + g_string_append_printf(sysblock, "sun-streams(\"%s\"", path); + if (door) + g_string_append_printf(sysblock, " door(\"%s\")", door); + g_string_append(sysblock, ");\n"); +} + +static void +system_sysblock_add_pipe(GString *sysblock, const gchar *path, gint pad_size) +{ + g_string_append_printf(sysblock, "pipe(\"%s\"", path); + if (pad_size >= 0) + g_string_append_printf(sysblock, " pad_size(%d)", pad_size); + g_string_append(sysblock, ");\n"); +} + +gboolean +system_generate_system(CfgLexer *lexer, gint type, const gchar *name, + CfgArgs *args, gpointer user_data) +{ + gchar buf[256]; + GString *sysblock; + struct utsname u; + + g_snprintf(buf, sizeof(buf), "source confgen system"); + + sysblock = g_string_sized_new(1024); + + if (uname(&u) != 0) + { + msg_error("system(): Cannot get information about the running kernel", + evt_tag_errno("error", errno), + NULL); + return FALSE; + } + + if (strcmp(u.sysname, "Linux") == 0) + { + char *log = "/dev/log"; + + if (getenv("LISTEN_FDS") != NULL) + { + struct stat sbuf; + + if (stat("/run/systemd/journal/syslog", &sbuf) == 0) + { + if (S_ISSOCK(sbuf.st_mode)) + log = "/run/systemd/journal/syslog"; + } + } + + system_sysblock_add_unix_dgram(sysblock, log, NULL); + system_sysblock_add_file(sysblock, "/proc/kmsg", -1, "kernel", "kernel"); + } + else if (strcmp(u.sysname, "SunOS") == 0) + { + system_sysblock_add_module(sysblock, "afstreams"); + + if (strcmp(u.release, "5.8") == 0) + system_sysblock_add_sun_streams(sysblock, "/dev/log", NULL); + else if (strcmp(u.release, "5.9") == 0) + system_sysblock_add_sun_streams(sysblock, "/dev/log", "/etc/.syslog_door"); + else + system_sysblock_add_sun_streams(sysblock, "/dev/log", "/var/run/syslog_door"); + } + else if (strcmp(u.sysname, "FreeBSD") == 0) + { + system_sysblock_add_unix_dgram(sysblock, "/var/run/log", NULL); + system_sysblock_add_unix_dgram(sysblock, "/var/run/logpriv", "0600"); + system_sysblock_add_file(sysblock, "/dev/klog", 0, "kernel", "no-parse"); + } + else if (strcmp(u.sysname, "GNU/kFreeBSD") == 0) + { + system_sysblock_add_unix_dgram(sysblock, "/var/run/log", NULL); + system_sysblock_add_file(sysblock, "/dev/klog", 0, "kernel", NULL); + } + else if (strcmp(u.sysname, "HP-UX") == 0) + { + system_sysblock_add_pipe(sysblock, "/dev/pipe", 2048); + } + else if (strcmp(u.sysname, "AIX") == 0 || + strcmp(u.sysname, "OSF1") == 0 || + strncmp(u.sysname, "CYGWIN", 6) == 0) + { + system_sysblock_add_unix_dgram(sysblock, "/dev/log", NULL); + } + else + { + msg_error("system(): Error detecting platform, unable to define the system() source. " + "Please send your system information to the developers!", + evt_tag_str("sysname", u.sysname), + evt_tag_str("release", u.release), + NULL); + return FALSE; + } + + if (!cfg_lexer_include_buffer(lexer, buf, sysblock->str, sysblock->len)) + { + g_string_free(sysblock, TRUE); + return FALSE; + } + + return TRUE; +} + +gboolean +system_source_module_init(GlobalConfig *cfg, CfgArgs *args) +{ + cfg_lexer_register_block_generator(cfg->lexer, + cfg_lexer_lookup_context_type_by_name("source"), + "system", system_generate_system, + NULL, NULL); + + return TRUE; +} + +const ModuleInfo module_info = +{ + .canonical_name = "system-source", + .version = VERSION, + .description = "The system-source module provides support for determining the system log sources at run time.", + .core_revision = SOURCE_REVISION, + .plugins = NULL, + .plugins_len = 0, +}; diff --git a/scl/system/generate-system-source.sh b/scl/system/generate-system-source.sh deleted file mode 100755 index 708a7a4..0000000 --- a/scl/system/generate-system-source.sh +++ /dev/null @@ -1,110 +0,0 @@ -#!/bin/sh -############################################################################# -# Copyright (c) 2010 BalaBit IT Ltd, Budapest, Hungary -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License version 2 as published -# by the Free Software Foundation, or (at your option) any later version. -# -# As an additional exemption you are allowed to compile & link against the -# OpenSSL libraries as published by the OpenSSL project. See the file -# COPYING for details. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA -# -############################################################################# - -# NOTE: the reason to set PATH and LD_LIBRARY_PATH/LIBPATH explicitly -# is that some OSes (notably AIX) can have multiple uname binaries, -# and the one on the PATH used by syslog-ng may depend on libraries -# that syslog-ng itself is distributing. However the one distributed -# by the syslog-ng binary is binary incompatible with the one required -# by the alternative uname, causing uname execution to fail. -# -# The alternative, better solution would be to patch syslog-ng to -# remove its own changes from LD_LIBRARY_PATH before executing user -# supplied scripts, but that was more involved than this simple change -# here. - -PATH=/bin:/usr/bin:$PATH -LIBPATH= -LD_LIBRARY_PATH= -export PATH LIBPATH LD_LIBRARY_PATH - -os=${UNAME_S:-`uname -s`} -osversion=${UNAME_R:-`uname -r`} - -case $os in - Linux) - DEVLOG="/dev/log" - if [ ! -z "$LISTEN_FDS" ] && [ -S "/run/systemd/journal/syslog" ]; then - DEVLOG="/run/systemd/journal/syslog" - fi - cat <<EOF -unix-dgram("${DEVLOG}"); -file("/proc/kmsg" program-override("kernel") flags(kernel)); -EOF - ;; - SunOS) - if [ "$osversion" = "5.8" ]; then - cat <<EOF -@module afstreams -sun-streams("/dev/log"); -EOF - elif [ "$osversion" = "5.9" ]; then - cat <<EOF -@module afstreams -sun-streams("/dev/log" door("/etc/.syslog_door")); -EOF - else - cat <<EOF -@module afstreams -sun-streams("/dev/log" door("/var/run/syslog_door")); -EOF - fi - ;; - FreeBSD) - cat <<EOF -unix-dgram("/var/run/log"); -unix-dgram("/var/run/logpriv" perm(0600)); -file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse)); -EOF - ;; - GNU/kFreeBSD) - cat <<EOF -unix-dgram("/var/run/log"); -file("/dev/klog" follow-freq(0) program-override("kernel")); -EOF - ;; - HP-UX) - cat <<EOF -pipe("/dev/log" pad_size(2048)); -EOF - ;; - AIX|OSF1|CYGWIN*) - cat <<EOF -unix-dgram("/dev/log"); -EOF - ;; - *) - # need to notify the user that something went terribly wrong... - (echo "system(): Error detecting platform, unable to define the system() source."; - echo ""; - echo "The script generating the syslog-ng configuration for the system() source failed."; - echo "It is caused either by an unsupported OS or the fact that the script could not"; - echo "execute uname(1)."; - echo ""; - echo " OS=${os}"; - echo " OS_VERSION=${osversion}"; - echo " SCRIPT=$0") >&2 - - exit 1 - ;; -esac diff --git a/scl/system/plugin.conf b/scl/system/plugin.conf index 5ba652e..494815a 100644 --- a/scl/system/plugin.conf +++ b/scl/system/plugin.conf @@ -1,23 +1,4 @@ -############################################################################# -# Copyright (c) 2010 BalaBit IT Ltd, Budapest, Hungary -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License version 2 as published -# by the Free Software Foundation, or (at your option) any later version. -# -# As an additional exemption you are allowed to compile & link against the -# OpenSSL libraries as published by the OpenSSL project. See the file -# COPYING for details. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA -# -############################################################################# - -@module confgen context(source) name(system) exec("`scl-root`/system/generate-system-source.sh") +# This loads the system-source module, which defines the system() block. +# If you'd like to define it yourself, comment out the following line, +# or don't include this file (via scl.conf) in your config. +@module system-source -- 1.7.10.4

1 0

How to prevent loss of logs.
by sagar naravane 27 Aug '12

27 Aug '12

Guys, My setup is like this APACHE EDGE SERVER ---> RELAY ---> SYSLOG-NG CENTRAL SERVER. Now what I've learned so far (i am still a newbie) is the log collection happens almost real time. However I want to build a setup where if any of the component above goes down the logs in between that particular duration is not lost. Say for example at 1:00 any of the component goes down and comes back up by 2:00 the log worth 1 hour should be passed to/collected on syslog-ng server ( i mean to say the logs on server shoulding be 12:59 xxxx 2:00 xxxx) I know this isnt a very well constructed question/doubt but initially i am looking for some tips. I'll put in details of my configs if needed :) -- Regards, Sagar Naravane

2 3