- syslog-ng - lists.balabit.hu

Can't seem to do a simple parse
by Tracy Dalzell 21 Sep '12

21 Sep '12

I am trying to do a very simple parsing test to reformat a message using syslog-ng 3.3.6 and I can't seem to use the value of my macro (mystring): patterndb (test.xml): <patterndb version='4' pub_date='2012-09-21'> <ruleset name='test' id='123456678'> <pattern>test</pattern> <rules> <rule provider='me' id='182437592347598' class='system'> <patterns> <pattern>@STRING:mystring: @</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> pdbtool results: [root@loghost01 syslog-ng]# pdbtool match -c -D -v -p test.xml -P test -M "INSERT INTO Test (id, value) VALUES (1, .044212)" Pattern matching part: @STRING:mystring=INSERT INTO Test @ Matching part: INSERT INTO Test Values: MESSAGE=INSERT INTO Test (id, value) VALUES (1, .044212) PROGRAM=test .classifier.class=unknown TAGS=.classifier.unknown relevent syslog-ng.conf lines: source s_remote { tcp(); }; parser p_test { db-parser(file("/etc/syslog-ng/test.xml")); }; destination d_test { file("/var/log/splunk/$HOST" template("Here is the message: ${R_DATE} ${mystring}\n")); }; log { source(s_remote); parser(p_test); destination(d_test); };\n")); }; Results from logger "INSERT INTO Test (id, value) VALUES (1, .044212)" on client are: Here is the message: Sep 21 16:46:53 Value of ${mystring} is null. What am I missing?

2 2

syslog-ng Insider - September 2012
by Peter Czanik 20 Sep '12

20 Sep '12

Dear syslog-ng users, This is the 16th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng 3.3.6 is released --------------------------- There was a new release for the syslog-ng stable line: version 3.3.6 There are mostly bugfixes in this release but also some smaller but long waited features: * the @include statement now accepts globs * upstream ivykis is used * better systemd support The change to upstream ivykis introduced a bug on FreeBSD and some other problems were also discovered. Version 3.3.7 is scheduled to arrive at the end of October, and there is now a 3.3.6.90, aka 3.3.7RC available for testing. Announcement for 3.3.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-August/000146.ht… Announcement for 3.3.6.90: https://lists.balabit.hu/pipermail/syslog-ng/2012-September/019365.html syslog-ng 3.4 news ------------------ Although there aren't any new alpha releases yet, there are already plenty of news for the development branch aswell! * The MongoDB destination received a major speed boost, and work is underway to make it even more performant. * The $(format-json) function now supports nested structures * There's a MAC address parser available for patterndb * A couple of new template functions were introduced, $(md5), $(sha1) and others. * The unit tests were improved upon greatly, and many many bugs were squashed aswell. For the complete list of changes, you can browse the commit messages at https://github.com/balabit/syslog-ng-3.4/commits/master And while there, check out the sources and give it a try, feedback is appreciated, help us make syslog-ng even better! SSB 3 LTS: wildcard search in logspaces --------------------------------------- The third important new feature in SSB 3 LTS is the ability to search within log messages using wildcard characters. It is indeed a small and simple change on the surface, but brings a lot of new possibilities. Read about the details at http://gyp.blogs.balabit.com/2012/07/new-features-in-ssb-3-lts-wildcard-sea… OTHER SHORT NEWS * syslog-ng ports in FreeBSD explained: https://czanik.blogs.balabit.com/2012/09/syslog-ng-rc/ * syslog-ng will be presented at two conferences in October: http://fedoraproject.org/wiki/FUDCon:Paris_2012 and http://conference.opensuse.org/ NEW RELEASES: * syslog-ng OSE 3.3.6: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-August/000146.ht… * syslog-ng OSE 3.3.6.90: https://lists.balabit.hu/pipermail/syslog-ng/2012-September/019365.html CASE STUDY: * University of Victoria: http://www.balabit.com/support/documentation/syslog-ng_cs_uvic_en.pdf ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

[PATCH 0/7] value-pairs dot notation walker
by Gergely Nagy 16 Sep '12

16 Sep '12

The patches that will follow this overview soon implement a 'walker' for value-pairs, a way to walk the internal flat structure syslog-ng uses as if it was a tree. This is important for two modules in particular: $(format-json) and the mongodb destination. The latter already supportes storing structured data, but it is using upserts (with all its downsides) to accomplish that. The former - up until now - was only able to format flat JSON. With these patches, both change: $(format-json) becomes able to emit structured data, and the mongodb destination starts to use inserts. The patches also merge the tfjson and jsonparser modules into a single json module, and implements our own JSON formatting for $(format-json). This means that json-glib support is completely removed, as it became unnecessary, and json-c is only used by the parser, but for practical reasons, required to build the plugin. That is, it is no longer possible to build $(format-json) only, but not json-parser, nor the other way around. The final patch is a bonus, that corrects a small error in jsonparser, and stops it from emitting an assertion warning on stderr in certain cases.

3 9

syslog-ng 3.3.7 RC1 (3.3.6.90) released
by Gergely Nagy 14 Sep '12

14 Sep '12

Hi! As promised earlier, I prepared a release candidate for the upcoming syslog-ng 3.3.7 release (due october 31), with the most pressing bugs already fixed. Testing this release candidate and reporting any and all issues found with it - be those regressions or anything else - would be greatly appreciated! The current state of the NEWS file is included below, and if anyone wants to track how the release is progressing and what needs to be fixed still before the final release, see the 3.3.7 target milestone on bugzilla: https://bugzilla.balabit.com/buglist.cgi?query_format=advanced&target_miles… The release is available from the following sources: GitHub ------ The release is tagged as v3.3.6.90 on the official repository at git://github.com/balabit/syslog-ng-3.3.git Tarball ------- A make dist tarball built from the same tag is available from my package repository: http://packages.madhouse-project.org/syslog-ng/3.3/3.3.6.90/syslog-ng-3.3.6… And finally, the NEWS file from the release: 3.3.6.90 (3.3.7 RC1) Thu, 06 Sep 2012 14:00:00 +0200 Bugfixes ======== * The bundled ivykis has been updated to fix two bugs found after the 3.3.6 release was tagged: * Fixed a Solaris-specific issue relating to TCP sources [#190] * Fixed a spinlock issue, triggered on at least FreeBSD [#193] * The glob-based configuration file inclusion was fixed to behave similarly to including a whole directory, to include files alphabetically. [#191] * Fixed a crash when trying to display the available modules in debug mode. [#189] * Fixed the building of afsocket-notls, so that it is correctly built without TLS support. [#188] * Fix compilation without spoof-source. [#192] Credits ======= syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balazs Scheidler <bazsi(a)balabit.hu> Cy Schubert <cy(a)FreeBSD.org> Dave Reisner <dreisner(a)archlinux.org> Gergely Nagy <algernon(a)balabit.hu> Jose Pedro Oliveira <jpo(a)di.uminho.pt> Lennert Buytenhek <buytenh(a)wantstofly.org> Marvin Nipper <marvin.nipper(a)stream.com> Peter Czanik <czanik(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> -- |8]

5 5

Syslog-ng configuration for network source (udp)
by Alex Clone 12 Sep '12

12 Sep '12

Hello, I am trying to "read" a log stream from interface eth0 and to save it in a file (/var/log/messages_tcp). I am newbie on the use of syslog-ng but I try to learn reading the official manual and a lot of configuration examples. Although, I have try a lot of different configurations, the result is always the same : Syslog-ng cannot read the streaming data. I hope that you can help me. That's the version of syslog-ng that I use: *~# /usr/local/syslog-ng/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Sep 6 2012 14:36:19 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: on Enable-Pacct: off* There is an incoming log stream in eth0 (udp), as you can see next : *~# tcpdump -i eth0 udp 'port 514' -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 06:05:58.669992 IP (tos 0x0, ttl 254, id 20209, offset 0, flags [none], proto UDP (17), length 82) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 54 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host WAN:172.30.254.4\0x0a 06:05:58.670134 IP (tos 0x0, ttl 254, id 52, offset 0, flags [none], proto UDP (17), length 85) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 57 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host outside:172.26.10.4\0x0a ………..* The syslog-ng.conf that I use is the following: * @version: 3.2 options { chain_hostnames(off); long_hostnames(off); use_dns(no); # We don't want DNS names, IP's only. flush_lines(0); stats_freq(43200); time_reopen(10); }; source s_network { # tcp(ip("172.30.6.70") port(514)); # udp(ip("172.30.6.70") port(514)); # syslog( # flags(no-multi-line) # ip(172.30.6.70) # keep-alive(yes) # keep_hostname(yes) # transport(udp) # TLS Options # tls() # ); tcp(); udp(); }; destination d_tcp { file("/var/log/messages_tcp" owner("root") group("adm") perm(0640)); }; log { source(s_network); destination(d_tcp); }; * When I start syslog-ng in debug mode (syslog-ng -Fevd) the results are the followings: *~# /usr/local/syslog-ng/sbin/syslog-ng -Fevdt Trying to open module; module='syslogformat', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libsyslogformat.so' Trying to open module; module='basicfuncs', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libbasicfuncs.so' Trying to open module; module='afsocket', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafsocket.so' Trying to open module; module='affile', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libaffile.so' Trying to open module; module='afprog', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafprog.so' Trying to open module; module='afuser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafuser.so' Trying to open module; module='dbparser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libdbparser.so' Trying to open module; module='csvparser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libcsvparser.so' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.2.4' ^CTermination requested via signal, terminating; syslog-ng shutting down; version='3.2.4' Closing log transport fd; fd='6' Running application hooks; hook='4'* As you can see there is no "Incoming log entry..:" line. I pressed Ctrl+C for the termination of program. Also listening to the network interface seems to be right: *~# netstat -tanpu | grep syslog tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 25950/syslog-ng udp 0 0 0.0.0.0:514 0.0.0.0:* 25950/syslog-ng * Please, can anyone help me with this problem? I don't know what could be blocking the packets, if it is because of the OS or because of syslog-ng configuration.What could it be wrong? I am waiting for your response. PS. Sorry for my "bad" English. Thanks is advance, Alex

3 4

6 identical servers, only 4 sending logs
by Brian Johnson 11 Sep '12

11 Sep '12

Hello all, I hope what I'm asking hasn't been covered previously, I tried some searches with no luck. If I'm duplicating something else, I apologize. My problem is, I have 6 DHCP servers with identical syslog-ng.conf and syslog.conf files on them. The set up is as so: dhcp-a-01 and dhcp-b-01 are a DHCP failover pair dhcp-a-02 and dhcp-b-02 are a DHCP failover pair dhcp-a-03 and dhcp-b-03 are a DHCP failover pair The 'dhcp-a' servers are in the A data center. 'dhcp-b' servers are in the B data center. Again, the syslog-ng.conf files on all of them are identical, checked with sha1sum. It is confirmed that all of them are using syslog-ng for logging. I have them all set to log to the same remote logging server. Logs from dhcp-[a,b]-01 and dhcp-[a,b]-03 are making it to the remote server with no issues. I can see it on the remote server and I can see it when doing a 'tcpdump port 514' on the servers themselves. For some reason, I'm not seeing any logs from dhcp-[a,b]-02 on the remote server and when I do 'tcpdump port 514' for a length of time, I get: dhcp-b-02:~# tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel when the other servers, done at the same time, show packets captured. I just did a "tail -f /var/log/syslog > /tmp/test" all of the servers between 11:43:26 and 11:45:38 (2m12s). In that time: dhcp-[a,b]-01 had roughly 2700 lines dhcp-[a-b]-02 had roughly 11000 lines dhcp-[a-b]-03 had roughly 1100 lines So to me it seems like there's some sort of throttling on the data that's able to be sent. There's ~5x more traffic on pair 2 than 1 (which will be rebalanced, just trying to get this working first) so that would make sense. The only thing that I could find that looks like it would help is the log_fifo_size option, but that doesn't seem to help -- I've made several adjustments to it, but it doesn't seem to make any difference. Can someone please let me know what I'm missing? Thanks! Brian

2 2

[RFC]: Pattern matching & corellation ideas
by Gergely Nagy 11 Sep '12

11 Sep '12

In the past few months, I talked a lot about patterndb and related things with colleagues - during coffee break, over a beer, etc -, and last night, I got as far as drafting a proof of concept tool that realizes some of the ideas we've had. Some of these might have been discussed on this list too, in the past, I really can't remember all the influences I'm afraid. Anyway! I like the concept of patterndb, but I absolutely hate XML. It's not a natural format for describing how to match patterns, and what to do with them. Not for me, anyway. I'd like something that is much closer to how I normally think, something that feels more like a programming language, a domain specific one, engineered for this task only, but still somewhat familiar. I also want it to be fast. For a long time, I also wanted to play with developing both a Domain Specific Language (DSL for short) and a compiler, but never had the opportunity. Pattern matching is one now! How about a small language, that we'd compile down to C, automatically add some boilerplate, and we'd get a syslog-ng parser plugin in return? That would mean the language is fairly easy to extend, it produces native code, which will hopefully run as fast - if not faster - than patterndb, and we skip the entire XML pile too! To demonstrate, this is what I've been thinking of: ,---- | (cond :message | (match "foo (bar) ([:number:])") | (do | (set! :bar "$1") | (set! :stuff "$2") | (conj! :tags "test stuff"))) | | (deftest message-match | "this is a foo bar 1234 message!" | | (== :bar "bar") | (== :stuff "1234") | (contains? :tags "test stuff")) `---- This would compile down to roughly the following C code (with the test excluded, for now): ,---- | gboolean | m_something(LogMsg *message, GString *subject) | { | GString *m1; /* ([:number:]) */ | | /* The subject must be at least as long as the static strings in the | pattern, if it's shorter, we don't match */ | if (subject->len < 8) | return FALSE; | | /* If we don't find part of the pattern, bail out. */ | if (strncmp(subject->str, "foo bar ") != 0) | return FALSE; | | if (!find_number(subject->str + 8, &m1)) | return FALSE; | | /* The whole stuff matched, yay! Lets fill in the fields. */ | | /* "bar" is a static string, fill it in as-is, no need to extract it | from the subject. */ | log_msg_set_value(msg, "bar", "bar", 3); | | /* :stuff is a number, so that needs to come from the | subject. Thankfully find_number() already did the extraction for | us. */ | log_msg_set_value(msg, "stuff", m1->str, m1->len); | | log_msg_set_tag_by_name(msg, "test stuff"); | | return TRUE; | } `---- I believe this is pretty efficient, the code generator can comment the the generated source nicely too. If we add named capture-groups, then the variables used can have meaningful names too! For example: (match "this (?<object>[:qstring:]) is good") In this case, the variable would be called m_object. However, the pattern-string is a bit awkward, when the rest is lispy, it also complicates the generator, so I was thinking of turning the pattern into a lispy syntax too: ,---- | (match "foo " (capture "bar") " " (capture :number)) `---- Or, with named capture groups added: ,---- | (match "this " (capture :qstring :as "object") " is good") `---- Of course, the action to take on a match can also contain another cond+match pair, so it can be nested as deep as one wishes to, the compiler will compile each cond into a separate function, and just call the appropriate one. Or perhaps inline them - that's an implementation detail, and doesn't really matter. The big advantage I see, is a DSL that is much closer to how I think, one that has the potential to produce a compact parser, one that is also easy to debug with conventional tools (gdb ;) because it compiles down to C. There is less run-time overhead too. Also, if implemented correctly, the generator would have the parser and the code generator entirely independent, so adding a different syntax would be as easy as writing a parser that produces the same abstract syntax tree the generator works with. This way, for those who're more familiar with C-like languages, the above matcher could be rewritten like this: ,---- | switch ($message) | { | case match("foo ", capture("bar"), " ", capture(:number:)) | { | set("bar", "$1"); | set("stuff", "$2"); | append($tags, "test stuff"); | } | } `---- And it would compile down to the exact same C code, accompanied by an appropriate autotools-based build system, so all you'd have to do in the end is to write the matcher, and issue the following commands: ,---- | $ matcher-generate test-patterns.pm | $ cd test-patterns | $ autoreconf -i && ./configure && make && make install `---- And finally, modify your syslog-ng.conf: ,---- | @module test-patterns | parser p_test { parser(test-patterns); }; `---- It does have downsides, though, namely that you need to regenerate & recompile the module and restart syslog-ng each time you modify the source, which is less convenient than just restarting syslog-ng itself. One also needs to learn a 'new' language to write pattern matchers in (but one has to learn patterndb too, anyway, so this isn't that big a disadvantage, especially since a more language-like thing is, in my opinion, easier to learn :). However, I believe that the advantages are worth it. For me, they certainly do, so I already started to hash out a proof of concept. So far, my PoC code can generate C functions, and supports a small subset of the DSL explained below. Do note that all this does not include corellation, because I believe that corellation should be separate from parsing, and a similar technique could be used to write advanced corellation setups - I will go into detail once I have a working proof of concept compiler for the parser. As a start, the DSL would support the following constructs: Top-level constructs: --------------------- * (cond :field condition action ...) Where :field can be any field, condition is a single condition function (see below) and action is a single action too (see even further below). Any number of condition-action paris can be specified, the first one matching will win, and the rest won't be tried. These two must always be paired together. * (deftest test-name "source string" test-conditions) Right now, lets ignore this. But in the long run, I want to be able to write down reasonably complex tests too. Not entirely sure yet what I need it to do though. Condition functions: -------------------- * (match pattern-spec) Matches a pattern-spec, simple as that. See below for the definition of the pattern-spec! * (not-match pattern-spec) The opposite of (match): action is triggered if the pattern does not match. * (exists?) Triggers the action if the field in the condition exists. * (not-exists?) Opposite of (exists?). * :default or (always) Always triggers, so one can do catch-all actions. Action functions: ----------------- * (set! :field value) * (clear! :field) These two should speak for themselves, I believe. * (conj! :field values) Conjoin (append) values to the specified field. Some fields (:tags) that need to be, will be treated specially, otherwise it just appends a separator (",") and the values. * (do actions...) Does all the specified actions. Pattern spec: ------------- The pattern can be built up from the sequence of the following things, in any order: * A plain string * (capture pattern-spec [:as name]) This just marks the pattern-spec as something to capture. While the implementation may produce code that captures things anyway, the only guarantee that something will be available for the actions, is to wrap it in (capture). The pattern-spec can be anything, it can contain nested captures. If :as name is specified, the capture will be named, and actions can refer to the captured thing by name. Otherwise, they need to refer it by a number. Each capture - named or anonymous - has a different number, starting from one, increasing with each occurrence of (capture). * Any of the following special keywords: * :number * :string * :qstring * :ipv4-address * :ipv6-address * :mac-address Future ideas ------------ Later, once the basics are ready and work, it would make sense to introduce a way to share common blocks of code: functions and perhaps variables. Conclusion ========== XML sucks, DSL rocks. Feedback appreciated, be that on the syntax, or the initially proposed features/functions/etc, or anything else. -- |8], who perhaps spent a little too much time near LISP-y stuff

4 8

Syslog-NG/Log Analyzer
by James McDonald 11 Sep '12

11 Sep '12

I finally got my syslog-ng servers to pass logs to the MySQL db, but was having some problems getting LogAnalzyer to access the db. I re-installed Log Analyzer but now I am getting an access denied error: Connection to the database server failed. Access denied for user 'syslog'@'FQDN_for_mysql_server.com' (using password: YES). Here is the output of the grants for the 'syslog' user: CODE: <http://kb.monitorware.com/> SELECT ALL mysql> show grants for 'syslog'@'FQDN_for_mysql_server.com'; +--------------------------------------------------------------------------- ---------+ | Grants for syslog(a)FQDN_for_mysql_server.com | +--------------------------------------------------------------------------- ---------+ | GRANT USAGE ON *.* TO 'syslog'@'FQDN_for_mysql_server.com' | | GRANT ALL PRIVILEGES ON `Syslog`.* TO 'syslog'@'FQDN_for_mysql_server.com' | +--------------------------------------------------------------------------- ---------+ 2 rows in set (0.00 sec) And here is what my config.php looks like: $CFG['DefaultSourceID'] = 'Source1'; $CFG['Sources']['Source1']['ID'] = 'Source1'; $CFG['Sources']['Source1']['Name'] = 'Syslog'; $CFG['Sources']['Source1']['ViewID'] = 'SYSLOG'; $CFG['Sources']['Source1']['SourceType'] = SOURCE_DB; $CFG['Sources']['Source1']['DBTableType'] = 'syslogng'; $CFG['Sources']['Source1']['DBType'] = DB_MYSQL; $CFG['Sources']['Source1']['DBServer'] = 'ustc-mysql'; $CFG['Sources']['Source1']['DBName'] = 'Syslog'; $CFG['Sources']['Source1']['DBUser'] = 'syslog'; $CFG['Sources']['Source1']['DBPassword'] = 'pTRl819ELI0viRmXaEUO'; $CFG['Sources']['Source1']['DBTableName'] = 'SystemEvents'; $CFG['Sources']['Source1']['DBEnableRowCounting'] = false; Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

2 3

[Bug 161] New: Syslog-ng stops processing after reload
by bugzilla＠bugzilla.balabit.com 10 Sep '12

10 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=161 Summary: Syslog-ng stops processing after reload Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bugzilla.syslogng(a)feystorm.net Type of the Report: --- Estimated Hours: 0.0 syslog-ng version: 3.3.4 OS: RHEL 6.2 Whenever I issue a reload to our syslog-ng instance syslog-ng stops processing messages and sits there doing nothing. The issue only seems to occur when the server is processing a high log volume. This server is part of a 2-node active/passive HA group. So we have 2 servers with the exact same configuration. Whenever I issue the reload on the server that is not currently active, it works fine. I can issue a failover and the server that was able to reload fine now hangs. The active server in this HA group processes around 35,000 messages per second. I havent posted the config because it is fairly large, however I can if desired. The most notable thing in it is that threading is enabled. Below is the backtrace of syslog-ng stuck in this state. I took a gcore, so I can gather any additional information that is needed. Also, if I try to do a full backtrace ('where full') in gdb on any thread other than thread 5, gdb segfaults. Not sure if that has any relevance or not. (gdb) info threads 5 Thread 0x7fb9c654ab00 (LWP 11040) 0x000000353cedd967 in writev () from /lib64/libc.so.6 4 Thread 0x7fb9c25fb700 (LWP 11049) 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 3 Thread 0x7fb9c2ffc700 (LWP 11048) 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 2 Thread 0x7fb9c39fd700 (LWP 11046) 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 * 1 Thread 0x7fb9c43fe700 (LWP 11045) 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 (gdb) where #0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c43fb9a0, msec=10000) at iv_method_epoll.c:73 #2 0x00007fb9c65cf1bb in iv_main () at iv_main.c:265 #3 0x00007fb9c65d3c36 in iv_work_thread (_thr=0x14fc610) at iv_work.c:196 #4 0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc6e0) at iv_thread.c:100 #5 0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000353cee570d in clone () from /lib64/libc.so.6 (gdb) thread 2 [Switching to thread 2 (Thread 0x7fb9c39fd700 (LWP 11046))]#0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 (gdb) where #0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c39fa9a0, msec=10000) at iv_method_epoll.c:73 #2 0x00007fb9c65cf1bb in iv_main () at iv_main.c:265 #3 0x00007fb9c65d3c36 in iv_work_thread (_thr=0x14fc8e0) at iv_work.c:196 #4 0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc9b0) at iv_thread.c:100 #5 0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000353cee570d in clone () from /lib64/libc.so.6 (gdb) thread 3 [Switching to thread 3 (Thread 0x7fb9c2ffc700 (LWP 11048))]#0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 (gdb) where #0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c2ff99a0, msec=10000) at iv_method_epoll.c:73 #2 0x00007fb9c65cf1bb in iv_main () at iv_main.c:265 #3 0x00007fb9c65d3c36 in iv_work_thread (_thr=0x15635c0) at iv_work.c:196 #4 0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x14fc380) at iv_thread.c:100 #5 0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000353cee570d in clone () from /lib64/libc.so.6 (gdb) thread 4 [Switching to thread 4 (Thread 0x7fb9c25fb700 (LWP 11049))]#0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 (gdb) where #0 0x000000353cee5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb9c65cfa3e in iv_epoll_poll (numfds=1, active=0x7fb9c25f89a0, msec=10000) at iv_method_epoll.c:73 #2 0x00007fb9c65cf1bb in iv_main () at iv_main.c:265 #3 0x00007fb9c65d3c36 in iv_work_thread (_thr=0x1566d20) at iv_work.c:196 #4 0x00007fb9c65d42a8 in iv_thread_handler (_thr=0x1566a70) at iv_thread.c:100 #5 0x000000353d2077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000353cee570d in clone () from /lib64/libc.so.6 (gdb) thread 5 [Switching to thread 5 (Thread 0x7fb9c654ab00 (LWP 11040))]#0 0x000000353cedd967 in writev () from /lib64/libc.so.6 (gdb) where #0 0x000000353cedd967 in writev () from /lib64/libc.so.6 #1 0x00007fb9c65820f8 in log_proto_file_writer_flush (s=0x171b5a0) at logproto.c:270 #2 0x00007fb9c658c403 in log_proto_flush (s=0x171b5a0) at logproto.h:86 #3 0x00007fb9c658ec3e in log_writer_flush (self=0x171adb0, flush_mode=LW_FLUSH_QUEUE) at logwriter.c:1032 #4 0x00007fb9c658f006 in log_writer_deinit (s=0x171adb0) at logwriter.c:1106 #5 0x00007fb9c63399d6 in log_pipe_deinit (s=0x171adb0) at ../../lib/logpipe.h:254 #6 0x00007fb9c633b3cd in affile_dw_deinit (s=0x171ac20) at affile.c:627 #7 0x00007fb9c63399d6 in log_pipe_deinit (s=0x171ac20) at ../../lib/logpipe.h:254 #8 0x00007fb9c633c211 in affile_dd_deinit_writer (key=0x171ad60, value=0x171ac20, user_data=0x0) at affile.c:974 #9 0x000000353e22a7a3 in g_hash_table_foreach () from /lib64/libglib-2.0.so.0 #10 0x00007fb9c633c349 in affile_dd_deinit (s=0x1539990) at affile.c:996 #11 0x00007fb9c6575d86 in log_pipe_deinit (s=0x1539990) at logpipe.h:254 #12 0x00007fb9c65761b4 in log_dest_group_deinit (s=0x1539ef0) at dgroup.c:76 #13 0x00007fb9c656f54c in log_pipe_deinit (s=0x1539ef0) at logpipe.h:254 #14 0x00007fb9c657078c in log_center_deinit (self=0x14f35c0) at center.c:576 #15 0x00007fb9c6571296 in cfg_deinit (cfg=0x14f1ca0) at cfg.c:261 #16 0x00007fb9c6590cac in main_loop_reload_config_apply () at mainloop.c:494 #17 0x00007fb9c6590aab in main_loop_io_worker_job_complete (self=0x156d160) at mainloop.c:417 #18 0x00007fb9c65d3ccc in iv_work_event (_pool=0x14f1bf0) at iv_work.c:221 #19 0x00007fb9c65d2862 in iv_event_run_pending_events (_dummy=0x0) at iv_event.c:67 #20 0x00007fb9c65d2cb5 in iv_event_raw_got_event (_this=0x7fb9c654aa18) at iv_event_raw.c:82 #21 0x00007fb9c65cefec in iv_run_active_list (active=0x7fff33ed17b0) at iv_main.c:219 #22 0x00007fb9c65cf1cc in iv_main () at iv_main.c:269 #23 0x00007fb9c65913e9 in main_loop_run () at mainloop.c:731 #24 0x00000000004016d0 in main () -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

[Bug 195] New: syslog 3.3.6.90 stats for src. unix-dgram do not show filesystem name
by bugzilla＠bugzilla.balabit.com 10 Sep '12

10 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=195 Summary: syslog 3.3.6.90 stats for src.unix-dgram do not show filesystem name Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 When using the syslog-ng-ctl to get statistics from syslog-ng the source stats look like; src.unix-dgram;local#1;;a;processed;2025 src.internal;local#0;;a;processed;54 src.file;local#2;/proc/kmsg;a;processed;273 for a configuration of source local { internal( tags("syslog.internal") ); unix-dgram("/dev/log" max_connections(10000) log_iw_size(1000000) ); file("/proc/kmsg" program_override("kernel") ); }; The file name of /proc/kmsg shows in the stats line for it, but the file name of /dev/log does not show in the stats line related to it. I think this is a bug and the /dev/log should show in the stats line. Evan. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 5