- syslog-ng - lists.balabit.hu

issue during start up
by Jason Kojro-Badziak 15 May '12

15 May '12

Hello all: I am afraid I am having another issue which I would like some help with. When I try to start syslog-ng, I am getting this error: Starting syslog-ng: Error initializing raw socket, spoof-source support disabled; Syslog-ng then starts, but it doesn't appear to be forwarding messages. This is extremely strange as I have syslog-ng installed and running with the exact same config file on another system without any issues. I am sure the problem is that I'm an idiot, but I would greatly appreciate any help with the error I am having. Thank you! Jason Kojro-Badziak Monolith Software Staff Engineer 311 North 2nd Street, Suite #302 St. Charles, IL 60174 Office: 312-957-6470 x3010 Email: jbadziak(a)monolith-software.com<mailto:jbadziak@monolith-software.com/jb.022411>

1 0

(no subject)
by pedro castro 13 May '12

13 May '12

Learn how to turn successfuI from home http://www.icarografica.it/kkimom.php?jdolpin=349 Sun, 13 May 2012 12:17:24 ____________ We DIDset him free--me and Tom. venize addyson

1 0

RFC: Applying transformations to a whole log message
by Gergely Nagy 10 May '12

10 May '12

Hi! In the GeoIP thread[1], I started to play with the idea of introducing another way to modify messages. So far, we have rewrite, which can set new values associated with a message, or change existing ones - one at a time. We also have template functions, which one can use how a specific value will be formatted. Again, pretty much one at a time. What syslog-ng lacks right now, is a way to apply a transformation to a message as a whole, a transformation that will take effect right there, right then, instead of making a modified copy like value-pairs() does. (value-pairs() also suffers from the problem that to be useful, it needs explicit support elsewhere: among the template functions, or within the destination driver). What I wish for, is to be able to apply any number of transformation functions to a whole LogMessage. Whether the transformations rewire the key names, or change values, I'd love to be able to just tell syslog-ng, that "here, take this message, go out and prosper, make it better, whatever it takes!" - and it would do just that. To give a few examples, I'd love to be able to do any and all of the following: * Ask syslog-ng to take a message, and look up every IP address associated with it (for simplicity's sake, lets assume every such address is stored within a key that ends with "_IP", and no other keys end with the same suffix) in a GeoIP database, and put the result in keys that have a "geo." prefix, followed by the original key name. * Ask syslog-ng to take a message, and rename the keys according to various rules I set - similar how value-pairs()' rekeying works, possibly following the same syntax. For example, I want to take all ".json.*" keys, remove the prefix, and uppercase the names. Then, I want to replace all leading dots with an underscore in whatever keys remain. * Ask syslog-ng to remove keys completely. I don't care about the DATE field, because I receive CEE-enabled messages only, and they come with a high-precision date field anyway, called a "timestamp". * I also want to drop every key where the value matches a certain pattern. Or perhaps not drop them, but anonymize the value.. For example, I might not like the word "plasson", so much so, that whatever key contains it, I never want to see it. I also want to pull a prank, because it just happens to be April 1st, so I want to replace every occurence of "Linux" with "Emacs" within a LogMessage, in every single key. * Since we're applying transformations, might aswell do what rewrite does too, and be able to set stuff - we already do subst. For example, I want to anonymize all the IP addresses. I don't mind the country-codes exposed, but I don't want the IPs in my logs. * I want to be able to compose all of the above, chain them together, so one gets executed after the other, and in the end, the LogMessage will end up with their combined result. For the above, I propose the following syntax: ,---- | map m_do_stuff { | geoip("*_IP", target-prefix("geo.")) | rekey(".json.*", | shift(6) uppercase()) | rekey(".*", replace(".", "_")) | filter-out(key("DATE")) | filter-out(value("plasson", type(substring))) | subst(value("Linux", "Emacs", type(substring))) | set(key("*_IP", "<anonymized>")) | }; `---- Of course, this differs a bit from the syntax used in rewrite, and to be honest, intentionally so. I could never learn to love rewrite's way of set("new-value", value("key-name")). Nevertheless, the syntax can be changed to be similar to rewrite, the functionality would remain the same even then. And how to use this? destination d_something { source(s_something); map(m_do_stuff); ... } And in case we want to tie it to a condition, then: map(m_do_stuff, condition(filter(f_filter_condition))) I don't think I'd want to support specifying maps in-line, but I suppose that could be done aswell. Basically, this would be rewrite on steroids, with the ability to modify keys and values in bulk aswell. The major advantage this would have is the ability to work not only on a single key, but apply transformations to any number of key-value pairs, changing either of them or both. If architectured well, it could even be fast, on par with rewrite if it has to do similar things. I mean, the following two should be equally fast: ,---- | map m_set_host { | set(key("HOST", "myhost")); | }; `---- ,---- | rewrite r_set_host { | set("myhost", value("HOST")); | }; `---- For this to work, and for optimisations to be made possible, the implementation will have to be clever, and able to take shortcuts. I have a few ideas about that too, but that's a topic for a later time: let's see first if the idea is deemed useful, and if the syntax I came up with makes sense to anyone else. What do YOU think? Would you have a use for a way to configure bulk transformations? If so, what other transformations would you find interesting? -- |8]

2 2

Connections not closing on syslog-ng 2.1.4 server
by Robert Nickel 10 May '12

10 May '12

I have a busy syslog-ng server that is collecting a large volume of logs. This problem is causing me grief and I am wondering if it is related to having flow_control enabled on the host. Any help would be greatly appreciated. The connection limit on the inbound tcp connection is slowly being exhausted by multiple connections from each client slowly building until the server stops working. The source is configured as such: source s_remote { tcp(ip(0.0.0.0) port(5140) log_iw_size(204800) max-connections(2048)); }; The bulk of the logs are being sent to other syslog-ng servers running on the same host. e.g.: destination child1 { tcp("127.0.0.1" port (5148)); }; All of the log lines have the same setup: log { source (s_remote); filter(f_q1); destination(d_q1); flags(flow-control,final); }; This setup is working very well with throughput in excess of 35000 messages per second but the whole thing blows up every couple of days due to running out of connections on the source. Before it dies, for a couple of days, I see a number of these errors in the logs: Number of allowed concurrent connections exceeded; num='2048', max='2048' Usually, the actual number of connections is a couple of hundred above the 2048 number listed here. (netstat -an | fgrep -v ESTABLISHED | fgrep :5140). The actual server count hovers around 1300 of which there are ~1000 actively logging. Options in use are: create_dirs(yes); dir_perm (0755); dns_cache_expire(28800); dns_cache (yes); flush_lines(10); flush_timeout(2048); frac_digits(3); keep_hostname(no); log_fetch_limit(100); log_fifo_size (2048); log_iw_size(100); long_hostnames(off); perm(0644); stats_freq(300); time_reopen(10); time_sleep(10); ts_format("iso"); use_dns(yes); use_fqdn(no); use_time_recvd(no); Thank you! --Robert

2 2

Log Directories created with IP and not hostnames
by James McDonald 10 May '12

10 May '12

Hi, I am new to syslog-ng, as well as this board. I posted a question before my account was verified, so I'm worried it didn't get posted, or I missed the responses. The Windows clients in my environment are creating their log directories named after their IP's and not their hostnames. The RHEL clients are using their hostnames. I would like to get the Windows to use their hostnames too. All the config files look to be configured correctly. Does anyone have any suggestions on what I can check or try to get the Windows clients to create their log directories using their hostnames? Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

1 0

[Bug 179] New: SDATA from Loggen Not Recorded
by bugzilla＠bugzilla.balabit.com 10 May '12

10 May '12

https://bugzilla.balabit.com/show_bug.cgi?id=179 Summary: SDATA from Loggen Not Recorded Product: syslog-ng Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bmehne(a)gmail.com Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=60) --> (https://bugzilla.balabit.com/attachment.cgi?id=60) Syslog-ng conf I have been trying to find some API to write SDATA to mongodb, but it seems that syslog-ng is not parsing the sdata correctly or loggen is not being called correctly. I am calling loggen with: loggen -r 5 -P -p "[syslog@0 uid=\"system\" id=\"4937903063198901248\" sid=\"-\" svchome=\"/run/cronsvc/01/1\" svcid=\"dummy\" name=\"JobExecutionStateManagementJob\" tid=\"__system__\" type=\"SYSTEM\" cid=\"-\" eid=\"-\"]" -S -s 1024 -i -I 2000 localhost 1000 My syslog-ng.conf is attached. I have compiled and run with syslog-ng 3.3.1 (from tar), 3.3.4 (as in debian repos), and 3.4 (from github). I have also tried with the mongodb driver patched to insert rather than upsert (from algernon github repo, flat-insert branch). My testing is taken from http://www.syslog.org/forum/index.php?topic=1233.0 My /var/log/messages.json spits out empty braces (e.g. { } ) for each message, and mongodb does record "SDATA" : "[meta sequenceId=\"70\"]",but that is not the intended sdata. Any help would be appreciated. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

if no host; do use_dns
by Lance Laursen 09 May '12

09 May '12

Hi list, With keep_hostname(yes), when the syslog-ng box receives a message which does not have a host field, you end up with the last hop IP as the hostname. Is there already a way to tell sysng to use_dns() when this situation is encountered? I'd like to continue using keep_hostname(yes) for all log messages which do have a hostname included, but in the instance that hostname is not included, have syslog-ng do a reverse dns lookup in order to resolve hostname. Thanks,

2 1

Windows clients using IP instead of hostname to create directories
by James McDonald 09 May '12

09 May '12

I have a number of Windows clients sending logs back to a central log server running RHEL 5.5. The Linux clients each create a directory for their logs using their respective hostnames as the name of the directory. The Windows clients, however, are using their IP addresses to name their respective directories. The hostnames and IP's are resolving correctly (tested via nslookup and ping) and, from what I can tell, the config files seem to be correct. Is there anything I can double-check or test that might help resolve this issue? I am using version 4.0.1 on the Windows clients and they are running Server 2008R2. Thanks, James McDonald System Administrator RHCSA, Security+ jmcdonald(a)lce.com

1 0

How do I get pdbtool to create version 4 patterndb.xml?
by Kurt Yoder 08 May '12

08 May '12

All, I'm trying to create patterndb version 4 XML so I can make use of its additional features over version 3. However pdbtool seems only to output version 3 XML. I verified that all of my component XML files are version 4, and used pdbtool from syslog-ng 3.3.5: $ /opt/syslog-ng/bin/pdbtool merge --pdb=/tmp/patterndb.xml --recursive --glob='*.pdb' --directory=/etc/syslog-ng/patterndb.d/ The generated file is version 3: $ head -2 /tmp/patterndb.xml <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2012-05-07'> How do I generate version 4 XML?

2 1

[Bug 177] New: crashes in syslog-ng-2.0.10-1 in Fedora i386
by bugzilla＠bugzilla.balabit.com 07 May '12

07 May '12

https://bugzilla.balabit.com/show_bug.cgi?id=177 Summary: crashes in syslog-ng-2.0.10-1 in Fedora i386 Product: syslog-ng Version: 2.0.x Platform: Other OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: rcp(a)sentientmeat.ca Type of the Report: bug Estimated Hours: 0.0 Let me start by apologizing for the age of the software and the platform. Our situation is that we need to support millisecond timestamps on an older branch of an embedded platform. This platform was previously happily running the Fedora Core 4 package syslog-ng-1.6.11-1.fc4.i386.rpm. We took the syslog-ng-2.0.10-1.fc10.src.rpm and eventlog-0.2.7-1.fc10.src.rpm packages from Fedora 10, and have recompiled them for Fedora 4. Unfortunately we've ended up with stability issues, in that the daemon crashes. We haven't yet found a recipe for reproducing the problem, but it happens daily in our lab. We'd like to know if these issues have been encountered before and if there are fixes available. We've attempted upgrading past the issue, by recompiling Fedora 12's syslog-ng-2.1.4-6.fc12.src.rpm and eventlog-0.2.7-4.fc12.src.rpm, and when that also suffered crashes, we added glib2-2.22.5-1.fc12.src.rpm. We've looked at syslog-ng 3, but the long list of dependancies that need to be updated for this platform is rather daunting. I'm including below the backtraces from our corefiles. We've seen the following many times. The environment included: export MALLOC_CHECK_=3 export LIBC_FATAL_STDERR_=1 export MALLOC_PERTURB_=171 core.2196.b50 warning: Can't read pathname for load map: Input/output error. Failed to read a valid object file image from memory. Core was generated by `/sbin/syslog-ng -p /var/run/syslogd.pid'. Program terminated with signal 6, Aborted. #0 0x002f0410 in ?? () (gdb) bt #0 0x002f0410 in ?? () #1 0xbff5de20 in ?? () #2 0x00000006 in ?? () #3 0x00000894 in ?? () #4 0x4ddda1f8 in raise () from /lib/tls/libc.so.6 #5 0x4dddb948 in abort () from /lib/tls/libc.so.6 #6 0x4de0f52a in __libc_message () from /lib/tls/libc.so.6 #7 0x4de15424 in _int_free () from /lib/tls/libc.so.6 #8 0x4de1595f in free () from /lib/tls/libc.so.6 #9 0x4dfb8b44 in g_free () from /usr/lib/libglib-2.0.so.0 #10 0x0806000a in log_reader_free (s=0x8ece698) at logreader.c:457 #11 0x08054274 in log_pipe_unref (self=0x8ece698) at logpipe.c:51 #12 0x0805f582 in log_reader_fd_finalize (source=0x8ecf560) at logreader.c:167 #13 0x4dfb0e7a in ?? () from /usr/lib/libglib-2.0.so.0 #14 0x4dfb250e in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 #15 0x4dfb53f6 in ?? () from /usr/lib/libglib-2.0.so.0 #16 0x4dfb58d8 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 #17 0x0804b73e in main_loop_run (cfg=0xbff5e940) at main.c:167 #18 0x0804c0e4 in main (argc=3, argv=0xbff5e9e4) at main.c:483 (gdb) frame 6 #6 0x4de0f52a in __libc_message () from /lib/tls/libc.so.6 (gdb) info registers eax 0x0 0 ecx 0x894 2196 edx 0x6 6 ebx 0x4ded7ff4 1307410420 esp 0xbff5df54 0xbff5df54 ebp 0xbff5e5a4 0xbff5e5a4 esi 0x0 0 edi 0xc 12 eip 0x4de0f52a 0x4de0f52a <__libc_fatal> eflags 0x202 [ IF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0xc010007b -1072693125 fs 0x0 0 gs 0x33 51 (gdb) x/32x 0xbff5df54 0xbff5df54: 0x0000000c 0xbff5e090 0x00000400 0x4decd3b4 0xbff5df64: 0x00000017 0xbff5eeab 0x0000000f 0x4decd3cd 0xbff5df74: 0x00000002 0x4decd46c 0x0000001f 0x4decd3d1 0xbff5df84: 0x00000004 0xbff5e607 0x00000008 0x4decd3d7 0xbff5df94: 0x00000005 0x00000000 0x4e01317c 0x08eca1b0 0xbff5dfa4: 0x4e01317c 0x00000000 0x00000000 0x4decd3d7 0xbff5dfb4: 0x00000005 0xbff5dfd0 0x00000025 0x00000000 0xbff5dfc4: 0x00000000 0x00000000 0xffffffff 0xbff5e607 (gdb) x/s 0x4decd3b4 0x4decd3b4: "*** glibc detected *** %s: %s: 0x%s ***\n" (gdb) x/s 0xbff5eeab 0xbff5eeab: "/sbin/syslog-ng" (gdb) x/s 0x4decd3cd 0x4decd3cd: ": %s: 0x%s ***\n" (gdb) x/s 0x4decd46c 0x4decd46c: "double free or corruption (out)" (gdb) x/s 0x4decd3d1 0x4decd3d1: ": 0x%s ***\n" (gdb) x/s 0xbff5e607 0xbff5e607: "08ee8bd0" # Same error, but without the Heap Consistency Checking environment variables: core.2229.a34 warning: Can't read pathname for load map: Input/output error. Failed to read a valid object file image from memory. Core was generated by `/sbin/syslog-ng -p /var/run/syslogd.pid'. Program terminated with signal 11, Segmentation fault. #0 0x4de15982 in free () from /lib/tls/libc.so.6 #0 0x4de15982 in free () from /lib/tls/libc.so.6 #1 0x4dfb8b44 in g_free () from /usr/lib/libglib-2.0.so.0 #2 0x0806000a in log_reader_free (s=0x9532ae8) at logreader.c:457 #3 0x08054274 in log_pipe_unref (self=0x9532ae8) at logpipe.c:51 #4 0x0805f582 in log_reader_fd_finalize (source=0x9538190) at logreader.c:167 #5 0x4dfb0e7a in ?? () from /usr/lib/libglib-2.0.so.0 #6 0x4dfb250e in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 #7 0x4dfb53f6 in ?? () from /usr/lib/libglib-2.0.so.0 #8 0x4dfb58d8 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 #9 0x0804b73e in main_loop_run (cfg=0xbfb8af40) at main.c:167 #10 0x0804c0e4 in main (argc=3, argv=0xbfb8afe4) at main.c:483 # This traceback was a one-time error: core.2351.b79 warning: Can't read pathname for load map: Input/output error. Failed to read a valid object file image from memory. Core was generated by `/sbin/syslog-ng -p /var/run/syslogd.pid'. Program terminated with signal 11, Segmentation fault. #0 0x080555c2 in log_center_queue (s=0x88c50d0, msg=0x88bb918, path_flags=0) at center.c:354 354 center.c: No such file or directory. in center.c #0 0x080555c2 in log_center_queue (s=0x88c50d0, msg=0x88bb918, path_flags=0) at center.c:354 #1 0x080547e2 in log_pipe_queue (s=0x88c50d0, msg=0x88bb918, path_flags=0) at logpipe.h:72 #2 0x080547a9 in log_source_group_queue (s=0x88c4568, msg=0x88bb918, path_flags=0) at sgroup.c:140 #3 0x080542c9 in log_pipe_queue (s=0x88c4568, msg=0x88bb918, path_flags=0) at logpipe.h:72 #4 0x080542a5 in log_pipe_forward_msg (self=0x88c5658, msg=0x88bb918, path_flags=0) at logpipe.c:60 #5 0x080599e6 in log_pipe_queue (s=0x88c5658, msg=0x88bb918, path_flags=0) at logpipe.h:72 #6 0x080599c2 in afsocket_sc_queue (s=0x88bd2c8, msg=0x88bb918, path_flags=0) at afsocket.c:261 #7 0x0805f0fc in log_pipe_queue (s=0x88bd2c8, msg=0x88bb918, path_flags=0) at logpipe.h:72 #8 0x0805f0d8 in log_source_queue (s=0x88bd2f8, msg=0x88bb918, path_flags=0) at logsource.c:49 #9 0x0805f72f in log_pipe_queue (s=0x88bd2f8, msg=0x88bb918, path_flags=0) at logpipe.h:72 #10 0x0805f700 in log_reader_handle_line (self=0x88bd2f8, line=0x88d9bd0 "<131>Apr 18 23:14:51 root[9796]: /usr/sw", ' ' <repeats 24 times>, "ipcMsg.cpp:297", ' ' <repeats 24 times>, "(BASE_IPC - 0x00000000) MplThread(4)@mgmtplane(9)", ' ' <repeats 21 times>, "ERROR /usr/sw/loa"..., length=307, saddr=0x0, parse_flags=4) at logreader.c:210 #11 0x0805f9c8 in log_reader_iterate_buf (self=0x88bd2f8, saddr=0x0, flush=0, msg_count=0xbfa7c230) at logreader.c:295 #12 0x0805fe58 in log_reader_fetch_log (self=0x88bd2f8, fd=0x88c46b0) at logreader.c:400 #13 0x0805f541 in log_reader_fd_dispatch (source=0x88b50c8, callback=0, user_data=0x0) at logreader.c:153 #14 0x4dfb23ee in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 #15 0x4dfb53f6 in ?? () from /usr/lib/libglib-2.0.so.0 #16 0x4dfb58d8 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 #17 0x0804b73e in main_loop_run (cfg=0xbfa7c4d0) at main.c:167 #18 0x0804c0e4 in main (argc=3, argv=0xbfa7c574) at main.c:483 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3