Hey everyone, Sorry to ask what is probably an easy question. I'm attempting to use the new db-parser feature in syslog-ng to parse Windows events generated by Snare. What I've attempted to do is write a simple XML file that will match one simple rule. This way I can test that everything is functioning properly and I can move on from there. Unfortunately, it's not working, so I'm unsure as to whether it is my rule or my installation. So I have a few basic questions that will hopefully clear up my confusion! - I did not install syslog-ng from source. I downloaded the .deb for version 3.01 and used that. Does this even include db-parser? (I'm assuming yes because it does not cause an error from the config file alone.) - My installation is in /opt/syslog-ng, so I've put the example XML files (http://www.balabit.com/downloads/files/patterndb/) in /opt/syslog-ng/var. Is this the right location? Other than that, here is my syslog-ng.conf file: @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { }; ###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); }; source s_net { udp(); tcp(); }; parser p_db { db-parser(); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_parsed { file("/var/log/remote/parsed.log"); }; log { source(s_local); destination(d_messages); }; log { source(s_net); destination(d_parsed); parser(p_db); }; And here is my windows.xml file: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-05-07'> <ruleset name='windows'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='nate' id='1' class='system'> <patterns> <pattern>540</pattern> </patterns> <description>This is a terrible terrible message to receive. Game over man! Game over!</description> </rule> </rules> </ruleset> </patterndb> As you see, to test I just want to match on the number '540' and put that log message in /var/log/remote/parsed.log. I have verified that messages that should match this are arriving at the machine when I send them. If i turn off the db-parser, the messages appear in the parsed.log file. If there is any documentation that explains the basics of this, sorry that I missed it. I've only been able to find some of the other messages on this mailing list and the blog entries on balabit. Thanks for any help! Nate
On Thu, 2009-05-07 at 16:26 -0400, Nate Hausrath wrote:
Hey everyone,
Sorry to ask what is probably an easy question.
I'm attempting to use the new db-parser feature in syslog-ng to parse Windows events generated by Snare. What I've attempted to do is write a simple XML file that will match one simple rule. This way I can test that everything is functioning properly and I can move on from there. Unfortunately, it's not working, so I'm unsure as to whether it is my rule or my installation. So I have a few basic questions that will hopefully clear up my confusion!
- I did not install syslog-ng from source. I downloaded the .deb for version 3.01 and used that. Does this even include db-parser? (I'm assuming yes because it does not cause an error from the config file alone.)
- My installation is in /opt/syslog-ng, so I've put the example XML files (http://www.balabit.com/downloads/files/patterndb/) in /opt/syslog-ng/var. Is this the right location?
the default location for the patterndb file is ${localstatedir}/patterndb.xml, but you can override that using the file option, e.g. db-parser(file(/path/to/patterndb)); the reason it is in var that in the future we assume that this file is going to be generated from several source files automatically.
Other than that, here is my syslog-ng.conf file:
@version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html #
options { };
###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); };
source s_net { udp(); tcp(); };
parser p_db { db-parser(); };
###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_parsed { file("/var/log/remote/parsed.log"); };
log { source(s_local); destination(d_messages); };
log { source(s_net); destination(d_parsed); parser(p_db); };
the log statements define a 'pipeline', thus your messages will reach the db-parser() only after having been written to the d_parsed destination.
And here is my windows.xml file:
<?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-05-07'> <ruleset name='windows'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='nate' id='1' class='system'> <patterns> <pattern>540</pattern> </patterns> <description>This is a terrible terrible message to receive. Game over man! Game over!</description> </rule> </rules> </ruleset> </patterndb>
As you see, to test I just want to match on the number '540' and put that log message in /var/log/remote/parsed.log. I have verified that messages that should match this are arriving at the machine when I send them. If i turn off the db-parser, the messages appear in the parsed.log file.
All messages should end up in your parsed.log file anyway. In order to only match those which matched any of the patterns, you need to put filters into your log statement.
If there is any documentation that explains the basics of this, sorry that I missed it. I've only been able to find some of the other messages on this mailing list and the blog entries on balabit.
Just feel free to ask about it here, I'm trying to give you timely answers, as other source of information about db-parser() is scarse. In the documentation you will find information about parsers in general, and db-parser() is one such parser, basically behaving quite similar to csv-parser(), except it uses a database to extract information from log messages. Also, you can find a working example in my presentation at OSDC: http://people.balabit.hu/bazsi/slides/osdc-2009-syslog-ng-3.0.odp -- Bazsi
Thanks for the response. It looks like there may be a problem with my install or a bug somewhere. With the config file below, I'm still not getting messages in my /var/log/remote/parsed.log file. However, when I comment out "parser(p_db);" in the log section of the config, messages begin showing up. After a bit more investigations, when I receive the first remote log message over UDP, the following appears in my /var/log/messages: kernel: : [89941.138626] syslog-ng[16473]: segfault at 00000010 eip 08063e49 esp bfa1a490 error 4 Even after this, the process is still running and the ports are still open. At this point, I'm not really sure how to diagnose the problem. -Nate
Other than that, here is my syslog-ng.conf file:
@version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html #
options { };
###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); };
source s_net { udp(); tcp(); };
parser p_db { db-parser(); };
###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_parsed { file("/var/log/remote/parsed.log"); };
log { source(s_local); destination(d_messages); };
log { source(s_net); destination(d_parsed); parser(p_db); };
the log statements define a 'pipeline', thus your messages will reach the db-parser() only after having been written to the d_parsed destination.
And here is my windows.xml file:
<?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-05-07'> <ruleset name='windows'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='nate' id='1' class='system'> <patterns> <pattern>540</pattern> </patterns> <description>This is a terrible terrible message to receive. Game over man! Game over!</description> </rule> </rules> </ruleset> </patterndb>
As you see, to test I just want to match on the number '540' and put that log message in /var/log/remote/parsed.log. I have verified that messages that should match this are arriving at the machine when I send them. If i turn off the db-parser, the messages appear in the parsed.log file.
All messages should end up in your parsed.log file anyway. In order to only match those which matched any of the patterns, you need to put filters into your log statement.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Fri, 2009-05-08 at 10:23 -0400, Nate Hausrath wrote:
Thanks for the response.
It looks like there may be a problem with my install or a bug somewhere. With the config file below, I'm still not getting messages in my /var/log/remote/parsed.log file. However, when I comment out "parser(p_db);" in the log section of the config, messages begin showing up.
After a bit more investigations, when I receive the first remote log message over UDP, the following appears in my /var/log/messages:
kernel: : [89941.138626] syslog-ng[16473]: segfault at 00000010 eip 08063e49 esp bfa1a490 error 4
Even after this, the process is still running and the ports are still open. At this point, I'm not really sure how to diagnose the problem.
Hmm... syslog-ng is automatically restarting itself. I'm working in doing a 3.0.2 release right now and there are really a lot of fixes in that. (just check out the NEWS file in the git repository) Hopefully I can finish this today, with shiny installers for all platforms. If the above problem persists, I can help diagnose it. -- Bazsi
Hi, Sorry for answering so slowly... On Fri, 2009-05-08 at 10:23 -0400, Nate Hausrath wrote:
Thanks for the response.
It looks like there may be a problem with my install or a bug somewhere. With the config file below, I'm still not getting messages in my /var/log/remote/parsed.log file. However, when I comment out "parser(p_db);" in the log section of the config, messages begin showing up.
After a bit more investigations, when I receive the first remote log message over UDP, the following appears in my /var/log/messages:
kernel: : [89941.138626] syslog-ng[16473]: segfault at 00000010 eip 08063e49 esp bfa1a490 error 4
Even after this, the process is still running and the ports are still open. At this point, I'm not really sure how to diagnose the problem.
Basically you have two problems, let's see one after the other. 1, You are using syslog-ng OSE which has support for db-parser pattern version 1 only while you have a version 2 pattern xml (version 2 is only included in the PE and the website provided xmls also in version 2). This way you end up without any actual rule and therefore no matching or parsing would occur. You can find an XML schema in the distribution which specifies the xml format to be used. I am currently working on porting the version 2 format to OSE and sorry for any inconvenience I might caused with the different versions. Meanwhile you can use the version 1 style xml: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='1' pub_date='2009-05-07'> <program name='windows'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='nate' id='1' class='system'> <pattern>540</pattern> <description>This is a terrible terrible message to receive. Game over man! Game over!</description> </rule> </rules> </program> </patterndb> 2, You have a segfault. I would need a backtrace or a core file to track down the problem. To get a core file you can use the --enable-core option to syslog-ng and gdb after that to get a backtrace. Sorry again for the version problem, I will push an update to my git tree with the new version 2 xml handling. best, Marton
-Nate
Other than that, here is my syslog-ng.conf file:
@version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html #
options { };
###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel: ")); };
source s_net { udp(); tcp(); };
parser p_db { db-parser(); };
###### # destinations destination d_messages { file("/var/log/messages"); }; destination d_parsed { file("/var/log/remote/parsed.log"); };
log { source(s_local); destination(d_messages); };
log { source(s_net); destination(d_parsed); parser(p_db); };
the log statements define a 'pipeline', thus your messages will reach the db-parser() only after having been written to the d_parsed destination.
And here is my windows.xml file:
<?xml version='1.0' encoding='UTF-8'?> <patterndb version='2' pub_date='2009-05-07'> <ruleset name='windows'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='nate' id='1' class='system'> <patterns> <pattern>540</pattern> </patterns> <description>This is a terrible terrible message to receive. Game over man! Game over!</description> </rule> </rules> </ruleset> </patterndb>
As you see, to test I just want to match on the number '540' and put that log message in /var/log/remote/parsed.log. I have verified that messages that should match this are arriving at the machine when I send them. If i turn off the db-parser, the messages appear in the parsed.log file.
All messages should end up in your parsed.log file anyway. In order to only match those which matched any of the patterns, you need to put filters into your log statement.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D
Thanks for the reply. Unfortunately, my gdb skills are absolutely terrible. Here is how I have tried to do it, which is probably wrong: # gdb /opt/syslog-ng/sbin/syslog-ng 2>&1 | tee gdb-syslog-ng.txt GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu"... (no debugging symbols found) (gdb) handle SIG33 pass nostop noprint Signal Stop Print Pass to program Description SIG33 No No Yes Real-time event 33 (gdb) set pagination 0 (gdb) run --no-caps --enable-core Starting program: /opt/syslog-ng/sbin/syslog-ng --no-caps --enable-core (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) Executing new program: /opt/syslog-ng/libexec/syslog-ng [Thread debugging using libthread_db enabled] Program exited normally. (gdb) backtrace full No stack. (gdb) info registers The program has no registers now. (gdb) Hopefully I am doing something that is simple to fix. Also, when I use --enable-core, where does it put the core file? Thanks! Nate On Mon, May 11, 2009 at 9:26 AM, ILLES, Marton <illes.marton@balabit.hu> wrote:
Hi,
Sorry for answering so slowly...
On Fri, 2009-05-08 at 10:23 -0400, Nate Hausrath wrote:
Thanks for the response.
It looks like there may be a problem with my install or a bug somewhere. With the config file below, I'm still not getting messages in my /var/log/remote/parsed.log file. However, when I comment out "parser(p_db);" in the log section of the config, messages begin showing up.
After a bit more investigations, when I receive the first remote log message over UDP, the following appears in my /var/log/messages:
kernel: : [89941.138626] syslog-ng[16473]: segfault at 00000010 eip 08063e49 esp bfa1a490 error 4
Even after this, the process is still running and the ports are still open. At this point, I'm not really sure how to diagnose the problem.
Basically you have two problems, let's see one after the other.
1, You are using syslog-ng OSE which has support for db-parser pattern version 1 only while you have a version 2 pattern xml (version 2 is only included in the PE and the website provided xmls also in version 2). This way you end up without any actual rule and therefore no matching or parsing would occur.
You can find an XML schema in the distribution which specifies the xml format to be used.
I am currently working on porting the version 2 format to OSE and sorry for any inconvenience I might caused with the different versions. Meanwhile you can use the version 1 style xml:
<?xml version='1.0' encoding='UTF-8'?> <patterndb version='1' pub_date='2009-05-07'> <program name='windows'> <pattern>MSWinEventLog</pattern> <rules> <rule provider='nate' id='1' class='system'> <pattern>540</pattern> <description>This is a terrible terrible message to receive. Game over man! Game over!</description> </rule> </rules> </program> </patterndb>
2, You have a segfault. I would need a backtrace or a core file to track down the problem. To get a core file you can use the --enable-core option to syslog-ng and gdb after that to get a backtrace.
Sorry again for the version problem, I will push an update to my git tree with the new version 2 xml handling.
best,
Marton
2, You have a segfault. I would need a backtrace or a core file to track down the problem. To get a core file you can use the --enable-core option to syslog-ng and gdb after that to get a backtrace.
Sorry again for the version problem, I will push an update to my git tree with the new version 2 xml handling.
best,
Marton
Hi Marton, I finally have the backtrace. If you need to core file as well, let me know. Core was generated by `/opt/syslog-ng-src/sbin/syslog-ng --no-caps --enable-core'. Program terminated with signal 11, Segmentation fault. [New process 5429] #0 0x08064117 in log_pipe_queue (s=0x0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 121 s->queue(s, msg, path_options); (gdb) bt full #0 0x08064117 in log_pipe_queue (s=0x0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #1 0x080640c5 in log_process_pipe_queue (s=0x80a0a38, msg=0x80a53c0, path_options=0xbfd749e4) at logprocess.c:73 self = (LogProcessPipe *) 0x80a0a38 #2 0x08054874 in log_pipe_queue (s=0x80a0a38, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #3 0x0805483c in log_pipe_forward_msg (self=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.c:72 No locals. #4 0x0806446c in log_multiplexer_queue (s=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logmpx.c:115 self = (LogMultiplexer *) 0x80a09c0 i = 1 local_options = {flow_control = 0, matched = 0xbfd7498c} matched = 1 delivered = 1 last_delivery = 0 fallback = 1 #5 0x08064490 in log_pipe_queue (s=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #6 0x080643f8 in log_multiplexer_queue (s=0x80a0990, msg=0x80a53c0, path_options=0xbfd74b28) at logmpx.c:105 next_hop = (LogPipe *) 0x80a09c0 self = (LogMultiplexer *) 0x80a0990 i = 0 local_options = {flow_control = 0, matched = 0xbfd749fc} matched = 1 delivered = 0 last_delivery = 1 fallback = 0 #7 0x0806b792 in log_pipe_queue (s=0x80a0990, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #8 0x0806b75d in log_source_group_queue (s=0x80a0370, msg=0x80a53c0, path_options=0xbfd74b28) at sgroup.c:97 self = (LogSourceGroup *) 0x80a0370 cfg = (GlobalConfig *) 0x809baa0 #9 0x08054874 in log_pipe_queue (s=0x80a0370, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #10 0x0805483c in log_pipe_forward_msg (self=0x80a01f0, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.c:72 No locals. #11 0x08054874 in log_pipe_queue (s=0x80a01f0, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #12 0x0805483c in log_pipe_forward_msg (self=0x80a3100, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.c:72 No locals. #13 0x0807a833 in log_pipe_queue (s=0x80a3100, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #14 0x0807a7b5 in log_source_queue (s=0x80a51d0, msg=0x80a53c0, path_options=0xbfd74b98) at logsource.c:167 self = (LogSource *) 0x80a51d0 local_options = {flow_control = -1, matched = 0x0} processed_counter = (guint32 *) 0x0 stamp = (guint32 *) 0x0 new = 13 handle = (StatsCounter *) 0x0 __PRETTY_FUNCTION__ = "log_source_queue" #15 0x0806c577 in log_pipe_queue (s=0x80a51d0, msg=0x80a53c0, path_options=0xbfd74b98) at logpipe.h:121 No locals. #16 0x0806c544 in log_reader_handle_line (self=0x80a51d0, line=0x80a31c8 "Apr 12 06:02:39 12log01 syslog-ng[7991]: syslog-ng starting up; version='2.0.5'\n\n", length=81, saddr=0x80a55e0, parse_flags=0) at logreader.c:267 m = (LogMessage *) 0x80a53c0 path_options = {flow_control = -1, matched = 0x0} #17 0x0806c79d in log_reader_fetch_log (self=0x80a51d0, proto=0x80a3168) at logreader.c:346 msg = ( const guchar *) 0x80a31c8 "Apr 12 06:02:39 12log01 syslog-ng[7991]: syslog-ng starting up; version='2.0.5'\n\n" msg_len = 81 status = LPS_SUCCESS sa = (GSockAddr *) 0x80a55e0 msg_count = 1 may_read = 1 parse_flags = 0 __PRETTY_FUNCTION__ = "log_reader_fetch_log" #18 0x0806c383 in log_reader_fd_dispatch (source=0x80a5248, callback=0, user_data=0x0) at logreader.c:207 self = (LogReaderWatch *) 0x80a5248 #19 0xb7e75cf6 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #20 0xb7e790b3 in ?? () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #21 0xb7e7966e in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #22 0x0804dff6 in main_loop_run (cfg=0xbfd74db0) at main.c:149 iters = 3 stats_timer_id = 5 #23 0x0804e64d in main (argc=1, argv=0xbfd74e54) at main.c:394 cfg = (GlobalConfig *) 0x809baa0 rc = 0 ctx = (GOptionContext *) 0x808ecc8 error = (GError *) 0x0 Thanks, Nate
On Wed, 2009-05-13 at 11:04 -0400, Nate Hausrath wrote:
2, You have a segfault. I would need a backtrace or a core file to track down the problem. To get a core file you can use the --enable-core option to syslog-ng and gdb after that to get a backtrace.
Sorry again for the version problem, I will push an update to my git tree with the new version 2 xml handling.
best,
Marton
Hi Marton,
I finally have the backtrace. If you need to core file as well, let me know.
Hi, Hmm, strange. Could you please send me the core and your configuration as well. What is the exact version you are using? thanks, Marton
Core was generated by `/opt/syslog-ng-src/sbin/syslog-ng --no-caps --enable-core'. Program terminated with signal 11, Segmentation fault. [New process 5429] #0 0x08064117 in log_pipe_queue (s=0x0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 121 s->queue(s, msg, path_options); (gdb) bt full #0 0x08064117 in log_pipe_queue (s=0x0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #1 0x080640c5 in log_process_pipe_queue (s=0x80a0a38, msg=0x80a53c0, path_options=0xbfd749e4) at logprocess.c:73 self = (LogProcessPipe *) 0x80a0a38 #2 0x08054874 in log_pipe_queue (s=0x80a0a38, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #3 0x0805483c in log_pipe_forward_msg (self=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.c:72 No locals. #4 0x0806446c in log_multiplexer_queue (s=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logmpx.c:115 self = (LogMultiplexer *) 0x80a09c0 i = 1 local_options = {flow_control = 0, matched = 0xbfd7498c} matched = 1 delivered = 1 last_delivery = 0 fallback = 1 #5 0x08064490 in log_pipe_queue (s=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #6 0x080643f8 in log_multiplexer_queue (s=0x80a0990, msg=0x80a53c0, path_options=0xbfd74b28) at logmpx.c:105 next_hop = (LogPipe *) 0x80a09c0 self = (LogMultiplexer *) 0x80a0990 i = 0 local_options = {flow_control = 0, matched = 0xbfd749fc} matched = 1 delivered = 0 last_delivery = 1 fallback = 0 #7 0x0806b792 in log_pipe_queue (s=0x80a0990, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #8 0x0806b75d in log_source_group_queue (s=0x80a0370, msg=0x80a53c0, path_options=0xbfd74b28) at sgroup.c:97 self = (LogSourceGroup *) 0x80a0370 cfg = (GlobalConfig *) 0x809baa0 #9 0x08054874 in log_pipe_queue (s=0x80a0370, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #10 0x0805483c in log_pipe_forward_msg (self=0x80a01f0, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.c:72 No locals. #11 0x08054874 in log_pipe_queue (s=0x80a01f0, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #12 0x0805483c in log_pipe_forward_msg (self=0x80a3100, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.c:72 No locals. #13 0x0807a833 in log_pipe_queue (s=0x80a3100, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #14 0x0807a7b5 in log_source_queue (s=0x80a51d0, msg=0x80a53c0, path_options=0xbfd74b98) at logsource.c:167 self = (LogSource *) 0x80a51d0 local_options = {flow_control = -1, matched = 0x0} processed_counter = (guint32 *) 0x0 stamp = (guint32 *) 0x0 new = 13 handle = (StatsCounter *) 0x0 __PRETTY_FUNCTION__ = "log_source_queue" #15 0x0806c577 in log_pipe_queue (s=0x80a51d0, msg=0x80a53c0, path_options=0xbfd74b98) at logpipe.h:121 No locals. #16 0x0806c544 in log_reader_handle_line (self=0x80a51d0, line=0x80a31c8 "Apr 12 06:02:39 12log01 syslog-ng[7991]: syslog-ng starting up; version='2.0.5'\n\n", length=81, saddr=0x80a55e0, parse_flags=0) at logreader.c:267 m = (LogMessage *) 0x80a53c0 path_options = {flow_control = -1, matched = 0x0} #17 0x0806c79d in log_reader_fetch_log (self=0x80a51d0, proto=0x80a3168) at logreader.c:346 msg = ( const guchar *) 0x80a31c8 "Apr 12 06:02:39 12log01 syslog-ng[7991]: syslog-ng starting up; version='2.0.5'\n\n" msg_len = 81 status = LPS_SUCCESS sa = (GSockAddr *) 0x80a55e0 msg_count = 1 may_read = 1 parse_flags = 0 __PRETTY_FUNCTION__ = "log_reader_fetch_log" #18 0x0806c383 in log_reader_fd_dispatch (source=0x80a5248, callback=0, user_data=0x0) at logreader.c:207 self = (LogReaderWatch *) 0x80a5248 #19 0xb7e75cf6 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #20 0xb7e790b3 in ?? () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #21 0xb7e7966e in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #22 0x0804dff6 in main_loop_run (cfg=0xbfd74db0) at main.c:149 iters = 3 stats_timer_id = 5 #23 0x0804e64d in main (argc=1, argv=0xbfd74e54) at main.c:394 cfg = (GlobalConfig *) 0x809baa0 rc = 0 ctx = (GOptionContext *) 0x808ecc8 error = (GError *) 0x0
Thanks, Nate ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D
Hi Nate, Thanks for the core and congratulation for your gdb skills :), though I could reproduce your problem meanwhile. In your configuration there is a problem that Bazsi has also spotted:
log { source(s_net); destination(d_parsed); parser(p_db); };
the log statements define a 'pipeline', thus your messages will reach the db-parser() only after having been written to the d_parsed destination.
It is not just a semantic error, but this is the cause of your segfault. As statements are evaluated one after an other the parser can not queue the message for the next statement (the pipeline ends unexpectedly as parser must always have a next pipe) and a null-pointer deref occurs and syslog-ng dies. It should be detected on configuration parsing time and should not die any latter. Changing your configuration would eliminate the segfault and would also make use of the parser. So you should have something like this: log { source(s_net); parser(p_db); destination(d_parsed); }; Hope it solves your problem. Marton On Wed, 2009-05-13 at 11:04 -0400, Nate Hausrath wrote:
2, You have a segfault. I would need a backtrace or a core file to track down the problem. To get a core file you can use the --enable-core option to syslog-ng and gdb after that to get a backtrace.
Sorry again for the version problem, I will push an update to my git tree with the new version 2 xml handling.
best,
Marton
Hi Marton,
I finally have the backtrace. If you need to core file as well, let me know.
Core was generated by `/opt/syslog-ng-src/sbin/syslog-ng --no-caps --enable-core'. Program terminated with signal 11, Segmentation fault. [New process 5429] #0 0x08064117 in log_pipe_queue (s=0x0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 121 s->queue(s, msg, path_options); (gdb) bt full #0 0x08064117 in log_pipe_queue (s=0x0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #1 0x080640c5 in log_process_pipe_queue (s=0x80a0a38, msg=0x80a53c0, path_options=0xbfd749e4) at logprocess.c:73 self = (LogProcessPipe *) 0x80a0a38 #2 0x08054874 in log_pipe_queue (s=0x80a0a38, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #3 0x0805483c in log_pipe_forward_msg (self=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.c:72 No locals. #4 0x0806446c in log_multiplexer_queue (s=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logmpx.c:115 self = (LogMultiplexer *) 0x80a09c0 i = 1 local_options = {flow_control = 0, matched = 0xbfd7498c} matched = 1 delivered = 1 last_delivery = 0 fallback = 1 #5 0x08064490 in log_pipe_queue (s=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #6 0x080643f8 in log_multiplexer_queue (s=0x80a0990, msg=0x80a53c0, path_options=0xbfd74b28) at logmpx.c:105 next_hop = (LogPipe *) 0x80a09c0 self = (LogMultiplexer *) 0x80a0990 i = 0 local_options = {flow_control = 0, matched = 0xbfd749fc} matched = 1 delivered = 0 last_delivery = 1 fallback = 0 #7 0x0806b792 in log_pipe_queue (s=0x80a0990, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #8 0x0806b75d in log_source_group_queue (s=0x80a0370, msg=0x80a53c0, path_options=0xbfd74b28) at sgroup.c:97 self = (LogSourceGroup *) 0x80a0370 cfg = (GlobalConfig *) 0x809baa0 #9 0x08054874 in log_pipe_queue (s=0x80a0370, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #10 0x0805483c in log_pipe_forward_msg (self=0x80a01f0, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.c:72 No locals. #11 0x08054874 in log_pipe_queue (s=0x80a01f0, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #12 0x0805483c in log_pipe_forward_msg (self=0x80a3100, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.c:72 No locals. #13 0x0807a833 in log_pipe_queue (s=0x80a3100, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #14 0x0807a7b5 in log_source_queue (s=0x80a51d0, msg=0x80a53c0, path_options=0xbfd74b98) at logsource.c:167 self = (LogSource *) 0x80a51d0 local_options = {flow_control = -1, matched = 0x0} processed_counter = (guint32 *) 0x0 stamp = (guint32 *) 0x0 new = 13 handle = (StatsCounter *) 0x0 __PRETTY_FUNCTION__ = "log_source_queue" #15 0x0806c577 in log_pipe_queue (s=0x80a51d0, msg=0x80a53c0, path_options=0xbfd74b98) at logpipe.h:121 No locals. #16 0x0806c544 in log_reader_handle_line (self=0x80a51d0, line=0x80a31c8 "Apr 12 06:02:39 12log01 syslog-ng[7991]: syslog-ng starting up; version='2.0.5'\n\n", length=81, saddr=0x80a55e0, parse_flags=0) at logreader.c:267 m = (LogMessage *) 0x80a53c0 path_options = {flow_control = -1, matched = 0x0} #17 0x0806c79d in log_reader_fetch_log (self=0x80a51d0, proto=0x80a3168) at logreader.c:346 msg = ( const guchar *) 0x80a31c8 "Apr 12 06:02:39 12log01 syslog-ng[7991]: syslog-ng starting up; version='2.0.5'\n\n" msg_len = 81 status = LPS_SUCCESS sa = (GSockAddr *) 0x80a55e0 msg_count = 1 may_read = 1 parse_flags = 0 __PRETTY_FUNCTION__ = "log_reader_fetch_log" #18 0x0806c383 in log_reader_fd_dispatch (source=0x80a5248, callback=0, user_data=0x0) at logreader.c:207 self = (LogReaderWatch *) 0x80a5248 #19 0xb7e75cf6 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #20 0xb7e790b3 in ?? () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #21 0xb7e7966e in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #22 0x0804dff6 in main_loop_run (cfg=0xbfd74db0) at main.c:149 iters = 3 stats_timer_id = 5 #23 0x0804e64d in main (argc=1, argv=0xbfd74e54) at main.c:394 cfg = (GlobalConfig *) 0x809baa0 rc = 0 ctx = (GOptionContext *) 0x808ecc8 error = (GError *) 0x0
Thanks, Nate ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D
That did it! It is working properly now. :) Thanks for your help! -Nate On Thu, May 14, 2009 at 2:21 PM, ILLES, Marton <illes.marton@balabit.hu> wrote:
Hi Nate,
Thanks for the core and congratulation for your gdb skills :), though I could reproduce your problem meanwhile.
In your configuration there is a problem that Bazsi has also spotted:
log { source(s_net); destination(d_parsed); parser(p_db); };
the log statements define a 'pipeline', thus your messages will reach the db-parser() only after having been written to the d_parsed destination.
It is not just a semantic error, but this is the cause of your segfault. As statements are evaluated one after an other the parser can not queue the message for the next statement (the pipeline ends unexpectedly as parser must always have a next pipe) and a null-pointer deref occurs and syslog-ng dies. It should be detected on configuration parsing time and should not die any latter.
Changing your configuration would eliminate the segfault and would also make use of the parser. So you should have something like this:
log { source(s_net); parser(p_db); destination(d_parsed); };
Hope it solves your problem.
Marton
On Wed, 2009-05-13 at 11:04 -0400, Nate Hausrath wrote:
2, You have a segfault. I would need a backtrace or a core file to track down the problem. To get a core file you can use the --enable-core option to syslog-ng and gdb after that to get a backtrace.
Sorry again for the version problem, I will push an update to my git tree with the new version 2 xml handling.
best,
Marton
Hi Marton,
I finally have the backtrace. If you need to core file as well, let me know.
Core was generated by `/opt/syslog-ng-src/sbin/syslog-ng --no-caps --enable-core'. Program terminated with signal 11, Segmentation fault. [New process 5429] #0 0x08064117 in log_pipe_queue (s=0x0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 121 s->queue(s, msg, path_options); (gdb) bt full #0 0x08064117 in log_pipe_queue (s=0x0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #1 0x080640c5 in log_process_pipe_queue (s=0x80a0a38, msg=0x80a53c0, path_options=0xbfd749e4) at logprocess.c:73 self = (LogProcessPipe *) 0x80a0a38 #2 0x08054874 in log_pipe_queue (s=0x80a0a38, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #3 0x0805483c in log_pipe_forward_msg (self=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.c:72 No locals. #4 0x0806446c in log_multiplexer_queue (s=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logmpx.c:115 self = (LogMultiplexer *) 0x80a09c0 i = 1 local_options = {flow_control = 0, matched = 0xbfd7498c} matched = 1 delivered = 1 last_delivery = 0 fallback = 1 #5 0x08064490 in log_pipe_queue (s=0x80a09c0, msg=0x80a53c0, path_options=0xbfd749e4) at logpipe.h:121 No locals. #6 0x080643f8 in log_multiplexer_queue (s=0x80a0990, msg=0x80a53c0, path_options=0xbfd74b28) at logmpx.c:105 next_hop = (LogPipe *) 0x80a09c0 self = (LogMultiplexer *) 0x80a0990 i = 0 local_options = {flow_control = 0, matched = 0xbfd749fc} matched = 1 delivered = 0 last_delivery = 1 fallback = 0 #7 0x0806b792 in log_pipe_queue (s=0x80a0990, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #8 0x0806b75d in log_source_group_queue (s=0x80a0370, msg=0x80a53c0, path_options=0xbfd74b28) at sgroup.c:97 self = (LogSourceGroup *) 0x80a0370 cfg = (GlobalConfig *) 0x809baa0 #9 0x08054874 in log_pipe_queue (s=0x80a0370, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #10 0x0805483c in log_pipe_forward_msg (self=0x80a01f0, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.c:72 No locals. #11 0x08054874 in log_pipe_queue (s=0x80a01f0, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #12 0x0805483c in log_pipe_forward_msg (self=0x80a3100, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.c:72 No locals. #13 0x0807a833 in log_pipe_queue (s=0x80a3100, msg=0x80a53c0, path_options=0xbfd74b28) at logpipe.h:121 No locals. #14 0x0807a7b5 in log_source_queue (s=0x80a51d0, msg=0x80a53c0, path_options=0xbfd74b98) at logsource.c:167 self = (LogSource *) 0x80a51d0 local_options = {flow_control = -1, matched = 0x0} processed_counter = (guint32 *) 0x0 stamp = (guint32 *) 0x0 new = 13 handle = (StatsCounter *) 0x0 __PRETTY_FUNCTION__ = "log_source_queue" #15 0x0806c577 in log_pipe_queue (s=0x80a51d0, msg=0x80a53c0, path_options=0xbfd74b98) at logpipe.h:121 No locals. #16 0x0806c544 in log_reader_handle_line (self=0x80a51d0, line=0x80a31c8 "Apr 12 06:02:39 12log01 syslog-ng[7991]: syslog-ng starting up; version='2.0.5'\n\n", length=81, saddr=0x80a55e0, parse_flags=0) at logreader.c:267 m = (LogMessage *) 0x80a53c0 path_options = {flow_control = -1, matched = 0x0} #17 0x0806c79d in log_reader_fetch_log (self=0x80a51d0, proto=0x80a3168) at logreader.c:346 msg = ( const guchar *) 0x80a31c8 "Apr 12 06:02:39 12log01 syslog-ng[7991]: syslog-ng starting up; version='2.0.5'\n\n" msg_len = 81 status = LPS_SUCCESS sa = (GSockAddr *) 0x80a55e0 msg_count = 1 may_read = 1 parse_flags = 0 __PRETTY_FUNCTION__ = "log_reader_fetch_log" #18 0x0806c383 in log_reader_fd_dispatch (source=0x80a5248, callback=0, user_data=0x0) at logreader.c:207 self = (LogReaderWatch *) 0x80a5248 #19 0xb7e75cf6 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #20 0xb7e790b3 in ?? () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #21 0xb7e7966e in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0 No symbol table info available. #22 0x0804dff6 in main_loop_run (cfg=0xbfd74db0) at main.c:149 iters = 3 stats_timer_id = 5 #23 0x0804e64d in main (argc=1, argv=0xbfd74e54) at main.c:394 cfg = (GlobalConfig *) 0x809baa0 rc = 0 ctx = (GOptionContext *) 0x808ecc8 error = (GError *) 0x0
Thanks, Nate ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
participants (4)
-
Balazs Scheidler
-
ILLES, Marton
-
Nate Hausrath
-
okapareeya＠hotmail.com