Re: [syslog-ng] Logging to db fails for multiple tables/destinations

1 Mar 2009

Balazs,

Thanks! That did the trick.

Liam

Balazs Scheidler wrote:
...
You are right. Only one of the destinations got initialized during
startup. This patch fixes it for me:

diff --git a/src/apphook.c b/src/apphook.c
index ab9cb02..6115b27 100644
--- a/src/apphook.c
+++ b/src/apphook.c
@@ -54,10 +54,10 @@ run_application_hook(gint type)
if (e->type == type)
         {
+          l_next = l->next;
           application_hooks = g_list_remove_link(application_hooks, l);
           e->func(type, e->user_data);
           g_free(e);
-          l_next = l->next;
           g_list_free_1(l);
         }
       else
On Mon, 2009-02-23 at 19:31 -0800, Liam Kirsher wrote:
...
Balazs,
I'm afraid this message may have gotten overlooked, and I'm hoping to
get this issue resolved soon so I can deal with my logging issues; so
I'm sending it again.  The debugging suggesting you made turned up
what I believe is a bug.
I have done as you suggested and run syslog-ng in debugging mode, same
syslog-ng.conf as before. It appears that the first entry line
(root.geocode_access) matches the filter but does not trigger the SQL
insert.  However, if I reverse the order of the log{} definitions,
then it does work and the other one doesn't!  I get different results
depending on the order of the two statements below.  It looks like the
SQL insert only happens for the log definition that is last.
log {
    source(s_sys);
    filter(f_geocode);
    parser(p_geocode);
    destination(d_geocode);
};
log {
    source(s_sys);
    filter(f_ut_access);
    parser(p_ut_access);
    destination(d_ut_access);
};
Would you please take a look?
Thanks,
Liam
/usr/local/sbin/syslog-ng --foreground  --verbose --debug --stderr
-p /var/run/syslogd.pid 
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.0.1'
Database thread started;
Incoming log entry; line='<14>obsidian: 2009-02-17
10:47:55,75.101.83.163,/hCi/KM35kk,root.geocode_access,INFO,san
francisco,"San Francisco, CA, US",37.77916,-122.420049\x0a'
Filter rule evaluation begins; filter_rule='f_filter2'
Filter node evaluation result; filter_result='match',
filter_type='level'
Filter node evaluation result; filter_result='match',
filter_type='facility'
Filter node evaluation result; filter_result='match',
filter_type='AND'
Filter rule evaluation result; filter_result='match',
filter_rule='f_filter2'
Initializing destination file writer; template='/var/log/messages',
filename='/var/log/messages'
Filter rule evaluation begins; filter_rule='f_filter3'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter3'
Filter rule evaluation begins; filter_rule='f_filter4'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter4'
Filter rule evaluation begins; filter_rule='f_filter5'
Filter node evaluation result; filter_result='not-match',
filter_type='level'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter5'
Filter rule evaluation begins; filter_rule='f_filter6'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter node evaluation result; filter_result='not-match',
filter_type='AND'
Filter node evaluation result; filter_result='not-match',
filter_type='OR'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter6'
Filter rule evaluation begins; filter_rule='f_filter7'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter7'
Filter rule evaluation begins; filter_rule='f_filter8'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter8'
Filter rule evaluation begins; filter_rule='f_geocode'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match',
filter_type='level'
Filter node evaluation result; filter_result='match',
filter_type='AND'
Filter node evaluation result; filter_result='match',
filter_type='filter(f_obsidian)'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match',
filter_type='AND'
Filter rule evaluation result; filter_result='match',
filter_rule='f_geocode'  ### Looks like a match, so SQL Insert should
go here, right?
Filter rule evaluation begins; filter_rule='f_ut_access'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match',
filter_type='level'
Filter node evaluation result; filter_result='match',
filter_type='AND'
Filter node evaluation result; filter_result='match',
filter_type='filter(f_obsidian)'
Filter node evaluation result; filter_result='not-match'
Filter node evaluation result; filter_result='not-match',
filter_type='AND'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_ut_access'
Incoming log entry; line='<14>obsidian: 2009-02-17
10:47:55,75.101.83.163,/hCi/KM35kk,root.ut_access,INFO,,,,,/v1/?loc=san+francisco&start=0&rows=10&f=html,,,37.77916,-122.420049\x0a'
Filter rule evaluation begins; filter_rule='f_filter2'
Filter node evaluation result; filter_result='match',
filter_type='level'
Filter node evaluation result; filter_result='match',
filter_type='facility'
Filter node evaluation result; filter_result='match',
filter_type='AND'
Filter rule evaluation result; filter_result='match',
filter_rule='f_filter2'
Filter rule evaluation begins; filter_rule='f_filter3'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter3'
Filter rule evaluation begins; filter_rule='f_filter4'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter4'
Filter rule evaluation begins; filter_rule='f_filter5'
Filter node evaluation result; filter_result='not-match',
filter_type='level'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter5'
Filter rule evaluation begins; filter_rule='f_filter6'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter node evaluation result; filter_result='not-match',
filter_type='AND'
Filter node evaluation result; filter_result='not-match',
filter_type='OR'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter6'
Filter rule evaluation begins; filter_rule='f_filter7'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter7'
Filter rule evaluation begins; filter_rule='f_filter8'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter8'
Filter rule evaluation begins; filter_rule='f_geocode'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match',
filter_type='level'
Filter node evaluation result; filter_result='match',
filter_type='AND'
Filter node evaluation result; filter_result='match',
filter_type='filter(f_obsidian)'
Filter node evaluation result; filter_result='not-match'
Filter node evaluation result; filter_result='not-match',
filter_type='AND'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_geocode'
Filter rule evaluation begins; filter_rule='f_ut_access'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match',
filter_type='level'
Filter node evaluation result; filter_result='match',
filter_type='AND'
Filter node evaluation result; filter_result='match',
filter_type='filter(f_obsidian)'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match',
filter_type='AND'
Filter rule evaluation result; filter_result='match',
filter_rule='f_ut_access'
Running SQL query; query='SELECT * FROM ut_access_log WHERE 0=1'
Running SQL query; query='INSERT INTO ut_access_log (datetime,
query_time, host, program, pid, request_id, level, ip, phone_id,
phone_type, software_version, client_version, query_string, art_id,
session_id, lat, lng) VALUES (\'2009-02-17T13:47:55-05:00\',
\'2009-02-17 10:47:55\', \'127.0.0.1\', \'obsidian\', \'\',
\'/hCi/KM35kk\', \'info\', \'75.101.83.163\', \'\', \'\', \'\', \'\',
\'/v1/?loc=san+francisco&start=0&rows=10&f=html\', \'\', \'\',
\'37.77916\', \'-122.420049\')'
Balazs Scheidler wrote:
...
On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
...
Hi --
I am /almost/ there, logging to Postgres database.  However, I've
discovered a puzzling and problematic behavior.This is probably just
some simple misunderstanding on my part, since this is my first foray
into syslog-ng.
I am logging to two different db tables.  Which table I log to is
determined by a regexp filter. The value is either root.ut_access or
root.geocode.
I can get either one to work, but not both at the same time.
If I comment out the log entry for the geocode, then ut_access works. 
However, if both log entries exist, only the gecocode_access_log table
gets a new row.  Nothing is logged to the ut_access_log table!  (Both
messages are logged to d_obsidian destination file, however.)
I've attached my config file.
Hmm.. could you post two example messages that should go to one or the
other destination?
Since you didn't specify flags(final) to either log statements, both
should be doing their job, independently from the other. The only thing
that should control whether one or the other destination is used is the
attached filter. You can get filter debugging by enabling the --debug /
--verbose options.
Be sure that you run syslog-ng in the foreground if you specify these as
these easily generate loops in the configuration unless the internal()
source is not present. (use --foreground for that, intenral() messages
will be printed on the standard error).
Judging the config I can't see an obvious problem, that's why I wanted
to test it, but I'd need a sample log message for that.
-- 
Liam Kirsher
PGP: http://liam.numenet.com/pgp/
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html
-- 
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

    