[syslog-ng] Logging to db fails for multiple tables/destinations

[Liam Kirsher]

Balazs,

I'm afraid this message may have gotten overlooked, and I'm hoping to
get this issue resolved soon so I can deal with my logging issues; so
I'm sending it again.  The debugging suggesting you made turned up what
I believe is a bug.

I have done as you suggested and run syslog-ng in debugging mode, same
syslog-ng.conf as before. It appears that the first entry line
(root.geocode_access) matches the filter but does not trigger the SQL
insert.  However, if I reverse the order of the log{} definitions, then
it does work and the other one doesn't!  I get different results
depending on the order of the two statements below.  It looks like the
SQL insert only happens for the log definition that is last.

log {
    source(s_sys);
    filter(f_geocode);
    parser(p_geocode);
    destination(d_geocode);
};

log {
    source(s_sys);
    filter(f_ut_access);
    parser(p_ut_access);
    destination(d_ut_access);
};

Would you please take a look?

Thanks,
Liam

 /usr/local/sbin/syslog-ng --foreground  --verbose --debug --stderr -p
/var/run/syslogd.pid
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.0.1'
Database thread started;
Incoming log entry; line='<14>obsidian: 2009-02-17
10:47:55,75.101.83.163,/hCi/KM35kk,root.geocode_access,INFO,san
francisco,"San Francisco, CA, US",37.77916,-122.420049\x0a'
Filter rule evaluation begins; filter_rule='f_filter2'
Filter node evaluation result; filter_result='match', filter_type='level'
Filter node evaluation result; filter_result='match', filter_type='facility'
Filter node evaluation result; filter_result='match', filter_type='AND'
Filter rule evaluation result; filter_result='match',
filter_rule='f_filter2'
Initializing destination file writer; template='/var/log/messages',
filename='/var/log/messages'
Filter rule evaluation begins; filter_rule='f_filter3'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter3'
Filter rule evaluation begins; filter_rule='f_filter4'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter4'
Filter rule evaluation begins; filter_rule='f_filter5'
Filter node evaluation result; filter_result='not-match',
filter_type='level'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter5'
Filter rule evaluation begins; filter_rule='f_filter6'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter node evaluation result; filter_result='not-match', filter_type='AND'
Filter node evaluation result; filter_result='not-match', filter_type='OR'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter6'
Filter rule evaluation begins; filter_rule='f_filter7'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter7'
Filter rule evaluation begins; filter_rule='f_filter8'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter8'
Filter rule evaluation begins; filter_rule='f_geocode'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match', filter_type='level'
Filter node evaluation result; filter_result='match', filter_type='AND'
Filter node evaluation result; filter_result='match',
filter_type='filter(f_obsidian)'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match', filter_type='AND'
Filter rule evaluation result; filter_result='match',
filter_rule='f_geocode'  ### Looks like a match, so SQL Insert should go
here, right?
Filter rule evaluation begins; filter_rule='f_ut_access'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match', filter_type='level'
Filter node evaluation result; filter_result='match', filter_type='AND'
Filter node evaluation result; filter_result='match',
filter_type='filter(f_obsidian)'
Filter node evaluation result; filter_result='not-match'
Filter node evaluation result; filter_result='not-match', filter_type='AND'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_ut_access'
Incoming log entry; line='<14>obsidian: 2009-02-17
10:47:55,75.101.83.163,/hCi/KM35kk,root.ut_access,INFO,,,,,/v1/?loc=san+francisco&start=0&rows=10&f=html,,,37.77916,-122.420049\x0a'
Filter rule evaluation begins; filter_rule='f_filter2'
Filter node evaluation result; filter_result='match', filter_type='level'
Filter node evaluation result; filter_result='match', filter_type='facility'
Filter node evaluation result; filter_result='match', filter_type='AND'
Filter rule evaluation result; filter_result='match',
filter_rule='f_filter2'
Filter rule evaluation begins; filter_rule='f_filter3'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter3'
Filter rule evaluation begins; filter_rule='f_filter4'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter4'
Filter rule evaluation begins; filter_rule='f_filter5'
Filter node evaluation result; filter_result='not-match',
filter_type='level'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter5'
Filter rule evaluation begins; filter_rule='f_filter6'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter node evaluation result; filter_result='not-match', filter_type='AND'
Filter node evaluation result; filter_result='not-match', filter_type='OR'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter6'
Filter rule evaluation begins; filter_rule='f_filter7'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter7'
Filter rule evaluation begins; filter_rule='f_filter8'
Filter node evaluation result; filter_result='not-match',
filter_type='facility'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_filter8'
Filter rule evaluation begins; filter_rule='f_geocode'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match', filter_type='level'
Filter node evaluation result; filter_result='match', filter_type='AND'
Filter node evaluation result; filter_result='match',
filter_type='filter(f_obsidian)'
Filter node evaluation result; filter_result='not-match'
Filter node evaluation result; filter_result='not-match', filter_type='AND'
Filter rule evaluation result; filter_result='not-match',
filter_rule='f_geocode'
Filter rule evaluation begins; filter_rule='f_ut_access'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match', filter_type='level'
Filter node evaluation result; filter_result='match', filter_type='AND'
Filter node evaluation result; filter_result='match',
filter_type='filter(f_obsidian)'
Filter node evaluation result; filter_result='match'
Filter node evaluation result; filter_result='match', filter_type='AND'
Filter rule evaluation result; filter_result='match',
filter_rule='f_ut_access'
Running SQL query; query='SELECT * FROM ut_access_log WHERE 0=1'
Running SQL query; query='INSERT INTO ut_access_log (datetime,
query_time, host, program, pid, request_id, level, ip, phone_id,
phone_type, software_version, client_version, query_string, art_id,
session_id, lat, lng) VALUES (\'2009-02-17T13:47:55-05:00\',
\'2009-02-17 10:47:55\', \'127.0.0.1\', \'obsidian\', \'\',
\'/hCi/KM35kk\', \'info\', \'75.101.83.163\', \'\', \'\', \'\', \'\',
\'/v1/?loc=san+francisco&start=0&rows=10&f=html\', \'\', \'\',
\'37.77916\', \'-122.420049\')'







Balazs Scheidler wrote:
> On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
>   
>> Hi --
>>
>> I am /almost/ there, logging to Postgres database.  However, I've
>> discovered a puzzling and problematic behavior.This is probably just
>> some simple misunderstanding on my part, since this is my first foray
>> into syslog-ng.
>> I am logging to two different db tables.  Which table I log to is
>> determined by a regexp filter. The value is either root.ut_access or
>> root.geocode.
>> I can get either one to work, but not both at the same time.
>> If I comment out the log entry for the geocode, then ut_access works. 
>> However, if both log entries exist, only the gecocode_access_log table
>> gets a new row.  Nothing is logged to the ut_access_log table!  (Both
>> messages are logged to d_obsidian destination file, however.)
>> I've attached my config file.
>>     
>
> Hmm.. could you post two example messages that should go to one or the
> other destination?
>
> Since you didn't specify flags(final) to either log statements, both
> should be doing their job, independently from the other. The only thing
> that should control whether one or the other destination is used is the
> attached filter. You can get filter debugging by enabling the --debug /
> --verbose options. 
>
> Be sure that you run syslog-ng in the foreground if you specify these as
> these easily generate loops in the configuration unless the internal()
> source is not present. (use --foreground for that, intenral() messages
> will be printed on the standard error).
>
> Judging the config I can't see an obvious problem, that's why I wanted
> to test it, but I'd need a sample log message for that.
>
>   

-- 
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20090223/8c3af33a/attachment.htm