[syslog-ng] Logging to db fails for multiple tables/destinations
Liam Kirsher
liamk at numenet.com
Sun Mar 1 22:24:54 CET 2009
Balazs,
Thanks! That did the trick.
Liam
Balazs Scheidler wrote:
> You are right. Only one of the destinations got initialized during
> startup. This patch fixes it for me:
>
> diff --git a/src/apphook.c b/src/apphook.c
> index ab9cb02..6115b27 100644
> --- a/src/apphook.c
> +++ b/src/apphook.c
> @@ -54,10 +54,10 @@ run_application_hook(gint type)
>
> if (e->type == type)
> {
> + l_next = l->next;
> application_hooks = g_list_remove_link(application_hooks, l);
> e->func(type, e->user_data);
> g_free(e);
> - l_next = l->next;
> g_list_free_1(l);
> }
> else
>
>
> On Mon, 2009-02-23 at 19:31 -0800, Liam Kirsher wrote:
>
>> Balazs,
>>
>> I'm afraid this message may have gotten overlooked, and I'm hoping to
>> get this issue resolved soon so I can deal with my logging issues; so
>> I'm sending it again. The debugging suggesting you made turned up
>> what I believe is a bug.
>>
>> I have done as you suggested and run syslog-ng in debugging mode, same
>> syslog-ng.conf as before. It appears that the first entry line
>> (root.geocode_access) matches the filter but does not trigger the SQL
>> insert. However, if I reverse the order of the log{} definitions,
>> then it does work and the other one doesn't! I get different results
>> depending on the order of the two statements below. It looks like the
>> SQL insert only happens for the log definition that is last.
>>
>> log {
>> source(s_sys);
>> filter(f_geocode);
>> parser(p_geocode);
>> destination(d_geocode);
>> };
>>
>> log {
>> source(s_sys);
>> filter(f_ut_access);
>> parser(p_ut_access);
>> destination(d_ut_access);
>> };
>>
>> Would you please take a look?
>>
>> Thanks,
>> Liam
>>
>> /usr/local/sbin/syslog-ng --foreground --verbose --debug --stderr
>> -p /var/run/syslogd.pid
>> Running application hooks; hook='1'
>> Running application hooks; hook='3'
>> syslog-ng starting up; version='3.0.1'
>> Database thread started;
>> Incoming log entry; line='<14>obsidian: 2009-02-17
>> 10:47:55,75.101.83.163,/hCi/KM35kk,root.geocode_access,INFO,san
>> francisco,"San Francisco, CA, US",37.77916,-122.420049\x0a'
>> Filter rule evaluation begins; filter_rule='f_filter2'
>> Filter node evaluation result; filter_result='match',
>> filter_type='level'
>> Filter node evaluation result; filter_result='match',
>> filter_type='facility'
>> Filter node evaluation result; filter_result='match',
>> filter_type='AND'
>> Filter rule evaluation result; filter_result='match',
>> filter_rule='f_filter2'
>> Initializing destination file writer; template='/var/log/messages',
>> filename='/var/log/messages'
>> Filter rule evaluation begins; filter_rule='f_filter3'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter3'
>> Filter rule evaluation begins; filter_rule='f_filter4'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter4'
>> Filter rule evaluation begins; filter_rule='f_filter5'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='level'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter5'
>> Filter rule evaluation begins; filter_rule='f_filter6'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='AND'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='OR'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter6'
>> Filter rule evaluation begins; filter_rule='f_filter7'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter7'
>> Filter rule evaluation begins; filter_rule='f_filter8'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter8'
>> Filter rule evaluation begins; filter_rule='f_geocode'
>> Filter node evaluation result; filter_result='match'
>> Filter node evaluation result; filter_result='match',
>> filter_type='level'
>> Filter node evaluation result; filter_result='match',
>> filter_type='AND'
>> Filter node evaluation result; filter_result='match',
>> filter_type='filter(f_obsidian)'
>> Filter node evaluation result; filter_result='match'
>> Filter node evaluation result; filter_result='match',
>> filter_type='AND'
>> Filter rule evaluation result; filter_result='match',
>> filter_rule='f_geocode' ### Looks like a match, so SQL Insert should
>> go here, right?
>> Filter rule evaluation begins; filter_rule='f_ut_access'
>> Filter node evaluation result; filter_result='match'
>> Filter node evaluation result; filter_result='match',
>> filter_type='level'
>> Filter node evaluation result; filter_result='match',
>> filter_type='AND'
>> Filter node evaluation result; filter_result='match',
>> filter_type='filter(f_obsidian)'
>> Filter node evaluation result; filter_result='not-match'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='AND'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_ut_access'
>> Incoming log entry; line='<14>obsidian: 2009-02-17
>> 10:47:55,75.101.83.163,/hCi/KM35kk,root.ut_access,INFO,,,,,/v1/?loc=san+francisco&start=0&rows=10&f=html,,,37.77916,-122.420049\x0a'
>> Filter rule evaluation begins; filter_rule='f_filter2'
>> Filter node evaluation result; filter_result='match',
>> filter_type='level'
>> Filter node evaluation result; filter_result='match',
>> filter_type='facility'
>> Filter node evaluation result; filter_result='match',
>> filter_type='AND'
>> Filter rule evaluation result; filter_result='match',
>> filter_rule='f_filter2'
>> Filter rule evaluation begins; filter_rule='f_filter3'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter3'
>> Filter rule evaluation begins; filter_rule='f_filter4'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter4'
>> Filter rule evaluation begins; filter_rule='f_filter5'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='level'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter5'
>> Filter rule evaluation begins; filter_rule='f_filter6'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='AND'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='OR'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter6'
>> Filter rule evaluation begins; filter_rule='f_filter7'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter7'
>> Filter rule evaluation begins; filter_rule='f_filter8'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='facility'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_filter8'
>> Filter rule evaluation begins; filter_rule='f_geocode'
>> Filter node evaluation result; filter_result='match'
>> Filter node evaluation result; filter_result='match',
>> filter_type='level'
>> Filter node evaluation result; filter_result='match',
>> filter_type='AND'
>> Filter node evaluation result; filter_result='match',
>> filter_type='filter(f_obsidian)'
>> Filter node evaluation result; filter_result='not-match'
>> Filter node evaluation result; filter_result='not-match',
>> filter_type='AND'
>> Filter rule evaluation result; filter_result='not-match',
>> filter_rule='f_geocode'
>> Filter rule evaluation begins; filter_rule='f_ut_access'
>> Filter node evaluation result; filter_result='match'
>> Filter node evaluation result; filter_result='match',
>> filter_type='level'
>> Filter node evaluation result; filter_result='match',
>> filter_type='AND'
>> Filter node evaluation result; filter_result='match',
>> filter_type='filter(f_obsidian)'
>> Filter node evaluation result; filter_result='match'
>> Filter node evaluation result; filter_result='match',
>> filter_type='AND'
>> Filter rule evaluation result; filter_result='match',
>> filter_rule='f_ut_access'
>> Running SQL query; query='SELECT * FROM ut_access_log WHERE 0=1'
>> Running SQL query; query='INSERT INTO ut_access_log (datetime,
>> query_time, host, program, pid, request_id, level, ip, phone_id,
>> phone_type, software_version, client_version, query_string, art_id,
>> session_id, lat, lng) VALUES (\'2009-02-17T13:47:55-05:00\',
>> \'2009-02-17 10:47:55\', \'127.0.0.1\', \'obsidian\', \'\',
>> \'/hCi/KM35kk\', \'info\', \'75.101.83.163\', \'\', \'\', \'\', \'\',
>> \'/v1/?loc=san+francisco&start=0&rows=10&f=html\', \'\', \'\',
>> \'37.77916\', \'-122.420049\')'
>>
>>
>>
>>
>>
>>
>>
>> Balazs Scheidler wrote:
>>
>>> On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
>>>
>>>
>>>> Hi --
>>>>
>>>> I am /almost/ there, logging to Postgres database. However, I've
>>>> discovered a puzzling and problematic behavior.This is probably just
>>>> some simple misunderstanding on my part, since this is my first foray
>>>> into syslog-ng.
>>>> I am logging to two different db tables. Which table I log to is
>>>> determined by a regexp filter. The value is either root.ut_access or
>>>> root.geocode.
>>>> I can get either one to work, but not both at the same time.
>>>> If I comment out the log entry for the geocode, then ut_access works.
>>>> However, if both log entries exist, only the gecocode_access_log table
>>>> gets a new row. Nothing is logged to the ut_access_log table! (Both
>>>> messages are logged to d_obsidian destination file, however.)
>>>> I've attached my config file.
>>>>
>>>>
>>> Hmm.. could you post two example messages that should go to one or the
>>> other destination?
>>>
>>> Since you didn't specify flags(final) to either log statements, both
>>> should be doing their job, independently from the other. The only thing
>>> that should control whether one or the other destination is used is the
>>> attached filter. You can get filter debugging by enabling the --debug /
>>> --verbose options.
>>>
>>> Be sure that you run syslog-ng in the foreground if you specify these as
>>> these easily generate loops in the configuration unless the internal()
>>> source is not present. (use --foreground for that, intenral() messages
>>> will be printed on the standard error).
>>>
>>> Judging the config I can't see an obvious problem, that's why I wanted
>>> to test it, but I'd need a sample log message for that.
>>>
>>>
>>>
>> --
>> Liam Kirsher
>> PGP: http://liam.numenet.com/pgp/
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/
More information about the syslog-ng
mailing list