Balazs, I'm afraid this message may have gotten overlooked, and I'm hoping to get this issue resolved soon so I can deal with my logging issues; so I'm sending it again. The debugging suggesting you made turned up what I believe is a bug. I have done as you suggested and run syslog-ng in debugging mode, same syslog-ng.conf as before. It appears that the first entry line (root.geocode_access) matches the filter but does not trigger the SQL insert. However, if I reverse the order of the log{} definitions, then it does work and the other one doesn't! I get different results depending on the order of the two statements below. It looks like the SQL insert only happens for the log definition that is last. log { source(s_sys); filter(f_geocode); parser(p_geocode); destination(d_geocode); }; log { source(s_sys); filter(f_ut_access); parser(p_ut_access); destination(d_ut_access); }; Would you please take a look? Thanks, Liam /usr/local/sbin/syslog-ng --foreground --verbose --debug --stderr -p /var/run/syslogd.pid Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.0.1' Database thread started; Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.geocode_access,INFO,san francisco,"San Francisco, CA, US",37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_geocode' ### Looks like a match, so SQL Insert should go here, right? Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_ut_access' Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.ut_access,INFO,,,,,/v1/?loc=san+francisco&start=0&rows=10&f=html,,,37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_geocode' Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_ut_access' Running SQL query; query='SELECT * FROM ut_access_log WHERE 0=1' Running SQL query; query='INSERT INTO ut_access_log (datetime, query_time, host, program, pid, request_id, level, ip, phone_id, phone_type, software_version, client_version, query_string, art_id, session_id, lat, lng) VALUES (\'2009-02-17T13:47:55-05:00\', \'2009-02-17 10:47:55\', \'127.0.0.1\', \'obsidian\', \'\', \'/hCi/KM35kk\', \'info\', \'75.101.83.163\', \'\', \'\', \'\', \'\', \'/v1/?loc=san+francisco&start=0&rows=10&f=html\', \'\', \'\', \'37.77916\', \'-122.420049\')' Balazs Scheidler wrote:
On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
Hi --
I am /almost/ there, logging to Postgres database. However, I've discovered a puzzling and problematic behavior.This is probably just some simple misunderstanding on my part, since this is my first foray into syslog-ng. I am logging to two different db tables. Which table I log to is determined by a regexp filter. The value is either root.ut_access or root.geocode. I can get either one to work, but not both at the same time. If I comment out the log entry for the geocode, then ut_access works. However, if both log entries exist, only the gecocode_access_log table gets a new row. Nothing is logged to the ut_access_log table! (Both messages are logged to d_obsidian destination file, however.) I've attached my config file.
Hmm.. could you post two example messages that should go to one or the other destination?
Since you didn't specify flags(final) to either log statements, both should be doing their job, independently from the other. The only thing that should control whether one or the other destination is used is the attached filter. You can get filter debugging by enabling the --debug / --verbose options.
Be sure that you run syslog-ng in the foreground if you specify these as these easily generate loops in the configuration unless the internal() source is not present. (use --foreground for that, intenral() messages will be printed on the standard error).
Judging the config I can't see an obvious problem, that's why I wanted to test it, but I'd need a sample log message for that.
-- Liam Kirsher PGP: http://liam.numenet.com/pgp/