Configuration question, logging to db not working.
Hi -- I am /almost/ there, logging to Postgres database. However, I've discovered a puzzling and problematic behavior.This is probably just some simple misunderstanding on my part, since this is my first foray into syslog-ng. I am logging to two different db tables. Which table I log to is determined by a regexp filter. The value is either root.ut_access or root.geocode. I can get either one to work, but not both at the same time. If I comment out the log entry for the geocode, then ut_access works. However, if both log entries exist, only the gecocode_access_log table gets a new row. Nothing is logged to the ut_access_log table! (Both messages are logged to d_obsidian destination file, however.) I've attached my config file. Any tips would be greatly appreciated!!! Liam ---------------------------------- @version:3.0 # syslog-ng configuration file. options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); tcp(ip(0.0.0.0) port(5000) max-connections(1000)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination d_obsidian { file("/var/log/django/$PROGRAM/$R_YEAR$R_MONTH$R_DAY.log"); }; destination d_ut_access { sql( type(pgsql) host("localhost") username("postgres") password("xxxxxx") database("prodlogs") table("ut_access_log") columns("datetime", "query_time", "host", "program", "pid", "request_id", "level", "ip", "phone_id", "phone_type", "software_version", "client_version", "query_string", "art_id", "session_id", "lat", "lng") values("$R_ISODATE", "${UT.QTIME}", "$HOST", "$PROGRAM", "$PID", "${UT.REQUEST_ID}", "$LEVEL", "${UT.IP}", "${UT.PHONE_ID}", "${UT.PHONE_TYPE}", "${UT.SOFTWARE_VERSION}", "${UT.CLIENT_VERSION}", "${UT.QUERY_STRING}", "${UT.ART_ID}", "${UT.SESSION_ID}", "${UT.LAT}", "${UT.LNG}") indexes("datetime", "host", "program", "ip", "phone_id") ); }; destination d_geocode { sql( type(pgsql) host("localhost") username("postgres") password("xxxxxx") database("prodlogs") table("geocode_access_log") columns("datetime", "querytime", "host", "program", "pid", "request_id", "level", "ip", "name", "place", "lat", "lng") values("$R_ISODATE", "${GEO.QTIME}", "$HOST", "$PROGRAM", "$PID", "${GEO.REQUEST_ID}", "$LEVEL", "${GEO.IP}", "${GEO.NAME}", "${GEO.PLACE}", "${GEO.LAT}", "${GEO.LNG}") indexes("datetime", "host", "program", "pid", "ip", "name", "place") ); }; parser p_ut_access { csv-parser( columns("UT.QTIME", "UT.IP", "UT.REQUEST_ID", "UT.CATEGORY", "UT.MYLEVEL", "UT.PHONE_ID", "UT.PHONE_TYPE", "UT.SOFTWARE_VERSION", "UT.CLIENT_VERSION", "UT.QUERY_STRING", "UT.ART_ID", "UT.SESSION_ID", "UT.LAT", "UT.LNG") delimiters(",") quote-pairs('""') flags(escape-double-char, strip-whitespace) ); }; parser p_geocode { csv-parser( columns("GEO.QTIME", "GEO.IP", "GEO.REQUEST_ID", "GEO.CATEGORY", "GEO.MYLEVEL", "GEO.NAME", "GEO.PLACE", "GEO.LAT", "GEO.LNG") delimiters(",") quote-pairs('""') flags(escape-double-char, strip-whitespace) ); }; #filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; filter f_obsidian { program("^obsidian$") and level(info); }; filter f_ut_access { filter(f_obsidian) and message("root\.ut_access"); }; filter f_geocode { filter(f_obsidian) and message("root\.geocode"); }; #log { source(s_sys); filter(f_filter1); destination(d_cons); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; log { source(s_sys); filter(f_ut_access); parser(p_ut_access); destination(d_ut_access); }; ### With this log entry commented out, logging to d_ut_access works. But if I uncomment it, nothing is logged to d_ut_access! ### What am I missing? #log { #source(s_sys); #filter(f_geocode); #parser(p_geocode); #destination(d_geocode); #}; log { source(s_sys); filter(f_obsidian); destination(d_obsidian); }; -- Liam Kirsher PGP: http://liam.numenet.com/pgp/
On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
Hi --
I am /almost/ there, logging to Postgres database. However, I've discovered a puzzling and problematic behavior.This is probably just some simple misunderstanding on my part, since this is my first foray into syslog-ng. I am logging to two different db tables. Which table I log to is determined by a regexp filter. The value is either root.ut_access or root.geocode. I can get either one to work, but not both at the same time. If I comment out the log entry for the geocode, then ut_access works. However, if both log entries exist, only the gecocode_access_log table gets a new row. Nothing is logged to the ut_access_log table! (Both messages are logged to d_obsidian destination file, however.) I've attached my config file.
Hmm.. could you post two example messages that should go to one or the other destination? Since you didn't specify flags(final) to either log statements, both should be doing their job, independently from the other. The only thing that should control whether one or the other destination is used is the attached filter. You can get filter debugging by enabling the --debug / --verbose options. Be sure that you run syslog-ng in the foreground if you specify these as these easily generate loops in the configuration unless the internal() source is not present. (use --foreground for that, intenral() messages will be printed on the standard error). Judging the config I can't see an obvious problem, that's why I wanted to test it, but I'd need a sample log message for that. -- Bazsi
Balazs, I have done as you suggested and run syslolg-ng in debugging mode, same syslog-ng.conf as before. It appears that the first entry line (root.geocode_access) matches the filter but does not trigger the SQL insert. However, if I reverse the order of the log{} definitions, then it does work and the other one doesn't! I get different results depending on the order of the two statements below. It looks like the SQL insert only happens for the log definition that is last. log { source(s_sys); filter(f_geocode); parser(p_geocode); destination(d_geocode); }; log { source(s_sys); filter(f_ut_access); parser(p_ut_access); destination(d_ut_access); }; I think this is a bug. Would you please take a look? Thanks, Liam /usr/local/sbin/syslog-ng --foreground --verbose --debug --stderr -p /var/run/syslogd.pid Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.0.1' Database thread started; Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.geocode_access,INFO,san francisco,"San Francisco, CA, US",37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_geocode' ### Looks like a match, so SQL Insert should go here, right? Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_ut_access' Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.ut_access,INFO,,,,,/v1/?loc=san+francisco&start=0&rows=10&f=html,,,37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_geocode' Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_ut_access' Running SQL query; query='SELECT * FROM ut_access_log WHERE 0=1' Running SQL query; query='INSERT INTO ut_access_log (datetime, query_time, host, program, pid, request_id, level, ip, phone_id, phone_type, software_version, client_version, query_string, art_id, session_id, lat, lng) VALUES (\'2009-02-17T13:47:55-05:00\', \'2009-02-17 10:47:55\', \'127.0.0.1\', \'obsidian\', \'\', \'/hCi/KM35kk\', \'info\', \'75.101.83.163\', \'\', \'\', \'\', \'\', \'/v1/?loc=san+francisco&start=0&rows=10&f=html\', \'\', \'\', \'37.77916\', \'-122.420049\')' Balazs Scheidler wrote:
On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
Hi --
I am /almost/ there, logging to Postgres database. However, I've discovered a puzzling and problematic behavior.This is probably just some simple misunderstanding on my part, since this is my first foray into syslog-ng. I am logging to two different db tables. Which table I log to is determined by a regexp filter. The value is either root.ut_access or root.geocode. I can get either one to work, but not both at the same time. If I comment out the log entry for the geocode, then ut_access works. However, if both log entries exist, only the gecocode_access_log table gets a new row. Nothing is logged to the ut_access_log table! (Both messages are logged to d_obsidian destination file, however.) I've attached my config file.
Hmm.. could you post two example messages that should go to one or the other destination?
Since you didn't specify flags(final) to either log statements, both should be doing their job, independently from the other. The only thing that should control whether one or the other destination is used is the attached filter. You can get filter debugging by enabling the --debug / --verbose options.
Be sure that you run syslog-ng in the foreground if you specify these as these easily generate loops in the configuration unless the internal() source is not present. (use --foreground for that, intenral() messages will be printed on the standard error).
Judging the config I can't see an obvious problem, that's why I wanted to test it, but I'd need a sample log message for that.
-- Liam Kirsher PGP: http://liam.numenet.com/pgp/
Balazs, I'm afraid this message may have gotten overlooked, and I'm hoping to get this issue resolved soon so I can deal with my logging issues; so I'm sending it again. The debugging suggesting you made turned up what I believe is a bug. I have done as you suggested and run syslog-ng in debugging mode, same syslog-ng.conf as before. It appears that the first entry line (root.geocode_access) matches the filter but does not trigger the SQL insert. However, if I reverse the order of the log{} definitions, then it does work and the other one doesn't! I get different results depending on the order of the two statements below. It looks like the SQL insert only happens for the log definition that is last. log { source(s_sys); filter(f_geocode); parser(p_geocode); destination(d_geocode); }; log { source(s_sys); filter(f_ut_access); parser(p_ut_access); destination(d_ut_access); }; Would you please take a look? Thanks, Liam /usr/local/sbin/syslog-ng --foreground --verbose --debug --stderr -p /var/run/syslogd.pid Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.0.1' Database thread started; Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.geocode_access,INFO,san francisco,"San Francisco, CA, US",37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_geocode' ### Looks like a match, so SQL Insert should go here, right? Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_ut_access' Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.ut_access,INFO,,,,,/v1/?loc=san+francisco&start=0&rows=10&f=html,,,37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_geocode' Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_ut_access' Running SQL query; query='SELECT * FROM ut_access_log WHERE 0=1' Running SQL query; query='INSERT INTO ut_access_log (datetime, query_time, host, program, pid, request_id, level, ip, phone_id, phone_type, software_version, client_version, query_string, art_id, session_id, lat, lng) VALUES (\'2009-02-17T13:47:55-05:00\', \'2009-02-17 10:47:55\', \'127.0.0.1\', \'obsidian\', \'\', \'/hCi/KM35kk\', \'info\', \'75.101.83.163\', \'\', \'\', \'\', \'\', \'/v1/?loc=san+francisco&start=0&rows=10&f=html\', \'\', \'\', \'37.77916\', \'-122.420049\')' Balazs Scheidler wrote:
On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
Hi --
I am /almost/ there, logging to Postgres database. However, I've discovered a puzzling and problematic behavior.This is probably just some simple misunderstanding on my part, since this is my first foray into syslog-ng. I am logging to two different db tables. Which table I log to is determined by a regexp filter. The value is either root.ut_access or root.geocode. I can get either one to work, but not both at the same time. If I comment out the log entry for the geocode, then ut_access works. However, if both log entries exist, only the gecocode_access_log table gets a new row. Nothing is logged to the ut_access_log table! (Both messages are logged to d_obsidian destination file, however.) I've attached my config file.
Hmm.. could you post two example messages that should go to one or the other destination?
Since you didn't specify flags(final) to either log statements, both should be doing their job, independently from the other. The only thing that should control whether one or the other destination is used is the attached filter. You can get filter debugging by enabling the --debug / --verbose options.
Be sure that you run syslog-ng in the foreground if you specify these as these easily generate loops in the configuration unless the internal() source is not present. (use --foreground for that, intenral() messages will be printed on the standard error).
Judging the config I can't see an obvious problem, that's why I wanted to test it, but I'd need a sample log message for that.
-- Liam Kirsher PGP: http://liam.numenet.com/pgp/
You are right. Only one of the destinations got initialized during startup. This patch fixes it for me: diff --git a/src/apphook.c b/src/apphook.c index ab9cb02..6115b27 100644 --- a/src/apphook.c +++ b/src/apphook.c @@ -54,10 +54,10 @@ run_application_hook(gint type) if (e->type == type) { + l_next = l->next; application_hooks = g_list_remove_link(application_hooks, l); e->func(type, e->user_data); g_free(e); - l_next = l->next; g_list_free_1(l); } else On Mon, 2009-02-23 at 19:31 -0800, Liam Kirsher wrote:
Balazs,
I'm afraid this message may have gotten overlooked, and I'm hoping to get this issue resolved soon so I can deal with my logging issues; so I'm sending it again. The debugging suggesting you made turned up what I believe is a bug.
I have done as you suggested and run syslog-ng in debugging mode, same syslog-ng.conf as before. It appears that the first entry line (root.geocode_access) matches the filter but does not trigger the SQL insert. However, if I reverse the order of the log{} definitions, then it does work and the other one doesn't! I get different results depending on the order of the two statements below. It looks like the SQL insert only happens for the log definition that is last.
log { source(s_sys); filter(f_geocode); parser(p_geocode); destination(d_geocode); };
log { source(s_sys); filter(f_ut_access); parser(p_ut_access); destination(d_ut_access); };
Would you please take a look?
Thanks, Liam
/usr/local/sbin/syslog-ng --foreground --verbose --debug --stderr -p /var/run/syslogd.pid Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.0.1' Database thread started; Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.geocode_access,INFO,san francisco,"San Francisco, CA, US",37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_geocode' ### Looks like a match, so SQL Insert should go here, right? Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_ut_access' Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.ut_access,INFO,,,,,/v1/?loc=san+francisco&start=0&rows=10&f=html,,,37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_geocode' Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_ut_access' Running SQL query; query='SELECT * FROM ut_access_log WHERE 0=1' Running SQL query; query='INSERT INTO ut_access_log (datetime, query_time, host, program, pid, request_id, level, ip, phone_id, phone_type, software_version, client_version, query_string, art_id, session_id, lat, lng) VALUES (\'2009-02-17T13:47:55-05:00\', \'2009-02-17 10:47:55\', \'127.0.0.1\', \'obsidian\', \'\', \'/hCi/KM35kk\', \'info\', \'75.101.83.163\', \'\', \'\', \'\', \'\', \'/v1/?loc=san+francisco&start=0&rows=10&f=html\', \'\', \'\', \'37.77916\', \'-122.420049\')'
Balazs Scheidler wrote:
On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
Hi --
I am /almost/ there, logging to Postgres database. However, I've discovered a puzzling and problematic behavior.This is probably just some simple misunderstanding on my part, since this is my first foray into syslog-ng. I am logging to two different db tables. Which table I log to is determined by a regexp filter. The value is either root.ut_access or root.geocode. I can get either one to work, but not both at the same time. If I comment out the log entry for the geocode, then ut_access works. However, if both log entries exist, only the gecocode_access_log table gets a new row. Nothing is logged to the ut_access_log table! (Both messages are logged to d_obsidian destination file, however.) I've attached my config file.
Hmm.. could you post two example messages that should go to one or the other destination?
Since you didn't specify flags(final) to either log statements, both should be doing their job, independently from the other. The only thing that should control whether one or the other destination is used is the attached filter. You can get filter debugging by enabling the --debug / --verbose options.
Be sure that you run syslog-ng in the foreground if you specify these as these easily generate loops in the configuration unless the internal() source is not present. (use --foreground for that, intenral() messages will be printed on the standard error).
Judging the config I can't see an obvious problem, that's why I wanted to test it, but I'd need a sample log message for that.
-- Liam Kirsher PGP: http://liam.numenet.com/pgp/ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
On Sat, 2009-02-28 at 09:41 +0100, Balazs Scheidler wrote:
You are right. Only one of the destinations got initialized during startup. This patch fixes it for me:
diff --git a/src/apphook.c b/src/apphook.c index ab9cb02..6115b27 100644 --- a/src/apphook.c +++ b/src/apphook.c @@ -54,10 +54,10 @@ run_application_hook(gint type)
if (e->type == type) { + l_next = l->next; application_hooks = g_list_remove_link(application_hooks, l); e->func(type, e->user_data); g_free(e); - l_next = l->next; g_list_free_1(l); } else
I just wanted to add that this should be in tomorrow's snapshot. -- Bazsi
Balazs, Thanks! That did the trick. Liam Balazs Scheidler wrote:
You are right. Only one of the destinations got initialized during startup. This patch fixes it for me:
diff --git a/src/apphook.c b/src/apphook.c index ab9cb02..6115b27 100644 --- a/src/apphook.c +++ b/src/apphook.c @@ -54,10 +54,10 @@ run_application_hook(gint type)
if (e->type == type) { + l_next = l->next; application_hooks = g_list_remove_link(application_hooks, l); e->func(type, e->user_data); g_free(e); - l_next = l->next; g_list_free_1(l); } else
On Mon, 2009-02-23 at 19:31 -0800, Liam Kirsher wrote:
Balazs,
I'm afraid this message may have gotten overlooked, and I'm hoping to get this issue resolved soon so I can deal with my logging issues; so I'm sending it again. The debugging suggesting you made turned up what I believe is a bug.
I have done as you suggested and run syslog-ng in debugging mode, same syslog-ng.conf as before. It appears that the first entry line (root.geocode_access) matches the filter but does not trigger the SQL insert. However, if I reverse the order of the log{} definitions, then it does work and the other one doesn't! I get different results depending on the order of the two statements below. It looks like the SQL insert only happens for the log definition that is last.
log { source(s_sys); filter(f_geocode); parser(p_geocode); destination(d_geocode); };
log { source(s_sys); filter(f_ut_access); parser(p_ut_access); destination(d_ut_access); };
Would you please take a look?
Thanks, Liam
/usr/local/sbin/syslog-ng --foreground --verbose --debug --stderr -p /var/run/syslogd.pid Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.0.1' Database thread started; Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.geocode_access,INFO,san francisco,"San Francisco, CA, US",37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Initializing destination file writer; template='/var/log/messages', filename='/var/log/messages' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_geocode' ### Looks like a match, so SQL Insert should go here, right? Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_ut_access' Incoming log entry; line='<14>obsidian: 2009-02-17 10:47:55,75.101.83.163,/hCi/KM35kk,root.ut_access,INFO,,,,,/v1/?loc=san+francisco&start=0&rows=10&f=html,,,37.77916,-122.420049\x0a' Filter rule evaluation begins; filter_rule='f_filter2' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='facility' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_filter2' Filter rule evaluation begins; filter_rule='f_filter3' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter3' Filter rule evaluation begins; filter_rule='f_filter4' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter4' Filter rule evaluation begins; filter_rule='f_filter5' Filter node evaluation result; filter_result='not-match', filter_type='level' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter5' Filter rule evaluation begins; filter_rule='f_filter6' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter node evaluation result; filter_result='not-match', filter_type='OR' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter6' Filter rule evaluation begins; filter_rule='f_filter7' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter7' Filter rule evaluation begins; filter_rule='f_filter8' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_filter8' Filter rule evaluation begins; filter_rule='f_geocode' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='not-match' Filter node evaluation result; filter_result='not-match', filter_type='AND' Filter rule evaluation result; filter_result='not-match', filter_rule='f_geocode' Filter rule evaluation begins; filter_rule='f_ut_access' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='level' Filter node evaluation result; filter_result='match', filter_type='AND' Filter node evaluation result; filter_result='match', filter_type='filter(f_obsidian)' Filter node evaluation result; filter_result='match' Filter node evaluation result; filter_result='match', filter_type='AND' Filter rule evaluation result; filter_result='match', filter_rule='f_ut_access' Running SQL query; query='SELECT * FROM ut_access_log WHERE 0=1' Running SQL query; query='INSERT INTO ut_access_log (datetime, query_time, host, program, pid, request_id, level, ip, phone_id, phone_type, software_version, client_version, query_string, art_id, session_id, lat, lng) VALUES (\'2009-02-17T13:47:55-05:00\', \'2009-02-17 10:47:55\', \'127.0.0.1\', \'obsidian\', \'\', \'/hCi/KM35kk\', \'info\', \'75.101.83.163\', \'\', \'\', \'\', \'\', \'/v1/?loc=san+francisco&start=0&rows=10&f=html\', \'\', \'\', \'37.77916\', \'-122.420049\')'
Balazs Scheidler wrote:
On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
Hi --
I am /almost/ there, logging to Postgres database. However, I've discovered a puzzling and problematic behavior.This is probably just some simple misunderstanding on my part, since this is my first foray into syslog-ng. I am logging to two different db tables. Which table I log to is determined by a regexp filter. The value is either root.ut_access or root.geocode. I can get either one to work, but not both at the same time. If I comment out the log entry for the geocode, then ut_access works. However, if both log entries exist, only the gecocode_access_log table gets a new row. Nothing is logged to the ut_access_log table! (Both messages are logged to d_obsidian destination file, however.) I've attached my config file.
Hmm.. could you post two example messages that should go to one or the other destination?
Since you didn't specify flags(final) to either log statements, both should be doing their job, independently from the other. The only thing that should control whether one or the other destination is used is the attached filter. You can get filter debugging by enabling the --debug / --verbose options.
Be sure that you run syslog-ng in the foreground if you specify these as these easily generate loops in the configuration unless the internal() source is not present. (use --foreground for that, intenral() messages will be printed on the standard error).
Judging the config I can't see an obvious problem, that's why I wanted to test it, but I'd need a sample log message for that.
-- Liam Kirsher PGP: http://liam.numenet.com/pgp/ ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Liam Kirsher PGP: http://liam.numenet.com/pgp/
On Fri, 2009-02-13 at 12:25 -0800, Liam Kirsher wrote:
Hi --
filter f_ut_access { filter(f_obsidian) and message("root\.ut_access"); }; filter f_geocode { filter(f_obsidian) and message("root\.geocode"); };
I've noticed one problem here, '\' is used by syslog-ng itself to quote characters, if you want to quote a regexp character use double backslashes: root\\.ut_access This may or may not solve your problem. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Liam Kirsher