syslog-ng is not reading level/facility properly
Hi List, After my last weeks note, I kept on doing some more digging. Perhaps some of this output will aid in finding a solution. I compiled from source, from the latest stable 2.0 line thinking maybe that was part of my problem since I installed via the epel repositories. Nope, still having the same problem. On a remote host, I have a nginx server, that is sending it's access logs to local5.notice and it's error logs to local5.crit. In the syslog.conf file I have the following entries. local5.notice @syslog-ng server local5.crit @syslog-ng server This data is definitely making it to the systlog server. I have done a tcpdump and it shows this with the following. Also I have it logging to local log files and they are working the way I expect. $ tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 16:16:40.725473 IP CLIENTIP.syslog > SYSLOGNGIP.syslog: SYSLOG local5.notice, length: 304 <-- coming in as local5.notice as I expect. Okay so they are hitting the server how I expect, but they aren't being filtered the way I expect. Below is my syslog-ng config. # Syslog-ng config options { use_dns(no); keep_hostname(yes); long_hostnames(off); chain_hostnames(off); sync(1); log_fifo_size(1024); create_dirs(yes); perm(0644); stats_freq(3600); mark_freq(600); }; # # Data Sources # # External Syslog source s_external { udp(ip("0.0.0.0") port(514) flags(no_parse)); }; # # Where to put Data # # Nginx / Lighttpd / Apache Access Logs destination d_access_log { file("/data01/syslog-ng/$HOST/ access.log"); }; # Nginx / Lighttpd / Apache Error Logs destination d_error_log { file("/data01/syslog-ng/$HOST/error.log"); }; # Everything else destination d_log { file("/data01/syslog-ng/$HOST/$FACILITY.$LEVEL"); }; # # Data Filters # # Facility local5 filter f_local5 { facility(local5); }; # Level info filter f_info { level(info); }; # Level notice filter f_notice { level(notice); }; # Level warn filter f_warn { level(warn); }; # Level crit filter f_crit { level(crit); }; # Level err filter f_err { level(err); }; # # Data Logging Locations # # Access Logs log { source(s_external); filter(f_local5); filter(f_notice); destination(d_access_log); }; # Error Logs log { source(s_external); filter(f_local5); filter(f_crit); destination(d_error_log); }; # Everything Else log { source(s_external); destination(d_log); }; Now the only place that ANY messages seem to go, is into /data01/ syslog-ng/$HOST/kern.emerg. Why is everything being marked as kern.emerg? And lastly here's some debug output that maybe able to shed some light to those who know more than I do. ./sbin/syslog-ng -d -v -e --cfgfile=./etc/syslog-ng.conf Running application hooks; hook='1' syslog-ng starting up; version='2.0.11' Incoming log entry; line='<156>logger: REMOTEIP www.domain.com - [08/ Sep/2010:17:17:44 -0400] "GET /2010/09/08-7912 HTTP/1.0" 200 83134 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) "\x0a' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' For some reason this isn't matching, but I'm officially out of experience on this to know what to do further. I should also note that I'm running splunk on this syslog-ng server and it's able to get my access logs properly. If anyone can throw me a bone here I'd really appreciate it, many thanks in advance. Charlie
See below On 9/8/2010 5:31 PM, Charlie Reddington wrote:
Hi List,
After my last weeks note, I kept on doing some more digging. Perhaps some of this output will aid in finding a solution.
I compiled from source, from the latest stable 2.0 line thinking maybe that was part of my problem since I installed via the epel repositories. Nope, still having the same problem.
On a remote host, I have a nginx server, that is sending it's access logs to local5.notice and it's error logs to local5.crit. In the syslog.conf file I have the following entries.
local5.notice @syslog-ng server local5.crit @syslog-ng server
This data is definitely making it to the systlog server. I have done a tcpdump and it shows this with the following. Also I have it logging to local log files and they are working the way I expect.
$ tcpdump port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 16:16:40.725473 IP CLIENTIP.syslog> SYSLOGNGIP.syslog: SYSLOG local5.notice, length: 304<-- coming in as local5.notice as I expect.
Okay so they are hitting the server how I expect, but they aren't being filtered the way I expect. Below is my syslog-ng config.
# Syslog-ng config
options { use_dns(no); keep_hostname(yes); long_hostnames(off); chain_hostnames(off); sync(1); log_fifo_size(1024); create_dirs(yes); perm(0644); stats_freq(3600); mark_freq(600); };
# # Data Sources #
# External Syslog source s_external { udp(ip("0.0.0.0") port(514) flags(no_parse)); };
# # Where to put Data #
# Nginx / Lighttpd / Apache Access Logs destination d_access_log { file("/data01/syslog-ng/$HOST/ access.log"); };
# Nginx / Lighttpd / Apache Error Logs destination d_error_log { file("/data01/syslog-ng/$HOST/error.log"); };
# Everything else destination d_log { file("/data01/syslog-ng/$HOST/$FACILITY.$LEVEL"); };
# # Data Filters #
# Facility local5 filter f_local5 { facility(local5); };
# Level info filter f_info { level(info); }; # Level notice filter f_notice { level(notice); }; # Level warn filter f_warn { level(warn); }; # Level crit filter f_crit { level(crit); }; # Level err filter f_err { level(err); };
# # Data Logging Locations #
# Access Logs log { source(s_external); filter(f_local5); filter(f_notice); destination(d_access_log); };
# Error Logs log { source(s_external); filter(f_local5); filter(f_crit); destination(d_error_log); };
# Everything Else log { source(s_external); destination(d_log); };
Now the only place that ANY messages seem to go, is into /data01/ syslog-ng/$HOST/kern.emerg.
Why is everything being marked as kern.emerg? It's not clear to me what is going to kern.emerg. Are the apache logs being put into the file, or something else? And lastly here's some debug output that maybe able to shed some light to those who know more than I do.
./sbin/syslog-ng -d -v -e --cfgfile=./etc/syslog-ng.conf
Running application hooks; hook='1' syslog-ng starting up; version='2.0.11'
Incoming log entry; line='<156>logger: REMOTEIP www.domain.com - [08/ Sep/2010:17:17:44 -0400] "GET /2010/09/08-7912 HTTP/1.0" 200 83134 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) "\x0a' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' Filter rule evaluation begins; filter_rule='f_local5' Filter node evaluation result; filter_result='not-match', filter_type='facility' Filter rule evaluation result; filter_result='not-match', filter_rule='f_local5' The priority is <156> on this log, for some reason. 156 equates to local3.warn, so not too surprising that the filter isn't matching.
For some reason this isn't matching, but I'm officially out of experience on this to know what to do further. I should also note that I'm running splunk on this syslog-ng server and it's able to get my access logs properly.
If anyone can throw me a bone here I'd really appreciate it, many thanks in advance.
Charlie
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
It would be interesting to see what you get if you created a destination like this: destination d_file { file(“/var/log/test” template(“$FACILITY:$PRIORITY $MSG\n”; template_escape(no))); }; and let everything flow into it. At least it would tell you what facility/priority the different messages have, which could help get to the next step in fixing the problem. Regards, Jerry
A second possibility on top of the items Jerry pointed out such as possibly using wrong facilities on the sources, etc. I see you have set the no_parse flag on your syslog socket. This instructs Syslog that your messages are non standard or should be handled in a raw fashion. This causes Syslog to be incapable of determining facility, severity, hostname, and tag information. Since it appears nginx is sending properly made messages this is likely not what you want. In order to confirm you should take a tcpdump with -s 0 to capture entire frames, and load this in Wireshark or tshark with -V option to look at the payload of the Syslog packets and compare it against what's written in these RFCs and documents: http://www.ietf.org/rfc/rfc3164.txt http://www.ietf.org/rfc/rfc5424.txt http://www.monitorware.com/common/en/articles/syslog-described.php http://www.networksorcery.com/enp/protocol/syslog.htm There are many more. You need to be sure you are handling the things you are receiving appropriately for the format which is being sent to you. If you have some messages which should be parsed and some which should not, these should be sent onto separate ports so you can have the right flags set on each port. HTH, Matthew. On Wed, Sep 08, 2010 at 06:03:33PM -0400, Jerry Bell wrote:
On 9/8/2010 5:31 PM, Charlie Reddington wrote:
# External Syslog source s_external { udp(ip("0.0.0.0") port(514) flags(no_parse)); };
You sir, are a god send. :) I should have formatted my question this way the first time. On Sep 8, 2010, at 5:16 PM, Matthew Hall wrote:
A second possibility on top of the items Jerry pointed out such as possibly using wrong facilities on the sources, etc.
I see you have set the no_parse flag on your syslog socket.
This instructs Syslog that your messages are non standard or should be handled in a raw fashion. This causes Syslog to be incapable of determining facility, severity, hostname, and tag information. Since it appears nginx is sending properly made messages this is likely not what you want.
This is actually exactly what was wrong with this. I had my doubts about this parse flag, but I was a bit unsure of what it did or didn't do. And we where using it in another location, so I was using that as my template. The way you explained it though, makes perfect sense. I removed it, and everything starting logging the way I expected it to.
There are many more. You need to be sure you are handling the things you are receiving appropriately for the format which is being sent to you.
If you have some messages which should be parsed and some which should not, these should be sent onto separate ports so you can have the right flags set on each port.
Duly noted, I'll make sure to do that when I bring the rest of our gear into the mix with this host. Thanks both of you for the help! Charlie
participants (3)
-
Charlie Reddington
-
Jerry Bell
-
Matthew Hall