May 2015 - syslog-ng - lists.balabit.hu

3.2.5 and Multiline(?) messages from Solaris
by Ray Van Dolson 12 May '15

12 May '15

Admittedly haven't done enough searching or testing on this, but am hoping someone might have a quick answer. Recently moved from the 2.x verions to 3.2.5 (as part of EPEL on RHEL6). Have noticed that we're no longer getting the full messages from some Solaris boxen using the tcp() and udp() source definitions. Messages like this: May 10 02:29:30 dev-zfs2 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): May 10 02:29:30 dev-zfs2 Log info 0x31080000 received for target 24. May 10 02:29:30 dev-zfs2 scsi_status=0x0, ioc_status=0x804b, scsi_state=0x0 Come through looking like this: May 10 02:29:30 dev-zfs2 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): (Only the initial line) However, messages like this one: May 9 04:12:57 dev-zfs2 scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): May 9 04:12:57 dev-zfs2 mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110610 .. do seem to be coming through "whole" (I do note that the priority is different in both). Relevant config items are as follows: log { source(remote); filter(syslog); destination(hosts_syslog); }; source remote { udp(); tcp(); # udp(ip(0.0.0.0) port(514)); # tcp(ip(0.0.0.0) port(514)); }; destination hosts_syslog { file("/logs/hosts/$HOST/$YEAR/$MONTH/syslog.$HOST.$YEAR.$MONTH.log" create_dirs(yes)); pipe("/logs/hosts/everything.fifo"); }; filter syslog { (not facility(mail) and not filter(f_ucgw) and not filter(f_esx)); }; Will try and do some packet captures to confirm Solaris is, in fact, sending the entire message (I believe it is since this worked on syslog-ng 2.x). Thanks, Ray

2 4

Error resolving reference; content='source', name='src', location='/etc/syslog-ng/syslog-ng.conf:26:7
by Rick Silacci 12 May '15

12 May '15

I can't figure out why I'm getting this message. Keep in mind, I just started using syslog. Here's the cfg: @version: 3.5 @include "scl.conf" @include "`scl-root`/system/tty10.conf" # Syslog-ng configuration file, compatible with default Debian syslogd # installation. # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; ######################## # Sources ######################## # This is the default behavior of sysklogd package # Logs may come from unix stream, but not from another machine. # #source s_src { # system(); # internal(); #}; destination mongodb { mongodb(); }; log { source(); destination(mongodb); }; # If you wish to get logs from remote machine you should uncomment # this and comment the above source line. # source s_net { tcp(ip(127.0.0.1) port(1000) keep-alive(yes)); }; ######################## # Destinations ######################## # First some standard logfile # destination d_auth { file("/var/log/auth.log"); }; destination d_cron { file("/var/log/cron.log"); }; destination d_daemon { file("/var/log/daemon.log"); }; destination d_kern { file("/var/log/kern.log"); }; destination d_lpr { file("/var/log/lpr.log"); }; destination d_mail { file("/var/log/mail.log"); }; destination d_syslog { file("/var/log/syslog"); }; destination d_user { file("/var/log/user.log"); }; destination d_uucp { file("/var/log/uucp.log"); }; #destination mongodb { file("/var/log/mongodb.log"); }; # This files are the log come from the mail subsystem. # destination d_mailinfo { file("/var/log/mail.info"); }; destination d_mailwarn { file("/var/log/mail.warn"); }; destination d_mailerr { file("/var/log/mail.err"); }; # Logging for INN news system # destination d_newscrit { file("/var/log/news/news.crit"); }; destination d_newserr { file("/var/log/news/news.err"); }; destination d_newsnotice { file("/var/log/news/news.notice"); }; # Some `catch-all' logfiles. # destination d_debug { file("/var/log/debug"); }; destination d_error { file("/var/log/error"); }; destination d_messages { file("/var/log/messages"); }; # The root's console. # destination d_console { usertty("root"); }; # Virtual console. # destination d_console_all { file(`tty10`); }; # The named pipe /dev/xconsole is for the nsole' utility. To use it, # you must invoke nsole' with the -file' option: # # $ xconsole -file /dev/xconsole [...] # destination d_xconsole { pipe("/dev/xconsole"); }; # Send the messages to an other host # #destination d_net { tcp("127.0.0.1" port(1000) log_fifo_size(1000)); }; # Debian only destination d_ppp { file("/var/log/ppp.log"); }; ######################## # Filters ######################## # Here's come the filter options. With this rules, we can set which # message go where. filter f_dbg { level(debug); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_err { level(err); }; filter f_crit { level(crit .. emerg); }; filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); }; filter f_error { level(err .. emerg) ; }; filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news); }; filter f_auth { facility(auth, authpriv) and not filter(f_debug); }; filter f_cron { facility(cron) and not filter(f_debug); }; filter f_daemon { facility(daemon) and not filter(f_debug); }; filter f_kern { facility(kern) and not filter(f_debug); }; filter f_lpr { facility(lpr) and not filter(f_debug); }; filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); }; filter f_mail { facility(mail) and not filter(f_debug); }; filter f_news { facility(news) and not filter(f_debug); }; filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_user { facility(user) and not filter(f_debug); }; filter f_uucp { facility(uucp) and not filter(f_debug); }; filter f_cnews { level(notice, err, crit) and facility(news); }; filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); }; filter f_ppp { facility(local2) and not filter(f_debug); }; filter f_console { level(warn .. emerg); }; ######################## # Log paths ######################## log { source(s_src); filter(f_auth); destination(d_auth); }; log { source(s_src); filter(f_cron); destination(d_cron); }; log { source(s_src); filter(f_daemon); destination(d_daemon); }; log { source(s_src); filter(f_kern); destination(d_kern); }; log { source(s_src); filter(f_lpr); destination(d_lpr); }; log { source(s_src); filter(f_syslog3); destination(d_syslog); }; log { source(s_src); filter(f_user); destination(d_user); }; log { source(s_src); filter(f_uucp); destination(d_uucp); }; log { source(s_src); filter(f_mail); destination(d_mail); }; #log { source(s_src); filter(f_mail); filter(f_info); destination(d_mailinfo); }; #log { source(s_src); filter(f_mail); filter(f_warn); destination(d_mailwarn); }; #log { source(s_src); filter(f_mail); filter(f_err); destination(d_mailerr); }; log { source(s_src); filter(f_news); filter(f_crit); destination(d_newscrit); }; log { source(s_src); filter(f_news); filter(f_err); destination(d_newserr); }; log { source(s_src); filter(f_news); filter(f_notice); destination(d_newsnotice); }; #log { source(s_src); filter(f_cnews); destination(d_console_all); }; #log { source(s_src); filter(f_cother); destination(d_console_all); }; #log { source(s_src); filter(f_ppp); destination(d_ppp); }; log { source(s_src); filter(f_debug); destination(d_debug); }; log { source(s_src); filter(f_error); destination(d_error); }; log { source(s_src); filter(f_messages); destination(d_messages); }; log { source(s_src); filter(f_console); destination(d_console_all); destination(d_xconsole); }; log { source(s_src); filter(f_crit); destination(d_console); }; # All messages send to a remote site # #log { source(s_src); destination(d_net); }; ### # Include all config files in /etc/syslog-ng/conf.d/ ### @include "/etc/syslog-ng/conf.d/*.conf"

2 6

Re: [syslog-ng] 3.2.5 and Multiline(?) messages from Solaris
by Jim Hendrick 12 May '15

12 May '15

You hit it I think. The first thing is to capture packets on at least one end to isolate where this is happening. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Ray Van Dolson <rvandolson(a)esri.com> Date: 05/11/2015 11:42 PM (GMT-05:00) To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] 3.2.5 and Multiline(?) messages from Solaris Admittedly haven't done enough searching or testing on this, but am hoping someone might have a quick answer. Recently moved from the 2.x verions to 3.2.5 (as part of EPEL on RHEL6). Have noticed that we're no longer getting the full messages from some Solaris boxen using the tcp() and udp() source definitions. Messages like this: May 10 02:29:30 dev-zfs2 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): May 10 02:29:30 dev-zfs2 Log info 0x31080000 received for target 24. May 10 02:29:30 dev-zfs2 scsi_status=0x0, ioc_status=0x804b, scsi_state=0x0 Come through looking like this: May 10 02:29:30 dev-zfs2 scsi: [ID 365881 kern.info] /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): (Only the initial line) However, messages like this one: May 9 04:12:57 dev-zfs2 scsi: [ID 243001 kern.warning] WARNING: /pci@0,0/pci8086,3410@9/pci15d9,400@0 (mpt_sas0): May 9 04:12:57 dev-zfs2 mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31110610 .. do seem to be coming through "whole" (I do note that the priority is different in both). Relevant config items are as follows: log { source(remote); filter(syslog); destination(hosts_syslog); }; source remote { udp(); tcp(); # udp(ip(0.0.0.0) port(514)); # tcp(ip(0.0.0.0) port(514)); }; destination hosts_syslog { file("/logs/hosts/$HOST/$YEAR/$MONTH/syslog.$HOST.$YEAR.$MONTH.log" create_dirs(yes)); pipe("/logs/hosts/everything.fifo"); }; filter syslog { (not facility(mail) and not filter(f_ucgw) and not filter(f_esx)); }; Will try and do some packet captures to confirm Solaris is, in fact, sending the entire message (I believe it is since this worked on syslog-ng 2.x). Thanks, Ray ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

1 0

unix-dgram for Linux syslog stream
by ZeroUno 09 May '15

09 May '15

Hi, I've read some old discussions about unix-dgram vs unix-stream for getting the /dev/log syslog stream on Linux (e.g. https://bugs.archlinux.org/task/22153) but some years have passed by and I'd like to be sure my configuration is safe. I'm using syslog-ng 3.2.5 on a system composed of different Linux (RHEL6) machines. I cannot install a different version. There is a central log facility, but each machine collects and sends its logs using the following source config: source src { file ("/proc/kmsg" program_override("kernel: ")); unix-dgram ("/dev/log" flags(no-multi-line)); internal(); }; It was originally using unix-stream(), but I need to change it to unix-dgram() because some custom applications are sending multiline messages which need to be converted into single line, and unix-stream() does not support this flag. Can I be sure that no messages risk to be lost due to this change? This is a rather critical application. Thank you very much. -- 01

2 2

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 06 May '15

06 May '15

Hi! I'm the mentor for the Riak destination for syslog-ng project, please allow me to answer the questions below: >>>>> "Fred" == Fred Dushin <fdushin(a)basho.com> writes: Fred> As far as I understand, you're talking about a mapping from keys to Fred> sets, but I'm unclear on a few things. The idea is to map a set of log messages to a Riak Set. Where both the key used for the set, and its contents are configurable by the user. There are no plans for a default at this time. There are many ways to configure a syslog-ng=>Riak setup with a destination like the one planned. One is to turn each log message (after parsing) to a Riak Map, and push those maps into a Riak Set. Another way is to format the parsed log messages (with all the extracted fiels, if any) into JSON, and push those into a set. So, for example, given the following syslog line: May 6 14:42:18 eowyn avahi-daemon[27812]: Invalid response packet from host fe80::5d0f:d53a:7b6:3680. We'd end up with a JSON like this: {"timestamp": "2015-05-06T14:42:18+02:00", "host": "eowyn", "program": "avahi-daemon", "pid": 27812, "message": "Invalid response packet from host fe80::5d0f:d53a:7b6:3680.", "avahi-daemon": { "type": "warning", "message": "Invalid response packet", "host": "fe80::5d0f:d53a:7b6:3680" } } We could either add that to a Riak set as-is, or turn it into a Riak map first. Fred> What are the keys you are thinking about? Time stamps? If Fred> timestamps, these are presumably the timestamps of the syslog Fred> event? Whatever the user configures. They may be time stamps (rounded, for predictable keys), or a combination of program name + current date (day granuality). Fred> Just a word of warning, if so. You might find a lot of Fred> variation in timestamp formats and granularity. Perhaps you Fred> can get something reliable out of syslog-ng, We get something sensible out of syslog-ng. But in the end, it is up to the user to configure the template used for keys. There may - and probably will - be examples, but no default. Fred> but that won't help you in the case where syslog-ng is Fred> functioning as a syslog relay, and you want to preserve the Fred> timestamp of the originator, which you should, if you want to Fred> preserve integrity of the logs (e.g, for compliance). In case of syslog-ng, we actually have access to a few kinds of timestamps: the timestamp from the log message (if any), the timestamp of receipt, and the current time. The granularity of timestamps is configurable to some extent. Fred> Or are you talking about a key being a (course grained) Fred> timestamp, say, an integral value in UTC seconds, for example? Fred> And the value(s) being all logs in that interval? Is that your Fred> motivation for sets? That's one way, yes. One could also use something like $PROGRAM/$YEAR-$MONTH-$DAY as key, if the program doesn't produce more than a megabyte of logs a day. So with the example above, our key in case of that log would be avahi-daemon/2015-05-06, and the message would be an element of the set underneath the key. Fred> How much of the syslog payload are you planning to parse? The destination itself is not going to do any parsing. Other parts of syslog-ng do that, and it is up to the user to set up a pipeline that feeds the destination. The source may be syslog, HTTP logs, the Journal, or any of the other sources syslog-ng supports. How much parsing is done, and what gets extracted, is no concern to the destination plugin. Fred> Another interesting problem is that the STRUCTURED-DATA element of Fred> 5424 uses OIDs to discriminate different data types that are encoded Fred> in the header. And while there is a kind of loosely coupled authority Fred> for OIDs, there is no infrastructure for determining a parsing Fred> strategy for these fields. They could really be anything, in the worst Fred> case. As far as I remember, syslog-ng treats all STRUCTURED-DATA elements as strings. But there are tools within syslog-ng to allow converting to other data types, but that must be done explicitly. Fred> But regardless of the deeply structured data, you could get some very Fred> interesting traction by just taking standard headers and indexing them Fred> through Yokozuna. Certainly, indexing the body of a syslog message is Fred> a great idea, as these messages are generally unstructured and fodder Fred> for lucene. This is something that Logstash/ElasticSearch can do Fred> pretty effectively today, and it would be cool to see the same in Riak Fred> + some syslog provider. Yep! When I proposed the idea, using Yokozuna is something I had in mind. Combine the parsing abilities of syslog-ng, Riak for archival purposes, and Yokozuna for searching. That sounds like a match made in heaven. Fred> Finally, it would be really nice if you could structure your plugin in Fred> such a way that they could eventually be ported to rsyslog [2]. The Fred> rsyslogd daemon is deployed by default on certain Linux favors and Fred> enjoys fairly widespread distribution. You might be able to get it Fred> supported in that community, as well. Part of the project is writing a small library to send data to Riak, From C. Just enough for syslog-ng's needs. That library could be used by rsyslog, too (like the MongoDB library originally written for syslog-ng's purposes is used by rsyslog too). But sharing more code than that is not practical, the two daemons work in widely different ways. -- |8]

1 0

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 05 May '15

05 May '15

>>>>> "Christopher" == Christopher Meiklejohn <cmeiklejohn(a)basho.com> writes: >> The intention is to store log messages in each element of the set: >> either as a string (syslog or json, or whatever else the user sees fit), >> or as a map of key-value pairs (where values themselves can be maps >> too). >> >> On average, the log messages are a few kilobytes in size. There may be >> exceptions, but >1mb ones are fairly rare. How much data the set would >> hold... now that's a question that can't really be answered. It is >> really up to the syslog-ng user to configure that. Christopher> I’m referring to the size of the entire set, not the objects that will be members of Christopher> the set. Therefore, the performance penalty seen when using large objects would Christopher> be observed as soon as the size of the entire set (or map) has reached ~1 MB. Christopher> Given that restriction, I’d imagine you would only be able to store a few messages Christopher> in each set. That granularity seems like you are no longer getting the benefits Christopher> of the set. The granularity is configurable by the user: if they have small (say, a few hundred byte long) messages, then we can store a reasonable amount of them in a single Set. For example, assuming an average length of 384 bytes / message (the longest line in today's logs on my laptop), a Set would be able to store about 2k messages. That's not too bad. Christopher> Additionally, the primary benefit of the data types in Riak is that they converge Christopher> deterministically when dealing with concurrent Christopher> operations. Not only that: using sets makes the keys predictable. If I want to retrieve the logs, with sets, I can retrieve the 2015-05-01T15:12:10-T15:12:15 key for example, and have all the logs from those 5 seconds. If I used one message per Riak object, it would be much harder to read the data back. There may be multiple threads adding to the same set, so the deterministic convergence is useful still. Christopher> I’m curious if the set is the right choice here; could Christopher> you just use a custom set format inside of a normal Christopher> Riak object That's an option worth considering, yes. Thanks! Christopher> (or store one message per Riak object, given the write will be an Christopher> immutable log entry?) That's the first goal of the project, because that's the easiest and most straightforward to implement. The downside of one message per Riak object is that it's hard to retrieve the data, because making the keys predictable is not going to be easy. Boxing a few hundred (or few thousand) together into a Set has the advantage of making the keys predictable at the cost of transferring more data to the client when looking for a subset of the logs within the set. -- |8]

1 0

Re: [syslog-ng] Feedback for GSoC project - RIak Destination for Syslog-ng
by Gergely Nagy 05 May '15

05 May '15

>>>>> "Christopher" == Christopher Meiklejohn <cmeiklejohn(a)basho.com> writes: Christopher> I’m a bit concerned with your use of the set embedded in the Christopher> map. The original idea was to use a Set directly. The Set-in-Map thing was just a thought experiment (Map-in-Set would make more sense). Christopher> Large objects have traditionally been a big problem in Riak due Christopher> to the use of distributed Erlang and head of line blocking. I’m Christopher> curious if you could elaborate on what type of data you will be Christopher> storing in the set: how big you expect each item to be, how big you Christopher> expect the map to be, and the overall layout of data inside of the Christopher> data structure. The intention is to store log messages in each element of the set: either as a string (syslog or json, or whatever else the user sees fit), or as a map of key-value pairs (where values themselves can be maps too). On average, the log messages are a few kilobytes in size. There may be exceptions, but >1mb ones are fairly rare. How much data the set would hold... now that's a question that can't really be answered. It is really up to the syslog-ng user to configure that. Hope that made it a little more clear. -- |8], mentor for the riak destination for syslog-ng GSoC project

1 0

Feedback for GSoC project - RIak Destination for Syslog-ng
by Parth Oberoi 02 May '15

02 May '15

Hello Everyone, My proposal[0] for adding Riak as a destination to Syslog-ng was accepted for GSoC2015, so before my coding period begins i.e. 25th May, I wanted to do some ground work and take some feedback from the potential users. The proposal[0] contains all the initial implementation details which might require improvement . Therefore if some of you could take a look and give me some feedback on it, would help me greatly. Below are a few links to important snippets of the proposal. [0] - https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Goals for the project :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Development Tools :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *Deliverables :-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… *My scheduled Plan:-* https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destinati… Thanks in advance. Regards, Parth Oberoi (@htrap)

1 0