[RFC (3.4) 0/5]: multiline & linux 3.5+ /dev/kmsg support
The next few patches gradually introduce two - in my opinion - interesting features for syslog-ng 3.4: support for one kind of multiline messages, where continuation lines start with whitespace; and support for the /dev/kmsg format introduced in linux 3.5. Neither implementation is complete yet, there are subtle issues with both (see later), but they're at a stage where taking them to a test drive would be useful, and code review wouldn't hurt either. The patches are also available on the feature/3.4/indented-multiline[1] branch of my git repository[2]. The branch may get rebased in the future, I do not make any promise of keeping it fast-forwardable. [1]: https://github.com/algernon/syslog-ng/commits/feature/3.4/indented-multiline [2]: git://github.com/algernon/syslog-ng.git linux 3.5 /dev/kmsg =================== The linux 3.5+ /dev/kmsg support is the easier to test: you only need a 3.5+ kernel, and when using the system() source, things will magically work! This includes parsing the relative timestamps in kmsg messages and also parsing any additional key-value pairs. This means that if we had an input line like this: ,---- | 6,802,65338577;ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex> | SUBSYSTEM=pci | DEVICE=+pci:0000:02:00.0 `---- Then we'll get back something like this: ,---- | { | "FILE_NAME" : "/dev/kmsg", | "MSGID" : "802", | "HOST" : "luthien", | "DATE" : "Oct 13 16:53:44", | "FACILITY" : "kern", | "SOURCEIP" : "127.0.0.1", | "TAGS" : ".source.s_kmsg", | "kernel" : { | "SUBSYSTEM" : "pci", | "DEVICE" : { | "name" : "0000:02:00.0", | "type" : "pci" | } | }, | "PRIORITY" : "info", | "HOST_FROM" : "luthien", | "MESSAGE" : "ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex>", | "PROGRAM" : "kernel" | } `---- For the above, the following template was used: $(format-json --scope selected-macros --scope nv-pairs --key .kernel.* --shift 1) (Mind you, program_override() is currently broken in 3.4, but that is unrelated to the /dev/kmsg support) Any name-value pairs the kernel supplies will be put into variables of the same name, prefixed with ".kernel." (so SUBSYSTEM becomes .kernel.SUBSYSTEM), and the value of DEVICE= will be further parsed, based on rules written down in the appropriate kernel docs. Indented multiline ================== The indented multiline support enhances the file() and tcp() sources with the ability to read multiline records (if the indented-multiline flag is set), where continuation lines start with whitespace. Such output is produced by the $(indent-multi-line) template function. Usage: source s_ml { tcp(port(12345) flags(indented-multiline, no-parse)); }; Then, you can test it like this: (cat <<EOF This is the first line - second - third EOF ) | nc localhost 12345 Using a JSON output, this would result in: {"MESSAGE":"This is the first line\n - second\n - third"} A record can be terminated in two ways: by another line that does not begin with whitespace, or by reacing EOF. This means that if netcat didn't close the connection, we wouldn't get the test record until another line arrives. Unfortunately, there's not much I can do about this limitation, but if anyone has a good idea how to work around or fix it, please let me know! Known issues ============ The /dev/kmsg handling is not reload/restart-safe: if syslog-ng gets reloaded or restarted, it will read the whole lot of /dev/kmsg over again, for example. There are possibly other subtle issues too, but all of those will be ironed out before I submit the patches for merging. The indented-multiline support has a more serious limitation: if data needs to be split (which happens in some circumstances when the incoming data is bigger than the internal message size (8192 by default)), it is not correctly reassembled, making indented-multiline less than useful for some tasks.
The indented multiline class is derived from LogProtoTextServer, the main difference is in determining what constitutes a full line: while the text server considers a newline as a terminator, the indented multiline server has stricter requirements: the newline must be followed by another line that does not start with whitespace. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- lib/Makefile.am | 2 + lib/logproto-builtins.c | 3 + lib/logproto-indented-multiline-server.c | 90 ++++++++++++++++++++++++++++++ lib/logproto-indented-multiline-server.h | 44 +++++++++++++++ lib/logproto-text-server.c | 37 +++++++----- lib/logproto-text-server.h | 8 +++ tests/unit/test_logproto.c | 45 +++++++++++++++ 7 files changed, 216 insertions(+), 13 deletions(-) create mode 100644 lib/logproto-indented-multiline-server.c create mode 100644 lib/logproto-indented-multiline-server.h diff --git a/lib/Makefile.am b/lib/Makefile.am index 65de69a..406cbd3 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -54,6 +54,7 @@ pkginclude_HEADERS = \ logproto-framed-server.h \ logproto-text-client.h \ logproto-text-server.h \ + logproto-indented-multiline-server.h \ logproto-record-server.h \ logproto-builtins.h \ logproto.h \ @@ -137,6 +138,7 @@ libsyslog_ng_la_SOURCES = \ logproto-framed-server.c \ logproto-text-client.c \ logproto-text-server.c \ + logproto-indented-multiline-server.c \ logproto-record-server.c \ logproto-builtins.c \ logqueue.c \ diff --git a/lib/logproto-builtins.c b/lib/logproto-builtins.c index bbe23ce..c1d674c 100644 --- a/lib/logproto-builtins.c +++ b/lib/logproto-builtins.c @@ -24,6 +24,7 @@ #include "logproto-dgram-server.h" #include "logproto-text-client.h" #include "logproto-text-server.h" +#include "logproto-indented-multiline-server.h" #include "logproto-framed-client.h" #include "logproto-framed-server.h" #include "plugin.h" @@ -35,6 +36,7 @@ DEFINE_LOG_PROTO_SERVER(log_proto_dgram); DEFINE_LOG_PROTO_CLIENT(log_proto_text); DEFINE_LOG_PROTO_SERVER(log_proto_text); +DEFINE_LOG_PROTO_SERVER(log_proto_indented_multiline); DEFINE_LOG_PROTO_CLIENT(log_proto_framed); DEFINE_LOG_PROTO_SERVER(log_proto_framed); @@ -43,6 +45,7 @@ static Plugin framed_server_plugins[] = LOG_PROTO_SERVER_PLUGIN(log_proto_dgram, "dgram"), LOG_PROTO_CLIENT_PLUGIN(log_proto_text, "text"), LOG_PROTO_SERVER_PLUGIN(log_proto_text, "text"), + LOG_PROTO_SERVER_PLUGIN(log_proto_indented_multiline, "indented-multiline"), LOG_PROTO_CLIENT_PLUGIN(log_proto_framed, "framed"), LOG_PROTO_SERVER_PLUGIN(log_proto_framed, "framed"), }; diff --git a/lib/logproto-indented-multiline-server.c b/lib/logproto-indented-multiline-server.c new file mode 100644 index 0000000..e3b52d3 --- /dev/null +++ b/lib/logproto-indented-multiline-server.c @@ -0,0 +1,90 @@ +/* + * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012 Gergely Nagy <algernon@balabit.hu> + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + */ +#include "logproto-indented-multiline-server.h" +#include "messages.h" + +#include <stdio.h> + +static gboolean +log_proto_indented_multiline_server_line_is_complete(LogProtoTextServer *self, + LogProtoBufferedServerState *state, + const guchar **eol) +{ + gsize pos; + + if (!*eol) + return FALSE; + + /* If we have eol, find more eols until we get one where the next + char is non-whitespace. */ + + pos = *eol - self->super.buffer; + + while (pos < state->pending_buffer_end - 1) + { + if (self->super.buffer[pos] == '\n') + *eol = self->super.buffer + pos; + else + { + pos++; + continue; + } + + if (self->super.buffer[pos + 1] != ' ' && + self->super.buffer[pos + 1] != '\t') + return TRUE; + + pos++; + } + + return FALSE; +} + +static void +log_proto_indented_multiline_server_line_flush (LogProtoTextServer *self, + LogProtoBufferedServerState *state, + const guchar **msg, gsize *msg_len, + const guchar *buffer_start, gsize buffer_bytes) +{ + *msg = buffer_start; + *msg_len = buffer_bytes; + if ((*msg)[buffer_bytes - 1] == '\n') + (*msg_len)--; + + state->pending_buffer_pos = state->pending_buffer_end; +} + +void +log_proto_indented_multiline_server_init(LogProtoIMultiLineServer *self, + LogTransport *transport, + const LogProtoServerOptions *options) +{ + log_proto_text_server_init(&self->super, transport, options); + self->super.is_line_complete = log_proto_indented_multiline_server_line_is_complete; + self->super.line_flush = log_proto_indented_multiline_server_line_flush; +} + +LogProtoServer * +log_proto_indented_multiline_server_new(LogTransport *transport, const LogProtoServerOptions *options) +{ + LogProtoIMultiLineServer *self = g_new0(LogProtoIMultiLineServer, 1); + + log_proto_indented_multiline_server_init(self, transport, options); + return &self->super.super.super; +} diff --git a/lib/logproto-indented-multiline-server.h b/lib/logproto-indented-multiline-server.h new file mode 100644 index 0000000..260d9b1 --- /dev/null +++ b/lib/logproto-indented-multiline-server.h @@ -0,0 +1,44 @@ +/* + * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012 Gergely Nagy <algernon@balabit.hu> + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + */ +#ifndef LOGPROTO_INDENTED_MULTILINE_SERVER_INCLUDED +#define LOGPROTO_INDENTED_MULTILINE_SERVER_INCLUDED + +#include "logproto-text-server.h" + +typedef struct _LogProtoIMultiLineServer LogProtoIMultiLineServer; +struct _LogProtoIMultiLineServer +{ + LogProtoTextServer super; +}; + +/* LogProtoIMultiLineServer + * + * This class processes indented multiline text files/streams. Each + * record consists of one line that starts with non-whitespace, with + * zero or more lines starting with whitespace. A record is terminated + * when we reach a line that starts with non-whitespace, or EOF. + */ +LogProtoServer *log_proto_indented_multiline_server_new(LogTransport *transport, + const LogProtoServerOptions *options); +void log_proto_indented_multiline_server_init(LogProtoIMultiLineServer *self, + LogTransport *transport, + const LogProtoServerOptions *options); + +#endif diff --git a/lib/logproto-text-server.c b/lib/logproto-text-server.c index ed1ed90..600c06d 100644 --- a/lib/logproto-text-server.c +++ b/lib/logproto-text-server.c @@ -185,16 +185,25 @@ log_proto_text_server_get_raw_size_of_buffer(LogProtoTextServer *self, const guc } } +static gboolean +log_proto_text_server_line_is_complete(LogProtoTextServer *self, + LogProtoBufferedServerState *state, + const guchar **eol) +{ + return !!(*eol); +} + +static void +log_proto_text_server_line_flush (LogProtoTextServer *self, + LogProtoBufferedServerState *state, + const guchar **msg, gsize *msg_len, + const guchar *buffer_start, gsize buffer_bytes) +{ + *msg = buffer_start; + *msg_len = buffer_bytes; + state->pending_buffer_pos = state->pending_buffer_end; +} -/** - * log_proto_text_server_fetch_from_buf: - * @self: LogReader instance - * @saddr: socket address to be assigned to new messages (consumed!) - * @flush: whether to flush the input buffer - * @msg_counter: the number of messages processed in the current poll iteration - * - * Returns TRUE if a message was found in the buffer, FALSE if we need to read again. - **/ static gboolean log_proto_text_server_fetch_from_buf(LogProtoBufferedServer *s, const guchar *buffer_start, gsize buffer_bytes, const guchar **msg, gsize *msg_len, gboolean flush_the_rest) { @@ -209,9 +218,7 @@ log_proto_text_server_fetch_from_buf(LogProtoBufferedServer *s, const guchar *bu * we are set to packet terminating mode or the connection is to * be teared down and we have partial data in our buffer. */ - *msg = buffer_start; - *msg_len = buffer_bytes; - state->pending_buffer_pos = state->pending_buffer_end; + self->line_flush (self, state, msg, msg_len, buffer_start, buffer_bytes); goto success; } @@ -235,7 +242,7 @@ log_proto_text_server_fetch_from_buf(LogProtoBufferedServer *s, const guchar *bu *msg = buffer_start; goto success; } - else if (!eol) + else if (!self->is_line_complete(self, state, &eol)) { gsize raw_split_size; @@ -328,6 +335,10 @@ log_proto_text_server_init(LogProtoTextServer *self, LogTransport *transport, co self->super.super.prepare = log_proto_text_server_prepare; self->super.super.free_fn = log_proto_text_server_free; self->super.fetch_from_buf = log_proto_text_server_fetch_from_buf; + + self->is_line_complete = log_proto_text_server_line_is_complete; + self->line_flush = log_proto_text_server_line_flush; + self->super.stream_based = TRUE; self->reverse_convert = (GIConv) -1; } diff --git a/lib/logproto-text-server.h b/lib/logproto-text-server.h index 720d2a6..012db89 100644 --- a/lib/logproto-text-server.h +++ b/lib/logproto-text-server.h @@ -34,6 +34,14 @@ struct _LogProtoTextServer gchar *reverse_buffer; gsize reverse_buffer_len; gint convert_scale; + + gboolean (*is_line_complete)(LogProtoTextServer *self, + LogProtoBufferedServerState *state, + const guchar **eol); + void (*line_flush)(LogProtoTextServer *self, + LogProtoBufferedServerState *state, + const guchar **msg, gsize *msg_len, + const guchar *buffer_start, gsize buffer_bytes); }; /* LogProtoTextServer diff --git a/tests/unit/test_logproto.c b/tests/unit/test_logproto.c index 2feec3d..51063f5 100644 --- a/tests/unit/test_logproto.c +++ b/tests/unit/test_logproto.c @@ -4,6 +4,7 @@ #include "logproto-text-server.h" #include "logproto-framed-server.h" #include "logproto-dgram-server.h" +#include "logproto-indented-multiline-server.h" #include "logproto-record-server.h" #include "apphook.h" @@ -507,6 +508,49 @@ test_log_proto_text_server(void) } /**************************************************************************************** + * LogProtoIMultiLineServer + ****************************************************************************************/ +static void +test_log_proto_indented_multiline_server_base(gboolean input_is_stream) +{ + LogProtoServer *proto; + + log_proto_testcase_begin("test_log_proto_indented_multiline_server_base"); + proto_server_options.max_msg_size = 32; + + proto = log_proto_indented_multiline_server_new( + /* 32 bytes max line length */ + (input_is_stream ? log_transport_mock_stream_new : log_transport_mock_records_new)( + "01234567\n" + /* line too long */ + //"0123456789ABCDEF0123456789ABCDEF01234567\n", -1, + /* multi-line */ + "0\n 1=2\n 3=4\nEND\n", -1, + + LTM_EOF), + get_inited_proto_server_options()); + + assert_proto_server_fetch(proto, "01234567", -1); + + /* input split due to an oversized input line */ + //assert_proto_server_fetch(proto, "0123456789ABCDEF0123456789ABCDEF", -1); + //assert_proto_server_fetch(proto, "01234567", -1); + + assert_proto_server_fetch(proto, "0\n 1=2\n 3=4", -1); + assert_proto_server_fetch(proto, "END", -1); + + log_proto_server_free(proto); + log_proto_testcase_end(); +} + +static void +test_log_proto_indented_multiline_server(void) +{ + test_log_proto_indented_multiline_server_base(FALSE); + test_log_proto_indented_multiline_server_base(TRUE); +} + +/**************************************************************************************** * LogProtoDGramServer ****************************************************************************************/ @@ -883,6 +927,7 @@ test_log_proto(void) test_log_proto_base(); test_log_proto_record_server(); test_log_proto_text_server(); + test_log_proto_indented_multiline_server(); test_log_proto_dgram_server(); test_log_proto_framed_server(); } -- 1.7.10.4
Teach logreader to accept "indented-multiline" as a flag, so that affile and afsocket can look for it, and set the proto to indented-multiline instead of text if the flag's found. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- lib/logreader.c | 1 + lib/logreader.h | 1 + modules/affile/affile.c | 3 +++ modules/afsocket/afinet.c | 5 ++++- 4 files changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/logreader.c b/lib/logreader.c index eb52330..e96e825 100644 --- a/lib/logreader.c +++ b/lib/logreader.c @@ -906,6 +906,7 @@ CfgFlagHandler log_reader_flag_handlers[] = { "kernel", CFH_SET, offsetof(LogReaderOptions, flags), LR_KERNEL }, { "empty-lines", CFH_SET, offsetof(LogReaderOptions, flags), LR_EMPTY_LINES }, { "threaded", CFH_SET, offsetof(LogReaderOptions, flags), LR_THREADED }, + { "indented-multiline", CFH_SET, offsetof(LogReaderOptions, flags), LR_INDENTED_ML }, { NULL }, }; diff --git a/lib/logreader.h b/lib/logreader.h index 5acb03b..df24c74 100644 --- a/lib/logreader.h +++ b/lib/logreader.h @@ -36,6 +36,7 @@ #define LR_SYSLOG_PROTOCOL 0x0010 #define LR_PREEMPT 0x0020 #define LR_THREADED 0x0040 +#define LR_INDENTED_ML 0x0080 /* options */ diff --git a/modules/affile/affile.c b/modules/affile/affile.c index 17be55c..d756070 100644 --- a/modules/affile/affile.c +++ b/modules/affile/affile.c @@ -30,6 +30,7 @@ #include "mainloop.h" #include "logproto-record-server.h" #include "logproto-text-server.h" +#include "logproto-indented-multiline-server.h" #include "logproto-text-client.h" #include "logproto-linux-proc-kmsg-reader.h" #include "logproto-file-writer.h" @@ -210,6 +211,8 @@ affile_sd_construct_proto(AFFileSourceDriver *self, gint fd) return log_proto_padded_record_server_new(transport, proto_options, self->pad_size); else if (affile_is_linux_proc_kmsg(self->filename->str)) return log_proto_linux_proc_kmsg_reader_new(transport, proto_options); + else if (self->reader_options.flags & LR_INDENTED_ML) + return log_proto_indented_multiline_server_new(transport, proto_options); else return log_proto_text_server_new(transport, proto_options); } diff --git a/modules/afsocket/afinet.c b/modules/afsocket/afinet.c index 542f3f4..abf9020 100644 --- a/modules/afsocket/afinet.c +++ b/modules/afsocket/afinet.c @@ -299,7 +299,10 @@ afinet_sd_apply_transport(AFSocketSourceDriver *s) else { default_bind_port = "514"; - self->super.logproto_name = "text"; + if (self->super.reader_options.flags & LR_INDENTED_ML) + self->super.logproto_name = "indented-multiline"; + else + self->super.logproto_name = "text"; } self->super.sock_type = SOCK_STREAM; } -- 1.7.10.4
With linux 3.5, /dev/kmsg was changed substantially, and it now emits structured kernel messages towards userspace. For its format, see Documentation/ABI/testing/dev-kmsg in a recent kernel source. This format plugin implements the parser for that format, including special parsing for the DEVICE keys, and resolving of the message timestamps on Linux. To use it on a 3.5+ kernel, something along these lines should replace the /proc/kmsg source: file("/dev/kmsg" flags(kernel, indented-multiline) program-override("kernel") format("linux-kmsg")); Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- configure.in | 2 + modules/Makefile.am | 2 +- modules/linux-kmsg-format/Makefile.am | 15 + .../linux-kmsg-format/linux-kmsg-format-plugin.c | 57 +++ modules/linux-kmsg-format/linux-kmsg-format.c | 470 ++++++++++++++++++++ modules/linux-kmsg-format/linux-kmsg-format.h | 30 ++ modules/linux-kmsg-format/tests/Makefile.am | 7 + .../tests/test_linux_format_kmsg.c | 179 ++++++++ 8 files changed, 761 insertions(+), 1 deletion(-) create mode 100644 modules/linux-kmsg-format/Makefile.am create mode 100644 modules/linux-kmsg-format/linux-kmsg-format-plugin.c create mode 100644 modules/linux-kmsg-format/linux-kmsg-format.c create mode 100644 modules/linux-kmsg-format/linux-kmsg-format.h create mode 100644 modules/linux-kmsg-format/tests/Makefile.am create mode 100644 modules/linux-kmsg-format/tests/test_linux_format_kmsg.c diff --git a/configure.in b/configure.in index 7757a87..91c61ca 100644 --- a/configure.in +++ b/configure.in @@ -1140,6 +1140,8 @@ AC_OUTPUT(dist.conf modules/confgen/Makefile modules/system-source/Makefile modules/syslogformat/Makefile + modules/linux-kmsg-format/Makefile + modules/linux-kmsg-format/tests/Makefile modules/pacctformat/Makefile modules/basicfuncs/Makefile modules/basicfuncs/tests/Makefile diff --git a/modules/Makefile.am b/modules/Makefile.am index 8f9d63b..ddddb66 100644 --- a/modules/Makefile.am +++ b/modules/Makefile.am @@ -1 +1 @@ -SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb afsmtp csvparser confgen system-source syslogformat pacctformat basicfuncs cryptofuncs dbparser json +SUBDIRS = afsocket afsql afstreams affile afprog afuser afmongodb afsmtp csvparser confgen system-source syslogformat linux-kmsg-format pacctformat basicfuncs cryptofuncs dbparser json diff --git a/modules/linux-kmsg-format/Makefile.am b/modules/linux-kmsg-format/Makefile.am new file mode 100644 index 0000000..8f18801 --- /dev/null +++ b/modules/linux-kmsg-format/Makefile.am @@ -0,0 +1,15 @@ +SUBDIRS = . tests + +moduledir = @moduledir@ +AM_CPPFLAGS = -I$(top_srcdir)/lib -I../../lib +export top_srcdir + +module_LTLIBRARIES := liblinux-kmsg-format.la +liblinux_kmsg_format_la_SOURCES = \ + linux-kmsg-format.c linux-kmsg-format.h linux-kmsg-format-plugin.c + +liblinux_kmsg_format_la_CPPFLAGS = $(AM_CPPFLAGS) +liblinux_kmsg_format_la_LIBADD = $(MODULE_DEPS_LIBS) +liblinux_kmsg_format_la_LDFLAGS = $(MODULE_LDFLAGS) + +include $(top_srcdir)/build/lex-rules.am diff --git a/modules/linux-kmsg-format/linux-kmsg-format-plugin.c b/modules/linux-kmsg-format/linux-kmsg-format-plugin.c new file mode 100644 index 0000000..bf91394 --- /dev/null +++ b/modules/linux-kmsg-format/linux-kmsg-format-plugin.c @@ -0,0 +1,57 @@ +/* + * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012 Gergely Nagy <algernon@balabit.hu> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "linux-kmsg-format.h" +#include "messages.h" +#include "plugin.h" + +static MsgFormatHandler linux_kmsg_handler = +{ + .parse = &linux_kmsg_format_handler +}; + +static MsgFormatHandler * +linux_kmsg_format_construct(Plugin *self, GlobalConfig *cfg, gint plugin_type, const gchar *plugin_name) +{ + return &linux_kmsg_handler; +} + +static Plugin linux_kmsg_format_plugin = +{ + .type = LL_CONTEXT_FORMAT, + .name = "linux-kmsg", + .construct = (gpointer (*)(Plugin *self, GlobalConfig *cfg, gint plugin_type, const gchar *plugin_name)) linux_kmsg_format_construct, +}; + +gboolean +linux_kmsg_format_module_init(GlobalConfig *cfg, CfgArgs *args) +{ + linux_msg_format_init(); + plugin_register(cfg, &linux_kmsg_format_plugin, 1); + return TRUE; +} + +const ModuleInfo module_info = +{ + .canonical_name = "linux-kmsg-format", + .version = VERSION, + .description = "The linux-kmsg-format module provides support for parsing linux 3.5+ /dev/kmsg-format messages.", + .core_revision = SOURCE_REVISION, + .plugins = &linux_kmsg_format_plugin, + .plugins_len = 1, +}; diff --git a/modules/linux-kmsg-format/linux-kmsg-format.c b/modules/linux-kmsg-format/linux-kmsg-format.c new file mode 100644 index 0000000..db1658c --- /dev/null +++ b/modules/linux-kmsg-format/linux-kmsg-format.c @@ -0,0 +1,470 @@ +/* + * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012 Gergely Nagy <algernon@balabit.hu> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "linux-kmsg-format.h" +#include "logmsg.h" +#include "messages.h" +#include "timeutils.h" +#include "misc.h" +#include "cfg.h" +#include "str-format.h" +#include "scratch-buffers.h" + +#include <ctype.h> +#include <string.h> +#include <fcntl.h> + +static NVHandle KMSG_LM_V_DEV_TYPE; +static NVHandle KMSG_LM_V_DEV_MINOR; +static NVHandle KMSG_LM_V_DEV_MAJOR; +static NVHandle KMSG_LM_V_DEV_NAME; +static NVHandle KMSG_LM_V_NETDEV_INDEX; +static struct timeval boot_time; + +/* + * The linux (3.5+) /dev/kmsg format looks like this: + * + * 6,802,65338577;ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex> + * SUBSYSTEM=pci + * DEVICE=+pci:0000:02:00.0 + * + * Where the first number is the syslog priority & facility, the + * second is a 64-bit sequence number, the third is a monotonic + * timestamp (where 0 is the boot time) in microseconds. + * + * Following the timestamp, there may be other items in the future, + * all comma-separated - these should be gracefully ignored. + * + * The message itself starts after the first ';', and lasts until the + * first newline. + * + * After the first newline, we have key-value pairs, each one line, + * starting with whitespace, the first '=' is the divisor. + * + * The "DEVICE" key can be further processed, its value is any of the + * following formats: + * + * - b1:2 - a block dev_t, with major:minor + * - c1:2 - a char dev_t, with major:minor + * - n1 - netdev with device index + * - +subsystem:device + * + * This implementation parses each of those, and also gracefully + * handles unknown formats. + */ + +static guint64 +kmsg_timeval_diff(struct timeval *t1, struct timeval *t2) +{ + return (t1->tv_sec - t2->tv_sec) * G_USEC_PER_SEC + (t1->tv_usec - t2->tv_usec); +} + +static void +kmsg_to_absolute_time(guint64 timestamp, LogStamp *dest) +{ + guint64 t; + + t = (boot_time.tv_sec + (timestamp / G_USEC_PER_SEC)) * G_USEC_PER_SEC + + boot_time.tv_usec + (timestamp % G_USEC_PER_SEC); + + dest->tv_sec = t / G_USEC_PER_SEC; + dest->tv_usec = t % G_USEC_PER_SEC; +} + +static gsize +kmsg_parse_prio(const guchar *data, gsize pos, gsize length, LogMessage *msg) +{ + gint pri = 0; + + while (pos < length && data[pos] != ',') + { + if (isdigit(data[pos])) + pri = pri * 10 + ((data[pos]) - '0'); + else + return -1; + pos++; + } + if (data[pos] != ',' || pos == length) + return -1; + + msg->pri = pri; + return pos; +} + +static gsize +kmsg_parse_seq(const guchar *data, gsize pos, gsize length, LogMessage *msg) +{ + gsize start = pos; + + while (pos < length && data[pos] != ',') + { + if (!isdigit(data[pos])) + return -1; + pos++; + } + if (data[pos] != ',' || pos == length) + return -1; + + log_msg_set_value(msg, LM_V_MSGID, (const gchar *)data + start, pos - start); + return pos; +} + +static gsize +kmsg_parse_timestamp(const guchar *data, gsize pos, gsize length, LogMessage *msg) +{ + guint64 timestamp = 0; + + while (pos < length && data[pos] != ',' && data[pos] != ';') + { + if (isdigit(data[pos])) + timestamp = timestamp * 10 + ((data[pos]) - '0'); + else + return -1; + pos++; + } + if ((data[pos] != ',' && data[pos] != ';') || pos == length) + return -1; + + kmsg_to_absolute_time(timestamp, &msg->timestamps[LM_TS_STAMP]); + msg->timestamps[LM_TS_STAMP].zone_offset = + get_local_timezone_ofs(msg->timestamps[LM_TS_STAMP].tv_sec); + + return pos; +} + +static gsize +kmsg_skip_to_message(const guchar *data, gsize pos, gsize length) +{ + while (pos < length && data[pos] != ';') + pos++; + + if (data[pos] != ';' || pos == length) + return -1; + return pos; +} + +static gsize +kmsg_parse_message(const guchar *data, gsize pos, gsize length, LogMessage *msg) +{ + gsize start = pos; + + while (pos < length && data[pos] != '\n') + pos++; + if (data[pos] != '\n') + return -1; + + log_msg_set_value(msg, LM_V_MESSAGE, (const gchar *)data + start, + pos - start); + return pos; +} + +static gboolean +kmsg_is_key_value_pair_device(const guchar *name, gsize length) +{ + if (strncmp((const char *)name, "DEVICE=", length + 1) == 0) + return TRUE; + return FALSE; +} + +static void +kmsg_parse_device_dev_t(const gchar *type, + const guchar *value, gsize length, + LogMessage *msg) +{ + gsize seppos = 0; + + log_msg_set_value(msg, KMSG_LM_V_DEV_TYPE, type, -1); + + while (seppos < length && value[seppos] != ':') + seppos++; + + log_msg_set_value(msg, KMSG_LM_V_DEV_MAJOR, + (const gchar *)value, seppos); + log_msg_set_value(msg, KMSG_LM_V_DEV_MINOR, + (const gchar *)value + seppos + 1, length - seppos - 1); +} + +static void +kmsg_parse_device_netdev(const guchar *value, gsize length, + LogMessage *msg) +{ + log_msg_set_value(msg, KMSG_LM_V_DEV_TYPE, "netdev", -1); + log_msg_set_value(msg, KMSG_LM_V_NETDEV_INDEX, + (const gchar *)value, length); +} + +static void +kmsg_parse_device_subsys(const guchar *value, gsize length, + LogMessage *msg) +{ + gsize seppos = 0; + + while (seppos < length && value[seppos] != ':') + seppos++; + + log_msg_set_value(msg, KMSG_LM_V_DEV_TYPE, + (const gchar *)value, seppos); + log_msg_set_value(msg, KMSG_LM_V_DEV_NAME, + (const gchar *)value + seppos + 1, length - seppos - 1); +} + +static void +kmsg_parse_device_unknown(const guchar *value, gsize length, + LogMessage *msg) +{ + log_msg_set_value(msg, KMSG_LM_V_DEV_TYPE, "<unknown>", -1); + log_msg_set_value(msg, KMSG_LM_V_DEV_NAME, + (const gchar *)value, length); + +} + +static void +kmsg_parse_device_key_value_pair(const guchar *value, gsize length, + LogMessage *msg) +{ + switch (value[0]) + { + case 'b': + kmsg_parse_device_dev_t("block", value + 1, length - 1, msg); + break; + case 'c': + kmsg_parse_device_dev_t("char", value + 1, length - 1, msg); + break; + case 'n': + kmsg_parse_device_netdev(value + 1, length - 1, msg); + break; + case '+': + kmsg_parse_device_subsys(value + 1, length - 1, msg); + break; + default: + kmsg_parse_device_unknown(value, length, msg); + break; + } +} + +static gsize +kmsg_parse_key_value_pair(const guchar *data, gsize pos, gsize length, + LogMessage *msg) +{ + gsize name_start, name_len, value_start, value_len; + ScratchBuffer *name; + + while (pos < length && (data[pos] == ' ' || data[pos] == '\t')) + pos++; + if (pos == length) + return -1; + name_start = pos; + + while (pos < length && data[pos] != '=') + pos++; + if (pos == length) + return -1; + name_len = pos - name_start; + value_start = ++pos; + + while (pos < length && data[pos] != '\n') + pos++; + if (data[pos] != '\n') + return -1; + value_len = pos - value_start; + + if (kmsg_is_key_value_pair_device(data + name_start, name_len)) + { + kmsg_parse_device_key_value_pair(data + value_start, value_len, + msg); + return pos; + } + + name = scratch_buffer_acquire(); + + g_string_assign(sb_string(name), ".kernel."); + g_string_append_len(sb_string(name), (const gchar *)data + name_start, name_len); + + log_msg_set_value(msg, + log_msg_get_value_handle(sb_string(name)->str), + (const gchar *)data + value_start, value_len); + scratch_buffer_release(name); + + return pos; +} + +static gboolean +log_msg_parse_kmsg(LogMessage *msg, const guchar *data, gsize length) +{ + gsize pos = 0; + + if ((pos = kmsg_parse_prio(data, pos, length, msg)) == -1) + return FALSE; + + if ((pos = kmsg_parse_seq(data, pos + 1, length, msg)) == -1) + return FALSE; + + if ((pos = kmsg_parse_timestamp(data, pos + 1, length, msg)) == -1) + return FALSE; + + if ((pos = kmsg_skip_to_message(data, pos, length)) == -1) + return FALSE; + + if ((pos = kmsg_parse_message(data, pos + 1, length, msg)) == -1) + return FALSE; + + if (pos + 1 >= length) + return TRUE; + + do + { + if ((pos = kmsg_parse_key_value_pair(data, pos + 1, length, msg)) == -1) + return FALSE; + } + while (pos < length); + + return TRUE; +} + +void +linux_kmsg_format_handler(MsgFormatOptions *parse_options, + const guchar *data, gsize length, + LogMessage *self) +{ + gboolean success; + + while (length > 0 && (data[length - 1] == '\n' || data[length - 1] == '\0')) + length--; + + if (parse_options->flags & LP_NOPARSE) + { + log_msg_set_value(self, LM_V_MESSAGE, (gchar *) data, length); + self->pri = parse_options->default_pri; + return; + } + + self->flags |= LF_UTF8; + + if (parse_options->flags & LP_LOCAL) + self->flags |= LF_LOCAL; + + self->initial_parse = TRUE; + + success = log_msg_parse_kmsg(self, data, length); + + self->initial_parse = FALSE; + + if (G_UNLIKELY(!success)) + { + gchar buf[2048]; + + self->timestamps[LM_TS_STAMP] = self->timestamps[LM_TS_RECVD]; + if ((self->flags & LF_STATE_OWN_PAYLOAD) && self->payload) + nv_table_unref(self->payload); + self->flags |= LF_STATE_OWN_PAYLOAD; + self->payload = nv_table_new(LM_V_MAX, 16, MAX(length * 2, 256)); + log_msg_set_value(self, LM_V_HOST, "", 0); + + g_snprintf(buf, sizeof(buf), "Error processing log message: %.*s", (gint) length, data); + log_msg_set_value(self, LM_V_MESSAGE, buf, -1); + log_msg_set_value(self, LM_V_PROGRAM, "syslog-ng", 9); + g_snprintf(buf, sizeof(buf), "%d", (int) getpid()); + log_msg_set_value(self, LM_V_PID, buf, -1); + + if (self->sdata) + { + g_free(self->sdata); + self->alloc_sdata = self->num_sdata = 0; + self->sdata = NULL; + } + self->pri = LOG_SYSLOG | LOG_ERR; + return; + } +} + +#ifdef __linux__ +static void +kmsg_init_boot_time(void) +{ + int fd, pos = 0; + gchar buf[1024]; + ssize_t rc; + struct timeval curr_time; + guint64 tdiff; + + if ((fd = open ("/proc/uptime", O_RDONLY)) == -1) + return; + + if ((rc = read (fd, buf, sizeof(buf))) <= 0) + { + close(fd); + return; + } + close(fd); + + gettimeofday(&curr_time, NULL); + + /* Read the seconds part */ + while (pos < rc && buf[pos] != '.') + { + if (isdigit(buf[pos])) + boot_time.tv_sec = boot_time.tv_sec * 10 + ((buf[pos]) - '0'); + else + { + boot_time.tv_sec = 0; + return; + } + pos++; + } + pos++; + + /* Then the microsecond part */ + while (pos < rc && buf[pos] != ' ') + { + if (isdigit(buf[pos])) + boot_time.tv_usec = boot_time.tv_usec * 10 + ((buf[pos]) - '0'); + else + { + boot_time.tv_sec = 0; + boot_time.tv_usec = 0; + return; + } + pos++; + } + + tdiff = kmsg_timeval_diff(&curr_time, &boot_time); + boot_time.tv_sec = tdiff / G_USEC_PER_SEC; + boot_time.tv_usec = tdiff % G_USEC_PER_SEC; +} +#endif + +void +linux_msg_format_init(void) +{ + static gboolean initialized = FALSE; + + if (!initialized) + { + KMSG_LM_V_DEV_TYPE = log_msg_get_value_handle(".kernel.DEVICE.type"); + KMSG_LM_V_DEV_MINOR = log_msg_get_value_handle(".kernel.DEVICE.minor"); + KMSG_LM_V_DEV_MAJOR = log_msg_get_value_handle(".kernel.DEVICE.major"); + KMSG_LM_V_DEV_NAME = log_msg_get_value_handle(".kernel.DEVICE.name"); + KMSG_LM_V_NETDEV_INDEX = log_msg_get_value_handle(".kernel.DEVICE.index"); + +#ifdef __linux__ + kmsg_init_boot_time(); +#endif + + initialized = TRUE; + } +} diff --git a/modules/linux-kmsg-format/linux-kmsg-format.h b/modules/linux-kmsg-format/linux-kmsg-format.h new file mode 100644 index 0000000..04ca939 --- /dev/null +++ b/modules/linux-kmsg-format/linux-kmsg-format.h @@ -0,0 +1,30 @@ +/* + * Copyright (c) 2012 BalaBit IT Ltd, Budapest, Hungary + * Copyright (c) 2012 Gergely Nagy <algernon@balabit.hu> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef LINUX_KMSG_FORMAT_H_INCLUDED +#define LINUX_KMSG_FORMAT_H_INCLUDED + +#include "msg-format.h" + +void linux_kmsg_format_handler(MsgFormatOptions *parse_options, + const guchar *data, gsize length, + LogMessage *self); + +void linux_msg_format_init (void); + +#endif diff --git a/modules/linux-kmsg-format/tests/Makefile.am b/modules/linux-kmsg-format/tests/Makefile.am new file mode 100644 index 0000000..d045a3e --- /dev/null +++ b/modules/linux-kmsg-format/tests/Makefile.am @@ -0,0 +1,7 @@ +AM_CFLAGS = -I$(top_srcdir)/lib -I../../../lib -I$(top_srcdir)/libtest -I../../../libtest -I$(top_srcdir)/modules/linux-kmsg-format -I.. + +AM_LDFLAGS = -dlpreopen ../../syslogformat/libsyslogformat.la -dlpreopen ../liblinux-kmsg-format.la +LDADD = $(top_builddir)/lib/libsyslog-ng.la $(top_builddir)/libtest/libsyslog-ng-test.a @TOOL_DEPS_LIBS@ + +check_PROGRAMS = test_linux_format_kmsg +TESTS = $(check_PROGRAMS) diff --git a/modules/linux-kmsg-format/tests/test_linux_format_kmsg.c b/modules/linux-kmsg-format/tests/test_linux_format_kmsg.c new file mode 100644 index 0000000..dd282d1 --- /dev/null +++ b/modules/linux-kmsg-format/tests/test_linux_format_kmsg.c @@ -0,0 +1,179 @@ +#include "testutils.h" +#include "msg_parse_lib.h" +#include "apphook.h" +#include "plugin.h" + +static LogMessage * +kmsg_parse_message(const gchar *raw_message_str) +{ + LogMessage *message; + GSockAddr *addr = g_sockaddr_inet_new("10.10.10.10", 1010); + + message = log_msg_new(raw_message_str, strlen(raw_message_str), addr, &parse_options); + + g_sockaddr_unref(addr); + return message; +} + +static void +assert_log_kmsg_value(LogMessage *message, const gchar *key, + const gchar *expected_value) +{ + const gchar *actual_value = log_msg_get_value(message, + log_msg_get_value_handle(key), + NULL); + assert_string(actual_value, expected_value, NULL); +} + +void +test_kmsg_single_line(void) +{ + gchar msg[] = "5,2,0;Linux version 3.5-trunk-amd64 (Debian 3.5.2-1~experimental.1) (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-1) ) #1 SMP Mon Aug 20 04:17:46 UTC 2012\n"; + LogMessage *parsed_message; + + testcase_begin("Testing single-line /dev/kmsg parsing; msg='%s'", msg); + + parsed_message = kmsg_parse_message(msg); + + assert_guint16(parsed_message->pri, 5, "Unexpected message priority"); + assert_log_message_value(parsed_message, LM_V_MSGID, "2"); + msg[sizeof(msg) - 2] = '\0'; + assert_log_message_value(parsed_message, LM_V_MESSAGE, msg + 6); + + log_msg_unref(parsed_message); + + testcase_end(); +} + +void +test_kmsg_multi_line(void) +{ + gchar msg[] = "6,202,98513;pci_root PNP0A08:00: host bridge window [io 0x0000-0x0cf7]\n" \ + " SUBSYSTEM=acpi\n" \ + " DEVICE=+acpi:PNP0A08:00\n"; + LogMessage *parsed_message; + + testcase_begin("Testing multi-line /dev/kmsg parsing; msg='%s'", msg); + + parsed_message = kmsg_parse_message(msg); + + assert_guint16(parsed_message->pri, 6, "Unexpected message priority"); + assert_log_message_value(parsed_message, LM_V_MSGID, "202"); + assert_log_message_value(parsed_message, LM_V_MESSAGE, "pci_root PNP0A08:00: host bridge window [io 0x0000-0x0cf7]"); + assert_log_kmsg_value(parsed_message, ".kernel.SUBSYSTEM", "acpi"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.type", "acpi"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.name", "PNP0A08:00"); + + log_msg_unref(parsed_message); + + testcase_end(); +} + +void +test_kmsg_with_extra_fields(void) +{ + gchar msg[] = "5,2,0,some extra field,3,4,5;And this is the real message\n"; + LogMessage *parsed_message; + + testcase_begin("Testing /dev/kmsg parsing, with extra fields; msg='%s'", msg); + + parsed_message = kmsg_parse_message(msg); + assert_guint16(parsed_message->pri, 5, "Unexpected message priority"); + assert_log_message_value(parsed_message, LM_V_MSGID, "2"); + assert_log_message_value(parsed_message, LM_V_MESSAGE, "And this is the real message"); + + log_msg_unref(parsed_message); + + testcase_end(); +} + +void +test_kmsg_device_parsing(void) +{ + gchar msg_subsys[] = "6,202,98513;pci_root PNP0A08:00: host bridge window [io 0x0000-0x0cf7]\n" \ + " SUBSYSTEM=acpi\n" \ + " DEVICE=+acpi:PNP0A08:00\n"; + gchar msg_block[] = "6,202,98513;Fake message\n" \ + " DEVICE=b12:1\n"; + gchar msg_char[] = "6,202,98513;Fake message\n" \ + " DEVICE=c3:4\n"; + gchar msg_netdev[] = "6,202,98513;Fake message\n" \ + " DEVICE=n8\n"; + gchar msg_unknown[] = "6,202,98513;Fake message\n" \ + " DEVICE=w12345\n"; + LogMessage *parsed_message; + + testcase_begin("Testing /dev/kmsg DEVICE= parsing"); + + parsed_message = kmsg_parse_message(msg_subsys); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.type", "acpi"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.name", "PNP0A08:00"); + log_msg_unref(parsed_message); + + parsed_message = kmsg_parse_message(msg_block); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.type", "block"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.major", "12"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.minor", "1"); + log_msg_unref(parsed_message); + + parsed_message = kmsg_parse_message(msg_char); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.type", "char"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.major", "3"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.minor", "4"); + log_msg_unref(parsed_message); + + parsed_message = kmsg_parse_message(msg_netdev); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.type", "netdev"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.index", "8"); + log_msg_unref(parsed_message); + + parsed_message = kmsg_parse_message(msg_unknown); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.type", "<unknown>"); + assert_log_kmsg_value(parsed_message, ".kernel.DEVICE.name", "w12345"); + log_msg_unref(parsed_message); + + testcase_end(); +} + +void +init_and_load_kmsgformat_module() +{ + Plugin *p; + + configuration = cfg_new(VERSION_VALUE); + plugin_load_module("linux-kmsg-format", configuration, NULL); + msg_format_options_defaults(&parse_options); + msg_format_options_init(&parse_options, configuration); + + parse_options.format = "linux-kmsg"; + p = plugin_find(configuration, LL_CONTEXT_FORMAT, parse_options.format); + parse_options.format_handler = plugin_construct(p, configuration, LL_CONTEXT_FORMAT, parse_options.format); +} + +void +deinit_kmsgformat_module() +{ + if (configuration) + cfg_free(configuration); + configuration = NULL; + parse_options.format = NULL; + parse_options.format_handler = NULL; +} + +int +main(void) +{ + app_startup(); + putenv("TZ=UTC"); + tzset(); + + init_and_load_kmsgformat_module(); + + test_kmsg_single_line(); + test_kmsg_multi_line(); + test_kmsg_with_extra_fields(); + test_kmsg_device_parsing(); + + deinit_kmsgformat_module(); + app_shutdown(); +} -- 1.7.10.4
On Linux, use a dgram reader for /dev/kmsg, because it works almost exactly like a dgram source anyway, and a dgram reader is more efficient than the indented-multiline one. Also, change the system() source to not enforce indentedn-multiline on /dev/kmsg anymore, since with the change to affile, there's no need to. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- modules/affile/affile.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/modules/affile/affile.c b/modules/affile/affile.c index d756070..265a3d5 100644 --- a/modules/affile/affile.c +++ b/modules/affile/affile.c @@ -28,6 +28,7 @@ #include "gprocess.h" #include "stats.h" #include "mainloop.h" +#include "logproto-dgram-server.h" #include "logproto-record-server.h" #include "logproto-text-server.h" #include "logproto-indented-multiline-server.h" @@ -55,6 +56,16 @@ affile_is_linux_proc_kmsg(const gchar *filename) } static inline gboolean +affile_is_linux_dev_kmsg(const gchar *filename) +{ +#ifdef __linux__ + if (strcmp(filename, "/dev/kmsg") == 0) + return TRUE; +#endif + return FALSE; +} + +static inline gboolean affile_is_device_node(const gchar *filename) { struct stat st; @@ -211,6 +222,8 @@ affile_sd_construct_proto(AFFileSourceDriver *self, gint fd) return log_proto_padded_record_server_new(transport, proto_options, self->pad_size); else if (affile_is_linux_proc_kmsg(self->filename->str)) return log_proto_linux_proc_kmsg_reader_new(transport, proto_options); + else if (affile_is_linux_dev_kmsg(self->filename->str)) + return log_proto_dgram_server_new(transport, proto_options); else if (self->reader_options.flags & LR_INDENTED_ML) return log_proto_indented_multiline_server_new(transport, proto_options); else -- 1.7.10.4
When on Linux, and /dev/kmsg is seekable, use that over /proc/kmsg, using indented-multiline and the linux-kmsg format. Signed-off-by: Gergely Nagy <algernon@balabit.hu> --- modules/system-source/system-source.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c index 929a5f8..8ad75b4 100644 --- a/modules/system-source/system-source.c +++ b/modules/system-source/system-source.c @@ -27,6 +27,7 @@ #include "messages.h" #include "plugin.h" +#include <fcntl.h> #include <sys/utsname.h> #include <sys/types.h> #include <sys/stat.h> @@ -48,7 +49,7 @@ system_sysblock_add_unix_dgram(GString *sysblock, const gchar *path, static void system_sysblock_add_file(GString *sysblock, const gchar *path, gint follow_freq, const gchar *prg_override, - const gchar *flags) + const gchar *flags, const gchar *format) { g_string_append_printf(sysblock, "file(\"%s\"", path); if (follow_freq >= 0) @@ -57,6 +58,8 @@ system_sysblock_add_file(GString *sysblock, const gchar *path, g_string_append_printf(sysblock, " program-override(\"%s\")", prg_override); if (flags) g_string_append_printf(sysblock, " flags(%s)", flags); + if (format) + g_string_append_printf(sysblock, " format(%s)", format); g_string_append(sysblock, ");\n"); } @@ -108,6 +111,9 @@ system_generate_system(CfgLexer *lexer, gint type, const gchar *name, if (strcmp(u.sysname, "Linux") == 0) { char *log = "/dev/log"; + gchar *kmsg = "/proc/kmsg"; + int fd; + gchar *format = NULL; if (getenv("LISTEN_FDS") != NULL) { @@ -121,7 +127,18 @@ system_generate_system(CfgLexer *lexer, gint type, const gchar *name, } system_sysblock_add_unix_dgram(sysblock, log, NULL); - system_sysblock_add_file(sysblock, "/proc/kmsg", -1, "kernel", "kernel"); + + if ((fd = open("/dev/kmsg", O_RDONLY)) != -1) + { + if (lseek (fd, 0, SEEK_END) != -1) + { + kmsg = "/dev/kmsg"; + format = "linux-kmsg"; + } + close (fd); + } + + system_sysblock_add_file(sysblock, kmsg, -1, "kernel", "kernel", format); } else if (strcmp(u.sysname, "SunOS") == 0) { @@ -138,12 +155,12 @@ system_generate_system(CfgLexer *lexer, gint type, const gchar *name, { system_sysblock_add_unix_dgram(sysblock, "/var/run/log", NULL); system_sysblock_add_unix_dgram(sysblock, "/var/run/logpriv", "0600"); - system_sysblock_add_file(sysblock, "/dev/klog", 0, "kernel", "no-parse"); + system_sysblock_add_file(sysblock, "/dev/klog", 0, "kernel", "no-parse", NULL); } else if (strcmp(u.sysname, "GNU/kFreeBSD") == 0) { system_sysblock_add_unix_dgram(sysblock, "/var/run/log", NULL); - system_sysblock_add_file(sysblock, "/dev/klog", 0, "kernel", NULL); + system_sysblock_add_file(sysblock, "/dev/klog", 0, "kernel", NULL, NULL); } else if (strcmp(u.sysname, "HP-UX") == 0) { -- 1.7.10.4
participants (1)
-
Gergely Nagy