<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000066">
<font color="#000000">Sent: Mon Jan 16 2012 08:47:26 GMT-0500 (EST)<br>
From: Balazs Scheidler <a class="moz-txt-link-rfc2396E" href="mailto:bazsi@balabit.hu"><bazsi@balabit.hu></a><br>
To: <a class="moz-txt-link-abbreviated" href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a> <br>
Subject: [syslog-ng] Bazsi's blog: syslog-ng flexibility
improvements</font>
<blockquote cite="mid:1326721646.2845.11.camel@bzorp" type="cite"><font
color="#000000"> <br>
<br>
</font>
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="GENERATOR" content="GtkHTML/4.1.92">
<title>@@weblog</title>
<h1>
<font color="#000000"><b><font size="6"><a
moz-do-not-send="true"
href="http://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/">syslog-ng
flexibility improvements</a></font></b></font>
</h1>
<font color="#000000">
syslog-ng is often referred as a very flexible application when
it comes to processing logs. Over the years however, I began to
feel that some things are a bit more difficult to achieve in the
configuration language than it should be. For instance it is
sometimes too rigid when you need a combination of parsers
(patterndb with db-parser) and rewrite rules to achieve the goal
you wanted. Parsers and rewrite rules are distinct part of the
configuration, it is not possible to combine them into a single
functionality. Also, declaring objects first and then
referencing them later, makes the configuration easy to read,
however sometimes it is quite cumbersome, when you only need to
invert the result of an already existing filter.<br>
<br>
To solve this situation, I’ve set out to implement an idea I had
on mind for some time now. It is quite difficult to describe the
feature in clear and concise words, as it is a combination of
various changes that together makes syslog-ng configuration more
flexible and easier to use, without sacrificing readability.
Curious? Please read on.<br>
<br>
<b>In-line objects</b><br>
<br>
Perhaps the simplest of all features is that you can now define
the contents of a given object right on the spot, without having
to use a separate statement. For example, earlier you had to
write:<br>
<br>
</font>
<pre><font color="#000000">log {
source(s_local);
filter(f_postfix);
destination(d_postfix);
};
</font></pre>
<font color="#000000">
<br>
Sometimes, f_postfix filter is only used once and is trivial.
This can now be written as:<br>
<br>
</font>
<pre><font color="#000000">log {
source(s_local);
filter { program("^postfix/"); };
destination(d_postfix);
};
</font></pre>
<font color="#000000">
<br>
Furthermore both the source() and destination() options can be
written in-line, you simply use braces instead of parentheses.
The same functionality applies to everything: sources,
destinations, filters, parsers and rewrite rules.<br>
<br>
<b>Junctions</b><br>
<br>
A limited form of junctions has been supported since syslog-ng
3.0 in the form of “embedded log statements”, which has been
generalized now. Within syslog-ng, when a message is received it
is dispatched to a log processing path or pipeline, which
carries out the task at hand. A junction is a point in the log
processing path where the processing is performed on multiple
independent branches, each doing its own specific thing with the
message.<br>
<br>
The limited functionality in 3.0 only allowed the processing
tree to split (or fork) into independent branches, each of the
branches was a “sink”, where processing also ended.
Configuration example:<br>
<br>
</font>
<pre><font color="#000000">log {
source(s_all); filter(f);
log { filter(f1); destination(d1); };
log { filter(f2); destination(d2); };
};
</font></pre>
<font color="#000000">
<br>
This sample forks the processing path into two branches starting
with the “log” keyword within the top-level log statement. The
first branch evaluates the filter f1 and the writes matching
messages to the d1 destination, effectively sending all messages
that match (f AND f1) to d1. Likewise, d2 receives all messages
that match (f AND f2).<br>
<br>
The limitation of the embedded log statement concept was simple:
it could only be listed at the very end of a log statement, and
the end-result of the branches couldn’t be processed further.
Effectively the message at the end of each branch “fell off”.
Junctions on the other hand makes it possible to do things to
messages once the branches converge to the same point again.
Repeating the sample above, it is now possible to write:<br>
<br>
</font>
<pre><font color="#000000">log {
source(s_all); filter(f);
junction {
log { filter(f1); destination(d1); };
log { filter(f2); destination(d2); };
};
destination(d_all);
};
</font></pre>
<font color="#000000">
<br>
The new thing is that you can now add processing <b>after</b>
the branches finish their processing. A bit more useful example
would be:<br>
<br>
</font>
<pre><font color="#000000">log {
source(s_apache_files);
source(s_syslog);
junction {
log { filter(f_apache_files); rewrite(r_apache_remove_file_header); parser(p_apache); flags(final); };
log { filter(f_apache_syslog); parser(p_apache); flags(final); };
};
destination(d_files);
};
</font></pre>
<font color="#000000">
<br>
This example does an alternative processing of incoming logs
based on where the message came from.<br>
<br>
<b>Everything is a log expression</b><br>
<br>
This feature is probably the most complicated, however provides
very nice properties and expressiveness to the configuration.
From now on, not just the well known log statement allows the
specification of log processing rules, but all the objects in
the syslog-ng configuration file can use the same expressive
power.<br>
<br>
It is now possible to use embedded log statements, junctions and
in-line object definitions within source, destination, filter,
rewrite and parser definitions. Huh, you could ask: what does it
bring to me as a benefit? Well, until now, objects of different
types were separate entities, connected using log statements,
with this change a source can also specify a rewrite rule and
that combination used as a log source in a log statement.<br>
<br>
For instance, a usual source definition looked like this:<br>
<br>
</font>
<pre><font color="#000000">source s_apache {
file("/var/log/apache/error.log");
};
</font></pre>
<font color="#000000">
<br>
If you wanted to process this log file in a specific way, you
needed to define the accompanying processing rules (parsers and
rewrite expressions) and combine them in a log statement. But
how about this:<br>
<br>
</font>
<pre><font color="#000000">source s_apache {
log {
source { file("/var/log/apache/error.log"); };
parser(p_apache_parser); };
};
};
log { source(s_apache); ... };
</font></pre>
<font color="#000000">
<br>
</font></blockquote>
<font color="#000000"><br>
This just doesnt feel right. I'm not quire sure how else to put it
:-)<br>
I mean that I think of `log` statements as output handlers for a
message. They control how the message leaves syslog-ng, whether it
be to a file, database, pipe, whatever. To me it seems to make
more sense if the `log` statement is called something else here
(inside the `source` block), though what I dont know. Maybe at the
least an alias, so that log and the alias are the exact same
things, but that reading the config would make it look more
logical. I dont know if I'm understanding the distinctions between
`log` and `junctions` properly, but it seems as if `log`s are a
serial execution of the statements within, and `junction`s are a
parallel execution of the statements within, so maybe names to
better reflect this?<br>
<br>
<br>
</font>
<blockquote cite="mid:1326721646.2845.11.camel@bzorp" type="cite"><font
color="#000000">
Can you see? The s_apache source used a file source and the
reference of a specific parser and all messages read from the
apache error log file would be processed by that parser. The log
statement is just as simple as if s_apache would be a “normal”
source definition. This feature allows pairing the essential log
preprocessing functionality very close to the source itself,
making it very easy to write and read the log statements. As an
added bonus, it becomes very easy to distribute application
specific source & parser definitions as an SCL configuration
snippet.<br>
<br>
<b>Where?</b><br>
<br>
This stuff is available in the <a moz-do-not-send="true"
href="http://github.com/bazsi/syslog-ng-3.4/">syslog-ng 3.4
git tree</a>, on master. It passes the included regression
test, so it is at least dogfoodable. The nice thing about the
implementation is that it only slightly increased the code size,
but brought a lot of new features. If you have trouble getting
the code from git, let me know, I’m willing to create an alpha
release, so that it becomes easier to play with it.<br>
<br>
<b>Feedback</b><br>
<br>
I see a lot of potential in this functionality, however my
examples may have not been the best ones. I would really
appreciate any kind of feedback, please be sure to send those to
the <a moz-do-not-send="true"
href="http://lists.balabit.hu/mailman/listinfo/syslog-ng/">syslog-ng
mailing list</a> or post me as a private email.<br>
</font></blockquote>
<font color="#000000"><br>
This does sound like its going to be some useful stuff, and will
definitely be keeping an eye on it.<br>
<br>
Thanks :-)<br>
<br>
-Patrick<br>
</font>
</body>
</html>