[syslog-ng] Simple config question

[Miguel Alvarez]

On Thu, Nov 3, 2011 at 10:36 AM, Gergely Nagy <algernon at balabit.hu> wrote:
> Miguel Alvarez <miguellvrz9 at gmail.com> writes:
>
>> Sorry, I should have elaborated on the tagging other files part.
>>
>> I have a system that runs snort as well as bro.  Snort writes its
>> alert file to /var/log/snort/alert .  Bro writes to a few files such
>> as /var/log/bro/conn.log and /var/log/bro/http.log .  Right now, I
>> have rsyslog tagging each with a description of the given alert file
>> so they can be filtered on the remote log server side.  "[SNORT]" for
>> snort, "[BRO-CONN]" and "[BRO-HTTP]" for bro's conn.log and http.log
>> respectively.  Here are the rsyslog configuration sections for those
>> three logs:
>
> Aha!
>
> Well, something similar is possible with syslog-ng aswell: (beware,
> completely untested, there might be typos!)
>
> ###
> # File sources
> # ------------
> #
> # These set up sources, and tag them appropriately. We'll use the tags
> # later in the rewrite rules.
> ###
>
> source s_snort_alert {
>  file("/var/log/snort/alert");
>  tag("snort");
> };
>
> source s_bro_conn {
>  file("/var/log/bro/conn.log");
>  tag("bro-conn");
> };
>
> source s_bro_http {
>  file("/var/log/bro/http.log");
>  tag("bro-http");
> };
>
> ###
> # Templates
> # ---------
> #
> # Templates are used similarly as in rsyslog (except our templates are
> # awesome, and theirs isn't. Sadly, this example is too simple to show
> # the power of syslog-ng templates. Oh well..).
> #
> # Anyway, in this case, the template will be similar to a normal
> # BSD legacy syslog format, with ${MSG_TAG} inserted between the
> # MSGHEADER and the message itself. If MSG_TAG is unset, nothing will be
> # inserted, and we'll get a standard format.
> ###
> template t_tagged {
>  template("${ISODATE} ${HOST} ${MSGHDR}${MSG_TAG}${MSG}");
> };
>
> ###
> # Destinations
> # ------------
> #
> # Ye olde TCP destination. You can replace tcp with upd, if so you
> # wish. It forwards everything that reaches the destination to the
> # specified host, on the given port, using the template we made above.
> ###
> destination d_remote_tagged {
>  tcp("192.168.1.1" port(1200) template(t_tagged));
> };
>
> ###
> # Rewrite
> # -------
> #
> # Rewrite rules! If we encounter a tag we care about, we set MSG_TAG
> # appropriately. That is all. If a message does not have the sought tag,
> # the rewrite does nothing.
> ###
> rewrite r_snort_tag {
>  rewrite(set("MSG_TAG", value("[SNORT] ") condition(tag("snort"))));
> };
>
> rewrite r_bro_conn_tag {
>  rewrite(set("MSG_TAG", value("[BRO-CONN] ") condition(tag("bro-conn"))));
> };
>
> rewrite r_bro_http_tag {
>  rewrite(set("MSG_TAG", value("[BRO-HTTP] ") condition(tag("bro-http"))));
> };
>
> ###
> # Logpath
> # -------
> #
> # Logpaths define how sources, filters, rewrite rules and destinations
> # are connected.
> #
> # In this case, this logpath will read from all three file sources
> # defined above, pass them through all three rewrite rules (remember:
> # those only do the rewrite if the appropriate tag matches), and
> # finally, send it over to the remote host.
> #
> # We also set a "final" flag, which means that if a message was caught
> # by this rule (ie, it came from any of the three files), it will not be
> # processed further by any other logpath.
> ###
>
> log {
>  source(s_snort_alert);
>  source(s_bro_conn);
>  source(s_bro_http);
>
>  rewrite (r_snort_tag);
>  rewrite (r_bro_conn_tag);
>  rewrite (r_bro_http_tag);
>
>  destination (d_remote_tagged);
>
>  flags(final);
> };
>
> And this is all you need to process the files.
>
> The rest of your rsyslog.conf is easier to translate to syslog-ng.conf
> style, and hence, I'm not going to describe it here. I'd suggest quickly
> skimming through the docs[1], and it should be reasonably
> straightforward.
>
>  [1]: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.2-guides/syslog-ng-ose-v3.2-guide-admin-en.html/bk01-toc.html
>
> The contrib/syslog2ng script in the syslog-ng sources should help with
> the translation too, as rsyslog.conf has parts that are compatible with
> old syslog.conf (and the rest of your config pretty much consist of such
> parts ;).
>
> Hope this helps!

Wow, thank you so much!  That actually looks pretty straight forward.

I initially had syslog-ng 3.2.4 installed but it was complaining about
the "source plugin tag not found".  I thought this was perhaps due to
it not being 3.3 so I built and installed 3.3.1 but am still seeing
it.  Is there something I'm missing from my build or not loading in my
config?

# /etc/init.d/syslog-ng start
Starting syslog-ng: Error parsing source, source plugin tag not found
in /etc/syslog-ng/syslog-ng.conf at line 62, column 1:

tag("snort");
^^^

I ran syslog-ng --version to see what modules were available and I
don't see anything with "tag" in it.  Do I need to pass something at
startup with the "--default-modules" flag?

# syslog-ng --version
syslog-ng 3.3.1
Installer-Version: 3.3.1
Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#3a736e62b27f7036ab23b91cf0839a95d0185e18
Compile-Date: Nov  3 2011 19:24:14
Default-Modules:
affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat
Available-Modules:
convertfuncs,afmongodb,affile,dummy,confgen,basicfuncs,csvparser,afsocket-tls,afuser,afsocket,dbparser,afprog,syslogformat
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: off
Enable-Pcre: on

Thank you!