[syslog-ng] Duplicate configuration items causing memory leak

[Balazs Scheidler]

On Wed, 2011-11-09 at 14:58 -0800, Dave Rawks wrote:
> On 11/09/2011 12:38 PM, Balazs Scheidler wrote:
> >> Does this patch break the behavior described in the documentation for
> >> 3.3.1?
> >>
> >>> If an object is defined twice (for example the original syslog-ng 
> >>> configuration file and the file imported into this configuration
> >>> file both define the same option, source, or other object), then the 
> >>> object that is defined later in the configuration file will be 
> >>> effective. For example, if you set a global option at the beginning 
> >>> of the configuration file, and later include a file that defines the 
> >>> same option with a different value, then the option defined in the 
> >>> imported file will be used.
> >>
> >> If you cannot overload previously defined objects then the ability to
> >> include config files becomes FAR less useful imho.
> > 
> > uhm, uhm. Yes it does. I didn't know this was a documented feature :)
> > 
> > This could be controlled (possibly by a configuration variable), but
> > before I do that could you explain your usecase, which makes include
> > files to be less useful?
> > 
> 
> Well, the standard case for having includes would be for use in
> distribution packaging. Where syslog-ng would ship with a default
> configuration and would then do an include against a directory of files,
> a "conf.d" style setup. Later other packages could drop bits of
> configuration into conf.d. Lets say you install some theoretical log
> collection and parsing application via a distribution provided package.
> Previously when overloading was supported that new package could drop a
> config chunk which overloads a "log object" from the default config so
> that now instead of outputting to a file it outputs to a destination
> object defined in this new config.

It is possible to create a catch-all log statement, where source
definitions need not be listed explicitly, e.g.

source aaa {};
source bbb {};

destination foo {};

log { destination(foo); flags(catch-all); };

This would direct syslog-ng to process all messages, regardless of the
source.

Also, conf.d style inclusion is supported since 3.0; so I would say that
distribution specific configuration can be implemented on what is
available right now.

> 
> Actually I was assuming that not only would this behavior persist into
> future versions of syslog-ng but I had also hoped that it would be
> enhanced to allow for things like partial overloads or concatenation of
> existing object definitions such as is available in software such as
> nagios. Since there is no function by which later defined objects can
> interrogate the state of earlier objects nor is there any capability for
> conditional object definition (afaik) there is some inherint limitation
> to the complexity of the configs which could be generated still, but
> this would be a huge step forward in config flexibility.

I understand what you mean, and I can imagine some use-cases, however I
still have my doubts that such a universal configuration file can be
created for syslog-ng that would include all possible needs so that a
package would only need to drop its own 'log definition' to interoperate
with that.

My vision right now is to create SCL blocks to cover applications, that
would:
  1) encapsulate all parameters required to track the logs of a given
application
  2) would make it very easy to create a syslog-ng configuration file
that uses these blocks

But otherwise how logs are processed are up to the user to specify.

Something along the lines of:

block source proftpd log_dir("/var/log/proftpd") {

	file("`log_dir`/transfer.log" follow-freq(1) tags('proftpd-transfer'));

}

source s_local {
	system();
	proftpd();
	...
};

E.g. instead of requiring a rigid structure in the configuration file
and then allow "plugins" to augment that with minor stuff, I'd like to
allow the admin (or the log management product using syslog-ng as a
transport) to create the processing rules she wants but still make it
relatively easy to pull in logs from various applications.

If we have an agent-like setup (e.g. pull logs from all applications and
send them to a central server/relay), you'd need something like:

source s_local {
	system();
	app1();
	app2();
	app3();
};

destination d_network { ... };

log { source(s_local); destination(d_network); };

However on the server, you'll most probably want something completely
different:

source s_network {
	syslog();
};

log { source(s_network); 
	log { filter(f_app1); parser(f_app1); destination(d_app1); };
	log { filter(f_app2); parser(f_app2); destination(d_app2); };
}


> 
> ------Separate but related thought ;)
> 
> Speaking of which, it would be nice if there was some method of bundling
> objects together. So that for instance you could define:
> 
> > source s_local {
> >         internal();
> >         unix-dgram("/dev/log");
> >         file("/proc/kmsg" program-override("kernel") flags(kernel));
> > };
> > source s_remote {
> >         syslog(
> >         transport("udp")
> >         so_rcvbuf(16777216)
> >         );
> > };
> > source s_everything {
> >         source(s_local)
> >         source(s_remote)
> >         );
> > };
> 
> I might be missing something, but afaict there is no easy method to do
> such a definition in the documentation.

You can't do this right now, except with catch-all, and using embedded
log statements (like in the example above).

I still can be convinced though, but could you explain how your "stock"
config would like, and how a package like proftpd would drop a config
snippet in the directory to "automatically" extend the local
configuration? And how the same would work in an agent / server like
setup?

I'm almost convinced to remove the error message about this case though,
even though I think that in most cases it is really an unintended config
error.

z

-- 
Bazsi