[syslog-ng] Duplicate configuration items causing memory leak

[Balazs Scheidler]

On Sat, 2011-11-12 at 17:51 +0100, Balazs Scheidler wrote:
> On Wed, 2011-11-09 at 14:58 -0800, Dave Rawks wrote:
> > On 11/09/2011 12:38 PM, Balazs Scheidler wrote:
> > >> Does this patch break the behavior described in the documentation for
> > >> 3.3.1?
> > >>
> > >>> If an object is defined twice (for example the original syslog-ng 
> > >>> configuration file and the file imported into this configuration
> > >>> file both define the same option, source, or other object), then the 
> > >>> object that is defined later in the configuration file will be 
> > >>> effective. For example, if you set a global option at the beginning 
> > >>> of the configuration file, and later include a file that defines the 
> > >>> same option with a different value, then the option defined in the 
> > >>> imported file will be used.
> > >>
> > >> If you cannot overload previously defined objects then the ability to
> > >> include config files becomes FAR less useful imho.
> > > 
> > > uhm, uhm. Yes it does. I didn't know this was a documented feature :)
> > > 
> > > This could be controlled (possibly by a configuration variable), but
> > > before I do that could you explain your usecase, which makes include
> > > files to be less useful?
> > > 
> > 
> > Well, the standard case for having includes would be for use in
> > distribution packaging. Where syslog-ng would ship with a default
> > configuration and would then do an include against a directory of files,
> > a "conf.d" style setup. Later other packages could drop bits of
> > configuration into conf.d. Lets say you install some theoretical log
> > collection and parsing application via a distribution provided package.
> > Previously when overloading was supported that new package could drop a
> > config chunk which overloads a "log object" from the default config so
> > that now instead of outputting to a file it outputs to a destination
> > object defined in this new config.
> 
> It is possible to create a catch-all log statement, where source
> definitions need not be listed explicitly, e.g.
> 
> source aaa {};
> source bbb {};
> 
> destination foo {};
> 
> log { destination(foo); flags(catch-all); };
> 
> This would direct syslog-ng to process all messages, regardless of the
> source.
> 
> Also, conf.d style inclusion is supported since 3.0; so I would say that
> distribution specific configuration can be implemented on what is
> available right now.
> 
> > 
> > Actually I was assuming that not only would this behavior persist into
> > future versions of syslog-ng but I had also hoped that it would be
> > enhanced to allow for things like partial overloads or concatenation of
> > existing object definitions such as is available in software such as
> > nagios. Since there is no function by which later defined objects can
> > interrogate the state of earlier objects nor is there any capability for
> > conditional object definition (afaik) there is some inherint limitation
> > to the complexity of the configs which could be generated still, but
> > this would be a huge step forward in config flexibility.
> 
> I understand what you mean, and I can imagine some use-cases, however I
> still have my doubts that such a universal configuration file can be
> created for syslog-ng that would include all possible needs so that a
> package would only need to drop its own 'log definition' to interoperate
> with that.
> 
> My vision right now is to create SCL blocks to cover applications, that
> would:
>   1) encapsulate all parameters required to track the logs of a given
> application
>   2) would make it very easy to create a syslog-ng configuration file
> that uses these blocks
> 
> But otherwise how logs are processed are up to the user to specify.
> 
> Something along the lines of:
> 
> block source proftpd log_dir("/var/log/proftpd") {
> 
> 	file("`log_dir`/transfer.log" follow-freq(1) tags('proftpd-transfer'));
> 
> }
> 
> source s_local {
> 	system();
> 	proftpd();
> 	...
> };
> 
> E.g. instead of requiring a rigid structure in the configuration file
> and then allow "plugins" to augment that with minor stuff, I'd like to
> allow the admin (or the log management product using syslog-ng as a
> transport) to create the processing rules she wants but still make it
> relatively easy to pull in logs from various applications.
> 
> If we have an agent-like setup (e.g. pull logs from all applications and
> send them to a central server/relay), you'd need something like:
> 
> source s_local {
> 	system();
> 	app1();
> 	app2();
> 	app3();
> };
> 
> destination d_network { ... };
> 
> log { source(s_local); destination(d_network); };
> 
> However on the server, you'll most probably want something completely
> different:
> 
> source s_network {
> 	syslog();
> };
> 
> log { source(s_network); 
> 	log { filter(f_app1); parser(f_app1); destination(d_app1); };
> 	log { filter(f_app2); parser(f_app2); destination(d_app2); };
> }
> 
> 
> > 
> > ------Separate but related thought ;)
> > 
> > Speaking of which, it would be nice if there was some method of bundling
> > objects together. So that for instance you could define:
> > 
> > > source s_local {
> > >         internal();
> > >         unix-dgram("/dev/log");
> > >         file("/proc/kmsg" program-override("kernel") flags(kernel));
> > > };
> > > source s_remote {
> > >         syslog(
> > >         transport("udp")
> > >         so_rcvbuf(16777216)
> > >         );
> > > };
> > > source s_everything {
> > >         source(s_local)
> > >         source(s_remote)
> > >         );
> > > };
> > 
> > I might be missing something, but afaict there is no easy method to do
> > such a definition in the documentation.
> 
> You can't do this right now, except with catch-all, and using embedded
> log statements (like in the example above).
> 
> I still can be convinced though, but could you explain how your "stock"
> config would like, and how a package like proftpd would drop a config
> snippet in the directory to "automatically" extend the local
> configuration? And how the same would work in an agent / server like
> setup?
> 
> I'm almost convinced to remove the error message about this case though,
> even though I think that in most cases it is really an unintended config
> error.

I've committed this patch for the time being (before 3.3.2):

commit c94867c78a3250795806a4b0eab69ea6be758c21
Author: Balazs Scheidler <bazsi at balabit.hu>
Date:   Sat Nov 12 20:46:22 2011 +0100

    configuration: allow configuration object duplicates
    
    Historically syslog-ng used to allow overriding configuration block simply
    by defining the same object twice. A previous patch has changed this
    to an error, however there seem to be people who rely on the old behaviour.
    
    In order to serve both groups of people (who get it wrong and who really
    wants to use this) a new global variable has been introduced.
    
    This line in the beginning of the config:
    
    @define allow-config-dups 1
    
    will restore the old behaviour, this is also printed to the console
    at startup.
    
    Reported-By: Dave Rawks <dave at pandora.com>
    Signed-off-by: Balazs Scheidler <bazsi at balabit.hu>




-- 
Bazsi