[syslog-ng] Internal error, duplicate configuration elements...

[David Hauck]

Hi Balazs,
 
On Tuesday, June 10, 2014 11:51 PM, Balazs Scheidler wrote:
> Hi,
> 
> the issue is that syslog-ng uses the destination IP address/port 
> combination as an identifier to recover the queue when reloading 
> syslog-ng, so it does have some issues (e.g. one borrowing the queue 
> of the other, or using the same queue after reload). This is not good.

Ah, OK, yes, this sounds like a problem. 
 
> This could be solved with a long-discussed, explicit id() parameter, 
> that would let the administrator assign a custom ID, which can be made 
> different for the conflicting drivers, but that hasn't happened yet.

Yup, good idea.

> Using junctions and log expressions could help you to reorganize your 
> configuration in a way that prevent this kind of conflict:
> 
> https://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvem
> en ts/

Thanks, I'll see what I can do.

BTW, would this "duplicate UDP destination" work any better if I had a policy where no re-transmissions occurred (i.e., I don't need any queuing for the purposes of re-transmits when the destinations are down)? If so, how would I configure this?

> Hope this helps.

Cheers,
-David

> On Fri, Jun 6, 2014 at 4:15 PM, David Hauck <davidh at netacquire.com>
> wrote:
> 
> 
> 	This morning Gergely Nagy wrote: 	> David Hauck <davidh at netacquire.com>
> writes: 	> 	>> Couple things here: 	>> 1. If this is an error why
> doesn't 'syslog-ng -s' indicate it as such? 	> 	> Because -s checks for
> syntax only, and this is not a syntax error, it's one level higher.
> 
> 
> 	OK.
> 
> 
> 	>> 2. Other than the error, things appear to function correctly.
> Why is this?
> 	>
> 	> That'd be a bit hard to explain without going into low-level details,
> 	> but I'll try: it works because of luck, mostly. UDP destinations will
> 	> not trip over each other when used like this. If you'd try that 
> with files, all hell would break loose.
> 
> 
> 	OK, thanks, having some understanding of how the low level 
> implementation works does help since I don't remember this constraint 
> being discussed in the manual/docs (but I may have missed something 
> there?). Initially it also wasn't clear that these destinations 
> weren't combined at a high level in the implementation so that they, 
> for example, operated within a "pipeline" and thereby enforced a 
> "single-writer" paradigm. Either way, and as you say, this is required 
> for file destinations, but certainly not for network destinations (even in the disconnect case).
> 
> 	In fact, my log() statements with the referenced destinations are 
> complex enough that I may need to rely on the multiple network 
> destination specification capability anyway since combining these as 
> you suggest would make maintenance of the configuration even more 
> difficult (there's a tangle of rules/conditions associated with one of the destination log() statements).
> 
> 
> 	>> 3. Other blocks do not require that their contents contain unique
> 	>> statements. For example, I can create filter blocks that have 	>>
> statements that intersect. Why not destination blocks? 	> 	> Because
> filters generally don't have state, so they can't conflict 	> with each
> other, they can work independently. For destinations, that's not the
> case. For example, if you'd have something like this: 	> 	> destination
> d_1 { file("/var/log/some.log"); udp(...); }; destination 	> d_2 {
> file("/var/log/some.log"); tcp(...); }; 	> 	> That would break horribly,
> as two threads would try to write to the 	> same file, and would write
> over each other, resulting in garbage. That 	> doesn't happen with
> network destinations, but those have different 	> issues. For example,
> if a network target goes down, and you had 	> duplicates in your config,
> then they'd notice the target being down separately, increasing the 
> chance you'll loose logs. If you had one target, and bound it to 
> various sources in a log{} statement instead, then you'd only have one 
> instance of the driver.
> 
> 
> 	I'm wondering if there might be other issues associated with multiple
> (common) network destinations (you only mention one as an example)?
> The example you mention isn't relevant in my case. Whether the network 
> destination is a combined statement/driver or not the payload includes 
> single packets, which are written separately, no? If this is the case 
> it wouldn't matter which packet send detected the network error
> - one or the other would get lost in both cases.
> 
> 	Thanks,
> 	-David
> 
> 
> __________________________________________________________
> ______________ ______ 	Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng 	Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng 	FAQ:
> http://www.balabit.com/wiki/syslog-ng-faq
> 
> 
> 
> 
> 
>