[syslog-ng] Internal error, duplicate configuration elements...

[Evan Rempel]

Reorganizing the log configuration with junctions helps in some circumstances. In my case I need to use a different template
depending on the source of the message (from a specific file source) so I need to use the same
destination IP/port, but defined with a different syslog-ng "destination" specification so that I can use a different template.

This results in the same problem that I can't work around by using a different junction/log structure.

Evan.

On 06/10/2014 11:51 PM, Balazs Scheidler wrote:
> Hi,
>
> the issue is that syslog-ng uses the destination IP address/port combination as an identifier to recover the queue when reloading syslog-ng, so it does have some issues (e.g. one borrowing the queue of the other, or using the same queue after reload). This is not good.
>
> This could be solved with a long-discussed, explicit id() parameter, that would let the administrator assign a custom ID, which can be made different for the conflicting drivers, but that hasn't happened yet.
>
> Using junctions and log expressions could help you to reorganize your configuration in a way that prevent this kind of conflict:
>
> https://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/
>
> Hope this helps.
>
>
> On Fri, Jun 6, 2014 at 4:15 PM, David Hauck <davidh at netacquire.com <mailto:davidh at netacquire.com>> wrote:
>
>     This morning Gergely Nagy wrote:
>      > David Hauck <davidh at netacquire.com <mailto:davidh at netacquire.com>> writes:
>      >
>      >> Couple things here:
>      >> 1. If this is an error why doesn't 'syslog-ng -s' indicate it as such?
>      >
>      > Because -s checks for syntax only, and this is not a syntax error, it's one level higher.
>
>     OK.
>
>      >> 2. Other than the error, things appear to function correctly. Why is this?
>      >
>      > That'd be a bit hard to explain without going into low-level details,
>      > but I'll try: it works because of luck, mostly. UDP destinations will
>      > not trip over each other when used like this. If you'd try that with files, all hell would break loose.
>
>     OK, thanks, having some understanding of how the low level implementation works does help since I don't remember this constraint being discussed in the manual/docs (but I may have missed something there?). Initially it also wasn't clear that these destinations weren't combined at a high level in the implementation so that they, for example, operated within a "pipeline" and thereby enforced a "single-writer" paradigm. Either way, and as you say, this is required for file destinations, but
>     certainly not for network destinations (even in the disconnect case).
>
>     In fact, my log() statements with the referenced destinations are complex enough that I may need to rely on the multiple network destination specification capability anyway since combining these as you suggest would make maintenance of the configuration even more difficult (there's a tangle of rules/conditions associated with one of the destination log() statements).
>
>      >> 3. Other blocks do not require that their contents contain unique
>      >> statements. For example, I can create filter blocks that have
>      >> statements that intersect. Why not destination blocks?
>      >
>      > Because filters generally don't have state, so they can't conflict
>      > with each other, they can work independently. For destinations, that's not the case. For example, if you'd have something like this:
>      >
>      > destination d_1 { file("/var/log/some.log"); udp(...); }; destination
>      > d_2 { file("/var/log/some.log"); tcp(...); };
>      >
>      > That would break horribly, as two threads would try to write to the
>      > same file, and would write over each other, resulting in garbage. That
>      > doesn't happen with network destinations, but those have different
>      > issues. For example, if a network target goes down, and you had
>      > duplicates in your config, then they'd notice the target being down separately, increasing the chance you'll loose logs. If you had one target, and bound it to various sources in a log{} statement instead, then you'd only have one instance of the driver.
>
>     I'm wondering if there might be other issues associated with multiple (common) network destinations (you only mention one as an example)? The example you mention isn't relevant in my case. Whether the network destination is a combined statement/driver or not the payload includes single packets, which are written separately, no? If this is the case it wouldn't matter which packet send detected the network error - one or the other would get lost in both cases.
>
>     Thanks,
>     -David
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>     FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>


-- 
Evan Rempel                                      erempel at uvic.ca
Senior Systems Administrator                        250.721.7691
Data Centre Services, University Systems, University of Victoria