[syslog-ng] deadlock when logging to SQL destination (PostgreSQL)

[Jim Hendrick]

Hi -

  Not directly similar, but somewhat analogous behaviour was observed
using syslog-ng to log to a remote elasticsearch destination.

The system exhibited the same symptoms - no ability to connect or login.

We have no logs or data to directly support this - but my suspicion is
that it had to do with TCP connections - possibly "maxing out" the
system and leaving it impossible to establish any new connections.

We could not (and cannot) conduct any real diagnostics on this system,
since it is a production server (kinda painful when it locks up)

But I would suggest looking at system level measurements & settings
(memory, TCP connections)

Also - you don't mention load - do you have the ability to control the
load and monitor the system as it increases?

You might try holding a small load for some time (minutes? hours?) while
measuring system behaviour and ramping up in a "step function" to see if
you can identify what is going on.

Good luck,
Jim

On 01/02/2015 09:24 AM, Tomáš Novosad wrote:
> Hello to everyone,
>
> i'd like to as if someone else experienced such thing before.
>
> I have a setup where i'm storing maillog and mailfilter log into
> PostgreSQL database via syslog-ng.
> Togethe with that, PostgreSQL is logging via syslog-ng as well.
>
> This setup creates some sort of deadlock:
> after some time of processing (varies)  syslog-ng freezes,
> thus completely blocking the server.
> It is not possible to do anything on the server as all calls to syslog
> just blocks.
> I'm not able to open new SSH session, as login is logged via syslog.
> Emails do not work as senmail is logging via syslog.
> These and all other applications just hung, whenever they try to write
> something into syslog.
>
> This deadlock is caused by rather "circular" setup, when syslog writes
> into Postgres and Postgres logs into syslog.
> I'm able to reproduce the problem in my testing environment.
> When Postgres does not log into syslog (only CSV logs are enabled,
> everything goes just fine.
> Same thing if syslog-ng stores nessages in Postgres on different host -
> everything is OK.
> But in scenario when postgres is on localhost and logs into syslog,
> i'am experiencing those deadlocks.
>
> Does anyone has similar experience?
>
> We're regulary using syslog-ng 3.4.6, but i have juste tested it on
> latest version 3.4.8 - same problem appeared.
> Postgres is of version 9.2.
> We're using our own CentOS-based distro (currently CentOS 6).
>
> I can provide configs, or reproduce the problem if needed.
>
> Thanks for any help or hints.