Hi - Not directly similar, but somewhat analogous behaviour was observed using syslog-ng to log to a remote elasticsearch destination. The system exhibited the same symptoms - no ability to connect or login. We have no logs or data to directly support this - but my suspicion is that it had to do with TCP connections - possibly "maxing out" the system and leaving it impossible to establish any new connections. We could not (and cannot) conduct any real diagnostics on this system, since it is a production server (kinda painful when it locks up) But I would suggest looking at system level measurements & settings (memory, TCP connections) Also - you don't mention load - do you have the ability to control the load and monitor the system as it increases? You might try holding a small load for some time (minutes? hours?) while measuring system behaviour and ramping up in a "step function" to see if you can identify what is going on. Good luck, Jim On 01/02/2015 09:24 AM, Tomáš Novosad wrote:
Hello to everyone,
i'd like to as if someone else experienced such thing before.
I have a setup where i'm storing maillog and mailfilter log into PostgreSQL database via syslog-ng. Togethe with that, PostgreSQL is logging via syslog-ng as well.
This setup creates some sort of deadlock: after some time of processing (varies) syslog-ng freezes, thus completely blocking the server. It is not possible to do anything on the server as all calls to syslog just blocks. I'm not able to open new SSH session, as login is logged via syslog. Emails do not work as senmail is logging via syslog. These and all other applications just hung, whenever they try to write something into syslog.
This deadlock is caused by rather "circular" setup, when syslog writes into Postgres and Postgres logs into syslog. I'm able to reproduce the problem in my testing environment. When Postgres does not log into syslog (only CSV logs are enabled, everything goes just fine. Same thing if syslog-ng stores nessages in Postgres on different host - everything is OK. But in scenario when postgres is on localhost and logs into syslog, i'am experiencing those deadlocks.
Does anyone has similar experience?
We're regulary using syslog-ng 3.4.6, but i have juste tested it on latest version 3.4.8 - same problem appeared. Postgres is of version 9.2. We're using our own CentOS-based distro (currently CentOS 6).
I can provide configs, or reproduce the problem if needed.
Thanks for any help or hints.