Re: [syslog-ng] Issues running syslog-ng-ctl program locally to determine logging issues
Also would like to throw out that I am open to using strace or some other debug hook program to figure out the cause here if that would help better, just would like a pointer as to how I would go about using it to identify dropped logs. I don't believe running syslog-ng with the debug setting on its own indicates if a logged event has been dropped or not (but I suppose it could indicate that the log never made its way over). Thanks, Andrew From: Andrew Bell Sent: Wednesday, April 15, 2015 11:55 AM To: syslog-ng@lists.balabit.hu Subject: Issues running syslog-ng-ctl program locally to determine logging issues Hello, Wondering if someone can help me with an issue that I am currently having. I am running syslog-ng as a local non-root user (I have it installed to a local instance under my home directory) and am trying to use the syslog-ng-ctl program in order to diagnose why certain log files are not passing through syslog-ng to their log destination (trying to rule out if it is a problem with the location where the logs originate OR if there is an issue with the syslog-ng setup and it is dropping or otherwise failing to forward those incoming logs). However, when I try to run syslog-ng-ctl under my local account I get the following error, where it appears to reference a control socket that is located in a directory that is owned and operated by the root account (I should note that I am unable to sudo as root with my setup on the system). /home/abell/syslog-ng-install/opt/syslog-ng/sbin$ ./syslog-ng-ctl verbose Error connecting control socket, socket='/opt/syslog-ng/var/run/syslog-ng.ctl', error='No such file or directory' This type of error above is also appearing when I run syslog-ng program itself with the debug ('-d -v') options set Error opening control socket, bind() failed; socket='/opt/syslog-ng/var/run/syslog-ng.ctl', error='No such file or directory (2)' I did some searching online and did find some manuals on syslog-ng-ctl that said I could specify a different directory where the control socket resides to one that isn't in the root directory using the "-c" flag. This mailing list post also mentioned that running a local instance of syslog-ng-ctl should automatically create a local instance of the control socket file located in the /var directory where the local instance resides (so in my case, I imagine it would be /home/abell/syslog-ng-install/var ???). Here's the reference link -> https://lists.balabit.hu/pipermail/syslog-ng/2009-June/013027.html - there weren't any next messages to this thread so it is difficult to tell if this solution worked for the original author here or not... So I created that "/var" directory as he suggested but I still don't see the .ctl file created when I go to stop and restart syslog-ng ... and so I don't really have anything to specify for the "-c" option as an alternative. I tried touching the control socket file in that directory as a zero byte file, but still nothing /home/abell/syslog-ng-install/opt/syslog-ng/var$ touch syslog-ng.ctl /home/abell/syslog-ng-install/opt/syslog-ng/var$ cd .. /home/abell/syslog-ng-install/opt/syslog-ng$ cd sbin /home/abell/syslog-ng-install/opt/syslog-ng/sbin$ ./syslog-ng-ctl verbose --set=on -c /home/abell/syslog-ng-install/opt/syslog-ng/var/syslog-ng.ctl Error connecting control socket, socket='/home/abell/syslog-ng-install/opt/syslog-ng/var/syslog-ng.ctl', error='Connection refused' Any ideas as to how I can get this control socket file created on my local syslog-ng - or if I can tell syslog-ng itself to specify spinning up the control socket file itself in a different location than the root one? I'd really like to be able to use this tool to figure out what is going on with my syslog-ng setup. Let me know if there's any other details or information that I can provide. Thanks, Andrew
On Wed, Apr 15, 2015 at 11:16 AM, Andrew Bell <abell@factset.com> wrote:
Also would like to throw out that I am open to using strace or some other debug hook program to figure out the cause here if that would help better, just would like a pointer as to how I would go about using it to identify dropped logs.
I wrote a small perl program (attached) to query syslog-ng-ctl and netstat to see buffer sizes and thus identify if logs are being dropped. It loops forever and once a second (or defined interval) it runs: syslog-ng-ctl to get the UDP messages processed syslog-ng-ctl to get the TLS (TCP) messages processed netstat to get the UDP errors netstat to get the UDP packets received It then prints out the rate of change of those four values and you can see if the "graph" is plateauing - which for the logs processed indicates some sort of bottleneck - or if there are peaks and valleys which indicate syslog-ng is able to handle the load. -m
On Wed, Apr 15, 2015 at 4:35 PM, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
On Wed, Apr 15, 2015 at 11:16 AM, Andrew Bell <abell@factset.com> wrote:
Also would like to throw out that I am open to using strace or some other debug hook program to figure out the cause here if that would help better, just would like a pointer as to how I would go about using it to identify dropped logs.
I wrote a small perl program (attached) to query syslog-ng-ctl and netstat to see buffer sizes and thus identify if logs are being dropped.
It loops forever and once a second (or defined interval) it runs:
syslog-ng-ctl to get the UDP messages processed syslog-ng-ctl to get the TLS (TCP) messages processed netstat to get the UDP errors netstat to get the UDP packets received
It then prints out the rate of change of those four values and you can see if the "graph" is plateauing - which for the logs processed indicates some sort of bottleneck - or if there are peaks and valleys which indicate syslog-ng is able to handle the load.
-m
Fantastic work Matt ... I am testing your script in a syslog-ng collector server: Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 2, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 2, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 Rate of UDP packet RX: 0, Rate of UDP packet RX errors: 0 .. and only shows me UDP stats. What about TCP stats?? I am receiving logs via TCP in this host ...
On Thu, Apr 16, 2015 at 1:57 AM, C. L. Martinez <carlopmart@gmail.com> wrote:
Fantastic work Matt ... I am testing your script in a syslog-ng collector server:
[...]
.. and only shows me UDP stats. What about TCP stats?? I am receiving logs via TCP in this host ...
You'll want to verify/tweak a few things: netstat -anus runs and yields something like: Udp: 1116688 packets received 7373 packets to unknown port received. 0 packet receive errors You'll want to make sure that the user that runs the script can also run: syslog-ng-ctl stats and you'll want to change the script and replace: source;s_udp;;a;processed; with a string that makes sense for your UDP source. Similarly, you'll want to change source;s_tls;;a;processed; to something that makes sense (for your TCP or TLS) logging. Again, this script just executes a couple commands and scrapes the data and shows how the data changes over time. Let me know if you've got further questions. -m
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 04/16/2015 04:29 PM, Matt Zagrabelny wrote:
On Thu, Apr 16, 2015 at 1:57 AM, C. L. Martinez <carlopmart@gmail.com> wrote:
Fantastic work Matt ... I am testing your script in a syslog-ng collector server:
[...]
.. and only shows me UDP stats. What about TCP stats?? I am receiving logs via TCP in this host ...
You'll want to verify/tweak a few things:
netstat -anus
runs and yields something like:
Udp: 1116688 packets received 7373 packets to unknown port received. 0 packet receive errors
You'll want to make sure that the user that runs the script can also run:
syslog-ng-ctl stats
and you'll want to change the script and replace:
source;s_udp;;a;processed;
with a string that makes sense for your UDP source.
Similarly, you'll want to change
source;s_tls;;a;processed;
to something that makes sense (for your TCP or TLS) logging.
Again, this script just executes a couple commands and scrapes the data and shows how the data changes over time.
Let me know if you've got further questions.
-m
Thanks Matt... It is very clear now.
participants (4)
-
Andrew Bell
-
C. L. Martinez
-
C.L. Martinez
-
Matt Zagrabelny