[syslog-ng] syslog-ng stats to ES ?

Scheidler, Balázs balazs.scheidler at balabit.com
Thu Oct 19 08:21:47 UTC 2017 

Looks that syslog-ng-ctl defaults to the default path (e.g.
/var/lib/syslog-ng/syslog-ng.ctl), but IIRC there is an option to specify
that to the syslog-ng-ctl client as well.

The new query interface for syslog-ng-ctl has changed a few things in
counters, whereas syslog-ng-ctl stats has been kept for backwards
compatibility.


-- 
Bazsi

On Wed, Oct 18, 2017 at 6:52 PM, Scot <scotrn at gmail.com> wrote:

> syslog-ng-ctl fails with
> * /usr/lib/systemd/system/syslog-ng.service *
> ExecStart=/usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F
> $SYSLOGNG_OPTS -p /var/run/syslogd.pid
>
> Looks like syslog-ng-ctl does NOT work when added as a CLI option.
> Probably need to be added to an env file /etc/sysconfig.
>
> *# ps -ef | grep syslog*
> root     14293     1 24 12:37 ?        00:00:01 /usr/sbin/syslog-ng
> --control /var/run/syslog-ng.ctl -F -p /var/run/syslogd.pid
> root     14316 14120  0 12:37 pts/0    00:00:00 grep --color=auto syslog
>
> *# lsof -p 14293  | grep ctl*
> syslog-ng 14293 root    5u     unix 0xffff880310fbd800       0t0 41294162
> /var/run/syslog-ng.ctl
> *# lsof /var/run/syslog-ng.ctl*
> COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
> syslog-ng 14293 root    5u  unix 0xffff880310fbd800      0t0 41294162
> /var/run/syslog-ng.ctl
> *# syslog-ng-ctl stats*
> Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl',
> error='Connection refused'
> *# ls -al /var/lib/syslog-ng/syslog-ng.ctl*
> srwxr-xr-x. 1 root root 0 Oct 17 17:19 /var/lib/syslog-ng/syslog-ng.ctl
> *# ls -al /var/run/syslog-ng.ctl*
> srwxr-xr-x. 1 root root 0 Oct 18 12:37 /var/run/syslog-ng.ctl
>
>
> I shutdown syslog-ng removed the file and the ctl file from the
> *syslog-ng.service  *
> Looks like the default is /var/lib/syslog-ng/syslog-ng.ctl  everything
> runs fine with that.
>
> *# lsof -p 27812  |grep ctl*
> syslog-ng 27812 root    5u     unix 0xffff88033269a400       0t0 41025397
> /var/lib/syslog-ng/syslog-ng.ctl
>
> One thing I did notice is query "*" adds .*written *metric for each
> destination which is missing from syslog-ng-ctl stats (maybe expected I
> didn't cross ref).
>
>
>
> On Wed, Oct 18, 2017 at 2:16 AM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
>> That socket should be opened, even without the control option. So if you
>> get that error, that means its somewhere else, or we didnt open it for some
>> reason
>> Can you run lsof on the syslog-ng process? That should display the
>> control socket.
>>
>> Also, syslog-ng-ctl reload would use the very same socket. Does that work?
>>
>> On Oct 17, 2017 21:01, "Scot" <scotrn at gmail.com> wrote:
>>
>>> CentOS 7
>>>
>>> I added --control /var/run/syslog-ng.ctl to
>>>
>>> * /usr/lib/systemd/system/syslog-ng.service *
>>> ExecStart=/usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F
>>> $SYSLOGNG_OPTS -p /var/run/syslogd.pid
>>>
>>>
>>>  syslog-ng-ctl query get /var/run/syslog-ng.ctl
>>> Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl',
>>> error='Connection refused'
>>>
>>>
>>>
>>> On Tue, Oct 17, 2017 at 2:44 PM, Scot <scotrn at gmail.com> wrote:
>>>
>>>> Where are the query options documented ?   Been looking Google, Balabit
>>>> for an hour.
>>>> man pages have nothing.
>>>>
>>>> On Tue, Oct 17, 2017 at 11:42 AM, Czanik, Péter <
>>>> peter.czanik at balabit.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> If you work with syslog-ng-ctl you can give "jo" ( JSON output:
>>>>> https://github.com/jpmens/jo ) a try. I only did some basic tests,
>>>>> but it seems to me that it can turn the output of "syslog-ng-ctl query"
>>>>> into JSON.
>>>>>
>>>>> Bye,
>>>>>
>>>>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>>>>> Balabit / syslog-ng upstream
>>>>> https://www.balabit.com/blog/author/peterczanik/
>>>>> https://twitter.com/PCzanik
>>>>>
>>>>> On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs <
>>>>> balazs.scheidler at balabit.com> wrote:
>>>>>
>>>>>> Difficult, the whole problem is naming of the name value pairs.
>>>>>>
>>>>>> The idea behind stats is to generate all name value pairs in one
>>>>>> message, and this simply does not scale. You are almost certainly
>>>>>> interested in a set of values or an aggregate of a set, and not everything.
>>>>>>
>>>>>> Just set stats-level() to 3, and look at the stats message.
>>>>>>
>>>>>> I am not saying its impossible, just that it requires some thought.
>>>>>>
>>>>>> On Oct 17, 2017 17:09, "Scot" <scotrn at gmail.com> wrote:
>>>>>>
>>>>>>> How about an output modifier ?
>>>>>>>
>>>>>>> On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs <
>>>>>>> balazs.scheidler at balabit.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> the issue with the internal stats() message is that if you have a
>>>>>>>> lot of counters that message is truncated. Also, it is pretty difficult to
>>>>>>>> parse.
>>>>>>>>
>>>>>>>> So I would vote for the "poll syslog-ng-ctl and generate messages"
>>>>>>>> solution.
>>>>>>>>
>>>>>>>> BTW: the internal PE team did something in this area, they created
>>>>>>>> some sort of internal source that does this polling, but I am not sure how
>>>>>>>> that works. Possibly there's documentation :)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bazsi
>>>>>>>>
>>>>>>>> On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Doesn't stats_freq() set an interval to log stats to syslog
>>>>>>>>> already?
>>>>>>>>>
>>>>>>>>> Description: The period between two STATS messages in seconds.
>>>>>>>>> STATS are log messages sent by syslog-ng, containing statistics
>>>>>>>>> about dropped log messages. Set to 0to disable the STATS messages.
>>>>>>>>>
>>>>>>>>> So
>>>>>>>>> internal_src -> format > elasticsearch -> syslog-ng_stats index ?
>>>>>>>>>
>>>>>>>>> On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel at uvic.ca>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I have a perl script that collects some stats and logs them to
>>>>>>>>>> syslog again. The syslog stream gets sent to ES, so they end up there, but
>>>>>>>>>> as a syslog line, not a specific statistic item for things like grafana.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/15/2017 05:57 PM, Scot wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>>   Looked around for a few hours and didn't see anything.
>>>>>>>>>>>
>>>>>>>>>>> Has anyone worked on sending syslog-ng stats to ES ?
>>>>>>>>>>> I see several ways I could but wondering if anyone has already.
>>>>>>>>>>> A push method directly from syslog-ng would be awesome.
>>>>>>>>>>>
>>>>>>>>>>> Scot
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> ____________________________________________________________
>>>>>>>>>> __________________
>>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>>> Documentation: http://www.balabit.com/support
>>>>>>>>>> /documentation/?product=syslog-ng
>>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ____________________________________________________________
>>>>>>>>> __________________
>>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>>> Documentation: http://www.balabit.com/support
>>>>>>>>> /documentation/?product=syslog-ng
>>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ____________________________________________________________
>>>>>>>> __________________
>>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>>> Documentation: http://www.balabit.com/support
>>>>>>>> /documentation/?product=syslog-ng
>>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ____________________________________________________________
>>>>>>> __________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation: http://www.balabit.com/support
>>>>>>> /documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ____________________________________________________________
>>>>>> __________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation: http://www.balabit.com/support
>>>>>> /documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ____________________________________________________________
>>>>> __________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support
>>>>> /documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20171019/8b14cc59/attachment-0001.html>

More information about the syslog-ng mailing list