Hi, Looked around for a few hours and didn't see anything. Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome. Scot
I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana. On 10/15/2017 05:57 PM, Scot wrote:
Hi,
Looked around for a few hours and didn't see anything.
Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome.
Scot
Doesn't stats_freq() set an interval to log stats to syslog already? Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages. So internal_src -> format > elasticsearch -> syslog-ng_stats index ? On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana.
On 10/15/2017 05:57 PM, Scot wrote:
Hi,
Looked around for a few hours and didn't see anything.
Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome.
Scot
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse. So I would vote for the "poll syslog-ng-ctl and generate messages" solution. BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :) -- Bazsi On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
Doesn't stats_freq() set an interval to log stats to syslog already?
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages.
So internal_src -> format > elasticsearch -> syslog-ng_stats index ?
On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana.
On 10/15/2017 05:57 PM, Scot wrote:
Hi,
Looked around for a few hours and didn't see anything.
Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome.
Scot
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
How about an output modifier ? On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse.
So I would vote for the "poll syslog-ng-ctl and generate messages" solution.
BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :)
-- Bazsi
On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
Doesn't stats_freq() set an interval to log stats to syslog already?
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages.
So internal_src -> format > elasticsearch -> syslog-ng_stats index ?
On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana.
On 10/15/2017 05:57 PM, Scot wrote:
Hi,
Looked around for a few hours and didn't see anything.
Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome.
Scot
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Difficult, the whole problem is naming of the name value pairs. The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything. Just set stats-level() to 3, and look at the stats message. I am not saying its impossible, just that it requires some thought. On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
How about an output modifier ?
On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse.
So I would vote for the "poll syslog-ng-ctl and generate messages" solution.
BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :)
-- Bazsi
On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
Doesn't stats_freq() set an interval to log stats to syslog already?
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages.
So internal_src -> format > elasticsearch -> syslog-ng_stats index ?
On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana.
On 10/15/2017 05:57 PM, Scot wrote:
Hi,
Looked around for a few hours and didn't see anything.
Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome.
Scot
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, If you work with syslog-ng-ctl you can give "jo" ( JSON output: https://github.com/jpmens/jo ) a try. I only did some basic tests, but it seems to me that it can turn the output of "syslog-ng-ctl query" into JSON. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Difficult, the whole problem is naming of the name value pairs.
The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything.
Just set stats-level() to 3, and look at the stats message.
I am not saying its impossible, just that it requires some thought.
On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
How about an output modifier ?
On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse.
So I would vote for the "poll syslog-ng-ctl and generate messages" solution.
BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :)
-- Bazsi
On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
Doesn't stats_freq() set an interval to log stats to syslog already?
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages.
So internal_src -> format > elasticsearch -> syslog-ng_stats index ?
On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana.
On 10/15/2017 05:57 PM, Scot wrote:
Hi,
Looked around for a few hours and didn't see anything.
Has anyone worked on sending syslog-ng stats to ES ? I see several ways I could but wondering if anyone has already. A push method directly from syslog-ng would be awesome.
Scot
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Where are the query options documented ? Been looking Google, Balabit for an hour. man pages have nothing. On Tue, Oct 17, 2017 at 11:42 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
If you work with syslog-ng-ctl you can give "jo" ( JSON output: https://github.com/jpmens/jo ) a try. I only did some basic tests, but it seems to me that it can turn the output of "syslog-ng-ctl query" into JSON.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Difficult, the whole problem is naming of the name value pairs.
The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything.
Just set stats-level() to 3, and look at the stats message.
I am not saying its impossible, just that it requires some thought.
On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
How about an output modifier ?
On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse.
So I would vote for the "poll syslog-ng-ctl and generate messages" solution.
BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :)
-- Bazsi
On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
Doesn't stats_freq() set an interval to log stats to syslog already?
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages.
So internal_src -> format > elasticsearch -> syslog-ng_stats index ?
On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana.
On 10/15/2017 05:57 PM, Scot wrote:
> Hi, > > Looked around for a few hours and didn't see anything. > > Has anyone worked on sending syslog-ng stats to ES ? > I see several ways I could but wondering if anyone has already. A > push method directly from syslog-ng would be awesome. > > Scot > > ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
CentOS 7 I added --control /var/run/syslog-ng.ctl to * /usr/lib/systemd/system/syslog-ng.service * ExecStart=/usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid syslog-ng-ctl query get /var/run/syslog-ng.ctl Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl', error='Connection refused' On Tue, Oct 17, 2017 at 2:44 PM, Scot <scotrn@gmail.com> wrote:
Where are the query options documented ? Been looking Google, Balabit for an hour. man pages have nothing.
On Tue, Oct 17, 2017 at 11:42 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
If you work with syslog-ng-ctl you can give "jo" ( JSON output: https://github.com/jpmens/jo ) a try. I only did some basic tests, but it seems to me that it can turn the output of "syslog-ng-ctl query" into JSON.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Difficult, the whole problem is naming of the name value pairs.
The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything.
Just set stats-level() to 3, and look at the stats message.
I am not saying its impossible, just that it requires some thought.
On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
How about an output modifier ?
On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse.
So I would vote for the "poll syslog-ng-ctl and generate messages" solution.
BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :)
-- Bazsi
On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
Doesn't stats_freq() set an interval to log stats to syslog already?
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages.
So internal_src -> format > elasticsearch -> syslog-ng_stats index ?
On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
> I have a perl script that collects some stats and logs them to > syslog again. The syslog stream gets sent to ES, so they end up there, but > as a syslog line, not a specific statistic item for things like grafana. > > > On 10/15/2017 05:57 PM, Scot wrote: > >> Hi, >> >> Looked around for a few hours and didn't see anything. >> >> Has anyone worked on sending syslog-ng stats to ES ? >> I see several ways I could but wondering if anyone has already. A >> push method directly from syslog-ng would be awesome. >> >> Scot >> >> > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support > /documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
That socket should be opened, even without the control option. So if you get that error, that means its somewhere else, or we didnt open it for some reason Can you run lsof on the syslog-ng process? That should display the control socket. Also, syslog-ng-ctl reload would use the very same socket. Does that work? On Oct 17, 2017 21:01, "Scot" <scotrn@gmail.com> wrote:
CentOS 7
I added --control /var/run/syslog-ng.ctl to
* /usr/lib/systemd/system/syslog-ng.service * ExecStart=/usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid
syslog-ng-ctl query get /var/run/syslog-ng.ctl Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl', error='Connection refused'
On Tue, Oct 17, 2017 at 2:44 PM, Scot <scotrn@gmail.com> wrote:
Where are the query options documented ? Been looking Google, Balabit for an hour. man pages have nothing.
On Tue, Oct 17, 2017 at 11:42 AM, Czanik, Péter <peter.czanik@balabit.com
wrote:
Hi,
If you work with syslog-ng-ctl you can give "jo" ( JSON output: https://github.com/jpmens/jo ) a try. I only did some basic tests, but it seems to me that it can turn the output of "syslog-ng-ctl query" into JSON.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Difficult, the whole problem is naming of the name value pairs.
The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything.
Just set stats-level() to 3, and look at the stats message.
I am not saying its impossible, just that it requires some thought.
On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
How about an output modifier ?
On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse.
So I would vote for the "poll syslog-ng-ctl and generate messages" solution.
BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :)
-- Bazsi
On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
> Doesn't stats_freq() set an interval to log stats to syslog already? > > Description: The period between two STATS messages in seconds. STATS are > log messages sent by syslog-ng, containing statistics about dropped > log messages. Set to 0to disable the STATS messages. > > So > internal_src -> format > elasticsearch -> syslog-ng_stats index ? > > On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> > wrote: > >> I have a perl script that collects some stats and logs them to >> syslog again. The syslog stream gets sent to ES, so they end up there, but >> as a syslog line, not a specific statistic item for things like grafana. >> >> >> On 10/15/2017 05:57 PM, Scot wrote: >> >>> Hi, >>> >>> Looked around for a few hours and didn't see anything. >>> >>> Has anyone worked on sending syslog-ng stats to ES ? >>> I see several ways I could but wondering if anyone has already. A >>> push method directly from syslog-ng would be awesome. >>> >>> Scot >>> >>> >> ____________________________________________________________ >> __________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support >> /documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support > /documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
syslog-ng-ctl fails with * /usr/lib/systemd/system/syslog-ng.service * ExecStart=/usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid Looks like syslog-ng-ctl does NOT work when added as a CLI option. Probably need to be added to an env file /etc/sysconfig. *# ps -ef | grep syslog* root 14293 1 24 12:37 ? 00:00:01 /usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F -p /var/run/syslogd.pid root 14316 14120 0 12:37 pts/0 00:00:00 grep --color=auto syslog *# lsof -p 14293 | grep ctl* syslog-ng 14293 root 5u unix 0xffff880310fbd800 0t0 41294162 /var/run/syslog-ng.ctl *# lsof /var/run/syslog-ng.ctl* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslog-ng 14293 root 5u unix 0xffff880310fbd800 0t0 41294162 /var/run/syslog-ng.ctl *# syslog-ng-ctl stats* Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl', error='Connection refused' *# ls -al /var/lib/syslog-ng/syslog-ng.ctl* srwxr-xr-x. 1 root root 0 Oct 17 17:19 /var/lib/syslog-ng/syslog-ng.ctl *# ls -al /var/run/syslog-ng.ctl* srwxr-xr-x. 1 root root 0 Oct 18 12:37 /var/run/syslog-ng.ctl I shutdown syslog-ng removed the file and the ctl file from the *syslog-ng.service * Looks like the default is /var/lib/syslog-ng/syslog-ng.ctl everything runs fine with that. *# lsof -p 27812 |grep ctl* syslog-ng 27812 root 5u unix 0xffff88033269a400 0t0 41025397 /var/lib/syslog-ng/syslog-ng.ctl One thing I did notice is query "*" adds .*written *metric for each destination which is missing from syslog-ng-ctl stats (maybe expected I didn't cross ref). On Wed, Oct 18, 2017 at 2:16 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
That socket should be opened, even without the control option. So if you get that error, that means its somewhere else, or we didnt open it for some reason Can you run lsof on the syslog-ng process? That should display the control socket.
Also, syslog-ng-ctl reload would use the very same socket. Does that work?
On Oct 17, 2017 21:01, "Scot" <scotrn@gmail.com> wrote:
CentOS 7
I added --control /var/run/syslog-ng.ctl to
* /usr/lib/systemd/system/syslog-ng.service * ExecStart=/usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid
syslog-ng-ctl query get /var/run/syslog-ng.ctl Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl', error='Connection refused'
On Tue, Oct 17, 2017 at 2:44 PM, Scot <scotrn@gmail.com> wrote:
Where are the query options documented ? Been looking Google, Balabit for an hour. man pages have nothing.
On Tue, Oct 17, 2017 at 11:42 AM, Czanik, Péter < peter.czanik@balabit.com> wrote:
Hi,
If you work with syslog-ng-ctl you can give "jo" ( JSON output: https://github.com/jpmens/jo ) a try. I only did some basic tests, but it seems to me that it can turn the output of "syslog-ng-ctl query" into JSON.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Difficult, the whole problem is naming of the name value pairs.
The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything.
Just set stats-level() to 3, and look at the stats message.
I am not saying its impossible, just that it requires some thought.
On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
How about an output modifier ?
On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
> Hi, > > the issue with the internal stats() message is that if you have a > lot of counters that message is truncated. Also, it is pretty difficult to > parse. > > So I would vote for the "poll syslog-ng-ctl and generate messages" > solution. > > BTW: the internal PE team did something in this area, they created > some sort of internal source that does this polling, but I am not sure how > that works. Possibly there's documentation :) > > > -- > Bazsi > > On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote: > >> Doesn't stats_freq() set an interval to log stats to syslog already? >> >> Description: The period between two STATS messages in seconds. >> STATS are log messages sent by syslog-ng, containing statistics >> about dropped log messages. Set to 0to disable the STATS messages. >> >> So >> internal_src -> format > elasticsearch -> syslog-ng_stats index ? >> >> On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> >> wrote: >> >>> I have a perl script that collects some stats and logs them to >>> syslog again. The syslog stream gets sent to ES, so they end up there, but >>> as a syslog line, not a specific statistic item for things like grafana. >>> >>> >>> On 10/15/2017 05:57 PM, Scot wrote: >>> >>>> Hi, >>>> >>>> Looked around for a few hours and didn't see anything. >>>> >>>> Has anyone worked on sending syslog-ng stats to ES ? >>>> I see several ways I could but wondering if anyone has already. A >>>> push method directly from syslog-ng would be awesome. >>>> >>>> Scot >>>> >>>> >>> ____________________________________________________________ >>> __________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: http://www.balabit.com/support >>> /documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >> >> ____________________________________________________________ >> __________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support >> /documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> > > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support > /documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > >
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Looks that syslog-ng-ctl defaults to the default path (e.g. /var/lib/syslog-ng/syslog-ng.ctl), but IIRC there is an option to specify that to the syslog-ng-ctl client as well. The new query interface for syslog-ng-ctl has changed a few things in counters, whereas syslog-ng-ctl stats has been kept for backwards compatibility. -- Bazsi On Wed, Oct 18, 2017 at 6:52 PM, Scot <scotrn@gmail.com> wrote:
syslog-ng-ctl fails with * /usr/lib/systemd/system/syslog-ng.service * ExecStart=/usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid
Looks like syslog-ng-ctl does NOT work when added as a CLI option. Probably need to be added to an env file /etc/sysconfig.
*# ps -ef | grep syslog* root 14293 1 24 12:37 ? 00:00:01 /usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F -p /var/run/syslogd.pid root 14316 14120 0 12:37 pts/0 00:00:00 grep --color=auto syslog
*# lsof -p 14293 | grep ctl* syslog-ng 14293 root 5u unix 0xffff880310fbd800 0t0 41294162 /var/run/syslog-ng.ctl *# lsof /var/run/syslog-ng.ctl* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslog-ng 14293 root 5u unix 0xffff880310fbd800 0t0 41294162 /var/run/syslog-ng.ctl *# syslog-ng-ctl stats* Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl', error='Connection refused' *# ls -al /var/lib/syslog-ng/syslog-ng.ctl* srwxr-xr-x. 1 root root 0 Oct 17 17:19 /var/lib/syslog-ng/syslog-ng.ctl *# ls -al /var/run/syslog-ng.ctl* srwxr-xr-x. 1 root root 0 Oct 18 12:37 /var/run/syslog-ng.ctl
I shutdown syslog-ng removed the file and the ctl file from the *syslog-ng.service * Looks like the default is /var/lib/syslog-ng/syslog-ng.ctl everything runs fine with that.
*# lsof -p 27812 |grep ctl* syslog-ng 27812 root 5u unix 0xffff88033269a400 0t0 41025397 /var/lib/syslog-ng/syslog-ng.ctl
One thing I did notice is query "*" adds .*written *metric for each destination which is missing from syslog-ng-ctl stats (maybe expected I didn't cross ref).
On Wed, Oct 18, 2017 at 2:16 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
That socket should be opened, even without the control option. So if you get that error, that means its somewhere else, or we didnt open it for some reason Can you run lsof on the syslog-ng process? That should display the control socket.
Also, syslog-ng-ctl reload would use the very same socket. Does that work?
On Oct 17, 2017 21:01, "Scot" <scotrn@gmail.com> wrote:
CentOS 7
I added --control /var/run/syslog-ng.ctl to
* /usr/lib/systemd/system/syslog-ng.service * ExecStart=/usr/sbin/syslog-ng --control /var/run/syslog-ng.ctl -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid
syslog-ng-ctl query get /var/run/syslog-ng.ctl Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl', error='Connection refused'
On Tue, Oct 17, 2017 at 2:44 PM, Scot <scotrn@gmail.com> wrote:
Where are the query options documented ? Been looking Google, Balabit for an hour. man pages have nothing.
On Tue, Oct 17, 2017 at 11:42 AM, Czanik, Péter < peter.czanik@balabit.com> wrote:
Hi,
If you work with syslog-ng-ctl you can give "jo" ( JSON output: https://github.com/jpmens/jo ) a try. I only did some basic tests, but it seems to me that it can turn the output of "syslog-ng-ctl query" into JSON.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Difficult, the whole problem is naming of the name value pairs.
The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything.
Just set stats-level() to 3, and look at the stats message.
I am not saying its impossible, just that it requires some thought.
On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
> How about an output modifier ? > > On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < > balazs.scheidler@balabit.com> wrote: > >> Hi, >> >> the issue with the internal stats() message is that if you have a >> lot of counters that message is truncated. Also, it is pretty difficult to >> parse. >> >> So I would vote for the "poll syslog-ng-ctl and generate messages" >> solution. >> >> BTW: the internal PE team did something in this area, they created >> some sort of internal source that does this polling, but I am not sure how >> that works. Possibly there's documentation :) >> >> >> -- >> Bazsi >> >> On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote: >> >>> Doesn't stats_freq() set an interval to log stats to syslog >>> already? >>> >>> Description: The period between two STATS messages in seconds. >>> STATS are log messages sent by syslog-ng, containing statistics >>> about dropped log messages. Set to 0to disable the STATS messages. >>> >>> So >>> internal_src -> format > elasticsearch -> syslog-ng_stats index ? >>> >>> On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> >>> wrote: >>> >>>> I have a perl script that collects some stats and logs them to >>>> syslog again. The syslog stream gets sent to ES, so they end up there, but >>>> as a syslog line, not a specific statistic item for things like grafana. >>>> >>>> >>>> On 10/15/2017 05:57 PM, Scot wrote: >>>> >>>>> Hi, >>>>> >>>>> Looked around for a few hours and didn't see anything. >>>>> >>>>> Has anyone worked on sending syslog-ng stats to ES ? >>>>> I see several ways I could but wondering if anyone has already. >>>>> A push method directly from syslog-ng would be awesome. >>>>> >>>>> Scot >>>>> >>>>> >>>> ____________________________________________________________ >>>> __________________ >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>> Documentation: http://www.balabit.com/support >>>> /documentation/?product=syslog-ng >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>> >>>> >>> >>> ____________________________________________________________ >>> __________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: http://www.balabit.com/support >>> /documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >>> >> >> ____________________________________________________________ >> __________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support >> /documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> >> > > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support > /documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Oops, this is also in OSE but found only in the PE documentation. I put Robert from the documentation team on CC to fix it. Right now you can read the PE documentation: https://www.balabit.com/documents/syslog-ng-pe-7.0-guides/en/syslog-ng-pe-gu... Basically: "syslog-ng-ctl query list" will give you a list of available fields. "syslog-ng-ctl query get '*'" will get all values, "get 'dst.*'" will get you only values where the name starts with "dst." Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Tue, Oct 17, 2017 at 8:44 PM, Scot <scotrn@gmail.com> wrote:
Where are the query options documented ? Been looking Google, Balabit for an hour. man pages have nothing.
On Tue, Oct 17, 2017 at 11:42 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
If you work with syslog-ng-ctl you can give "jo" ( JSON output: https://github.com/jpmens/jo ) a try. I only did some basic tests, but it seems to me that it can turn the output of "syslog-ng-ctl query" into JSON.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Difficult, the whole problem is naming of the name value pairs.
The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything.
Just set stats-level() to 3, and look at the stats message.
I am not saying its impossible, just that it requires some thought.
On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
How about an output modifier ?
On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse.
So I would vote for the "poll syslog-ng-ctl and generate messages" solution.
BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :)
-- Bazsi
On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
Doesn't stats_freq() set an interval to log stats to syslog already?
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages.
So internal_src -> format > elasticsearch -> syslog-ng_stats index ?
On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
> I have a perl script that collects some stats and logs them to > syslog again. The syslog stream gets sent to ES, so they end up there, but > as a syslog line, not a specific statistic item for things like grafana. > > > On 10/15/2017 05:57 PM, Scot wrote: > >> Hi, >> >> Looked around for a few hours and didn't see anything. >> >> Has anyone worked on sending syslog-ng stats to ES ? >> I see several ways I could but wondering if anyone has already. A >> push method directly from syslog-ng would be awesome. >> >> Scot >> >> > ____________________________________________________________ > __________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support > /documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I worked a bit more on this: https://www.balabit.com/blog/collecting-syslog-ng-statistics-to-graphite/ Probably not the nicest solution, but it works :) Sending to Elasticsearch is pretty similar. I chose the Graphite way, as for Elasticsearch / Kibana one also needs to configure mapping, so values are stored as numbers instead of strings. I still need to get more familiar with mapping... (on my ToDo list). Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Tue, Oct 17, 2017 at 5:42 PM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
If you work with syslog-ng-ctl you can give "jo" ( JSON output: https://github.com/jpmens/jo ) a try. I only did some basic tests, but it seems to me that it can turn the output of "syslog-ng-ctl query" into JSON.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Oct 17, 2017 at 5:20 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Difficult, the whole problem is naming of the name value pairs.
The idea behind stats is to generate all name value pairs in one message, and this simply does not scale. You are almost certainly interested in a set of values or an aggregate of a set, and not everything.
Just set stats-level() to 3, and look at the stats message.
I am not saying its impossible, just that it requires some thought.
On Oct 17, 2017 17:09, "Scot" <scotrn@gmail.com> wrote:
How about an output modifier ?
On Tue, Oct 17, 2017 at 11:02 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Hi,
the issue with the internal stats() message is that if you have a lot of counters that message is truncated. Also, it is pretty difficult to parse.
So I would vote for the "poll syslog-ng-ctl and generate messages" solution.
BTW: the internal PE team did something in this area, they created some sort of internal source that does this polling, but I am not sure how that works. Possibly there's documentation :)
-- Bazsi
On Tue, Oct 17, 2017 at 4:37 PM, Scot <scotrn@gmail.com> wrote:
Doesn't stats_freq() set an interval to log stats to syslog already?
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0to disable the STATS messages.
So internal_src -> format > elasticsearch -> syslog-ng_stats index ?
On Mon, Oct 16, 2017 at 11:01 AM, Evan Rempel <erempel@uvic.ca> wrote:
I have a perl script that collects some stats and logs them to syslog again. The syslog stream gets sent to ES, so they end up there, but as a syslog line, not a specific statistic item for things like grafana.
On 10/15/2017 05:57 PM, Scot wrote:
> Hi, > > Looked around for a few hours and didn't see anything. > > Has anyone worked on sending syslog-ng stats to ES ? > I see several ways I could but wondering if anyone has already. A > push method directly from syslog-ng would be awesome. > > Scot > > ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Czanik, Péter
-
Evan Rempel
-
Scheidler, Balázs
-
Scot