How to get list of IPs
Hi All I have syslog Ng server running on Solaris platform and configured to receive or accept events from all unix servers in one single log file. If I want to collect list of servers reporting to the log server , I have to manually extract list from a single log file which is too much time consuming . What would the best way ? Regards Sathish
someone correct me if i’m wrong, but i don’t think there’s a good way to get this information directly from the syslog-ng daemon, especially if it’s restarted. a more reliable way is to pull the data off disk. depending on your log volume putting them into a database will allow you to easily do this with a single query. i write about 20 million log lines into a mysql database every day so this should cover most use cases. a query would look like one of these two (depends if you want a line count or not): mysql> select distinct host from syslog_table_20170511; mysql> select distinct host,count(*) as count from syslog_table_20170511 group by host order by count desc; pulling from log files on disk is obviously takes longer to run but can be done with a one-liner like this: $ cat file.log | awk '{print $1}' | sort | uniq -c | sort -nr another option is to write one log file per host and just run an ls to see which hosts have sent logs for the day, for example: /loghost/YYYYMMDD/hostname.prv.log the dynamic path can be setup using syslog-ng.
On May 12, 2017, at 1:11 PM, Sathish Sundaravel <sathish.sundaravel@gmail.com> wrote:
Hi All
I have syslog Ng server running on Solaris platform and configured to receive or accept events from all unix servers in one single log file.
If I want to collect list of servers reporting to the log server , I have to manually extract list from a single log file which is too much time consuming . What would the best way ?
Regards Sathish ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Since the syslog messages might come from UDP sources, you can not look at connections. You have to look at all of the host names in all of the log lines. The way we address this is to have a program destination that we send just the hostname to for EVERY log line. That program keeps a running hash and every 5 minutes logs its own line 2017-05-12T00:00:23.225-07:00 local@syslogserver.comp.uvic.ca local0.info flare-heartbeat[14903]: got syslog messages from syslogclient.comp.uvic.ca Your program could update a database or any other repository on a 5 minute interval. If that load is too much, just switch to a 10 minute interval. We feed these "got syslog messages" lines into another program that manages a heartbeat timer for every host, and if the "got syslog messages" line is missing for a host, we alert ourselves to the fact that syslog messages have stopped arriving for a host. We wrote our program in perl, which can update hashes very quickly. Our test shows we can handle 1 million events per second and track them all via this approach. We are only tracking 1000+ hosts via this mechanism. I hope that is helpful. Evan. On 05/12/2017 10:54 AM, Nik Ambrosch wrote:
someone correct me if i’m wrong, but i don’t think there’s a good way to get this information directly from the syslog-ng daemon, especially if it’s restarted. a more reliable way is to pull the data off disk.
depending on your log volume putting them into a database will allow you to easily do this with a single query. i write about 20 million log lines into a mysql database every day so this should cover most use cases. a query would look like one of these two (depends if you want a line count or not):
mysql> select distinct host from syslog_table_20170511; mysql> select distinct host,count(*) as count from syslog_table_20170511 group by host order by count desc;
pulling from log files on disk is obviously takes longer to run but can be done with a one-liner like this:
$ cat file.log | awk '{print $1}' | sort | uniq -c | sort -nr
another option is to write one log file per host and just run an ls to see which hosts have sent logs for the day, for example:
/loghost/YYYYMMDD/hostname.prv.log
the dynamic path can be setup using syslog-ng.
On May 12, 2017, at 1:11 PM, Sathish Sundaravel <sathish.sundaravel@gmail.com> wrote:
Hi All
I have syslog Ng server running on Solaris platform and configured to receive or accept events from all unix servers in one single log file.
If I want to collect list of servers reporting to the log server , I have to manually extract list from a single log file which is too much time consuming . What would the best way ?
Regards Sathish ______________________________________________________________________________
Auto split your logs using the $HOST macro ? Watch and count as time progresses since some systems may only log something daily. On Fri, May 12, 2017 at 2:06 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since the syslog messages might come from UDP sources, you can not look at connections. You have to look at all of the host names in all of the log lines.
The way we address this is to have a program destination that we send just the hostname to for EVERY log line. That program keeps a running hash and every 5 minutes logs its own line
2017-05-12T00:00:23.225-07:00 local@syslogserver.comp.uvic.ca local0.info flare-heartbeat[14903]: got syslog messages from syslogclient.comp.uvic.ca
Your program could update a database or any other repository on a 5 minute interval. If that load is too much, just switch to a 10 minute interval.
We feed these "got syslog messages" lines into another program that manages a heartbeat timer for every host, and if the "got syslog messages" line is missing for a host, we alert ourselves to the fact that syslog messages have stopped arriving for a host.
We wrote our program in perl, which can update hashes very quickly. Our test shows we can handle 1 million events per second and track them all via this approach. We are only tracking 1000+ hosts via this mechanism.
I hope that is helpful.
Evan.
On 05/12/2017 10:54 AM, Nik Ambrosch wrote:
someone correct me if i’m wrong, but i don’t think there’s a good way to get this information directly from the syslog-ng daemon, especially if it’s restarted. a more reliable way is to pull the data off disk.
depending on your log volume putting them into a database will allow you to easily do this with a single query. i write about 20 million log lines into a mysql database every day so this should cover most use cases. a query would look like one of these two (depends if you want a line count or not):
mysql> select distinct host from syslog_table_20170511; mysql> select distinct host,count(*) as count from syslog_table_20170511 group by host order by count desc;
pulling from log files on disk is obviously takes longer to run but can be done with a one-liner like this:
$ cat file.log | awk '{print $1}' | sort | uniq -c | sort -nr
another option is to write one log file per host and just run an ls to see which hosts have sent logs for the day, for example:
/loghost/YYYYMMDD/hostname.prv.log
the dynamic path can be setup using syslog-ng.
On May 12, 2017, at 1:11 PM, Sathish Sundaravel <
sathish.sundaravel@gmail.com> wrote:
Hi All
I have syslog Ng server running on Solaris platform and configured to receive or accept events from all unix servers in one single log file.
If I want to collect list of servers reporting to the log server , I have to manually extract list from a single log file which is too much time consuming . What would the best way ?
Regards Sathish ____________________________________________________________ __________________
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Nik, Thank you . Currently I don't have such database option enabled. I have gathered host information from log file at this moment. Enabling log using database and host name format sounds good idea. Any other thoughts ? Regards Sathish On Sat, 13 May 2017 at 1:55 AM, Nik Ambrosch <nik@ambrosch.com> wrote:
someone correct me if i’m wrong, but i don’t think there’s a good way to get this information directly from the syslog-ng daemon, especially if it’s restarted. a more reliable way is to pull the data off disk.
depending on your log volume putting them into a database will allow you to easily do this with a single query. i write about 20 million log lines into a mysql database every day so this should cover most use cases. a query would look like one of these two (depends if you want a line count or not):
mysql> select distinct host from syslog_table_20170511; mysql> select distinct host,count(*) as count from syslog_table_20170511 group by host order by count desc;
pulling from log files on disk is obviously takes longer to run but can be done with a one-liner like this:
$ cat file.log | awk '{print $1}' | sort | uniq -c | sort -nr
another option is to write one log file per host and just run an ls to see which hosts have sent logs for the day, for example:
/loghost/YYYYMMDD/hostname.prv.log
the dynamic path can be setup using syslog-ng.
On May 12, 2017, at 1:11 PM, Sathish Sundaravel < sathish.sundaravel@gmail.com> wrote:
Hi All
I have syslog Ng server running on Solaris platform and configured to receive or accept events from all unix servers in one single log file.
If I want to collect list of servers reporting to the log server , I have to manually extract list from a single log file which is too much time consuming . What would the best way ?
Regards Sathish
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Syslog-ng has a stats mechanism. If you sez stats-level(3), syslog-ng starts tracking host and program counters, and a timestamp for the last message received. These are lost when syslog-ng is restarted, but you can collect this every few minutes. You can query these counters using syslog-ng-ctl stats. The stats subsystem is being worked on to extend it, there were numerous pull requests recently. Cheers Bazsi On May 13, 2017 4:13 AM, "Sathish Sundaravel" <sathish.sundaravel@gmail.com> wrote:
Hi Nik,
Thank you . Currently I don't have such database option enabled. I have gathered host information from log file at this moment. Enabling log using database and host name format sounds good idea.
Any other thoughts ?
Regards Sathish
On Sat, 13 May 2017 at 1:55 AM, Nik Ambrosch <nik@ambrosch.com> wrote:
someone correct me if i’m wrong, but i don’t think there’s a good way to get this information directly from the syslog-ng daemon, especially if it’s restarted. a more reliable way is to pull the data off disk.
depending on your log volume putting them into a database will allow you to easily do this with a single query. i write about 20 million log lines into a mysql database every day so this should cover most use cases. a query would look like one of these two (depends if you want a line count or not):
mysql> select distinct host from syslog_table_20170511; mysql> select distinct host,count(*) as count from syslog_table_20170511 group by host order by count desc;
pulling from log files on disk is obviously takes longer to run but can be done with a one-liner like this:
$ cat file.log | awk '{print $1}' | sort | uniq -c | sort -nr
another option is to write one log file per host and just run an ls to see which hosts have sent logs for the day, for example:
/loghost/YYYYMMDD/hostname.prv.log
the dynamic path can be setup using syslog-ng.
On May 12, 2017, at 1:11 PM, Sathish Sundaravel < sathish.sundaravel@gmail.com> wrote:
Hi All
I have syslog Ng server running on Solaris platform and configured to receive or accept events from all unix servers in one single log file.
If I want to collect list of servers reporting to the log server , I have to manually extract list from a single log file which is too much time consuming . What would the best way ?
Regards Sathish ____________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thank you so much Balazs and Nik . On Sat, 13 May 2017 at 2:05 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
Syslog-ng has a stats mechanism. If you sez stats-level(3), syslog-ng starts tracking host and program counters, and a timestamp for the last message received.
These are lost when syslog-ng is restarted, but you can collect this every few minutes.
You can query these counters using syslog-ng-ctl stats.
The stats subsystem is being worked on to extend it, there were numerous pull requests recently.
Cheers Bazsi
On May 13, 2017 4:13 AM, "Sathish Sundaravel" < sathish.sundaravel@gmail.com> wrote:
Hi Nik,
Thank you . Currently I don't have such database option enabled. I have gathered host information from log file at this moment. Enabling log using database and host name format sounds good idea.
Any other thoughts ?
Regards Sathish
On Sat, 13 May 2017 at 1:55 AM, Nik Ambrosch <nik@ambrosch.com> wrote:
someone correct me if i’m wrong, but i don’t think there’s a good way to get this information directly from the syslog-ng daemon, especially if it’s restarted. a more reliable way is to pull the data off disk.
depending on your log volume putting them into a database will allow you to easily do this with a single query. i write about 20 million log lines into a mysql database every day so this should cover most use cases. a query would look like one of these two (depends if you want a line count or not):
mysql> select distinct host from syslog_table_20170511; mysql> select distinct host,count(*) as count from syslog_table_20170511 group by host order by count desc;
pulling from log files on disk is obviously takes longer to run but can be done with a one-liner like this:
$ cat file.log | awk '{print $1}' | sort | uniq -c | sort -nr
another option is to write one log file per host and just run an ls to see which hosts have sent logs for the day, for example:
/loghost/YYYYMMDD/hostname.prv.log
the dynamic path can be setup using syslog-ng.
On May 12, 2017, at 1:11 PM, Sathish Sundaravel < sathish.sundaravel@gmail.com> wrote:
Hi All
I have syslog Ng server running on Solaris platform and configured to receive or accept events from all unix servers in one single log file.
If I want to collect list of servers reporting to the log server , I have to manually extract list from a single log file which is too much time consuming . What would the best way ?
Regards Sathish
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Balazs Scheidler
-
Evan Rempel
-
Nik Ambrosch
-
Sathish Sundaravel
-
Scot