[syslog-ng] How to get list of IPs

[Evan Rempel]

Since the syslog messages might come from UDP sources, you can not look at connections. You have to look at all of the host names in all of the log lines.

The way we address this is to have a program destination that we send just the hostname to for EVERY log line. That program keeps a running hash and every 5 minutes logs its own line

2017-05-12T00:00:23.225-07:00 local at syslogserver.comp.uvic.ca local0.info flare-heartbeat[14903]: got syslog messages from syslogclient.comp.uvic.ca

Your program could update a database or any other repository on a 5 minute interval. If that load is too much, just switch to a 10 minute interval.

We feed these "got syslog messages" lines into another program that manages a heartbeat timer for every host, and if the "got syslog messages" line is missing
for a host, we alert ourselves to the fact that syslog messages have stopped arriving for a host.

We wrote our program in perl, which can update hashes very quickly. Our test shows we can handle 1 million events per second and track them all via this approach.
We are only tracking 1000+ hosts via this mechanism.


I hope that is helpful.

Evan.


On 05/12/2017 10:54 AM, Nik Ambrosch wrote:
> someone correct me if i’m wrong, but i don’t think there’s a good way to get this information directly from the syslog-ng daemon, especially if it’s restarted.  a more reliable way is to pull the data off disk.
>
> depending on your log volume putting them into a database will allow you to easily do this with a single query.  i write about 20 million log lines into a mysql database every day so this should cover most use cases.  a query would look like one of these two (depends if you want a line count or not):
>
>    mysql> select distinct host from syslog_table_20170511;
>    mysql> select distinct host,count(*) as count from syslog_table_20170511 group by host order by count desc;
>
> pulling from log files on disk is obviously takes longer to run but can be done with a one-liner like this:
>
>    $ cat file.log | awk '{print $1}' | sort | uniq -c | sort -nr
>
> another option is to write one log file per host and just run an ls to see which hosts have sent logs for the day, for example:
>
> /loghost/YYYYMMDD/hostname.prv.log
>
> the dynamic path can be setup using syslog-ng.
>
>
>> On May 12, 2017, at 1:11 PM, Sathish Sundaravel <sathish.sundaravel at gmail.com> wrote:
>>
>> Hi All
>>
>> I have syslog Ng server running on Solaris platform and configured to receive or accept events from all unix servers in one single log file.
>>
>> If I want to collect list of servers reporting to the log server , I have to manually extract list from a single log file which is too much time consuming . What would the best way ?
>>
>> Regards
>> Sathish
>> ______________________________________________________________________________