[syslog-ng] How to get list of IPs

[Scot]

Auto split your logs using the $HOST macro ?
Watch and count as time progresses since some systems may only log
something daily.


On Fri, May 12, 2017 at 2:06 PM, Evan Rempel <erempel at uvic.ca> wrote:

> Since the syslog messages might come from UDP sources, you can not look at
> connections. You have to look at all of the host names in all of the log
> lines.
>
> The way we address this is to have a program destination that we send just
> the hostname to for EVERY log line. That program keeps a running hash and
> every 5 minutes logs its own line
>
> 2017-05-12T00:00:23.225-07:00 local at syslogserver.comp.uvic.ca local0.info
> flare-heartbeat[14903]: got syslog messages from syslogclient.comp.uvic.ca
>
> Your program could update a database or any other repository on a 5 minute
> interval. If that load is too much, just switch to a 10 minute interval.
>
> We feed these "got syslog messages" lines into another program that
> manages a heartbeat timer for every host, and if the "got syslog messages"
> line is missing
> for a host, we alert ourselves to the fact that syslog messages have
> stopped arriving for a host.
>
> We wrote our program in perl, which can update hashes very quickly. Our
> test shows we can handle 1 million events per second and track them all via
> this approach.
> We are only tracking 1000+ hosts via this mechanism.
>
>
> I hope that is helpful.
>
> Evan.
>
>
>
> On 05/12/2017 10:54 AM, Nik Ambrosch wrote:
>
>> someone correct me if i’m wrong, but i don’t think there’s a good way to
>> get this information directly from the syslog-ng daemon, especially if it’s
>> restarted.  a more reliable way is to pull the data off disk.
>>
>> depending on your log volume putting them into a database will allow you
>> to easily do this with a single query.  i write about 20 million log lines
>> into a mysql database every day so this should cover most use cases.  a
>> query would look like one of these two (depends if you want a line count or
>> not):
>>
>>    mysql> select distinct host from syslog_table_20170511;
>>    mysql> select distinct host,count(*) as count from
>> syslog_table_20170511 group by host order by count desc;
>>
>> pulling from log files on disk is obviously takes longer to run but can
>> be done with a one-liner like this:
>>
>>    $ cat file.log | awk '{print $1}' | sort | uniq -c | sort -nr
>>
>> another option is to write one log file per host and just run an ls to
>> see which hosts have sent logs for the day, for example:
>>
>> /loghost/YYYYMMDD/hostname.prv.log
>>
>> the dynamic path can be setup using syslog-ng.
>>
>>
>> On May 12, 2017, at 1:11 PM, Sathish Sundaravel <
>>> sathish.sundaravel at gmail.com> wrote:
>>>
>>> Hi All
>>>
>>> I have syslog Ng server running on Solaris platform and configured to
>>> receive or accept events from all unix servers in one single log file.
>>>
>>> If I want to collect list of servers reporting to the log server , I
>>> have to manually extract list from a single log file which is too much time
>>> consuming . What would the best way ?
>>>
>>> Regards
>>> Sathish
>>> ____________________________________________________________
>>> __________________
>>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=
> syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170512/ed466a55/attachment.html>