[syslog-ng]Syslog-ng replay script for centralized syslog data

[Bill Nash]

On Thu, 28 Oct 2004, Dave Johnson wrote:

> bill--
>
> I'm only offering suggestions, and working on a limited subset of
> knowledge about your architecture.  Your mileage may vary...

Hehe, not *my* architecture. I was responding to the initial question. I'm 
just playing devil's advocate. I currently run 140 hosts with three 
analyzers, using syslog-ng, and pondering your solution as applied to my 
meager deployment.. well. I wouldn't want to maintain it. =)

- billn

>
>
> On Thu, 28 Oct 2004 13:52:55 -0700 (PDT), Bill Nash <billn@billn.net> wrote:
>>
>> That's a dramatic increase in complexity, however. The strongest lifeform
>> is often the simplest. Troubleshooting, or even implementing such a setup,
>> may not be feasible and would likely require kernel recompiles to even
>> enable the features, depending on the existing implementation. Scale is
>> another factor that would make this option less attractive. A QoS option
>> would be better implemented at the network level across all tcp/514
>> traffic.
>>
>> - billn
>>
>>
>>
>> On Thu, 28 Oct 2004, Dave Johnson wrote:
>>
>>> Just another thought, (which isn't as easy as the other suggestion) --
>>>
>>> * Set up ratelimiting on your remote servers to the central server's
>>> IP and just syslog-ng with tcp to the central server.
>>>  - Make sure you have a decent sized queue on the remote server so
>>> you can queue up packets
>>>  - setting up ratelimiting on linux and getting the results just
>>> right might take some time.
>>>
>>> (you can google search for /etc/init.d/cbq scripts)  and make sure you
>>> have class base queueing enabled in your kernel.
>>>
>>> ---
>>>
>>> * You can create another ip on your central server if your going to be
>>> doing admin tasks from that box.  (IE you don't want your ssh to be in
>>> the same ratelimiting rule as the syslog traffic).
>>>
>>> * If compression is important (due to the small link size), you could
>>> leverage ssh to do this.
>>>
>>> This approach is a little more complicated, but your logs would show up sooner.
>>>
>>> Depending on important this data is, you may want the backup ftp/rsync
>>> method anyways...
>>>
>>>
>>> On Thu, 28 Oct 2004 15:02:33 -0500, Dave Johnson <davejjohnson@gmail.com> wrote:
>>>> You can do it many ways, one way (quick and easy):
>>>>
>>>> remote nodes <every ten minutes cron>
>>>> log, bzip2 in directory "A"
>>>> run rsyncd for directory "A"
>>>> ---
>>>> central node <every ten minutes +1 minute> <or just do it every 2 mins, etc..>
>>>> run script:
>>>> 1) rsync --bwlimit 9k -u get from remote node's "A"
>>>> 2) bunzip2 files
>>>> 3) cat file into /dev/log (or local platform's way of injecting into syslog)
>>>> ---------
>>>> http://samba.anu.edu.au/rsync/
>>>>
>>>>
>>>>
>>>> On Thu, 28 Oct 2004 12:03:53 -0700 (PDT), LEROY ISAAC
>>>> <lisaac01@yahoo.com> wrote:
>>>>>
>>>>>
>>>>> I have a need to retrieve syslog data from various
>>>>> remote nodes, and the smallest network link to the
>>>>> remote nodes is 19K. The syslog traffic for the link
>>>>> cannot exceed 9K.
>>>>>
>>>>> I plan to setup a configuration which generates new
>>>>> log files every 10 minutes. These files are then
>>>>> compressed, zipped, and transfered to a centralized
>>>>> loghost.
>>>>>
>>>>> The files are then unzipped, uncompressed, and the
>>>>> data is inserted into the syslog-ng data stream on the
>>>>> central syslog-ng host.
>>>>>
>>>>> Is there a script or utility which will accomplish
>>>>> this task? If not, then does any one have any
>>>>> suggestions on products which may accomplish this same
>>>>> task.
>>>>>
>>>>> LeRoy Isaac
>>>>> --- DTrinh71@aol.com wrote:
>>>>>
>>>>>> OK. Thanks.
>>>>>>
>>>>>> So, what does Ray want? Suggestions?
>>>>>>
>>>>>> David
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> syslog-ng maillist  -  syslog-ng@lists.balabit.hu
>>>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> syslog-ng maillist  -  syslog-ng@lists.balabit.hu
>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>>>
>> _______________________________________________
>>
>>
>> syslog-ng maillist  -  syslog-ng@lists.balabit.hu
>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>>
>>
> _______________________________________________
> syslog-ng maillist  -  syslog-ng@lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>