Syslog-ng replay script for centralized syslog data
I have a need to retrieve syslog data from various remote nodes, and the smallest network link to the remote nodes is 19K. The syslog traffic for the link cannot exceed 9K. I plan to setup a configuration which generates new log files every 10 minutes. These files are then compressed, zipped, and transfered to a centralized loghost. The files are then unzipped, uncompressed, and the data is inserted into the syslog-ng data stream on the central syslog-ng host. Is there a script or utility which will accomplish this task? If not, then does any one have any suggestions on products which may accomplish this same task. LeRoy Isaac --- DTrinh71@aol.com wrote:
OK. Thanks.
So, what does Ray want? Suggestions?
David
Without doing hugely customized hacking, I'd suggest tweaking the logrotate scripts to do your rotation functions for you (adjust cron interval as well). As for combining it with your master feed, that could just be as simple as concatenating your logs by date. Hard to speculate. - billn On Thu, 28 Oct 2004, LEROY ISAAC wrote:
I have a need to retrieve syslog data from various remote nodes, and the smallest network link to the remote nodes is 19K. The syslog traffic for the link cannot exceed 9K.
I plan to setup a configuration which generates new log files every 10 minutes. These files are then compressed, zipped, and transfered to a centralized loghost.
The files are then unzipped, uncompressed, and the data is inserted into the syslog-ng data stream on the central syslog-ng host.
Is there a script or utility which will accomplish this task? If not, then does any one have any suggestions on products which may accomplish this same task.
LeRoy Isaac --- DTrinh71@aol.com wrote:
OK. Thanks.
So, what does Ray want? Suggestions?
David
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
You can do it many ways, one way (quick and easy): remote nodes <every ten minutes cron> log, bzip2 in directory "A" run rsyncd for directory "A" --- central node <every ten minutes +1 minute> <or just do it every 2 mins, etc..> run script: 1) rsync --bwlimit 9k -u get from remote node's "A" 2) bunzip2 files 3) cat file into /dev/log (or local platform's way of injecting into syslog) --------- http://samba.anu.edu.au/rsync/ On Thu, 28 Oct 2004 12:03:53 -0700 (PDT), LEROY ISAAC <lisaac01@yahoo.com> wrote:
I have a need to retrieve syslog data from various remote nodes, and the smallest network link to the remote nodes is 19K. The syslog traffic for the link cannot exceed 9K.
I plan to setup a configuration which generates new log files every 10 minutes. These files are then compressed, zipped, and transfered to a centralized loghost.
The files are then unzipped, uncompressed, and the data is inserted into the syslog-ng data stream on the central syslog-ng host.
Is there a script or utility which will accomplish this task? If not, then does any one have any suggestions on products which may accomplish this same task.
LeRoy Isaac --- DTrinh71@aol.com wrote:
OK. Thanks.
So, what does Ray want? Suggestions?
David
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Just another thought, (which isn't as easy as the other suggestion) -- * Set up ratelimiting on your remote servers to the central server's IP and just syslog-ng with tcp to the central server. - Make sure you have a decent sized queue on the remote server so you can queue up packets - setting up ratelimiting on linux and getting the results just right might take some time. (you can google search for /etc/init.d/cbq scripts) and make sure you have class base queueing enabled in your kernel. --- * You can create another ip on your central server if your going to be doing admin tasks from that box. (IE you don't want your ssh to be in the same ratelimiting rule as the syslog traffic). * If compression is important (due to the small link size), you could leverage ssh to do this. This approach is a little more complicated, but your logs would show up sooner. Depending on important this data is, you may want the backup ftp/rsync method anyways... On Thu, 28 Oct 2004 15:02:33 -0500, Dave Johnson <davejjohnson@gmail.com> wrote:
You can do it many ways, one way (quick and easy):
remote nodes <every ten minutes cron> log, bzip2 in directory "A" run rsyncd for directory "A" --- central node <every ten minutes +1 minute> <or just do it every 2 mins, etc..> run script: 1) rsync --bwlimit 9k -u get from remote node's "A" 2) bunzip2 files 3) cat file into /dev/log (or local platform's way of injecting into syslog) --------- http://samba.anu.edu.au/rsync/
On Thu, 28 Oct 2004 12:03:53 -0700 (PDT), LEROY ISAAC <lisaac01@yahoo.com> wrote:
I have a need to retrieve syslog data from various remote nodes, and the smallest network link to the remote nodes is 19K. The syslog traffic for the link cannot exceed 9K.
I plan to setup a configuration which generates new log files every 10 minutes. These files are then compressed, zipped, and transfered to a centralized loghost.
The files are then unzipped, uncompressed, and the data is inserted into the syslog-ng data stream on the central syslog-ng host.
Is there a script or utility which will accomplish this task? If not, then does any one have any suggestions on products which may accomplish this same task.
LeRoy Isaac --- DTrinh71@aol.com wrote:
OK. Thanks.
So, what does Ray want? Suggestions?
David
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
That's a dramatic increase in complexity, however. The strongest lifeform is often the simplest. Troubleshooting, or even implementing such a setup, may not be feasible and would likely require kernel recompiles to even enable the features, depending on the existing implementation. Scale is another factor that would make this option less attractive. A QoS option would be better implemented at the network level across all tcp/514 traffic. - billn On Thu, 28 Oct 2004, Dave Johnson wrote:
Just another thought, (which isn't as easy as the other suggestion) --
* Set up ratelimiting on your remote servers to the central server's IP and just syslog-ng with tcp to the central server. - Make sure you have a decent sized queue on the remote server so you can queue up packets - setting up ratelimiting on linux and getting the results just right might take some time.
(you can google search for /etc/init.d/cbq scripts) and make sure you have class base queueing enabled in your kernel.
---
* You can create another ip on your central server if your going to be doing admin tasks from that box. (IE you don't want your ssh to be in the same ratelimiting rule as the syslog traffic).
* If compression is important (due to the small link size), you could leverage ssh to do this.
This approach is a little more complicated, but your logs would show up sooner.
Depending on important this data is, you may want the backup ftp/rsync method anyways...
On Thu, 28 Oct 2004 15:02:33 -0500, Dave Johnson <davejjohnson@gmail.com> wrote:
You can do it many ways, one way (quick and easy):
remote nodes <every ten minutes cron> log, bzip2 in directory "A" run rsyncd for directory "A" --- central node <every ten minutes +1 minute> <or just do it every 2 mins, etc..> run script: 1) rsync --bwlimit 9k -u get from remote node's "A" 2) bunzip2 files 3) cat file into /dev/log (or local platform's way of injecting into syslog) --------- http://samba.anu.edu.au/rsync/
On Thu, 28 Oct 2004 12:03:53 -0700 (PDT), LEROY ISAAC <lisaac01@yahoo.com> wrote:
I have a need to retrieve syslog data from various remote nodes, and the smallest network link to the remote nodes is 19K. The syslog traffic for the link cannot exceed 9K.
I plan to setup a configuration which generates new log files every 10 minutes. These files are then compressed, zipped, and transfered to a centralized loghost.
The files are then unzipped, uncompressed, and the data is inserted into the syslog-ng data stream on the central syslog-ng host.
Is there a script or utility which will accomplish this task? If not, then does any one have any suggestions on products which may accomplish this same task.
LeRoy Isaac --- DTrinh71@aol.com wrote:
OK. Thanks.
So, what does Ray want? Suggestions?
David
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
bill-- I'm only offering suggestions, and working on a limited subset of knowledge about your architecture. Your mileage may vary... On Thu, 28 Oct 2004 13:52:55 -0700 (PDT), Bill Nash <billn@billn.net> wrote:
That's a dramatic increase in complexity, however. The strongest lifeform is often the simplest. Troubleshooting, or even implementing such a setup, may not be feasible and would likely require kernel recompiles to even enable the features, depending on the existing implementation. Scale is another factor that would make this option less attractive. A QoS option would be better implemented at the network level across all tcp/514 traffic.
- billn
On Thu, 28 Oct 2004, Dave Johnson wrote:
Just another thought, (which isn't as easy as the other suggestion) --
* Set up ratelimiting on your remote servers to the central server's IP and just syslog-ng with tcp to the central server. - Make sure you have a decent sized queue on the remote server so you can queue up packets - setting up ratelimiting on linux and getting the results just right might take some time.
(you can google search for /etc/init.d/cbq scripts) and make sure you have class base queueing enabled in your kernel.
---
* You can create another ip on your central server if your going to be doing admin tasks from that box. (IE you don't want your ssh to be in the same ratelimiting rule as the syslog traffic).
* If compression is important (due to the small link size), you could leverage ssh to do this.
This approach is a little more complicated, but your logs would show up sooner.
Depending on important this data is, you may want the backup ftp/rsync method anyways...
On Thu, 28 Oct 2004 15:02:33 -0500, Dave Johnson <davejjohnson@gmail.com> wrote:
You can do it many ways, one way (quick and easy):
remote nodes <every ten minutes cron> log, bzip2 in directory "A" run rsyncd for directory "A" --- central node <every ten minutes +1 minute> <or just do it every 2 mins, etc..> run script: 1) rsync --bwlimit 9k -u get from remote node's "A" 2) bunzip2 files 3) cat file into /dev/log (or local platform's way of injecting into syslog) --------- http://samba.anu.edu.au/rsync/
On Thu, 28 Oct 2004 12:03:53 -0700 (PDT), LEROY ISAAC <lisaac01@yahoo.com> wrote:
I have a need to retrieve syslog data from various remote nodes, and the smallest network link to the remote nodes is 19K. The syslog traffic for the link cannot exceed 9K.
I plan to setup a configuration which generates new log files every 10 minutes. These files are then compressed, zipped, and transfered to a centralized loghost.
The files are then unzipped, uncompressed, and the data is inserted into the syslog-ng data stream on the central syslog-ng host.
Is there a script or utility which will accomplish this task? If not, then does any one have any suggestions on products which may accomplish this same task.
LeRoy Isaac --- DTrinh71@aol.com wrote:
OK. Thanks.
So, what does Ray want? Suggestions?
David
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________
syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Thu, 28 Oct 2004, Dave Johnson wrote:
bill--
I'm only offering suggestions, and working on a limited subset of knowledge about your architecture. Your mileage may vary...
Hehe, not *my* architecture. I was responding to the initial question. I'm just playing devil's advocate. I currently run 140 hosts with three analyzers, using syslog-ng, and pondering your solution as applied to my meager deployment.. well. I wouldn't want to maintain it. =) - billn
On Thu, 28 Oct 2004 13:52:55 -0700 (PDT), Bill Nash <billn@billn.net> wrote:
That's a dramatic increase in complexity, however. The strongest lifeform is often the simplest. Troubleshooting, or even implementing such a setup, may not be feasible and would likely require kernel recompiles to even enable the features, depending on the existing implementation. Scale is another factor that would make this option less attractive. A QoS option would be better implemented at the network level across all tcp/514 traffic.
- billn
On Thu, 28 Oct 2004, Dave Johnson wrote:
Just another thought, (which isn't as easy as the other suggestion) --
* Set up ratelimiting on your remote servers to the central server's IP and just syslog-ng with tcp to the central server. - Make sure you have a decent sized queue on the remote server so you can queue up packets - setting up ratelimiting on linux and getting the results just right might take some time.
(you can google search for /etc/init.d/cbq scripts) and make sure you have class base queueing enabled in your kernel.
---
* You can create another ip on your central server if your going to be doing admin tasks from that box. (IE you don't want your ssh to be in the same ratelimiting rule as the syslog traffic).
* If compression is important (due to the small link size), you could leverage ssh to do this.
This approach is a little more complicated, but your logs would show up sooner.
Depending on important this data is, you may want the backup ftp/rsync method anyways...
On Thu, 28 Oct 2004 15:02:33 -0500, Dave Johnson <davejjohnson@gmail.com> wrote:
You can do it many ways, one way (quick and easy):
remote nodes <every ten minutes cron> log, bzip2 in directory "A" run rsyncd for directory "A" --- central node <every ten minutes +1 minute> <or just do it every 2 mins, etc..> run script: 1) rsync --bwlimit 9k -u get from remote node's "A" 2) bunzip2 files 3) cat file into /dev/log (or local platform's way of injecting into syslog) --------- http://samba.anu.edu.au/rsync/
On Thu, 28 Oct 2004 12:03:53 -0700 (PDT), LEROY ISAAC <lisaac01@yahoo.com> wrote:
I have a need to retrieve syslog data from various remote nodes, and the smallest network link to the remote nodes is 19K. The syslog traffic for the link cannot exceed 9K.
I plan to setup a configuration which generates new log files every 10 minutes. These files are then compressed, zipped, and transfered to a centralized loghost.
The files are then unzipped, uncompressed, and the data is inserted into the syslog-ng data stream on the central syslog-ng host.
Is there a script or utility which will accomplish this task? If not, then does any one have any suggestions on products which may accomplish this same task.
LeRoy Isaac --- DTrinh71@aol.com wrote:
OK. Thanks.
So, what does Ray want? Suggestions?
David
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________
syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
participants (3)
-
Bill Nash
-
Dave Johnson
-
LEROY ISAAC