hi there bit of a problem with too many logs being generated and i'm not sure what to do. I'm using a iptables firewall setup like this: $IPTABLES -t filter -N ACCEPTLOG $IPTABLES -t filter -A ACCEPTLOG -j LOG --log-prefix "iptables:" --log-level\ debug $IPTABLES -t filter -A ACCEPTLOG -j ACCEPT the firewall is also a masquerading NAT gateway for about 50 clients. I want to record all traffic flowing through the gateway, so along with the firewall rules stated above i have this in my syslog-ng.conf : destination iptables { file("/var/log/iptables"); }; filter iptables { match("^iptables:"); }; log { source(src); filter(iptables); destination(iptables); }; all very nice, except for the fact that within 24 hours the log grew to 500MB's, which is a little too big. Of course i've configured logrotate to g-zip the file every day which greatly reduces the size, but i'd much prefer to have smaller logs but with the necessary information still there. Here's an example from one client just for a simple request to port 445: Dec 17 11:33:18 iptables:IN=eth1 OUT=eth0 SRC=192.168.1.220 DST=213.220.xxx.xxx LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=974 DF PROTO=TCP SPT=32900 DPT=445 WINDOW=10136 RES=0x00 ACK URGP=0 this entry is repeated 10 times per second !! i've tried different log levels in my firewall but it doesnt seem to change anything. Would be grateful for any help. cheers, garvald
Am Fr, den 17.12.2004 schrieb garvald@bluemail.ch um 11:37:
hi there
Hi
bit of a problem with too many logs being generated and i'm not sure what to do. I'm using a iptables firewall setup like this:
$IPTABLES -t filter -N ACCEPTLOG $IPTABLES -t filter -A ACCEPTLOG -j LOG --log-prefix "iptables:" --log-level\ debug $IPTABLES -t filter -A ACCEPTLOG -j ACCEPT
the firewall is also a masquerading NAT gateway for about 50 clients. I want to record all traffic flowing through the gateway,[...]
I do something similar but limit the amount of packets being logged by iptables -A INPUT -i ppp0 -m state --state NEW,INVALID -j LOG ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ so I get only one entry (the first packet) per connection; used mainly to do a statistic on what ports are being knocked on.
[...] but i'd much prefer to have smaller logs but with the necessary information still there. ^^^^^^^^^^^^^^^^^^^^^ Which leads to my main question:
What exactly do you do with the logged data? (If you don't mind telling) I currently whitepaper the use of syslog-ng to build a syslog infrastructure (collect logs on a central loghost, dump them into a relational DB, get useful information out of the DB). The most interesting part so far is the latter, getting something useful out of the logs, so i'm very curious what you do with those 500MB+ per day.
[...] i've tried different log levels in my firewall but it doesnt seem to change anything. Would be grateful for any help.
The '--log-level debug' parameter you use specifies the priority the message gets tagged with, it doesn't change the behaviour of the packet filter in any way.
cheers, garvald
Wolfgang -- Wolfgang Braun <wolfgang.braun@gmx.de>, Dipl. Inform. (FH) gpg-key: 1024D/4B32CE55
Are you logging _all_ your traffic through the firewall? If you have requirements to log good traffic (and/or report on it), then try and only log initial connection setups (this is all done in iptables configs and is out of the scope of syslog-ng). However, to point you in the right direction-- IE: maybe try a chain that sequentially: - connection established rules without logging - log new connections - allow these ports/new connections - deny log - deny On Fri, 17 Dec 2004 17:14:09 +0100, Wolfgang Braun <wolfgang.braun@gmx.de> wrote:
Am Fr, den 17.12.2004 schrieb garvald@bluemail.ch um 11:37:
hi there
Hi
bit of a problem with too many logs being generated and i'm not sure what to do. I'm using a iptables firewall setup like this:
$IPTABLES -t filter -N ACCEPTLOG $IPTABLES -t filter -A ACCEPTLOG -j LOG --log-prefix "iptables:" --log-level\ debug $IPTABLES -t filter -A ACCEPTLOG -j ACCEPT
the firewall is also a masquerading NAT gateway for about 50 clients. I want to record all traffic flowing through the gateway,[...]
I do something similar but limit the amount of packets being logged by
iptables -A INPUT -i ppp0 -m state --state NEW,INVALID -j LOG ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
so I get only one entry (the first packet) per connection; used mainly to do a statistic on what ports are being knocked on.
[...] but i'd much prefer to have smaller logs but with the necessary information still there. ^^^^^^^^^^^^^^^^^^^^^ Which leads to my main question:
What exactly do you do with the logged data? (If you don't mind telling)
I currently whitepaper the use of syslog-ng to build a syslog infrastructure (collect logs on a central loghost, dump them into a relational DB, get useful information out of the DB). The most interesting part so far is the latter, getting something useful out of the logs, so i'm very curious what you do with those 500MB+ per day.
[...] i've tried different log levels in my firewall but it doesnt seem to change anything. Would be grateful for any help.
The '--log-level debug' parameter you use specifies the priority the message gets tagged with, it doesn't change the behaviour of the packet filter in any way.
cheers, garvald
Wolfgang
-- Wolfgang Braun <wolfgang.braun@gmx.de>, Dipl. Inform. (FH) gpg-key: 1024D/4B32CE55
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Fri, 17 Dec 2004 11:37:16 +0100 garvald@bluemail.ch wrote:
bit of a problem with too many logs being generated and i'm not sure what to do. I'm using a iptables firewall setup like this:
Log sampling might be a nice experimental feature to have in this and other related cases. In some cases such as logging all possible filter denies or logging all queries in busy DNS servers, getting a copy of every single log may not be necessary and due to quantity, impractical. In the simplest case, every N messages would be logged to disk, but in more complex scenarios it could based on message content or specific source and use a more advanced algorithm rather than logging every N messages. John
participants (4)
-
Dave Johnson
-
garvaldï¼ bluemail.ch
-
John Kristoff
-
Wolfgang Braun