I want to forward Windows server 2008 R2 logs to Linux syslog-ng.
Hello all. I have a Windows Box that want to forward all Even logs to my Linux box. I install Snare on Windows(172.30.10.19) and configure it to forward logs to Linux and my Linux box receive it properly. When I use " tcpdump udp "port 514" ", Tcpdump show me that Snare sending Logs to Linux but Syslog-ng can't write it to log files :(. I paste my syslog-ng configure : options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); #udp(ip(0.0.0.0) port(514)); }; #source s_net { #udp (ip(172.30.10.19) port(514)); #}; source s_net { udp(); }; filter f_openwrt { host("172.30.10.19");}; destination df_openwrt { file("/var/log/winlog/win.log"); }; log { source ( s_net ); filter( f_openwrt ); destination ( df_openwrt ); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: Can you tell me how can I solve this problem? Cheers.
One way to check would be to have syslog-ng use the host macro as part of the directory (or file) name and let it create directories (or files) for every host it hears from like this: change create_dirs(yes) and create a destination that will use information it parses out of the received logs in the filenames: destination d_separatedbyhosts { file( "/var/log/$HOST/$HOST.$FACILITY.$SEVERITY.$YEAR.$MONTH.$DAY" ); } then don't filter at all - just let syslog-ng create at will. While this may not be what you want eventually, it would let syslog-ng create files for any host it hears from, and that might show you how the $HOST macro is being parsed. Jim On 08/24/2014 01:33 PM, Jason Long wrote:
Hello all. I have a Windows Box that want to forward all Even logs to my Linux box. I install Snare on Windows(172.30.10.19) and configure it to forward logs to Linux and my Linux box receive it properly. When I use " tcpdump udp "port 514" ", Tcpdump show me that Snare sending Logs to Linux but Syslog-ng can't write it to log files :(. I paste my syslog-ng configure :
options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); };
source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); #udp(ip(0.0.0.0) port(514)); };
#source s_net { #udp (ip(172.30.10.19) port(514)); #}; source s_net { udp(); }; filter f_openwrt { host("172.30.10.19");}; destination df_openwrt { file("/var/log/winlog/win.log"); }; log { source ( s_net ); filter( f_openwrt ); destination ( df_openwrt ); };
destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); };
filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); };
#log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); };
# vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
Can you tell me how can I solve this problem?
Cheers.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Jim. Thank you so much for your reply. Excuse me, Can you write a config file for me that collect Windows log? On Monday, August 25, 2014 1:18 AM, Jim Hendrick <jrhendri@roadrunner.com> wrote: One way to check would be to have syslog-ng use the host macro as part of the directory (or file) name and let it create directories (or files) for every host it hears from like this: change create_dirs(yes) and create a destination that will use information it parses out of the received logs in the filenames: destination d_separatedbyhosts { file( "/var/log/$HOST/$HOST.$FACILITY.$SEVERITY.$YEAR.$MONTH.$DAY" ); } then don't filter at all - just let syslog-ng create at will. While this may not be what you want eventually, it would let syslog-ng create files for any host it hears from, and that might show you how the $HOST macro is being parsed. Jim On 08/24/2014 01:33 PM, Jason Long wrote: Hello all.
I have a Windows Box that want to forward all Even logs to my Linux box. I install Snare on Windows(172.30.10.19) and configure it to forward logs to Linux and my Linux box receive it properly. When I use " tcpdump udp "port 514" ", Tcpdump show me that Snare sending Logs to Linux but Syslog-ng can't write it to log files :(. I paste my syslog-ng configure :
options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); };
source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); #udp(ip(0.0.0.0) port(514)); };
#source s_net { #udp (ip(172.30.10.19) port(514)); #}; source s_net { udp(); }; filter f_openwrt { host("172.30.10.19");}; destination df_openwrt { file("/var/log/winlog/win.log"); }; log { source ( s_net ); filter( f_openwrt ); destination ( df_openwrt ); };
destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); };
filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); };
#log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); };
# vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
Can you tell me how can I solve this problem?
Cheers.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Jason - All I am able to provide is this (which is the base that I use in most cases). Obviously the group needs to exist (i.e. "logadmin") I find this is pretty solid, and the directory structure allows easy searching by date and/or server. You will need to look up the options for yourself at the Balabit site (they have excellent documentation) The one thing that I will say is that $HOST and $HOST_FROM differ in that $HOST is parsed from the data in the syslog messages themselves, where $HOST_FROM is the server that sent the packets over the network. I find it more reliable especially when parsing "non-standard" events. Best of luck, Jim # @version: 3.5 @include "scl.conf" options { check_hostname(yes); # check if the hostname contains valid characters use_dns(no); # do not resolve names for speed dns_cache(no); # no DNS cache since we do not resolve names keep_hostname(yes); # keep hostnames to enable related macros chain_hostnames(no); # do not track / forward syslog forwarder chain # options related to file and directories dir_owner("root"); dir_group("logadmin"); owner("root"); group("logadmin"); perm(0640); dir_perm(0750); create_dirs(yes); }; source s_local { system(); internal( ); }; source s_network { udp(); tcp(); }; destination d_separatedbyhosts { file("/data/syslog-ng/$YEAR/$MONTH/$DAY/$HOST_FROM/$HOST_FROM.$FACILITY.$PRIORITY.$YEAR.$MONTH.$DAY"); }; log { source(s_local); source(s_network); destination(d_separatedbyhosts); }; ---- Jason Long <hack3rcon@yahoo.com> wrote:
Hi Jim. Thank you so much for your reply. Excuse me, Can you write a config file for me that collect Windows log?
On Monday, August 25, 2014 1:18 AM, Jim Hendrick <jrhendri@roadrunner.com> wrote: One way to check would be to have syslog-ng use the host macro as part of the directory (or file) name and let it create directories (or files) for every host it hears from like this: change create_dirs(yes) and create a destination that will use information it parses out of the received logs in the filenames: destination d_separatedbyhosts { file( "/var/log/$HOST/$HOST.$FACILITY.$SEVERITY.$YEAR.$MONTH.$DAY" ); } then don't filter at all - just let syslog-ng create at will. While this may not be what you want eventually, it would let syslog-ng create files for any host it hears from, and that might show you how the $HOST macro is being parsed. Jim On 08/24/2014 01:33 PM, Jason Long wrote: Hello all.
I have a Windows Box that want to forward all Even logs to my Linux box. I install Snare on Windows(172.30.10.19) and configure it to forward logs to Linux and my Linux box receive it properly. When I use " tcpdump udp "port 514" ", Tcpdump show me that Snare sending Logs to Linux but Syslog-ng can't write it to log files :(. I paste my syslog-ng configure :
options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); };
source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); #udp(ip(0.0.0.0) port(514)); };
#source s_net { #udp (ip(172.30.10.19) port(514)); #}; source s_net { udp(); }; filter f_openwrt { host("172.30.10.19");}; destination df_openwrt { file("/var/log/winlog/win.log"); }; log { source ( s_net ); filter( f_openwrt ); destination ( df_openwrt ); };
destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); };
filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); };
#log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); };
# vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
Can you tell me how can I solve this problem?
Cheers.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
If a log file is renamed syslog-ng does not write a new file until restarted. Is the data received during that time lost and is there a conf option for this. Can syslog-ng rotate based on size ? What is recommended to manage fast growing files .
Scot, You fail to mention what version of syslog-ng you are using and on which platform.
If a log file is renamed syslog-ng does not write a new file until restarted.
Correct. Renaming a file on a unix system is just a change to the parent directory. Processes reading from or writing to the file which keep the file open will know nothing about the change.
Is the data received during that time lost
No. The process will continue to write to the same file which now has a new name.
and is there a conf option for this.
It's not clear what "this" is. There are lots of log rotation tools and they have various options to handle rotation. Two common approaches are 1) Signal (usually HUP) process(es) after rotation 2) Copy and null See the documentation and examples for your log rotation tool or better yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing messages in plain-text files"[1].
Can syslog-ng rotate based on size ?
Not directly in the way rsyslogd does with max-size, for example, however many log rotation tools have size parameters if this is a requirement.
What is recommended to manage fast growing files .
See e.g. 17.5. "Configuring log rotation"[2]. In general you need to know your log data and your requirements for keeping it. Your syslog-ng and/or log rotation tool configuration should implement these requirements. Typically in a two tier environment the clients log only recent data on local storage while transmitting some or all log data over the network to the loghost(s) for archive, analysis, etc. Depending on how fast "Fast" is, there may also be performance considerations, but start with requirements. [1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide... [2] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide...
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- -Andrew J. Caines- Unix Systems Engineer A.J.Caines@halplant.com "Machines take me by surprise with great frequency" - Alan Turing
Thanks Andrew, Version is 3.5. Maybe it would be clearer this way. We started to send firewall session data to syslog-ng. The end goal is to track firewall sessions to build/update/audit firewall rules. So our logs increased, not a big deal, I write to $YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall logs files that are time consuming to compress and parse. One admin wants to use log-rotate to move logs over $(SIZE) but that could result in many syslog-ng restarts a day and still involves a lot of post processing. I could use an $HOUR macro as well but I that still creates some pretty large files. NEW approach: Has anyone used the $MSG parsers to accomplish a similar task in line ? http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-gu... <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/csv-parser.html> I don’t need to log every session created just unique hits to the highlighted area of this sample in counter format like the stats output has. "SrcRule:IP/PORT DstRule:IP/PORT COUNT Dec 24 11:02:42 192.168.X.X : %FWX_X: Built outbound UDP connection ####### for RuleName:SRCIP/PORT (SRCIP/PORT) to RuleName:DSTIP/PORT (DSTIP/PORT)
On Dec 24, 2014, at 12:08 PM, Andrew J. Caines <A.J.Caines@halplant.com> wrote:
Scot,
You fail to mention what version of syslog-ng you are using and on which platform.
If a log file is renamed syslog-ng does not write a new file until restarted.
Correct. Renaming a file on a unix system is just a change to the parent directory. Processes reading from or writing to the file which keep the file open will know nothing about the change.
Is the data received during that time lost
No. The process will continue to write to the same file which now has a new name.
and is there a conf option for this.
It's not clear what "this" is.
There are lots of log rotation tools and they have various options to handle rotation. Two common approaches are
1) Signal (usually HUP) process(es) after rotation 2) Copy and null
See the documentation and examples for your log rotation tool or better yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing messages in plain-text files"[1].
Can syslog-ng rotate based on size ?
Not directly in the way rsyslogd does with max-size, for example, however many log rotation tools have size parameters if this is a requirement.
What is recommended to manage fast growing files .
See e.g. 17.5. "Configuring log rotation"[2].
In general you need to know your log data and your requirements for keeping it. Your syslog-ng and/or log rotation tool configuration should implement these requirements.
Typically in a two tier environment the clients log only recent data on local storage while transmitting some or all log data over the network to the loghost(s) for archive, analysis, etc.
Depending on how fast "Fast" is, there may also be performance considerations, but start with requirements.
[1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide... [2] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide...
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- -Andrew J. Caines- Unix Systems Engineer A.J.Caines@halplant.com "Machines take me by surprise with great frequency" - Alan Turing ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
We generate a fair number of firewall logs daily and use the $HOUR macro to store the flat files. A nightly cron does a "find -mtime -exec gzip {} \;" to keep older filed zipped and another deletes them after a suitable period. As far as parsing - how do you parse the logs? For what? Do you process them in a SIEM or do you use other programs / scripts? We use multiple destinations to store logs in local files, send logs to a SIEM, etc. I'm sure the list can provide lots of (hopefully) useful suggestions. Jim On 12/24/2014 01:45 PM, Scot Needy wrote:
Thanks Andrew,
Version is 3.5. Maybe it would be clearer this way.
We started to send firewall session data to syslog-ng. The end goal is to track firewall sessions to build/update/audit firewall rules. So our logs increased, not a big deal, I write to $YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall logs files that are time consuming to compress and parse.
One admin wants to use log-rotate to move logs over $(SIZE) but that could result in many syslog-ng restarts a day and still involves a lot of post processing. I could use an $HOUR macro as well but I that still creates some pretty large files.
NEW approach:
Has anyone used the $MSG parsers to accomplish a similar task in line ? http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-gu...
I don’t need to log every session created just unique hits to the highlighted area of this sample in counter format like the stats output has. "SrcRule:IP/PORT DstRule:IP/PORT COUNT
Dec 24 11:02:42 192.168.X.X : %*FWX_X*: Built outbound UDP connection ####### for *RuleName:SRCIP/PORT* (SRCIP/PORT) to *RuleName:DSTIP/PORT* (DSTIP/PORT)
On Dec 24, 2014, at 12:08 PM, Andrew J. Caines <A.J.Caines@halplant.com <mailto:A.J.Caines@halplant.com>> wrote:
Scot,
You fail to mention what version of syslog-ng you are using and on which platform.
If a log file is renamed syslog-ng does not write a new file until restarted.
Correct. Renaming a file on a unix system is just a change to the parent directory. Processes reading from or writing to the file which keep the file open will know nothing about the change.
Is the data received during that time lost
No. The process will continue to write to the same file which now has a new name.
and is there a conf option for this.
It's not clear what "this" is.
There are lots of log rotation tools and they have various options to handle rotation. Two common approaches are
1) Signal (usually HUP) process(es) after rotation 2) Copy and null
See the documentation and examples for your log rotation tool or better yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing messages in plain-text files"[1].
Can syslog-ng rotate based on size ?
Not directly in the way rsyslogd does with max-size, for example, however many log rotation tools have size parameters if this is a requirement.
What is recommended to manage fast growing files .
See e.g. 17.5. "Configuring log rotation"[2].
In general you need to know your log data and your requirements for keeping it. Your syslog-ng and/or log rotation tool configuration should implement these requirements.
Typically in a two tier environment the clients log only recent data on local storage while transmitting some or all log data over the network to the loghost(s) for archive, analysis, etc.
Depending on how fast "Fast" is, there may also be performance considerations, but start with requirements.
[1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide... [2] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide...
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- -Andrew J. Caines- Unix Systems Engineer A.J.Caines@halplant.com "Machines take me by surprise with great frequency" - Alan Turing ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I just noticed the part about tracking firewall sessions. Do you (or would you consider) collecting netflows ? Much more space efficient and designed specifically for that kind of analysis. Take a look at the SiLK tools for an excellent suite that might give you some good ideas. Jim On 12/24/2014 04:57 PM, Jim Hendrick wrote:
We generate a fair number of firewall logs daily and use the $HOUR macro to store the flat files.
A nightly cron does a "find -mtime -exec gzip {} \;" to keep older filed zipped and another deletes them after a suitable period.
As far as parsing - how do you parse the logs? For what? Do you process them in a SIEM or do you use other programs / scripts?
We use multiple destinations to store logs in local files, send logs to a SIEM, etc.
I'm sure the list can provide lots of (hopefully) useful suggestions.
Jim
On 12/24/2014 01:45 PM, Scot Needy wrote:
Thanks Andrew,
Version is 3.5. Maybe it would be clearer this way.
We started to send firewall session data to syslog-ng. The end goal is to track firewall sessions to build/update/audit firewall rules. So our logs increased, not a big deal, I write to $YEAR$MONTH$DAY.$HOST.log but that produced a few 20Gb firewall logs files that are time consuming to compress and parse.
One admin wants to use log-rotate to move logs over $(SIZE) but that could result in many syslog-ng restarts a day and still involves a lot of post processing. I could use an $HOUR macro as well but I that still creates some pretty large files.
NEW approach:
Has anyone used the $MSG parsers to accomplish a similar task in line ? http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-gu...
I don’t need to log every session created just unique hits to the highlighted area of this sample in counter format like the stats output has. "SrcRule:IP/PORT DstRule:IP/PORT COUNT
Dec 24 11:02:42 192.168.X.X : %*FWX_X*: Built outbound UDP connection ####### for *RuleName:SRCIP/PORT* (SRCIP/PORT) to *RuleName:DSTIP/PORT* (DSTIP/PORT)
On Dec 24, 2014, at 12:08 PM, Andrew J. Caines <A.J.Caines@halplant.com <mailto:A.J.Caines@halplant.com>> wrote:
Scot,
You fail to mention what version of syslog-ng you are using and on which platform.
If a log file is renamed syslog-ng does not write a new file until restarted.
Correct. Renaming a file on a unix system is just a change to the parent directory. Processes reading from or writing to the file which keep the file open will know nothing about the change.
Is the data received during that time lost
No. The process will continue to write to the same file which now has a new name.
and is there a conf option for this.
It's not clear what "this" is.
There are lots of log rotation tools and they have various options to handle rotation. Two common approaches are
1) Signal (usually HUP) process(es) after rotation 2) Copy and null
See the documentation and examples for your log rotation tool or better yet, use syslog-ng's native log naming capabilities. See 7.2. "Storing messages in plain-text files"[1].
Can syslog-ng rotate based on size ?
Not directly in the way rsyslogd does with max-size, for example, however many log rotation tools have size parameters if this is a requirement.
What is recommended to manage fast growing files .
See e.g. 17.5. "Configuring log rotation"[2].
In general you need to know your log data and your requirements for keeping it. Your syslog-ng and/or log rotation tool configuration should implement these requirements.
Typically in a two tier environment the clients log only recent data on local storage while transmitting some or all log data over the network to the loghost(s) for archive, analysis, etc.
Depending on how fast "Fast" is, there may also be performance considerations, but start with requirements.
[1] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide... [2] http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guide...
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- -Andrew J. Caines- Unix Systems Engineer A.J.Caines@halplant.com "Machines take me by surprise with great frequency" - Alan Turing ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Andrew J. Caines
-
Jason Long
-
Jim Hendrick
-
jrhendri＠roadrunner.com
-
Scot Needy