need help debugging some network received logs that aren't writing to files
Hello. First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it. I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious. I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name. I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch. I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device. Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool.. Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); }; # ===================== # UDP Packet Source source s_udp { udp(); }; # ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); }; # ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); }; # ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; ===== I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything. Any thoughts? Cheers, -Chris
When I’m not seeing any data I start with tcpdump to make sure it’s actually getting to the log host and review the packet dump for a possible issue. There is also a debug, trace function in syslog-ng but in heavy traffic environments like yours I start with tcpdump with a mac or ip filter. On Feb 18, 2014, at 3:12 PM, Chris Moody <chris@node-nine.com> wrote:
Hello.
First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it.
I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious.
I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name.
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool..
Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); };
# ===================== # UDP Packet Source source s_udp { udp(); };
# ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); };
# ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); };
# ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; =====
I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything.
Any thoughts?
Cheers, -Chris ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Chris, Are there *any* folders/files being created at all? There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST... - Austin On 2/18/2014 2:12 PM, Chris Moody wrote:
Hello.
First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it.
I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious.
I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name.
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool..
Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); };
# ===================== # UDP Packet Source source s_udp { udp(); };
# ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); };
# ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); };
# ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; =====
I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything.
Any thoughts?
Cheers, -Chris ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip. Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot) Just debugging a host-device that we're not seeing logs accounted for. -Chris On 2/18/14 3:51 PM, Austin Jorden wrote:
Hi Chris,
Are there *any* folders/files being created at all?
There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST...
- Austin
On 2/18/2014 2:12 PM, Chris Moody wrote:
Hello.
First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it.
I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious.
I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name.
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool..
Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); };
# ===================== # UDP Packet Source source s_udp { udp(); };
# ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); };
# ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); };
# ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; =====
I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything.
Any thoughts?
Cheers, -Chris ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
We had a parsing problem on our ASA where the log contained an extra date so the Host looked like “Feb”. There was a syslog option in the ASA not to send the date in the header. On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com> wrote:
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip.
Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot)
Just debugging a host-device that we're not seeing logs accounted for.
-Chris
On 2/18/14 3:51 PM, Austin Jorden wrote:
Hi Chris,
Are there *any* folders/files being created at all?
There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST...
- Austin
On 2/18/2014 2:12 PM, Chris Moody wrote:
Hello.
First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it.
I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious.
I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name.
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool..
Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); };
# ===================== # UDP Packet Source source s_udp { udp(); };
# ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); };
# ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); };
# ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; =====
I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything.
Any thoughts?
Cheers, -Chris ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hmm... that's a thought. The troublesome device is an IOS system. I'llgive 'er a gander to see if there are any other options. I don't recall there being any that controlled the 'hostname' header field though. -Chris On 2/18/14 5:10 PM, Scot Needy wrote:
We had a parsing problem on our ASA where the log contained an extra date so the Host looked like “Feb”.
There was a syslog option in the ASA not to send the date in the header.
On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com> wrote:
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip.
Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot)
Just debugging a host-device that we're not seeing logs accounted for.
-Chris
On 2/18/14 3:51 PM, Austin Jorden wrote:
Hi Chris,
Are there *any* folders/files being created at all?
There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST...
- Austin
On 2/18/2014 2:12 PM, Chris Moody wrote:
Hello.
First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it.
I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious.
I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name.
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool..
Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); };
# ===================== # UDP Packet Source source s_udp { udp(); };
# ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); };
# ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); };
# ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; =====
I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything.
Any thoughts?
Cheers, -Chris ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
It wasn’t adding the data to the hostname just adding extra header data that broke the RFC format. On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com> wrote:
Hmm... that's a thought. The troublesome device is an IOS system. I'llgive 'er a gander to see if there are any other options. I don't recall there being any that controlled the 'hostname' header field though.
-Chris
On 2/18/14 5:10 PM, Scot Needy wrote:
We had a parsing problem on our ASA where the log contained an extra date so the Host looked like “Feb”.
There was a syslog option in the ASA not to send the date in the header.
On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com> wrote:
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip.
Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot)
Just debugging a host-device that we're not seeing logs accounted for.
-Chris
On 2/18/14 3:51 PM, Austin Jorden wrote:
Hi Chris,
Are there *any* folders/files being created at all?
There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST...
- Austin
On 2/18/2014 2:12 PM, Chris Moody wrote:
Hello.
First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it.
I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious.
I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name.
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool..
Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); };
# ===================== # UDP Packet Source source s_udp { udp(); };
# ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); };
# ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); };
# ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; =====
I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything.
Any thoughts?
Cheers, -Chris ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Latest syslog-ng versions parse some of the cisco extensions. Which version do you run? On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn@gmail.com> wrote:
It wasn't adding the data to the hostname just adding extra header data that broke the RFC format.
On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com> wrote:
Hmm... that's a thought. The troublesome device is an IOS system. I'llgive 'er a gander to see if there are any other options. I don't recall there being any that controlled the 'hostname' header field though.
-Chris
On 2/18/14 5:10 PM, Scot Needy wrote:
We had a parsing problem on our ASA where the log contained an extra date so the Host looked like "Feb".
There was a syslog option in the ASA not to send the date in the header.
On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com> wrote:
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip.
Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot)
Just debugging a host-device that we're not seeing logs accounted for.
-Chris
On 2/18/14 3:51 PM, Austin Jorden wrote:
Hi Chris,
Are there *any* folders/files being created at all?
There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST...
- Austin
On 2/18/2014 2:12 PM, Chris Moody wrote:
Hello.
First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it.
I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious.
I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name.
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool..
Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); };
# ===================== # UDP Packet Source source s_udp { udp(); };
# ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); };
# ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); };
# ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; =====
I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything.
Any thoughts?
Cheers, -Chris
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
ok - sorry for the latent reply folks - got wrangled into other troubleshooting efforts. Anyway, I've taken a deeper look at this and I'm experiencing this behavior when receiving logs from IOS, Nexus, and other device types as well..so it's not just some weird version/bug type issue with the received messages. The log hosts are both running: === syslog-ng 3.2.5 === These are RHEL 6.2 VirtualMachines, each logging to their own dedicated 5-TB NFS mount. Plenty of CPU & Memory overhead. I'm not seeing any issues with I/O-wait to the NFS mounts. These hosts are -very- busy (from loads of firewall logs) but have been working beautifully in the past. This seems to be a relatively recent development...not sure how long though unfortunately. There are about 1000 current active spools on each host that are updating just fine when new messages come in. I'm just experiencing where some of the spool files have gone stale and don't get written to any longer (despite messages being received) as well as no new spools get created. It's like the destination directive isn't being adhered-to. === destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0755) ); }; === This should be creating a new log spool per-host upon receipt of logs over the wire...and it's working...but not 100% any longer it seems. I've done tcpdump captures of the nodes that I'm having trouble with and I do see the source-IP in all the 'hostname' fields...so it's not like these are coming through as malformed which was a good first-pass thought. Even tried toggling 'keep_hostname' to no (currently 'yes') and that doesn't seem to help. I have tried sending logs from some new systems to these aggregators and they are not creating new spools either. I've run tcpdump to confirm the message receipt, but no new files are being written for new devices either. It's almost behaving like perhaps there's too many files being written to...too many open filehandles... something along these lines perhaps. Hoping for some other outside-perspective ideas of things I can check or debug as I've been trying to debug this too long...and am most likely staring at the issue right in the face. -Chris On 2/22/14 6:44 AM, Balazs Scheidler wrote:
Latest syslog-ng versions parse some of the cisco extensions. Which version do you run?
On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
It wasn't adding the data to the hostname just adding extra header data that broke the RFC format.
On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com <mailto:chris@node-nine.com>> wrote:
> Hmm... that's a thought. The troublesome device is an IOS system. > I'llgive 'er a gander to see if there are any other options. I don't > recall there being any that controlled the 'hostname' header field though. > > -Chris > > On 2/18/14 5:10 PM, Scot Needy wrote: >> We had a parsing problem on our ASA where the log contained an extra date so the Host looked like "Feb". >> >> There was a syslog option in the ASA not to send the date in the header. >> >> On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com <mailto:chris@node-nine.com>> wrote: >> >>> yes - there are tons of spool files being created successfully. As any >>> new network device starts logging we see a new log-spool get created for >>> it's source-ip. >>> >>> Tons of free disk space - almost a Tb of free room. Loads of >>> processor/mem overhead. Nothing glaring in syslog-ng's logs (like >>> unable to write or whatnot) >>> >>> Just debugging a host-device that we're not seeing logs accounted for. >>> >>> -Chris >>> >>> On 2/18/14 3:51 PM, Austin Jorden wrote: >>>> Hi Chris, >>>> >>>> Are there *any* folders/files being created at all? >>>> >>>> There's one thing I noticed that isn't specified... which is the >>>> "createdirs = Yes" option. It appears (well, I assume) that you're >>>> wanting it to create a separate text file for each $HOST, not a separate >>>> directory named $HOST... >>>> >>>> - Austin >>>> >>>> On 2/18/2014 2:12 PM, Chris Moody wrote: >>>>> Hello. >>>>> >>>>> First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome >>>>> code for years now. I've built all sorts of logging infrastructure with >>>>> it. >>>>> >>>>> I seem to have hit on something though that's got me scratching my head >>>>> and lacking for explanation. Perhaps I've just been staring at it and >>>>> debugging it too long and am missing something obvious. >>>>> >>>>> I've got an installation with a couple thousand network devices logging >>>>> successfully to output spools on our log aggretor. This is rockin' and >>>>> works beautifully. I've got things configured whereby each network >>>>> source logs to it's own individual spool file with the source-ip as the >>>>> spool name. >>>>> >>>>> I'm running into a case though where I have a Cisco switch sending logs >>>>> to my log aggregator but the log-server isn't writing the output to the >>>>> device's spool file. It is working however for many many more devices >>>>> just like this switch. >>>>> >>>>> I've confirmed via tcpdump that this log traffic does actually hit the >>>>> box, but it never gets recorded into the log spool for that network device. >>>>> >>>>> Since the host is -super- busy receiving logs from other gear >>>>> enterprise-wide, I have to treat it very gingerly, so can't enable too >>>>> much debugging...but I'm really confused why the logs wouldn't show up >>>>> in the log spool.. >>>>> >>>>> Here's some bits of the config that are relevant: >>>>> ===== >>>>> options { >>>>> keep_hostname(yes); >>>>> use_dns(no); >>>>> use_fqdn(no); >>>>> stats_freq(600); >>>>> stats_level(2); >>>>> # Allow large messages >>>>> log_msg_size(65536); >>>>> }; >>>>> >>>>> # ===================== >>>>> # UDP Packet Source >>>>> source s_udp { >>>>> udp(); >>>>> }; >>>>> >>>>> # ===================== >>>>> # TCP Packet Source >>>>> source s_tcp { >>>>> tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); >>>>> }; >>>>> >>>>> # ===================== >>>>> destination net_perhost { >>>>> file("/data/log/per-host/$HOST" >>>>> owner(root) >>>>> group(nwadmin) >>>>> perm(0775) >>>>> ); >>>>> }; >>>>> >>>>> # ===================== >>>>> log { >>>>> source(s_tcp); >>>>> source(s_udp); >>>>> destination(net_perhost); >>>>> }; >>>>> ===== >>>>> >>>>> I've checked around for perhaps a different spool name, thinking perhaps >>>>> the data was getting recognized as something other than it's source-ip, >>>>> but haven't seen anything. >>>>> >>>>> Any thoughts? >>>>> >>>>> Cheers, >>>>> -Chris >>>>> ______________________________________________________________________________ >>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>> >>>> ______________________________________________________________________________ >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Do you have any log rotation? Is it possible that the log files got rotated/deleted but syslog still has the file handle open and continues to write to it. Since it never gets closed, it never needs to reopen it so it never detects the missing filename and never creates a new one. you can use lsof and search for "delete" Evan Rempel 250.271.7691 University Systems, University of Victoria Chris Moody <chris@node-nine.com> wrote: ok - sorry for the latent reply folks - got wrangled into other troubleshooting efforts. Anyway, I've taken a deeper look at this and I'm experiencing this behavior when receiving logs from IOS, Nexus, and other device types as well..so it's not just some weird version/bug type issue with the received messages. The log hosts are both running: === syslog-ng 3.2.5 === These are RHEL 6.2 VirtualMachines, each logging to their own dedicated 5-TB NFS mount. Plenty of CPU & Memory overhead. I'm not seeing any issues with I/O-wait to the NFS mounts. These hosts are -very- busy (from loads of firewall logs) but have been working beautifully in the past. This seems to be a relatively recent development...not sure how long though unfortunately. There are about 1000 current active spools on each host that are updating just fine when new messages come in. I'm just experiencing where some of the spool files have gone stale and don't get written to any longer (despite messages being received) as well as no new spools get created. It's like the destination directive isn't being adhered-to. === destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0755) ); }; === This should be creating a new log spool per-host upon receipt of logs over the wire...and it's working...but not 100% any longer it seems. I've done tcpdump captures of the nodes that I'm having trouble with and I do see the source-IP in all the 'hostname' fields...so it's not like these are coming through as malformed which was a good first-pass thought. Even tried toggling 'keep_hostname' to no (currently 'yes') and that doesn't seem to help. I have tried sending logs from some new systems to these aggregators and they are not creating new spools either. I've run tcpdump to confirm the message receipt, but no new files are being written for new devices either. It's almost behaving like perhaps there's too many files being written to...too many open filehandles... something along these lines perhaps. Hoping for some other outside-perspective ideas of things I can check or debug as I've been trying to debug this too long...and am most likely staring at the issue right in the face. -Chris On 2/22/14 6:44 AM, Balazs Scheidler wrote: Latest syslog-ng versions parse some of the cisco extensions. Which version do you run? On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn@gmail.com<mailto:scotrn@gmail.com>> wrote: It wasn’t adding the data to the hostname just adding extra header data that broke the RFC format. On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com<mailto:chris@node-nine.com>> wrote:
Hmm... that's a thought. The troublesome device is an IOS system. I'llgive 'er a gander to see if there are any other options. I don't recall there being any that controlled the 'hostname' header field though.
-Chris
On 2/18/14 5:10 PM, Scot Needy wrote:
We had a parsing problem on our ASA where the log contained an extra date so the Host looked like “Feb”.
There was a syslog option in the ASA not to send the date in the header.
On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com<mailto:chris@node-nine.com>> wrote:
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip.
Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot)
Just debugging a host-device that we're not seeing logs accounted for.
-Chris
On 2/18/14 3:51 PM, Austin Jorden wrote:
Hi Chris,
Are there *any* folders/files being created at all?
There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST...
- Austin
On 2/18/2014 2:12 PM, Chris Moody wrote:
Hello.
First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome code for years now. I've built all sorts of logging infrastructure with it.
I seem to have hit on something though that's got me scratching my head and lacking for explanation. Perhaps I've just been staring at it and debugging it too long and am missing something obvious.
I've got an installation with a couple thousand network devices logging successfully to output spools on our log aggretor. This is rockin' and works beautifully. I've got things configured whereby each network source logs to it's own individual spool file with the source-ip as the spool name.
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Since the host is -super- busy receiving logs from other gear enterprise-wide, I have to treat it very gingerly, so can't enable too much debugging...but I'm really confused why the logs wouldn't show up in the log spool..
Here's some bits of the config that are relevant: ===== options { keep_hostname(yes); use_dns(no); use_fqdn(no); stats_freq(600); stats_level(2); # Allow large messages log_msg_size(65536); };
# ===================== # UDP Packet Source source s_udp { udp(); };
# ===================== # TCP Packet Source source s_tcp { tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); };
# ===================== destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0775) ); };
# ===================== log { source(s_tcp); source(s_udp); destination(net_perhost); }; =====
I've checked around for perhaps a different spool name, thinking perhaps the data was getting recognized as something other than it's source-ip, but haven't seen anything.
Any thoughts?
Cheers, -Chris ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
So here was the issue. having udp(so_rcvbuf(1024000) in the syslog-ng conf was creating the issue. when i commented this it would have visibility only to the logs specificly forwarded to it.. And talking about log rotation, Any ideas other than using $YEAR/$MONTH/ basically date parameters to rotate logs on and successfully have crrons to delete them? regards Shashank On Thu, Mar 6, 2014 at 9:41 AM, Evan Rempel <erempel@uvic.ca> wrote:
Do you have any log rotation? Is it possible that the log files got rotated/deleted but syslog still has the file handle open and continues to write to it. Since it never gets closed, it never needs to reopen it so it never detects the missing filename and never creates a new one.
you can use lsof and search for "delete"
Evan Rempel 250.271.7691 University Systems, University of Victoria
Chris Moody <chris@node-nine.com> wrote:
ok - sorry for the latent reply folks - got wrangled into other troubleshooting efforts.
Anyway, I've taken a deeper look at this and I'm experiencing this behavior when receiving logs from IOS, Nexus, and other device types as well..so it's not just some weird version/bug type issue with the received messages.
The log hosts are both running: === syslog-ng 3.2.5 ===
These are RHEL 6.2 VirtualMachines, each logging to their own dedicated 5-TB NFS mount. Plenty of CPU & Memory overhead. I'm not seeing any issues with I/O-wait to the NFS mounts. These hosts are -very- busy (from loads of firewall logs) but have been working beautifully in the past. This seems to be a relatively recent development...not sure how long though unfortunately.
There are about 1000 current active spools on each host that are updating just fine when new messages come in. I'm just experiencing where some of the spool files have gone stale and don't get written to any longer (despite messages being received) as well as no new spools get created. It's like the destination directive isn't being adhered-to. === destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0755) ); }; === This should be creating a new log spool per-host upon receipt of logs over the wire...and it's working...but not 100% any longer it seems.
I've done tcpdump captures of the nodes that I'm having trouble with and I do see the source-IP in all the 'hostname' fields...so it's not like these are coming through as malformed which was a good first-pass thought. Even tried toggling 'keep_hostname' to no (currently 'yes') and that doesn't seem to help.
I have tried sending logs from some new systems to these aggregators and they are not creating new spools either. I've run tcpdump to confirm the message receipt, but no new files are being written for new devices either.
It's almost behaving like perhaps there's too many files being written to...too many open filehandles... something along these lines perhaps.
Hoping for some other outside-perspective ideas of things I can check or debug as I've been trying to debug this too long...and am most likely staring at the issue right in the face.
-Chris
On 2/22/14 6:44 AM, Balazs Scheidler wrote:
Latest syslog-ng versions parse some of the cisco extensions. Which version do you run? On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn@gmail.com> wrote:
It wasn't adding the data to the hostname just adding extra header data that broke the RFC format.
On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com> wrote:
Hmm... that's a thought. The troublesome device is an IOS system. I'llgive 'er a gander to see if there are any other options. I don't recall there being any that controlled the 'hostname' header field though.
-Chris
On 2/18/14 5:10 PM, Scot Needy wrote:
We had a parsing problem on our ASA where the log contained an extra date so the Host looked like "Feb".
There was a syslog option in the ASA not to send the date in the header.
On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com> wrote:
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip.
Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot)
Just debugging a host-device that we're not seeing logs accounted for.
-Chris
On 2/18/14 3:51 PM, Austin Jorden wrote:
Hi Chris,
Are there *any* folders/files being created at all?
There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST...
- Austin
On 2/18/2014 2:12 PM, Chris Moody wrote: > Hello. > > First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome > code for years now. I've built all sorts of logging infrastructure with > it. > > I seem to have hit on something though that's got me scratching my head > and lacking for explanation. Perhaps I've just been staring at it and > debugging it too long and am missing something obvious. > > I've got an installation with a couple thousand network devices logging > successfully to output spools on our log aggretor. This is rockin' and > works beautifully. I've got things configured whereby each network > source logs to it's own individual spool file with the source-ip as the > spool name. > > I'm running into a case though where I have a Cisco switch sending logs > to my log aggregator but the log-server isn't writing the output to the > device's spool file. It is working however for many many more devices > just like this switch. > > I've confirmed via tcpdump that this log traffic does actually hit the > box, but it never gets recorded into the log spool for that network device. > > Since the host is -super- busy receiving logs from other gear > enterprise-wide, I have to treat it very gingerly, so can't enable too > much debugging...but I'm really confused why the logs wouldn't show up > in the log spool.. > > Here's some bits of the config that are relevant: > ===== > options { > keep_hostname(yes); > use_dns(no); > use_fqdn(no); > stats_freq(600); > stats_level(2); > # Allow large messages > log_msg_size(65536); > }; > > # ===================== > # UDP Packet Source > source s_udp { > udp(); > }; > > # ===================== > # TCP Packet Source > source s_tcp { > tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); > }; > > # ===================== > destination net_perhost { > file("/data/log/per-host/$HOST" > owner(root) > group(nwadmin) > perm(0775) > ); > }; > > # ===================== > log { > source(s_tcp); > source(s_udp); > destination(net_perhost); > }; > ===== > > I've checked around for perhaps a different spool name, thinking perhaps > the data was getting recognized as something other than it's source-ip, > but haven't seen anything. > > Any thoughts? > > Cheers, > -Chris >
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Oops! Apologies replied to the wrong thread! Please ignore On Thu, Mar 6, 2014 at 9:47 AM, Shashank Rohatgi <srck85@gmail.com> wrote:
So here was the issue. having udp(so_rcvbuf(1024000) in the syslog-ng conf was creating the issue. when i commented this it would have visibility only to the logs specificly forwarded to it..
And talking about log rotation, Any ideas other than using $YEAR/$MONTH/ basically date parameters to rotate logs on and successfully have crrons to delete them?
regards Shashank
On Thu, Mar 6, 2014 at 9:41 AM, Evan Rempel <erempel@uvic.ca> wrote:
Do you have any log rotation? Is it possible that the log files got rotated/deleted but syslog still has the file handle open and continues to write to it. Since it never gets closed, it never needs to reopen it so it never detects the missing filename and never creates a new one.
you can use lsof and search for "delete"
Evan Rempel 250.271.7691 University Systems, University of Victoria
Chris Moody <chris@node-nine.com> wrote:
ok - sorry for the latent reply folks - got wrangled into other troubleshooting efforts.
Anyway, I've taken a deeper look at this and I'm experiencing this behavior when receiving logs from IOS, Nexus, and other device types as well..so it's not just some weird version/bug type issue with the received messages.
The log hosts are both running: === syslog-ng 3.2.5 ===
These are RHEL 6.2 VirtualMachines, each logging to their own dedicated 5-TB NFS mount. Plenty of CPU & Memory overhead. I'm not seeing any issues with I/O-wait to the NFS mounts. These hosts are -very- busy (from loads of firewall logs) but have been working beautifully in the past. This seems to be a relatively recent development...not sure how long though unfortunately.
There are about 1000 current active spools on each host that are updating just fine when new messages come in. I'm just experiencing where some of the spool files have gone stale and don't get written to any longer (despite messages being received) as well as no new spools get created. It's like the destination directive isn't being adhered-to. === destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0755) ); }; === This should be creating a new log spool per-host upon receipt of logs over the wire...and it's working...but not 100% any longer it seems.
I've done tcpdump captures of the nodes that I'm having trouble with and I do see the source-IP in all the 'hostname' fields...so it's not like these are coming through as malformed which was a good first-pass thought. Even tried toggling 'keep_hostname' to no (currently 'yes') and that doesn't seem to help.
I have tried sending logs from some new systems to these aggregators and they are not creating new spools either. I've run tcpdump to confirm the message receipt, but no new files are being written for new devices either.
It's almost behaving like perhaps there's too many files being written to...too many open filehandles... something along these lines perhaps.
Hoping for some other outside-perspective ideas of things I can check or debug as I've been trying to debug this too long...and am most likely staring at the issue right in the face.
-Chris
On 2/22/14 6:44 AM, Balazs Scheidler wrote:
Latest syslog-ng versions parse some of the cisco extensions. Which version do you run? On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn@gmail.com> wrote:
It wasn't adding the data to the hostname just adding extra header data that broke the RFC format.
On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com> wrote:
Hmm... that's a thought. The troublesome device is an IOS system. I'llgive 'er a gander to see if there are any other options. I don't recall there being any that controlled the 'hostname' header field though.
-Chris
On 2/18/14 5:10 PM, Scot Needy wrote:
We had a parsing problem on our ASA where the log contained an extra date so the Host looked like "Feb".
There was a syslog option in the ASA not to send the date in the header.
On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com> wrote:
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip.
Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot)
Just debugging a host-device that we're not seeing logs accounted for.
-Chris
On 2/18/14 3:51 PM, Austin Jorden wrote: > Hi Chris, > > Are there *any* folders/files being created at all? > > There's one thing I noticed that isn't specified... which is the > "createdirs = Yes" option. It appears (well, I assume) that you're > wanting it to create a separate text file for each $HOST, not a separate > directory named $HOST... > > - Austin > > On 2/18/2014 2:12 PM, Chris Moody wrote: >> Hello. >> >> First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome >> code for years now. I've built all sorts of logging infrastructure with >> it. >> >> I seem to have hit on something though that's got me scratching my head >> and lacking for explanation. Perhaps I've just been staring at it and >> debugging it too long and am missing something obvious. >> >> I've got an installation with a couple thousand network devices logging >> successfully to output spools on our log aggretor. This is rockin' and >> works beautifully. I've got things configured whereby each network >> source logs to it's own individual spool file with the source-ip as the >> spool name. >> >> I'm running into a case though where I have a Cisco switch sending logs >> to my log aggregator but the log-server isn't writing the output to the >> device's spool file. It is working however for many many more devices >> just like this switch. >> >> I've confirmed via tcpdump that this log traffic does actually hit the >> box, but it never gets recorded into the log spool for that network device. >> >> Since the host is -super- busy receiving logs from other gear >> enterprise-wide, I have to treat it very gingerly, so can't enable too >> much debugging...but I'm really confused why the logs wouldn't show up >> in the log spool.. >> >> Here's some bits of the config that are relevant: >> ===== >> options { >> keep_hostname(yes); >> use_dns(no); >> use_fqdn(no); >> stats_freq(600); >> stats_level(2); >> # Allow large messages >> log_msg_size(65536); >> }; >> >> # ===================== >> # UDP Packet Source >> source s_udp { >> udp(); >> }; >> >> # ===================== >> # TCP Packet Source >> source s_tcp { >> tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); >> }; >> >> # ===================== >> destination net_perhost { >> file("/data/log/per-host/$HOST" >> owner(root) >> group(nwadmin) >> perm(0775) >> ); >> }; >> >> # ===================== >> log { >> source(s_tcp); >> source(s_udp); >> destination(net_perhost); >> }; >> ===== >> >> I've checked around for perhaps a different spool name, thinking perhaps >> the data was getting recognized as something other than it's source-ip, >> but haven't seen anything. >> >> Any thoughts? >> >> Cheers, >> -Chris >>
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
one other thought (basic and probably unrelated) we run about a dozen syslog-ng log aggregators for thousands of devices in multiple data centers, and found performance was much faster / more reliable by using $HOST_FROM instead of $HOST. "If the /|keep_hostname()|/ option is enabled (/|keep_hostname(yes)|/), the value of the $HOST macro will be the hostname retrieved from the log message" there might be parsing options with the log format that would be bypassed by using $HOST_FROM which is" "Name of the host that sent the message to syslog-ng, as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain." and since you have use_dns(no) it will be the IP address (unless for some reason the receiving server has the name without using DNS - like its own) anyway this seems to work well for us. Jim On 03/05/2014 11:11 PM, Evan Rempel wrote:
Do you have any log rotation? Is it possible that the log files got rotated/deleted but syslog still has the file handle open and continues to write to it. Since it never gets closed, it never needs to reopen it so it never detects the missing filename and never creates a new one.
you can use lsof and search for "delete"
Evan Rempel 250.271.7691 University Systems, University of Victoria
Chris Moody <chris@node-nine.com> wrote:
ok - sorry for the latent reply folks - got wrangled into other troubleshooting efforts.
Anyway, I've taken a deeper look at this and I'm experiencing this behavior when receiving logs from IOS, Nexus, and other device types as well..so it's not just some weird version/bug type issue with the received messages.
The log hosts are both running: === syslog-ng 3.2.5 ===
These are RHEL 6.2 VirtualMachines, each logging to their own dedicated 5-TB NFS mount. Plenty of CPU & Memory overhead. I'm not seeing any issues with I/O-wait to the NFS mounts. These hosts are -very- busy (from loads of firewall logs) but have been working beautifully in the past. This seems to be a relatively recent development...not sure how long though unfortunately.
There are about 1000 current active spools on each host that are updating just fine when new messages come in. I'm just experiencing where some of the spool files have gone stale and don't get written to any longer (despite messages being received) as well as no new spools get created. It's like the destination directive isn't being adhered-to. === destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0755) ); }; === This should be creating a new log spool per-host upon receipt of logs over the wire...and it's working...but not 100% any longer it seems.
I've done tcpdump captures of the nodes that I'm having trouble with and I do see the source-IP in all the 'hostname' fields...so it's not like these are coming through as malformed which was a good first-pass thought. Even tried toggling 'keep_hostname' to no (currently 'yes') and that doesn't seem to help.
I have tried sending logs from some new systems to these aggregators and they are not creating new spools either. I've run tcpdump to confirm the message receipt, but no new files are being written for new devices either.
It's almost behaving like perhaps there's too many files being written to...too many open filehandles... something along these lines perhaps.
Hoping for some other outside-perspective ideas of things I can check or debug as I've been trying to debug this too long...and am most likely staring at the issue right in the face.
-Chris
On 2/22/14 6:44 AM, Balazs Scheidler wrote:
Latest syslog-ng versions parse some of the cisco extensions. Which version do you run?
On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
It wasn't adding the data to the hostname just adding extra header data that broke the RFC format.
On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com <mailto:chris@node-nine.com>> wrote:
> Hmm... that's a thought. The troublesome device is an IOS system. > I'llgive 'er a gander to see if there are any other options. I don't > recall there being any that controlled the 'hostname' header field though. > > -Chris > > On 2/18/14 5:10 PM, Scot Needy wrote: >> We had a parsing problem on our ASA where the log contained an extra date so the Host looked like "Feb". >> >> There was a syslog option in the ASA not to send the date in the header. >> >> On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com <mailto:chris@node-nine.com>> wrote: >> >>> yes - there are tons of spool files being created successfully. As any >>> new network device starts logging we see a new log-spool get created for >>> it's source-ip. >>> >>> Tons of free disk space - almost a Tb of free room. Loads of >>> processor/mem overhead. Nothing glaring in syslog-ng's logs (like >>> unable to write or whatnot) >>> >>> Just debugging a host-device that we're not seeing logs accounted for. >>> >>> -Chris >>> >>> On 2/18/14 3:51 PM, Austin Jorden wrote: >>>> Hi Chris, >>>> >>>> Are there *any* folders/files being created at all? >>>> >>>> There's one thing I noticed that isn't specified... which is the >>>> "createdirs = Yes" option. It appears (well, I assume) that you're >>>> wanting it to create a separate text file for each $HOST, not a separate >>>> directory named $HOST... >>>> >>>> - Austin >>>> >>>> On 2/18/2014 2:12 PM, Chris Moody wrote: >>>>> Hello. >>>>> >>>>> First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome >>>>> code for years now. I've built all sorts of logging infrastructure with >>>>> it. >>>>> >>>>> I seem to have hit on something though that's got me scratching my head >>>>> and lacking for explanation. Perhaps I've just been staring at it and >>>>> debugging it too long and am missing something obvious. >>>>> >>>>> I've got an installation with a couple thousand network devices logging >>>>> successfully to output spools on our log aggretor. This is rockin' and >>>>> works beautifully. I've got things configured whereby each network >>>>> source logs to it's own individual spool file with the source-ip as the >>>>> spool name. >>>>> >>>>> I'm running into a case though where I have a Cisco switch sending logs >>>>> to my log aggregator but the log-server isn't writing the output to the >>>>> device's spool file. It is working however for many many more devices >>>>> just like this switch. >>>>> >>>>> I've confirmed via tcpdump that this log traffic does actually hit the >>>>> box, but it never gets recorded into the log spool for that network device. >>>>> >>>>> Since the host is -super- busy receiving logs from other gear >>>>> enterprise-wide, I have to treat it very gingerly, so can't enable too >>>>> much debugging...but I'm really confused why the logs wouldn't show up >>>>> in the log spool.. >>>>> >>>>> Here's some bits of the config that are relevant: >>>>> ===== >>>>> options { >>>>> keep_hostname(yes); >>>>> use_dns(no); >>>>> use_fqdn(no); >>>>> stats_freq(600); >>>>> stats_level(2); >>>>> # Allow large messages >>>>> log_msg_size(65536); >>>>> }; >>>>> >>>>> # ===================== >>>>> # UDP Packet Source >>>>> source s_udp { >>>>> udp(); >>>>> }; >>>>> >>>>> # ===================== >>>>> # TCP Packet Source >>>>> source s_tcp { >>>>> tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); >>>>> }; >>>>> >>>>> # ===================== >>>>> destination net_perhost { >>>>> file("/data/log/per-host/$HOST" >>>>> owner(root) >>>>> group(nwadmin) >>>>> perm(0775) >>>>> ); >>>>> }; >>>>> >>>>> # ===================== >>>>> log { >>>>> source(s_tcp); >>>>> source(s_udp); >>>>> destination(net_perhost); >>>>> }; >>>>> ===== >>>>> >>>>> I've checked around for perhaps a different spool name, thinking perhaps >>>>> the data was getting recognized as something other than it's source-ip, >>>>> but haven't seen anything. >>>>> >>>>> Any thoughts? >>>>> >>>>> Cheers, >>>>> -Chris >>>>> ______________________________________________________________________________ >>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> >>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>> >>>> ______________________________________________________________________________ >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> >>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
That's a great idea Jim. I'll give that some testing. Thank you for the suggestion. Be back in touch soon. -Chris On 3/6/14 7:05 AM, Jim Hendrick wrote:
one other thought (basic and probably unrelated)
we run about a dozen syslog-ng log aggregators for thousands of devices in multiple data centers, and found performance was much faster / more reliable by using $HOST_FROM instead of $HOST. "If the /|keep_hostname()|/ option is enabled (/|keep_hostname(yes)|/), the value of the $HOST macro will be the hostname retrieved from the log message"
there might be parsing options with the log format that would be bypassed by using $HOST_FROM which is" "Name of the host that sent the message to syslog-ng, as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain."
and since you have use_dns(no) it will be the IP address (unless for some reason the receiving server has the name without using DNS - like its own)
anyway this seems to work well for us.
Jim
On 03/05/2014 11:11 PM, Evan Rempel wrote:
Do you have any log rotation? Is it possible that the log files got rotated/deleted but syslog still has the file handle open and continues to write to it. Since it never gets closed, it never needs to reopen it so it never detects the missing filename and never creates a new one.
you can use lsof and search for "delete"
Evan Rempel 250.271.7691 University Systems, University of Victoria
Chris Moody<chris@node-nine.com> wrote:
ok - sorry for the latent reply folks - got wrangled into other troubleshooting efforts.
Anyway, I've taken a deeper look at this and I'm experiencing this behavior when receiving logs from IOS, Nexus, and other device types as well..so it's not just some weird version/bug type issue with the received messages.
The log hosts are both running: === syslog-ng 3.2.5 ===
These are RHEL 6.2 VirtualMachines, each logging to their own dedicated 5-TB NFS mount. Plenty of CPU & Memory overhead. I'm not seeing any issues with I/O-wait to the NFS mounts. These hosts are -very- busy (from loads of firewall logs) but have been working beautifully in the past. This seems to be a relatively recent development...not sure how long though unfortunately.
There are about 1000 current active spools on each host that are updating just fine when new messages come in. I'm just experiencing where some of the spool files have gone stale and don't get written to any longer (despite messages being received) as well as no new spools get created. It's like the destination directive isn't being adhered-to. === destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0755) ); }; === This should be creating a new log spool per-host upon receipt of logs over the wire...and it's working...but not 100% any longer it seems.
I've done tcpdump captures of the nodes that I'm having trouble with and I do see the source-IP in all the 'hostname' fields...so it's not like these are coming through as malformed which was a good first-pass thought. Even tried toggling 'keep_hostname' to no (currently 'yes') and that doesn't seem to help.
I have tried sending logs from some new systems to these aggregators and they are not creating new spools either. I've run tcpdump to confirm the message receipt, but no new files are being written for new devices either.
It's almost behaving like perhaps there's too many files being written to...too many open filehandles... something along these lines perhaps.
Hoping for some other outside-perspective ideas of things I can check or debug as I've been trying to debug this too long...and am most likely staring at the issue right in the face.
-Chris
On 2/22/14 6:44 AM, Balazs Scheidler wrote:
Latest syslog-ng versions parse some of the cisco extensions. Which version do you run?
On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
It wasn't adding the data to the hostname just adding extra header data that broke the RFC format.
On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com <mailto:chris@node-nine.com>> wrote:
> Hmm... that's a thought. The troublesome device is an IOS system. > I'llgive 'er a gander to see if there are any other options. I don't > recall there being any that controlled the 'hostname' header field though. > > -Chris > > On 2/18/14 5:10 PM, Scot Needy wrote: >> We had a parsing problem on our ASA where the log contained an extra date so the Host looked like "Feb". >> >> There was a syslog option in the ASA not to send the date in the header. >> >> On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com <mailto:chris@node-nine.com>> wrote: >> >>> yes - there are tons of spool files being created successfully. As any >>> new network device starts logging we see a new log-spool get created for >>> it's source-ip. >>> >>> Tons of free disk space - almost a Tb of free room. Loads of >>> processor/mem overhead. Nothing glaring in syslog-ng's logs (like >>> unable to write or whatnot) >>> >>> Just debugging a host-device that we're not seeing logs accounted for. >>> >>> -Chris >>> >>> On 2/18/14 3:51 PM, Austin Jorden wrote: >>>> Hi Chris, >>>> >>>> Are there *any* folders/files being created at all? >>>> >>>> There's one thing I noticed that isn't specified... which is the >>>> "createdirs = Yes" option. It appears (well, I assume) that you're >>>> wanting it to create a separate text file for each $HOST, not a separate >>>> directory named $HOST... >>>> >>>> - Austin >>>> >>>> On 2/18/2014 2:12 PM, Chris Moody wrote: >>>>> Hello. >>>>> >>>>> First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome >>>>> code for years now. I've built all sorts of logging infrastructure with >>>>> it. >>>>> >>>>> I seem to have hit on something though that's got me scratching my head >>>>> and lacking for explanation. Perhaps I've just been staring at it and >>>>> debugging it too long and am missing something obvious. >>>>> >>>>> I've got an installation with a couple thousand network devices logging >>>>> successfully to output spools on our log aggretor. This is rockin' and >>>>> works beautifully. I've got things configured whereby each network >>>>> source logs to it's own individual spool file with the source-ip as the >>>>> spool name. >>>>> >>>>> I'm running into a case though where I have a Cisco switch sending logs >>>>> to my log aggregator but the log-server isn't writing the output to the >>>>> device's spool file. It is working however for many many more devices >>>>> just like this switch. >>>>> >>>>> I've confirmed via tcpdump that this log traffic does actually hit the >>>>> box, but it never gets recorded into the log spool for that network device. >>>>> >>>>> Since the host is -super- busy receiving logs from other gear >>>>> enterprise-wide, I have to treat it very gingerly, so can't enable too >>>>> much debugging...but I'm really confused why the logs wouldn't show up >>>>> in the log spool.. >>>>> >>>>> Here's some bits of the config that are relevant: >>>>> ===== >>>>> options { >>>>> keep_hostname(yes); >>>>> use_dns(no); >>>>> use_fqdn(no); >>>>> stats_freq(600); >>>>> stats_level(2); >>>>> # Allow large messages >>>>> log_msg_size(65536); >>>>> }; >>>>> >>>>> # ===================== >>>>> # UDP Packet Source >>>>> source s_udp { >>>>> udp(); >>>>> }; >>>>> >>>>> # ===================== >>>>> # TCP Packet Source >>>>> source s_tcp { >>>>> tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); >>>>> }; >>>>> >>>>> # ===================== >>>>> destination net_perhost { >>>>> file("/data/log/per-host/$HOST" >>>>> owner(root) >>>>> group(nwadmin) >>>>> perm(0775) >>>>> ); >>>>> }; >>>>> >>>>> # ===================== >>>>> log { >>>>> source(s_tcp); >>>>> source(s_udp); >>>>> destination(net_perhost); >>>>> }; >>>>> ===== >>>>> >>>>> I've checked around for perhaps a different spool name, thinking perhaps >>>>> the data was getting recognized as something other than it's source-ip, >>>>> but haven't seen anything. >>>>> >>>>> Any thoughts? >>>>> >>>>> Cheers, >>>>> -Chris >>>>> ______________________________________________________________________________ >>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> >>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>>> >>>> ______________________________________________________________________________ >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> >>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng FAQ:http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng FAQ:http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
It may very well be that the HOST_FROM value is more consistent. Performance wise it should be the same. On Mar 6, 2014 1:05 PM, "Jim Hendrick" <jrhendri@roadrunner.com> wrote:
one other thought (basic and probably unrelated)
we run about a dozen syslog-ng log aggregators for thousands of devices in multiple data centers, and found performance was much faster / more reliable by using $HOST_FROM instead of $HOST. "If the *keep_hostname()* option is enabled (*keep_hostname(yes)*), the value of the $HOST macro will be the hostname retrieved from the log message"
there might be parsing options with the log format that would be bypassed by using $HOST_FROM which is" "Name of the host that sent the message to syslog-ng, as resolved by syslog-ng using DNS. If the message traverses several hosts, this is the last host in the chain."
and since you have use_dns(no) it will be the IP address (unless for some reason the receiving server has the name without using DNS - like its own)
anyway this seems to work well for us.
Jim
On 03/05/2014 11:11 PM, Evan Rempel wrote:
Do you have any log rotation? Is it possible that the log files got rotated/deleted but syslog still has the file handle open and continues to write to it. Since it never gets closed, it never needs to reopen it so it never detects the missing filename and never creates a new one.
you can use lsof and search for "delete"
Evan Rempel 250.271.7691 University Systems, University of Victoria
Chris Moody <chris@node-nine.com> <chris@node-nine.com> wrote:
ok - sorry for the latent reply folks - got wrangled into other troubleshooting efforts.
Anyway, I've taken a deeper look at this and I'm experiencing this behavior when receiving logs from IOS, Nexus, and other device types as well..so it's not just some weird version/bug type issue with the received messages.
The log hosts are both running: === syslog-ng 3.2.5 ===
These are RHEL 6.2 VirtualMachines, each logging to their own dedicated 5-TB NFS mount. Plenty of CPU & Memory overhead. I'm not seeing any issues with I/O-wait to the NFS mounts. These hosts are -very- busy (from loads of firewall logs) but have been working beautifully in the past. This seems to be a relatively recent development...not sure how long though unfortunately.
There are about 1000 current active spools on each host that are updating just fine when new messages come in. I'm just experiencing where some of the spool files have gone stale and don't get written to any longer (despite messages being received) as well as no new spools get created. It's like the destination directive isn't being adhered-to. === destination net_perhost { file("/data/log/per-host/$HOST" owner(root) group(nwadmin) perm(0755) ); }; === This should be creating a new log spool per-host upon receipt of logs over the wire...and it's working...but not 100% any longer it seems.
I've done tcpdump captures of the nodes that I'm having trouble with and I do see the source-IP in all the 'hostname' fields...so it's not like these are coming through as malformed which was a good first-pass thought. Even tried toggling 'keep_hostname' to no (currently 'yes') and that doesn't seem to help.
I have tried sending logs from some new systems to these aggregators and they are not creating new spools either. I've run tcpdump to confirm the message receipt, but no new files are being written for new devices either.
It's almost behaving like perhaps there's too many files being written to...too many open filehandles... something along these lines perhaps.
Hoping for some other outside-perspective ideas of things I can check or debug as I've been trying to debug this too long...and am most likely staring at the issue right in the face.
-Chris
On 2/22/14 6:44 AM, Balazs Scheidler wrote:
Latest syslog-ng versions parse some of the cisco extensions. Which version do you run? On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn@gmail.com> wrote:
It wasn't adding the data to the hostname just adding extra header data that broke the RFC format.
On Feb 18, 2014, at 5:14 PM, Chris Moody <chris@node-nine.com> wrote:
Hmm... that's a thought. The troublesome device is an IOS system. I'llgive 'er a gander to see if there are any other options. I don't recall there being any that controlled the 'hostname' header field though.
-Chris
On 2/18/14 5:10 PM, Scot Needy wrote:
We had a parsing problem on our ASA where the log contained an extra date so the Host looked like "Feb".
There was a syslog option in the ASA not to send the date in the header.
On Feb 18, 2014, at 4:59 PM, Chris Moody <chris@node-nine.com> wrote:
yes - there are tons of spool files being created successfully. As any new network device starts logging we see a new log-spool get created for it's source-ip.
Tons of free disk space - almost a Tb of free room. Loads of processor/mem overhead. Nothing glaring in syslog-ng's logs (like unable to write or whatnot)
Just debugging a host-device that we're not seeing logs accounted for.
-Chris
On 2/18/14 3:51 PM, Austin Jorden wrote:
Hi Chris,
Are there *any* folders/files being created at all?
There's one thing I noticed that isn't specified... which is the "createdirs = Yes" option. It appears (well, I assume) that you're wanting it to create a separate text file for each $HOST, not a separate directory named $HOST...
- Austin
On 2/18/2014 2:12 PM, Chris Moody wrote: > Hello. > > First off, thanks a __TON__ for syslog-ng. I've sworn by this awesome > code for years now. I've built all sorts of logging infrastructure with > it. > > I seem to have hit on something though that's got me scratching my head > and lacking for explanation. Perhaps I've just been staring at it and > debugging it too long and am missing something obvious. > > I've got an installation with a couple thousand network devices logging > successfully to output spools on our log aggretor. This is rockin' and > works beautifully. I've got things configured whereby each network > source logs to it's own individual spool file with the source-ip as the > spool name. > > I'm running into a case though where I have a Cisco switch sending logs > to my log aggregator but the log-server isn't writing the output to the > device's spool file. It is working however for many many more devices > just like this switch. > > I've confirmed via tcpdump that this log traffic does actually hit the > box, but it never gets recorded into the log spool for that network device. > > Since the host is -super- busy receiving logs from other gear > enterprise-wide, I have to treat it very gingerly, so can't enable too > much debugging...but I'm really confused why the logs wouldn't show up > in the log spool.. > > Here's some bits of the config that are relevant: > ===== > options { > keep_hostname(yes); > use_dns(no); > use_fqdn(no); > stats_freq(600); > stats_level(2); > # Allow large messages > log_msg_size(65536); > }; > > # ===================== > # UDP Packet Source > source s_udp { > udp(); > }; > > # ===================== > # TCP Packet Source > source s_tcp { > tcp(ip(aaa.bbb.ccc.ddd) port(514) max-connections(50000)); > }; > > # ===================== > destination net_perhost { > file("/data/log/per-host/$HOST" > owner(root) > group(nwadmin) > perm(0775) > ); > }; > > # ===================== > log { > source(s_tcp); > source(s_udp); > destination(net_perhost); > }; > ===== > > I've checked around for perhaps a different spool name, thinking perhaps > the data was getting recognized as something other than it's source-ip, > but haven't seen anything. > > Any thoughts? > > Cheers, > -Chris >
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq >
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 18.02.2014 21:12, Chris Moody wrote:
I'm running into a case though where I have a Cisco switch sending logs to my log aggregator but the log-server isn't writing the output to the device's spool file. It is working however for many many more devices just like this switch.
I've confirmed via tcpdump that this log traffic does actually hit the box, but it never gets recorded into the log spool for that network device.
Most likely your Cisco device sends logs which are not conforming to syslog standard (as in: format). Try pointing this device to a source() with "flags(no-parse)" set. See Administrator Guide for more details. HTH. -- Jakub Jankowski|shasta@toxcorp.com|http://toxcorp.com/ GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
participants (8)
-
Austin Jorden
-
Balazs Scheidler
-
Chris Moody
-
Evan Rempel
-
Jakub Jankowski
-
Jim Hendrick
-
Scot Needy
-
Shashank Rohatgi