<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>ok - sorry for the latent reply folks - got wrangled into other
troubleshooting efforts.<br>
<br>
Anyway, I've taken a deeper look at this and I'm experiencing this
behavior when receiving logs from IOS, Nexus, and other device
types as well..so it's not just some weird version/bug type issue
with the received messages.<br>
<br>
The log hosts are both running:<br>
===<br>
syslog-ng 3.2.5<br>
===<br>
<br>
These are RHEL 6.2 VirtualMachines, each logging to their own
dedicated 5-TB NFS mount. Plenty of CPU & Memory overhead.
I'm not seeing any issues with I/O-wait to the NFS mounts. These
hosts are -very- busy (from loads of firewall logs) but have been
working beautifully in the past. This seems to be a relatively
recent development...not sure how long though unfortunately.<br>
<br>
There are about 1000 current active spools on each host that are
updating just fine when new messages come in. I'm just
experiencing where some of the spool files have gone stale and
don't get written to any longer (despite messages being received)
as well as no new spools get created. It's like the destination
directive isn't being adhered-to.<br>
===<br>
destination net_perhost {<br>
file("/data/log/per-host/$HOST"<br>
owner(root)<br>
group(nwadmin)<br>
perm(0755)<br>
);<br>
};<br>
===<br>
This should be creating a new log spool per-host upon receipt of
logs over the wire...and it's working...but not 100% any longer it
seems.<br>
<br>
I've done tcpdump captures of the nodes that I'm having trouble
with and I do see the source-IP in all the 'hostname' fields...so
it's not like these are coming through as malformed which was a
good first-pass thought. Even tried toggling 'keep_hostname' to
no (currently 'yes') and that doesn't seem to help.<br>
<br>
I have tried sending logs from some new systems to these
aggregators and they are not creating new spools either. I've run
tcpdump to confirm the message receipt, but no new files are being
written for new devices either.<br>
<br>
It's almost behaving like perhaps there's too many files being
written to...too many open filehandles... something along these
lines perhaps. <br>
<br>
Hoping for some other outside-perspective ideas of things I can
check or debug as I've been trying to debug this too long...and am
most likely staring at the issue right in the face.<br>
<br>
-Chris<br>
<br>
</tt>
<div class="moz-cite-prefix">On 2/22/14 6:44 AM, Balazs Scheidler
wrote:<br>
</div>
<blockquote
cite="mid:CAKcfE+Z1guCXJFSWuzU3w6XD-XjG_ize4ac=vW69MjMvpvcPkw@mail.gmail.com"
type="cite">
<p dir="ltr">Latest syslog-ng versions parse some of the cisco
extensions. Which version do you run?</p>
<div class="gmail_quote">On Feb 18, 2014 11:20 PM, "Scot Needy"
<<a moz-do-not-send="true" href="mailto:scotrn@gmail.com">scotrn@gmail.com</a>>
wrote:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
It wasn’t adding the data to the hostname just adding extra
header data that broke the RFC format.<br>
<br>
<br>
On Feb 18, 2014, at 5:14 PM, Chris Moody <<a
moz-do-not-send="true" href="mailto:chris@node-nine.com">chris@node-nine.com</a>>
wrote:<br>
<br>
> Hmm... that's a thought. The troublesome device is an
IOS system.<br>
> I'llgive 'er a gander to see if there are any other
options. I don't<br>
> recall there being any that controlled the 'hostname'
header field though.<br>
><br>
> -Chris<br>
><br>
> On 2/18/14 5:10 PM, Scot Needy wrote:<br>
>> We had a parsing problem on our ASA where the log
contained an extra date so the Host looked like “Feb”.<br>
>><br>
>> There was a syslog option in the ASA not to send the
date in the header.<br>
>><br>
>> On Feb 18, 2014, at 4:59 PM, Chris Moody <<a
moz-do-not-send="true" href="mailto:chris@node-nine.com">chris@node-nine.com</a>>
wrote:<br>
>><br>
>>> yes - there are tons of spool files being created
successfully. As any<br>
>>> new network device starts logging we see a new
log-spool get created for<br>
>>> it's source-ip.<br>
>>><br>
>>> Tons of free disk space - almost a Tb of free
room. Loads of<br>
>>> processor/mem overhead. Nothing glaring in
syslog-ng's logs (like<br>
>>> unable to write or whatnot)<br>
>>><br>
>>> Just debugging a host-device that we're not
seeing logs accounted for.<br>
>>><br>
>>> -Chris<br>
>>><br>
>>> On 2/18/14 3:51 PM, Austin Jorden wrote:<br>
>>>> Hi Chris,<br>
>>>><br>
>>>> Are there *any* folders/files being created
at all?<br>
>>>><br>
>>>> There's one thing I noticed that isn't
specified... which is the<br>
>>>> "createdirs = Yes" option. It appears (well,
I assume) that you're<br>
>>>> wanting it to create a separate text file for
each $HOST, not a separate<br>
>>>> directory named $HOST...<br>
>>>><br>
>>>> - Austin<br>
>>>><br>
>>>> On 2/18/2014 2:12 PM, Chris Moody wrote:<br>
>>>>> Hello.<br>
>>>>><br>
>>>>> First off, thanks a __TON__ for
syslog-ng. I've sworn by this awesome<br>
>>>>> code for years now. I've built all sorts
of logging infrastructure with<br>
>>>>> it.<br>
>>>>><br>
>>>>> I seem to have hit on something though
that's got me scratching my head<br>
>>>>> and lacking for explanation. Perhaps
I've just been staring at it and<br>
>>>>> debugging it too long and am missing
something obvious.<br>
>>>>><br>
>>>>> I've got an installation with a couple
thousand network devices logging<br>
>>>>> successfully to output spools on our log
aggretor. This is rockin' and<br>
>>>>> works beautifully. I've got things
configured whereby each network<br>
>>>>> source logs to it's own individual spool
file with the source-ip as the<br>
>>>>> spool name.<br>
>>>>><br>
>>>>> I'm running into a case though where I
have a Cisco switch sending logs<br>
>>>>> to my log aggregator but the log-server
isn't writing the output to the<br>
>>>>> device's spool file. It is working
however for many many more devices<br>
>>>>> just like this switch.<br>
>>>>><br>
>>>>> I've confirmed via tcpdump that this log
traffic does actually hit the<br>
>>>>> box, but it never gets recorded into the
log spool for that network device.<br>
>>>>><br>
>>>>> Since the host is -super- busy receiving
logs from other gear<br>
>>>>> enterprise-wide, I have to treat it very
gingerly, so can't enable too<br>
>>>>> much debugging...but I'm really confused
why the logs wouldn't show up<br>
>>>>> in the log spool..<br>
>>>>><br>
>>>>> Here's some bits of the config that are
relevant:<br>
>>>>> =====<br>
>>>>> options {<br>
>>>>> keep_hostname(yes);<br>
>>>>> use_dns(no);<br>
>>>>> use_fqdn(no);<br>
>>>>> stats_freq(600);<br>
>>>>> stats_level(2);<br>
>>>>> # Allow large messages<br>
>>>>> log_msg_size(65536);<br>
>>>>> };<br>
>>>>><br>
>>>>> # =====================<br>
>>>>> # UDP Packet Source<br>
>>>>> source s_udp {<br>
>>>>> udp();<br>
>>>>> };<br>
>>>>><br>
>>>>> # =====================<br>
>>>>> # TCP Packet Source<br>
>>>>> source s_tcp {<br>
>>>>> tcp(ip(aaa.bbb.ccc.ddd)
port(514) max-connections(50000));<br>
>>>>> };<br>
>>>>><br>
>>>>> # =====================<br>
>>>>> destination net_perhost {<br>
>>>>> file("/data/log/per-host/$HOST"<br>
>>>>> owner(root)<br>
>>>>> group(nwadmin)<br>
>>>>> perm(0775)<br>
>>>>> );<br>
>>>>> };<br>
>>>>><br>
>>>>> # =====================<br>
>>>>> log {<br>
>>>>> source(s_tcp);<br>
>>>>> source(s_udp);<br>
>>>>> destination(net_perhost);<br>
>>>>> };<br>
>>>>> =====<br>
>>>>><br>
>>>>> I've checked around for perhaps a
different spool name, thinking perhaps<br>
>>>>> the data was getting recognized as
something other than it's source-ip,<br>
>>>>> but haven't seen anything.<br>
>>>>><br>
>>>>> Any thoughts?<br>
>>>>><br>
>>>>> Cheers,<br>
>>>>> -Chris<br>
>>>>>
______________________________________________________________________________<br>
>>>>> Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
>>>>> Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
>>>>> FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
>>>>><br>
>>>>
______________________________________________________________________________<br>
>>>> Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
>>>> Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
>>>> FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
>>>><br>
>>>
______________________________________________________________________________<br>
>>> Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
>>> Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
>>> FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
>>><br>
>>
______________________________________________________________________________<br>
>> Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
>> Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
>> FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
>><br>
><br>
>
______________________________________________________________________________<br>
> Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
><br>
<br>
______________________________________________________________________________<br>
Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a moz-do-not-send="true"
href="http://www.balabit.com/wiki/syslog-ng-faq"
target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</body>
</html>