[syslog-ng] need help debugging some network received logs that aren't writing to files
Chris Moody
chris at node-nine.com
Thu Mar 6 03:56:50 CET 2014
ok - sorry for the latent reply folks - got wrangled into other
troubleshooting efforts.
Anyway, I've taken a deeper look at this and I'm experiencing this
behavior when receiving logs from IOS, Nexus, and other device types as
well..so it's not just some weird version/bug type issue with the
received messages.
The log hosts are both running:
===
syslog-ng 3.2.5
===
These are RHEL 6.2 VirtualMachines, each logging to their own dedicated
5-TB NFS mount. Plenty of CPU & Memory overhead. I'm not seeing any
issues with I/O-wait to the NFS mounts. These hosts are -very- busy
(from loads of firewall logs) but have been working beautifully in the
past. This seems to be a relatively recent development...not sure how
long though unfortunately.
There are about 1000 current active spools on each host that are
updating just fine when new messages come in. I'm just experiencing
where some of the spool files have gone stale and don't get written to
any longer (despite messages being received) as well as no new spools
get created. It's like the destination directive isn't being adhered-to.
===
destination net_perhost {
file("/data/log/per-host/$HOST"
owner(root)
group(nwadmin)
perm(0755)
);
};
===
This should be creating a new log spool per-host upon receipt of logs
over the wire...and it's working...but not 100% any longer it seems.
I've done tcpdump captures of the nodes that I'm having trouble with and
I do see the source-IP in all the 'hostname' fields...so it's not like
these are coming through as malformed which was a good first-pass
thought. Even tried toggling 'keep_hostname' to no (currently 'yes')
and that doesn't seem to help.
I have tried sending logs from some new systems to these aggregators and
they are not creating new spools either. I've run tcpdump to confirm
the message receipt, but no new files are being written for new devices
either.
It's almost behaving like perhaps there's too many files being written
to...too many open filehandles... something along these lines perhaps.
Hoping for some other outside-perspective ideas of things I can check or
debug as I've been trying to debug this too long...and am most likely
staring at the issue right in the face.
-Chris
On 2/22/14 6:44 AM, Balazs Scheidler wrote:
>
> Latest syslog-ng versions parse some of the cisco extensions. Which
> version do you run?
>
> On Feb 18, 2014 11:20 PM, "Scot Needy" <scotrn at gmail.com
> <mailto:scotrn at gmail.com>> wrote:
>
> It wasn't adding the data to the hostname just adding extra header
> data that broke the RFC format.
>
>
> On Feb 18, 2014, at 5:14 PM, Chris Moody <chris at node-nine.com
> <mailto:chris at node-nine.com>> wrote:
>
> > Hmm... that's a thought. The troublesome device is an IOS system.
> > I'llgive 'er a gander to see if there are any other options. I
> don't
> > recall there being any that controlled the 'hostname' header
> field though.
> >
> > -Chris
> >
> > On 2/18/14 5:10 PM, Scot Needy wrote:
> >> We had a parsing problem on our ASA where the log contained an
> extra date so the Host looked like "Feb".
> >>
> >> There was a syslog option in the ASA not to send the date in
> the header.
> >>
> >> On Feb 18, 2014, at 4:59 PM, Chris Moody <chris at node-nine.com
> <mailto:chris at node-nine.com>> wrote:
> >>
> >>> yes - there are tons of spool files being created
> successfully. As any
> >>> new network device starts logging we see a new log-spool get
> created for
> >>> it's source-ip.
> >>>
> >>> Tons of free disk space - almost a Tb of free room. Loads of
> >>> processor/mem overhead. Nothing glaring in syslog-ng's logs (like
> >>> unable to write or whatnot)
> >>>
> >>> Just debugging a host-device that we're not seeing logs
> accounted for.
> >>>
> >>> -Chris
> >>>
> >>> On 2/18/14 3:51 PM, Austin Jorden wrote:
> >>>> Hi Chris,
> >>>>
> >>>> Are there *any* folders/files being created at all?
> >>>>
> >>>> There's one thing I noticed that isn't specified... which is the
> >>>> "createdirs = Yes" option. It appears (well, I assume) that
> you're
> >>>> wanting it to create a separate text file for each $HOST, not
> a separate
> >>>> directory named $HOST...
> >>>>
> >>>> - Austin
> >>>>
> >>>> On 2/18/2014 2:12 PM, Chris Moody wrote:
> >>>>> Hello.
> >>>>>
> >>>>> First off, thanks a __TON__ for syslog-ng. I've sworn by
> this awesome
> >>>>> code for years now. I've built all sorts of logging
> infrastructure with
> >>>>> it.
> >>>>>
> >>>>> I seem to have hit on something though that's got me
> scratching my head
> >>>>> and lacking for explanation. Perhaps I've just been staring
> at it and
> >>>>> debugging it too long and am missing something obvious.
> >>>>>
> >>>>> I've got an installation with a couple thousand network
> devices logging
> >>>>> successfully to output spools on our log aggretor. This is
> rockin' and
> >>>>> works beautifully. I've got things configured whereby each
> network
> >>>>> source logs to it's own individual spool file with the
> source-ip as the
> >>>>> spool name.
> >>>>>
> >>>>> I'm running into a case though where I have a Cisco switch
> sending logs
> >>>>> to my log aggregator but the log-server isn't writing the
> output to the
> >>>>> device's spool file. It is working however for many many
> more devices
> >>>>> just like this switch.
> >>>>>
> >>>>> I've confirmed via tcpdump that this log traffic does
> actually hit the
> >>>>> box, but it never gets recorded into the log spool for that
> network device.
> >>>>>
> >>>>> Since the host is -super- busy receiving logs from other gear
> >>>>> enterprise-wide, I have to treat it very gingerly, so can't
> enable too
> >>>>> much debugging...but I'm really confused why the logs
> wouldn't show up
> >>>>> in the log spool..
> >>>>>
> >>>>> Here's some bits of the config that are relevant:
> >>>>> =====
> >>>>> options {
> >>>>> keep_hostname(yes);
> >>>>> use_dns(no);
> >>>>> use_fqdn(no);
> >>>>> stats_freq(600);
> >>>>> stats_level(2);
> >>>>> # Allow large messages
> >>>>> log_msg_size(65536);
> >>>>> };
> >>>>>
> >>>>> # =====================
> >>>>> # UDP Packet Source
> >>>>> source s_udp {
> >>>>> udp();
> >>>>> };
> >>>>>
> >>>>> # =====================
> >>>>> # TCP Packet Source
> >>>>> source s_tcp {
> >>>>> tcp(ip(aaa.bbb.ccc.ddd) port(514)
> max-connections(50000));
> >>>>> };
> >>>>>
> >>>>> # =====================
> >>>>> destination net_perhost {
> >>>>> file("/data/log/per-host/$HOST"
> >>>>> owner(root)
> >>>>> group(nwadmin)
> >>>>> perm(0775)
> >>>>> );
> >>>>> };
> >>>>>
> >>>>> # =====================
> >>>>> log {
> >>>>> source(s_tcp);
> >>>>> source(s_udp);
> >>>>> destination(net_perhost);
> >>>>> };
> >>>>> =====
> >>>>>
> >>>>> I've checked around for perhaps a different spool name,
> thinking perhaps
> >>>>> the data was getting recognized as something other than it's
> source-ip,
> >>>>> but haven't seen anything.
> >>>>>
> >>>>> Any thoughts?
> >>>>>
> >>>>> Cheers,
> >>>>> -Chris
> >>>>>
> ______________________________________________________________________________
> >>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>>>> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>>>
> >>>>
> ______________________________________________________________________________
> >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>>> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>>
> >>>
> ______________________________________________________________________________
> >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>
> >>
> ______________________________________________________________________________
> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140305/af415a8c/attachment-0001.htm
More information about the syslog-ng
mailing list