<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    one other thought (basic and probably unrelated)<br>
    <br>
    we run about a dozen syslog-ng log aggregators for thousands of
    devices in multiple data centers, and found performance was much
    faster / more reliable by using $HOST_FROM instead of $HOST.<br>
    "If the <em class="parameter"><code>keep_hostname()</code></em>
    option is enabled (<em class="parameter"><code>keep_hostname(yes)</code></em>),
    the value of the $HOST macro will be the hostname retrieved from the
    log message"<br>
    <br>
    there might be parsing options with the log format that would be
    bypassed by using $HOST_FROM which is"<br>
    "Name of the host that sent the message to syslog-ng, as resolved by
    syslog-ng using DNS. If the message traverses several hosts, this is
    the
    last host in the chain."<br>
    <br>
    and since you have use_dns(no) it will be the IP address (unless for
    some reason the receiving server has the name without using DNS -
    like its own)<br>
    <br>
    anyway this seems to work well for us.<br>
    <br>
    Jim<br>
    <br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 03/05/2014 11:11 PM, Evan Rempel
      wrote:<br>
    </div>
    <blockquote
      cite="mid:tf0anrgu13rjgwt6sank7qd1.1394079081925@email.android.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=ISO-8859-1">
      <pre style="word-wrap:break-word; font-size:10.0pt; font-family:Tahoma; color:black">Do you have any log rotation?
Is it possible that the log files got rotated/deleted but syslog still has the file handle open and continues to write to it. Since it never gets closed, it never needs to reopen it so it never detects the missing filename and never creates a new one.

you can use lsof and search for "delete"



Evan Rempel   250.271.7691
University Systems, University of Victoria

Chris Moody <a class="moz-txt-link-rfc2396E" href="mailto:chris@node-nine.com">&lt;chris@node-nine.com&gt;</a> wrote:

</pre>
      <div><tt>ok - sorry for the latent reply folks - got wrangled into
          other troubleshooting efforts.<br>
          <br>
          Anyway, I've taken a deeper look at this and I'm experiencing
          this behavior when receiving logs from IOS, Nexus, and other
          device types as well..so it's not just some weird version/bug
          type issue with the received messages.<br>
          <br>
          The log hosts are both running:<br>
          ===<br>
          syslog-ng 3.2.5<br>
          ===<br>
          <br>
          These are RHEL 6.2 VirtualMachines, each logging to their own
          dedicated 5-TB NFS mount.&nbsp; Plenty of CPU &amp; Memory
          overhead.&nbsp; I'm not seeing any issues with I/O-wait to the NFS
          mounts.&nbsp; These hosts are -very- busy (from loads of firewall
          logs) but have been working beautifully in the past.&nbsp; This
          seems to be a relatively recent development...not sure how
          long though unfortunately.<br>
          <br>
          There are about 1000 current active spools on each host that
          are updating just fine when new messages come in.&nbsp; I'm just
          experiencing where some of the spool files have gone stale and
          don't get written to any longer (despite messages being
          received) as well as no new spools get created.&nbsp; It's like the
          destination directive isn't being adhered-to.<br>
          ===<br>
          destination net_perhost {<br>
          &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; file("/data/log/per-host/$HOST"<br>
          &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; owner(root)<br>
          &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; group(nwadmin)<br>
          &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; perm(0755)<br>
          &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; );<br>
          };<br>
          ===<br>
          This should be creating a new log spool per-host upon receipt
          of logs over the wire...and it's working...but not 100% any
          longer it seems.<br>
          <br>
          I've done tcpdump captures of the nodes that I'm having
          trouble with and I do see the source-IP in all the 'hostname'
          fields...so it's not like these are coming through as
          malformed which was a good first-pass thought.&nbsp; Even tried
          toggling 'keep_hostname' to no (currently 'yes') and that
          doesn't seem to help.<br>
          <br>
          I have tried sending logs from some new systems to these
          aggregators and they are not creating new spools either.&nbsp; I've
          run tcpdump to confirm the message receipt, but no new files
          are being written for new devices either.<br>
          <br>
          It's almost behaving like perhaps there's too many files being
          written to...too many open filehandles... something along
          these lines perhaps.&nbsp;
          <br>
          <br>
          Hoping for some other outside-perspective ideas of things I
          can check or debug as I've been trying to debug this too
          long...and am most likely staring at the issue right in the
          face.<br>
          <br>
          -Chris<br>
          <br>
        </tt>
        <div class="moz-cite-prefix">On 2/22/14 6:44 AM, Balazs
          Scheidler wrote:<br>
        </div>
        <blockquote type="cite">
          <p dir="ltr">Latest syslog-ng versions parse some of the cisco
            extensions. Which version do you run?</p>
          <div class="gmail_quote">On Feb 18, 2014 11:20 PM, "Scot
            Needy" &lt;<a moz-do-not-send="true"
              href="mailto:scotrn@gmail.com">scotrn@gmail.com</a>&gt;
            wrote:<br type="attribution">
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;
              border-left:1px #ccc solid; padding-left:1ex">
              It wasn&#8217;t adding the data to the hostname just adding
              extra header data that broke the RFC &nbsp;format.<br>
              <br>
              <br>
              On Feb 18, 2014, at 5:14 PM, Chris Moody &lt;<a
                moz-do-not-send="true" href="mailto:chris@node-nine.com">chris@node-nine.com</a>&gt;
              wrote:<br>
              <br>
              &gt; Hmm... that's a thought. &nbsp; The troublesome device is
              an IOS system.<br>
              &gt; I'llgive 'er a gander to see if there are any other
              options. &nbsp;I don't<br>
              &gt; recall there being any that controlled the 'hostname'
              header field though.<br>
              &gt;<br>
              &gt; -Chris<br>
              &gt;<br>
              &gt; On 2/18/14 5:10 PM, Scot Needy wrote:<br>
              &gt;&gt; We had a parsing problem on our ASA where the log
              contained an extra date so the Host looked like &#8220;Feb&#8221;.<br>
              &gt;&gt;<br>
              &gt;&gt; There was a syslog option in the ASA not to send
              the date in the header.<br>
              &gt;&gt;<br>
              &gt;&gt; On Feb 18, 2014, at 4:59 PM, Chris Moody &lt;<a
                moz-do-not-send="true" href="mailto:chris@node-nine.com">chris@node-nine.com</a>&gt;
              wrote:<br>
              &gt;&gt;<br>
              &gt;&gt;&gt; yes - there are tons of spool files being
              created successfully. As any<br>
              &gt;&gt;&gt; new network device starts logging we see a
              new log-spool get created for<br>
              &gt;&gt;&gt; it's source-ip.<br>
              &gt;&gt;&gt;<br>
              &gt;&gt;&gt; Tons of free disk space - almost a Tb of free
              room. &nbsp;Loads of<br>
              &gt;&gt;&gt; processor/mem overhead. &nbsp;Nothing glaring in
              syslog-ng's logs (like<br>
              &gt;&gt;&gt; unable to write or whatnot)<br>
              &gt;&gt;&gt;<br>
              &gt;&gt;&gt; Just debugging a host-device that we're not
              seeing logs accounted for.<br>
              &gt;&gt;&gt;<br>
              &gt;&gt;&gt; -Chris<br>
              &gt;&gt;&gt;<br>
              &gt;&gt;&gt; On 2/18/14 3:51 PM, Austin Jorden wrote:<br>
              &gt;&gt;&gt;&gt; Hi Chris,<br>
              &gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt; Are there *any* folders/files being
              created at all?<br>
              &gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt; There's one thing I noticed that isn't
              specified... which is the<br>
              &gt;&gt;&gt;&gt; "createdirs = Yes" option. It appears
              (well, I assume) that you're<br>
              &gt;&gt;&gt;&gt; wanting it to create a separate text file
              for each $HOST, not a separate<br>
              &gt;&gt;&gt;&gt; directory named $HOST...<br>
              &gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt; - Austin<br>
              &gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt; On 2/18/2014 2:12 PM, Chris Moody wrote:<br>
              &gt;&gt;&gt;&gt;&gt; Hello.<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; First off, thanks a __TON__ for
              syslog-ng. &nbsp;I've sworn by this awesome<br>
              &gt;&gt;&gt;&gt;&gt; code for years now. &nbsp;I've built all
              sorts of logging infrastructure with<br>
              &gt;&gt;&gt;&gt;&gt; it.<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; I seem to have hit on something
              though that's got me scratching my head<br>
              &gt;&gt;&gt;&gt;&gt; and lacking for explanation. &nbsp;Perhaps
              I've just been staring at it and<br>
              &gt;&gt;&gt;&gt;&gt; debugging it too long and am missing
              something obvious.<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; I've got an installation with a
              couple thousand network devices logging<br>
              &gt;&gt;&gt;&gt;&gt; successfully to output spools on our
              log aggretor. &nbsp;This is rockin' and<br>
              &gt;&gt;&gt;&gt;&gt; works beautifully. &nbsp;I've got things
              configured whereby each network<br>
              &gt;&gt;&gt;&gt;&gt; source logs to it's own individual
              spool file with the source-ip as the<br>
              &gt;&gt;&gt;&gt;&gt; spool name.<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; I'm running into a case though where
              I have a Cisco switch sending logs<br>
              &gt;&gt;&gt;&gt;&gt; to my log aggregator but the
              log-server isn't writing the output to the<br>
              &gt;&gt;&gt;&gt;&gt; device's spool file. &nbsp;It is working
              however for many many more devices<br>
              &gt;&gt;&gt;&gt;&gt; just like this switch.<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; I've confirmed via tcpdump that this
              log traffic does actually hit the<br>
              &gt;&gt;&gt;&gt;&gt; box, but it never gets recorded into
              the log spool for that network device.<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; Since the host is -super- busy
              receiving logs from other gear<br>
              &gt;&gt;&gt;&gt;&gt; enterprise-wide, I have to treat it
              very gingerly, so can't enable too<br>
              &gt;&gt;&gt;&gt;&gt; much debugging...but I'm really
              confused why the logs wouldn't show up<br>
              &gt;&gt;&gt;&gt;&gt; in the log spool..<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; Here's some bits of the config that
              are relevant:<br>
              &gt;&gt;&gt;&gt;&gt; =====<br>
              &gt;&gt;&gt;&gt;&gt; options {<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; keep_hostname(yes);<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; use_dns(no);<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; use_fqdn(no);<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; stats_freq(600);<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; stats_level(2);<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Allow large messages<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; log_msg_size(65536);<br>
              &gt;&gt;&gt;&gt;&gt; };<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; # =====================<br>
              &gt;&gt;&gt;&gt;&gt; # UDP Packet Source<br>
              &gt;&gt;&gt;&gt;&gt; source s_udp {<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; udp();<br>
              &gt;&gt;&gt;&gt;&gt; };<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; # =====================<br>
              &gt;&gt;&gt;&gt;&gt; # TCP Packet Source<br>
              &gt;&gt;&gt;&gt;&gt; source s_tcp {<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;tcp(ip(aaa.bbb.ccc.ddd)
              port(514) max-connections(50000));<br>
              &gt;&gt;&gt;&gt;&gt; };<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; # =====================<br>
              &gt;&gt;&gt;&gt;&gt; destination net_perhost {<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
              file("/data/log/per-host/$HOST"<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; owner(root)<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; group(nwadmin)<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; perm(0775)<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; );<br>
              &gt;&gt;&gt;&gt;&gt; };<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; # =====================<br>
              &gt;&gt;&gt;&gt;&gt; log {<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; source(s_tcp);<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; source(s_udp);<br>
              &gt;&gt;&gt;&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; destination(net_perhost);<br>
              &gt;&gt;&gt;&gt;&gt; };<br>
              &gt;&gt;&gt;&gt;&gt; =====<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; I've checked around for perhaps a
              different spool name, thinking perhaps<br>
              &gt;&gt;&gt;&gt;&gt; the data was getting recognized as
              something other than it's source-ip,<br>
              &gt;&gt;&gt;&gt;&gt; but haven't seen anything.<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; Any thoughts?<br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;&gt; Cheers,<br>
              &gt;&gt;&gt;&gt;&gt; -Chris<br>
              &gt;&gt;&gt;&gt;&gt;
______________________________________________________________________________<br>
              &gt;&gt;&gt;&gt;&gt; Member info: <a
                moz-do-not-send="true"
                href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                target="_blank">
                https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
              &gt;&gt;&gt;&gt;&gt; Documentation: <a
                moz-do-not-send="true"
                href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
              &gt;&gt;&gt;&gt;&gt; FAQ: <a moz-do-not-send="true"
                href="http://www.balabit.com/wiki/syslog-ng-faq"
                target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
              &gt;&gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;&gt;
______________________________________________________________________________<br>
              &gt;&gt;&gt;&gt; Member info: <a moz-do-not-send="true"
                href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                target="_blank">
                https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
              &gt;&gt;&gt;&gt; Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
              &gt;&gt;&gt;&gt; FAQ: <a moz-do-not-send="true"
                href="http://www.balabit.com/wiki/syslog-ng-faq"
                target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
              &gt;&gt;&gt;&gt;<br>
              &gt;&gt;&gt;
______________________________________________________________________________<br>
              &gt;&gt;&gt; Member info: <a moz-do-not-send="true"
                href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                target="_blank">
                https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
              &gt;&gt;&gt; Documentation: <a moz-do-not-send="true"
                href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
              &gt;&gt;&gt; FAQ: <a moz-do-not-send="true"
                href="http://www.balabit.com/wiki/syslog-ng-faq"
                target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
              &gt;&gt;&gt;<br>
              &gt;&gt;
______________________________________________________________________________<br>
              &gt;&gt; Member info: <a moz-do-not-send="true"
                href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                target="_blank">
                https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
              &gt;&gt; Documentation: <a moz-do-not-send="true"
                href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
              &gt;&gt; FAQ: <a moz-do-not-send="true"
                href="http://www.balabit.com/wiki/syslog-ng-faq"
                target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
              &gt;&gt;<br>
              &gt;<br>
              &gt;
______________________________________________________________________________<br>
              &gt; Member info: <a moz-do-not-send="true"
                href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                target="_blank">
                https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
              &gt; Documentation: <a moz-do-not-send="true"
                href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
              &gt; FAQ: <a moz-do-not-send="true"
                href="http://www.balabit.com/wiki/syslog-ng-faq"
                target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
              &gt;<br>
              <br>
______________________________________________________________________________<br>
              Member info: <a moz-do-not-send="true"
                href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
                target="_blank">
                https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
              Documentation: <a moz-do-not-send="true"
                href="http://www.balabit.com/support/documentation/?product=syslog-ng"
                target="_blank">
http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
              FAQ: <a moz-do-not-send="true"
                href="http://www.balabit.com/wiki/syslog-ng-faq"
                target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
              <br>
            </blockquote>
          </div>
          <br>
          <fieldset class="mimeAttachmentHeader"></fieldset>
          <br>
          <pre>______________________________________________________________________________
Member info: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>

</pre>
        </blockquote>
        <br>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>

</pre>
    </blockquote>
    <br>
  </body>
</html>