Lance<br><br>Thanks for your insight. This hackjob wouldn't really be worth the time since its only a few logs and only from the server manager.<br>Most DELL logs are rfc compliant just not a few lines now and then.<br>
I hope I don't encounter any more serious non-standard programs but if I do I have a better idea now what I could do.<br>I guess the problem is isolating the specific log source.<br><br>My solution works pretty well otherwise.<br>
Btw. any idea on whether syslog-ng can do "gethostbyname" call ? DNS works but seems awkward considering that I only wanna look up my own hostname and shouldn't need a dns lookup for that. <br>I don't use /etc/hosts at all but I did turn on the dns cache option. I assume this means syslog-ng does a lookup once at startup per each client and then caches the result ?<br>
<br><div class="gmail_quote">On Thu, Sep 2, 2010 at 11:26 PM, Lance Laursen <span dir="ltr"><<a href="mailto:lance@demonware.net">lance@demonware.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hey,<div><br></div><div>Sorry I didn't read your original email hard enough. So basically, out of the box syslog-ng is attempting to parse your non-rfc-standard log entries, and as a result the "Server" column after the time and date is being assigned the $HOST macro, whilst the rest of the message is getting thrown into $MSG. Out of the box syslog-ng tries its best to parse messages appropriately, but in this case it's close enough to RFC standard that it's just not getting it right. Dell stuff is a bit too windowy.</div>
<div>Setting flags(no-parse) will solve your problem, except that if you assign it to the /dev/log source, all of your other logs are going to be affected. This does suck. Can you get dell openmanage to send to a network location? If so, you could make a network source (<a href="http://127.0.0.1:12345" target="_blank">127.0.0.1:12345</a>) and set the flags(no-parse) on that. After you set no-parse, you'll have to either use csv-parser or patterndb to pull out the fields (ie: the date, then the rest of the message), and write out/relay the message using a template. Isolating this source will negate any issues with other logs. I would recommend googling how to parse apache logs (or another common app with non-rfc logs) with syslog-ng for examples on templating and the no-parse option.<div>
<div></div><div class="h5"><br>
<br><div class="gmail_quote">On Thu, Sep 2, 2010 at 7:52 PM, stucky <span dir="ltr"><<a href="mailto:stucky101@gmail.com" target="_blank">stucky101@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Guys<br><br>Let me clarify. This is not on the syslog server but the clients. I need to make 100% sure clients have the correct hostname set _before_ they even send the message to the log server.<br>So this is for the /dev/log, proc/kmsg source. I need the correct hostname field set by the time the server gets it since I log to a SNAT VIP so the server thinks everything comes from<br>
the loadbalancer hence dns is out.<br>Having said that I could try your approach on any src not only a tcp source I assume. Not sure if the regex is worth the pain. It might introduce more issues.<br><br>So the answer is out of the box syslog only "overwrites" the hostfield. It doesn't squeeze it in right ?<div>
<div></div><div><br>
<br><br><div class="gmail_quote">On Thu, Sep 2, 2010 at 7:09 PM, <span dir="ltr"><<a href="mailto:syslogng@feystorm.net" target="_blank">syslogng@feystorm.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff" text="#0050d0">
<font size="-1"><font face="Helvetica, Arial, sans-serif">I think you
would be able to do this. You can set the no-parse flag on the tcp
source the bad messages come in on, and then use a filter on the $MSG
macro to grab things out. Like a pcre filter that does
'(?<PROGRAM>some.regex). I'm not certain if filters can set
macros such as PROGRAM though, but worth a shot.<br>
</font></font><br>
Sent: Thursday, September 02, 2010 7:40:38 PM<br>
From: stucky <a href="mailto:stucky101@gmail.com" target="_blank"><stucky101@gmail.com></a><div><br>
To: Syslog-ng users' and developers' mailing list
<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank"><syslog-ng@lists.balabit.hu></a> <br></div>
Subject: Re: [syslog-ng] Insert hostname instead of overwrite ?
<blockquote type="cite"><div><div></div><div>Guys<br>
<br>
We're not on the same page here. I have already addressed the missing
hostname by forcing syslog-ng to use dns to lookup its own hostname and
then insert it.<br>
All I was asking is if I can make syslog truly "insert" the hostname.
Currently it simply overwrites whatever is in this field (This this
case the word "Server")<br>
and replaces it with the correct hostname.<br>
I was simply saying that this field which was just overwritten might
have contained important loginfo - that's all. It doesn't in this case
but what if it did.<br>
So to make this clear syslog can do this<br>
<br>
Replace "Server Administrator" with "{hostname} Administrator"<br>
<br>
I was wondering if it could instead do this :<br>
<br>
Replace "Server Administrator" with "{hostname} Server Administrator"
in order not to truncate the log content.<br>
<br>
On a side node instead of using dns wouldn't it be great if syslog
could do a "gethostbyname" instead to figure out its own hostname ?
Should be much more efficient<br>
for local log source like this.<br>
<br>
<div class="gmail_quote">On Thu, Sep 2, 2010 at 5:28 PM, Lance
Laursen <span dir="ltr"><<a href="mailto:lance@demonware.net" target="_blank">lance@demonware.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hmm.
Well, if you can't put an intermediary syslog server with use_dns
enabled between your Dell app and the load balancer, I think you're
outta luck. If the log message doesn't contain a hostname, and the
sending IP is that of the load balancer, then syslog really has no way
to know where the message came from. You could write Dell and ask them
to conform to RFC syslog standards but I don't think that's going to
happen any time soon :).
<div>The only other thing I can think of is that if you only have
one dell openManage box, you could filter for something specific to
those logs then apply a static hostname using a template. But that
method sucks and doesn't work as soon as you have two openManage boxes
forwarding syslogs.</div>
<div>
<div>
<div><br>
<br>
<div class="gmail_quote">On Thu, Sep 2, 2010 at 3:39 PM, stucky <span dir="ltr"><<a href="mailto:stucky101@gmail.com" target="_blank">stucky101@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
That's exactly the problem. I cannot keep a hostname that was never
written in the first place.<br>
The DELL server administrator doesn't send it. As per my email below it
sends this :
<div><br>
<br>
Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242>
The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)<br>
<br>
</div>
So If I do a "keep_hostname" syslog-ng assumes that the server is
called "Server" which is of course wrong.
<div>
<div><br>
<br>
<div class="gmail_quote">On Thu, Sep 2, 2010 at 8:21 AM, Balazs
Scheidler <span dir="ltr"><<a href="mailto:bazsi@balabit.hu" target="_blank">bazsi@balabit.hu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div>On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:<br>
> Guys<br>
><br>
> I'm trying to log to a loadbalanced VIP. It seems to work ok except<br>
> that the loadbalancer uses SNAT so I loose my source IP.<br>
> This means I cannot use dns or even the source ip to get the source<br>
> host as all logs appear to come from the same source (the<br>
> loadbalancer).<br>
> This means I have no choice but to rely on the hostname field which<br>
> works about 98% of the time but some stuff like Dell OpenManage
skips<br>
> the hostname field.<br>
> So I'd get logs like this on host "cage" f.e.<br>
><br>
> Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242<br>
> The Patrol Read has started.: Controller 0 (PERC 5/i Integrated)<br>
><br>
> I fixed that by telling the syslog-ng client to force itself to
figure<br>
> out a proper hostname and now the log looks like this<br>
><br>
> Aug 17 13:51:10 cage Administrator[]: Instrumentation Service
EventID:<br>
> 1000 Server Administrator starting<br>
><br>
> I thought syslog-ng inserts the hostname but by the looks of it it<br>
> simply replaces whatever is in the expected field with the
hostname it<br>
> has just figured out.<br>
> As you can see it overwrote the entry "Server".<br>
> No biggie in the above case but what if this field contained
valuable<br>
> information ? I'd loose that.<br>
> Any way to squeeze in the hostname so to speak ?<br>
<br>
</div>
</div>
what about keep_hostname(yes) ?<br>
<div>
<div><br>
<br>
--<br>
Bazsi<br>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
</div>
</div>
-- <br>
<font color="#888888">stucky<br>
</font><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
</div>
</div>
Lance Laursen<br>
Demonware Systems Engineer<br>
</div>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
stucky<br>
</div></div><pre><hr size="4" width="90%"><div>
______________________________________________________________________________
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a>
</div></pre>
</blockquote>
</div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
<br></blockquote></div><br><br clear="all"><br>-- <br>stucky<br>
</div></div><br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Lance Laursen<br>Demonware Systems Engineer<br>
</div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
<br></blockquote></div><br><br clear="all"><br>-- <br>stucky<br>