[syslog-ng] Insert hostname instead of overwrite ?

stucky stucky101 at gmail.com
Fri Sep 3 10:06:48 CEST 2010 

Lance

Thanks for your insight. This hackjob wouldn't really be worth the time
since its only a few logs and only from the server manager.
Most DELL logs are rfc compliant just not a few lines now and then.
I hope I don't encounter any more serious non-standard programs but if I do
I have a better idea now what I could do.
I guess the problem is isolating the specific log source.

My solution works pretty well otherwise.
Btw. any idea on whether syslog-ng can do "gethostbyname" call ? DNS works
but seems awkward considering that I only wanna look up my own hostname and
shouldn't need a dns lookup for that.
I don't use /etc/hosts at all but I did turn on the dns cache option. I
assume this means syslog-ng does a lookup once at startup per each client
and then caches the result ?

On Thu, Sep 2, 2010 at 11:26 PM, Lance Laursen <lance at demonware.net> wrote:

> Hey,
>
> Sorry I didn't read your original email hard enough. So basically, out of
> the box syslog-ng is attempting to parse your non-rfc-standard log entries,
> and as a result the "Server" column after the time and date is being
> assigned the $HOST macro, whilst the rest of the message is getting thrown
> into $MSG. Out of the box syslog-ng tries its best to parse messages
> appropriately, but in this case it's close enough to RFC standard that it's
> just not getting it right. Dell stuff is a bit too windowy.
> Setting flags(no-parse) will solve your problem, except that if you assign
> it to the /dev/log source, all of your other logs are going to be affected.
> This does suck. Can you get dell openmanage to send to a network location?
> If so, you could make a network source (127.0.0.1:12345) and set the
> flags(no-parse) on that. After you set no-parse, you'll have to either use
> csv-parser or patterndb to pull out the fields (ie: the date, then the rest
> of the message), and write out/relay the message using a template. Isolating
> this source will negate any issues with other logs. I would recommend
> googling how to parse apache logs (or another common app with non-rfc logs)
> with syslog-ng for examples on templating and the no-parse option.
>
>
> On Thu, Sep 2, 2010 at 7:52 PM, stucky <stucky101 at gmail.com> wrote:
>
>> Guys
>>
>> Let me clarify. This is not on the syslog server but the clients. I need
>> to make 100% sure clients have the correct hostname set _before_ they even
>> send the message to the log server.
>> So this is for the /dev/log, proc/kmsg source. I need the correct hostname
>> field set by the time the server gets it since I log to a SNAT VIP so the
>> server thinks everything comes from
>> the loadbalancer hence dns is out.
>> Having said that I could try your approach on any src not only a tcp
>> source I assume. Not sure if the regex is worth the pain. It might introduce
>> more issues.
>>
>> So the answer is out of the box syslog only "overwrites" the hostfield. It
>> doesn't squeeze it in right ?
>>
>>
>>
>> On Thu, Sep 2, 2010 at 7:09 PM, <syslogng at feystorm.net> wrote:
>>
>>>  I think you would be able to do this. You can set the no-parse flag on
>>> the tcp source the bad messages come in on, and then use a filter on the
>>> $MSG macro to grab things out. Like a pcre filter that does
>>> '(?<PROGRAM>some.regex). I'm not certain if filters can set macros such as
>>> PROGRAM though, but worth a shot.
>>>
>>> Sent: Thursday, September 02, 2010 7:40:38 PM
>>> From: stucky <stucky101 at gmail.com> <stucky101 at gmail.com>
>>>
>>> To: Syslog-ng users' and developers' mailing list
>>> <syslog-ng at lists.balabit.hu> <syslog-ng at lists.balabit.hu>
>>> Subject: Re: [syslog-ng] Insert hostname instead of overwrite ?
>>>
>>> Guys
>>>
>>> We're not on the same page here. I have already addressed the missing
>>> hostname by forcing syslog-ng to use dns to lookup its own hostname and then
>>> insert it.
>>> All I was asking is if I can make syslog truly "insert" the hostname.
>>> Currently it simply overwrites whatever is in this field (This this case the
>>> word "Server")
>>> and replaces it with the correct hostname.
>>> I was simply saying that this field which was just overwritten might have
>>> contained important loginfo - that's all. It doesn't in this case but what
>>> if it did.
>>> So to make this clear syslog can do this
>>>
>>> Replace "Server Administrator" with "{hostname} Administrator"
>>>
>>> I was wondering if it could instead do this :
>>>
>>> Replace "Server Administrator" with "{hostname} Server Administrator" in
>>> order not to truncate the log content.
>>>
>>> On a side node instead of using dns wouldn't it be great if syslog could
>>> do a "gethostbyname" instead to figure out its own hostname ? Should be much
>>> more efficient
>>> for local log source like this.
>>>
>>> On Thu, Sep 2, 2010 at 5:28 PM, Lance Laursen <lance at demonware.net>wrote:
>>>
>>>> Hmm. Well, if you can't put an intermediary syslog server with use_dns
>>>> enabled between your Dell app and the load balancer, I think you're outta
>>>> luck. If the log message doesn't contain a hostname, and the sending IP is
>>>> that of the load balancer, then syslog really has no way to know where the
>>>> message came from. You could write Dell and ask them to conform to RFC
>>>> syslog standards but I don't think that's going to happen any time soon :).
>>>> The only other thing I can think of is that if you only have one dell
>>>> openManage box, you could filter for something specific to those logs then
>>>> apply a static hostname using a template. But that method sucks and doesn't
>>>> work as soon as you have two openManage boxes forwarding syslogs.
>>>>
>>>>
>>>> On Thu, Sep 2, 2010 at 3:39 PM, stucky <stucky101 at gmail.com> wrote:
>>>>
>>>>> That's exactly the problem. I cannot keep a hostname that was never
>>>>> written in the first place.
>>>>> The DELL server administrator doesn't send it. As per my email below it
>>>>> sends this :
>>>>>
>>>>>
>>>>> Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242>
>>>>> The Patrol Read has started.:  Controller 0 (PERC 5/i Integrated)
>>>>>
>>>>>  So If I do a "keep_hostname" syslog-ng assumes that the server is
>>>>> called "Server" which is of course wrong.
>>>>>
>>>>>
>>>>> On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi at balabit.hu>wrote:
>>>>>
>>>>>>  On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
>>>>>> > Guys
>>>>>> >
>>>>>> > I'm trying to log to a loadbalanced VIP. It seems to work ok except
>>>>>> > that the loadbalancer uses SNAT so I loose my source IP.
>>>>>> > This means I cannot use dns or even the source ip to get the source
>>>>>> > host as all logs appear to come from the same source (the
>>>>>> > loadbalancer).
>>>>>> > This means I have no choice but to rely on the hostname field which
>>>>>> > works about 98% of the time but some stuff like Dell OpenManage
>>>>>> skips
>>>>>> > the hostname field.
>>>>>> > So I'd get logs like this on host "cage" f.e.
>>>>>> >
>>>>>> > Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242
>>>>>> > The Patrol Read has started.:  Controller 0 (PERC 5/i Integrated)
>>>>>> >
>>>>>> > I fixed that by telling the syslog-ng client to force itself to
>>>>>> figure
>>>>>> > out a proper hostname and now the log looks like this
>>>>>> >
>>>>>> > Aug 17 13:51:10 cage Administrator[]: Instrumentation Service
>>>>>> EventID:
>>>>>> > 1000  Server Administrator starting
>>>>>> >
>>>>>> > I thought syslog-ng inserts the hostname but by the looks of it it
>>>>>> > simply replaces whatever is in the expected field with the hostname
>>>>>> it
>>>>>> > has just figured out.
>>>>>> > As you can see it overwrote the entry "Server".
>>>>>> > No biggie in the above case but what if this field contained
>>>>>> valuable
>>>>>> > information ? I'd loose that.
>>>>>> > Any way to squeeze in the hostname so to speak ?
>>>>>>
>>>>>>  what about keep_hostname(yes) ?
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Bazsi
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>> stucky
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>  Lance Laursen
>>>> Demonware Systems Engineer
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> stucky
>>>
>>> ------------------------------
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>
>>>
>>
>>
>> --
>> stucky
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>>
>
>
> --
> Lance Laursen
> Demonware Systems Engineer
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>


-- 
stucky
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100903/a6859150/attachment-0001.htm

More information about the syslog-ng mailing list