[syslog-ng] Insert hostname instead of overwrite ?

Lance Laursen lance at demonware.net
Fri Sep 3 08:26:19 CEST 2010 

Hey,

Sorry I didn't read your original email hard enough. So basically, out of
the box syslog-ng is attempting to parse your non-rfc-standard log entries,
and as a result the "Server" column after the time and date is being
assigned the $HOST macro, whilst the rest of the message is getting thrown
into $MSG. Out of the box syslog-ng tries its best to parse messages
appropriately, but in this case it's close enough to RFC standard that it's
just not getting it right. Dell stuff is a bit too windowy.
Setting flags(no-parse) will solve your problem, except that if you assign
it to the /dev/log source, all of your other logs are going to be affected.
This does suck. Can you get dell openmanage to send to a network location?
If so, you could make a network source (127.0.0.1:12345) and set the
flags(no-parse) on that. After you set no-parse, you'll have to either use
csv-parser or patterndb to pull out the fields (ie: the date, then the rest
of the message), and write out/relay the message using a template. Isolating
this source will negate any issues with other logs. I would recommend
googling how to parse apache logs (or another common app with non-rfc logs)
with syslog-ng for examples on templating and the no-parse option.

On Thu, Sep 2, 2010 at 7:52 PM, stucky <stucky101 at gmail.com> wrote:

> Guys
>
> Let me clarify. This is not on the syslog server but the clients. I need to
> make 100% sure clients have the correct hostname set _before_ they even send
> the message to the log server.
> So this is for the /dev/log, proc/kmsg source. I need the correct hostname
> field set by the time the server gets it since I log to a SNAT VIP so the
> server thinks everything comes from
> the loadbalancer hence dns is out.
> Having said that I could try your approach on any src not only a tcp source
> I assume. Not sure if the regex is worth the pain. It might introduce more
> issues.
>
> So the answer is out of the box syslog only "overwrites" the hostfield. It
> doesn't squeeze it in right ?
>
>
>
> On Thu, Sep 2, 2010 at 7:09 PM, <syslogng at feystorm.net> wrote:
>
>>  I think you would be able to do this. You can set the no-parse flag on
>> the tcp source the bad messages come in on, and then use a filter on the
>> $MSG macro to grab things out. Like a pcre filter that does
>> '(?<PROGRAM>some.regex). I'm not certain if filters can set macros such as
>> PROGRAM though, but worth a shot.
>>
>> Sent: Thursday, September 02, 2010 7:40:38 PM
>> From: stucky <stucky101 at gmail.com> <stucky101 at gmail.com>
>>
>> To: Syslog-ng users' and developers' mailing list
>> <syslog-ng at lists.balabit.hu> <syslog-ng at lists.balabit.hu>
>> Subject: Re: [syslog-ng] Insert hostname instead of overwrite ?
>>
>> Guys
>>
>> We're not on the same page here. I have already addressed the missing
>> hostname by forcing syslog-ng to use dns to lookup its own hostname and then
>> insert it.
>> All I was asking is if I can make syslog truly "insert" the hostname.
>> Currently it simply overwrites whatever is in this field (This this case the
>> word "Server")
>> and replaces it with the correct hostname.
>> I was simply saying that this field which was just overwritten might have
>> contained important loginfo - that's all. It doesn't in this case but what
>> if it did.
>> So to make this clear syslog can do this
>>
>> Replace "Server Administrator" with "{hostname} Administrator"
>>
>> I was wondering if it could instead do this :
>>
>> Replace "Server Administrator" with "{hostname} Server Administrator" in
>> order not to truncate the log content.
>>
>> On a side node instead of using dns wouldn't it be great if syslog could
>> do a "gethostbyname" instead to figure out its own hostname ? Should be much
>> more efficient
>> for local log source like this.
>>
>> On Thu, Sep 2, 2010 at 5:28 PM, Lance Laursen <lance at demonware.net>wrote:
>>
>>> Hmm. Well, if you can't put an intermediary syslog server with use_dns
>>> enabled between your Dell app and the load balancer, I think you're outta
>>> luck. If the log message doesn't contain a hostname, and the sending IP is
>>> that of the load balancer, then syslog really has no way to know where the
>>> message came from. You could write Dell and ask them to conform to RFC
>>> syslog standards but I don't think that's going to happen any time soon :).
>>> The only other thing I can think of is that if you only have one dell
>>> openManage box, you could filter for something specific to those logs then
>>> apply a static hostname using a template. But that method sucks and doesn't
>>> work as soon as you have two openManage boxes forwarding syslogs.
>>>
>>>
>>> On Thu, Sep 2, 2010 at 3:39 PM, stucky <stucky101 at gmail.com> wrote:
>>>
>>>> That's exactly the problem. I cannot keep a hostname that was never
>>>> written in the first place.
>>>> The DELL server administrator doesn't send it. As per my email below it
>>>> sends this :
>>>>
>>>>
>>>> Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242> The
>>>> Patrol Read has started.:  Controller 0 (PERC 5/i Integrated)
>>>>
>>>>  So If I do a "keep_hostname" syslog-ng assumes that the server is
>>>> called "Server" which is of course wrong.
>>>>
>>>>
>>>> On Thu, Sep 2, 2010 at 8:21 AM, Balazs Scheidler <bazsi at balabit.hu>wrote:
>>>>
>>>>>  On Tue, 2010-08-17 at 18:26 -0700, stucky wrote:
>>>>> > Guys
>>>>> >
>>>>> > I'm trying to log to a loadbalanced VIP. It seems to work ok except
>>>>> > that the loadbalancer uses SNAT so I loose my source IP.
>>>>> > This means I cannot use dns or even the source ip to get the source
>>>>> > host as all logs appear to come from the same source (the
>>>>> > loadbalancer).
>>>>> > This means I have no choice but to rely on the hostname field which
>>>>> > works about 98% of the time but some stuff like Dell OpenManage skips
>>>>> > the hostname field.
>>>>> > So I'd get logs like this on host "cage" f.e.
>>>>> >
>>>>> > Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242
>>>>> > The Patrol Read has started.:  Controller 0 (PERC 5/i Integrated)
>>>>> >
>>>>> > I fixed that by telling the syslog-ng client to force itself to
>>>>> figure
>>>>> > out a proper hostname and now the log looks like this
>>>>> >
>>>>> > Aug 17 13:51:10 cage Administrator[]: Instrumentation Service
>>>>> EventID:
>>>>> > 1000  Server Administrator starting
>>>>> >
>>>>> > I thought syslog-ng inserts the hostname but by the looks of it it
>>>>> > simply replaces whatever is in the expected field with the hostname
>>>>> it
>>>>> > has just figured out.
>>>>> > As you can see it overwrote the entry "Server".
>>>>> > No biggie in the above case but what if this field contained valuable
>>>>> > information ? I'd loose that.
>>>>> > Any way to squeeze in the hostname so to speak ?
>>>>>
>>>>>  what about keep_hostname(yes) ?
>>>>>
>>>>>
>>>>> --
>>>>> Bazsi
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>>>
>>>>>
>>>>
>>>>
>>>>  --
>>>> stucky
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>  Lance Laursen
>>> Demonware Systems Engineer
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>
>>>
>>
>>
>> --
>> stucky
>>
>> ------------------------------
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>>
>
>
> --
> stucky
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>


-- 
Lance Laursen
Demonware Systems Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100902/fc522f01/attachment-0001.htm

More information about the syslog-ng mailing list