[syslog-ng] Insert hostname instead of overwrite ?
Balazs Scheidler
bazsi at balabit.hu
Fri Sep 3 13:33:37 CEST 2010
On Fri, 2010-09-03 at 01:06 -0700, stucky wrote:
> Lance
>
> Thanks for your insight. This hackjob wouldn't really be worth the
> time since its only a few logs and only from the server manager.
> Most DELL logs are rfc compliant just not a few lines now and then.
> I hope I don't encounter any more serious non-standard programs but if
> I do I have a better idea now what I could do.
> I guess the problem is isolating the specific log source.
>
> My solution works pretty well otherwise.
> Btw. any idea on whether syslog-ng can do "gethostbyname" call ? DNS
> works but seems awkward considering that I only wanna look up my own
> hostname and shouldn't need a dns lookup for that.
gethostbyname() automatically does a DNS lookup through NSS (if the name
is not in /etc/hosts), and yes syslog-ng uses that to get the hostname
information from the IP address.
Its own hostname is found out by using gethostname(), so no DNS lookup
in that case.
> I don't use /etc/hosts at all but I did turn on the dns cache option.
> I assume this means syslog-ng does a lookup once at startup per each
> client and then caches the result ?
Once when it encounters it, yes. The dns cache size is limited but can
be tuned, as the time a given entry is assumed to be valid in the cache
(just in order to follow DNS name changes).
>
> On Thu, Sep 2, 2010 at 11:26 PM, Lance Laursen <lance at demonware.net>
> wrote:
> Hey,
>
>
> Sorry I didn't read your original email hard enough. So
> basically, out of the box syslog-ng is attempting to parse
> your non-rfc-standard log entries, and as a result the
> "Server" column after the time and date is being assigned the
> $HOST macro, whilst the rest of the message is getting thrown
> into $MSG. Out of the box syslog-ng tries its best to parse
> messages appropriately, but in this case it's close enough to
> RFC standard that it's just not getting it right. Dell stuff
> is a bit too windowy.
> Setting flags(no-parse) will solve your problem, except that
> if you assign it to the /dev/log source, all of your other
> logs are going to be affected. This does suck. Can you get
> dell openmanage to send to a network location? If so, you
> could make a network source (127.0.0.1:12345) and set the
> flags(no-parse) on that. After you set no-parse, you'll have
> to either use csv-parser or patterndb to pull out the fields
> (ie: the date, then the rest of the message), and write
> out/relay the message using a template. Isolating this source
> will negate any issues with other logs. I would recommend
> googling how to parse apache logs (or another common app with
> non-rfc logs) with syslog-ng for examples on templating and
> the no-parse option.
>
>
>
> On Thu, Sep 2, 2010 at 7:52 PM, stucky <stucky101 at gmail.com>
> wrote:
> Guys
>
> Let me clarify. This is not on the syslog server but
> the clients. I need to make 100% sure clients have the
> correct hostname set _before_ they even send the
> message to the log server.
> So this is for the /dev/log, proc/kmsg source. I need
> the correct hostname field set by the time the server
> gets it since I log to a SNAT VIP so the server thinks
> everything comes from
> the loadbalancer hence dns is out.
> Having said that I could try your approach on any src
> not only a tcp source I assume. Not sure if the regex
> is worth the pain. It might introduce more issues.
>
> So the answer is out of the box syslog only
> "overwrites" the hostfield. It doesn't squeeze it in
> right ?
>
>
>
>
> On Thu, Sep 2, 2010 at 7:09 PM,
> <syslogng at feystorm.net> wrote:
> I think you would be able to do this. You can
> set the no-parse flag on the tcp source the
> bad messages come in on, and then use a filter
> on the $MSG macro to grab things out. Like a
> pcre filter that does '(?<PROGRAM>some.regex).
> I'm not certain if filters can set macros such
> as PROGRAM though, but worth a shot.
>
> Sent: Thursday, September 02, 2010 7:40:38 PM
> From: stucky <stucky101 at gmail.com>
>
> To: Syslog-ng users' and developers' mailing
> list <syslog-ng at lists.balabit.hu>
>
> Subject: Re: [syslog-ng] Insert hostname
> instead of overwrite ?
> >
> > Guys
> >
> > We're not on the same page here. I have
> > already addressed the missing hostname by
> > forcing syslog-ng to use dns to lookup its
> > own hostname and then insert it.
> > All I was asking is if I can make syslog
> > truly "insert" the hostname. Currently it
> > simply overwrites whatever is in this field
> > (This this case the word "Server")
> > and replaces it with the correct hostname.
> > I was simply saying that this field which
> > was just overwritten might have contained
> > important loginfo - that's all. It doesn't
> > in this case but what if it did.
> > So to make this clear syslog can do this
> >
> > Replace "Server Administrator" with
> > "{hostname} Administrator"
> >
> > I was wondering if it could instead do
> > this :
> >
> > Replace "Server Administrator" with
> > "{hostname} Server Administrator" in order
> > not to truncate the log content.
> >
> > On a side node instead of using dns wouldn't
> > it be great if syslog could do a
> > "gethostbyname" instead to figure out its
> > own hostname ? Should be much more efficient
> > for local log source like this.
> >
> > On Thu, Sep 2, 2010 at 5:28 PM, Lance
> > Laursen <lance at demonware.net> wrote:
> > Hmm. Well, if you can't put an
> > intermediary syslog server with
> > use_dns enabled between your Dell
> > app and the load balancer, I think
> > you're outta luck. If the log
> > message doesn't contain a hostname,
> > and the sending IP is that of the
> > load balancer, then syslog really
> > has no way to know where the message
> > came from. You could write Dell and
> > ask them to conform to RFC syslog
> > standards but I don't think that's
> > going to happen any time soon :).
> > The only other thing I can think of
> > is that if you only have one dell
> > openManage box, you could filter for
> > something specific to those logs
> > then apply a static hostname using a
> > template. But that method sucks and
> > doesn't work as soon as you have two
> > openManage boxes forwarding syslogs.
> >
> >
> > On Thu, Sep 2, 2010 at 3:39 PM,
> > stucky <stucky101 at gmail.com> wrote:
> > That's exactly the problem.
> > I cannot keep a hostname
> > that was never written in
> > the first place.
> > The DELL server
> > administrator doesn't send
> > it. As per my email below it
> > sends this :
> >
> >
> > Aug 16 21:47:22 Server
> > Administrator: Storage
> > Service EventID: 2242> The
> > Patrol Read has started.:
> > Controller 0 (PERC 5/i
> > Integrated)
> >
> >
> > So If I do a "keep_hostname"
> > syslog-ng assumes that the
> > server is called "Server"
> > which is of course wrong.
> >
> >
> > On Thu, Sep 2, 2010 at 8:21
> > AM, Balazs Scheidler
> > <bazsi at balabit.hu> wrote:
> > On Tue, 2010-08-17
> > at 18:26 -0700,
> > stucky wrote:
> > > Guys
> > >
> > > I'm trying to log
> > to a loadbalanced
> > VIP. It seems to
> > work ok except
> > > that the
> > loadbalancer uses
> > SNAT so I loose my
> > source IP.
> > > This means I
> > cannot use dns or
> > even the source ip
> > to get the source
> > > host as all logs
> > appear to come from
> > the same source (the
> > > loadbalancer).
> > > This means I have
> > no choice but to
> > rely on the hostname
> > field which
> > > works about 98% of
> > the time but some
> > stuff like Dell
> > OpenManage skips
> > > the hostname
> > field.
> > > So I'd get logs
> > like this on host
> > "cage" f.e.
> > >
> > > Aug 16 21:47:22
> > Server
> > Administrator:
> > Storage Service
> > EventID: 2242
> > > The Patrol Read
> > has started.:
> > Controller 0 (PERC
> > 5/i Integrated)
> > >
> > > I fixed that by
> > telling the
> > syslog-ng client to
> > force itself to
> > figure
> > > out a proper
> > hostname and now the
> > log looks like this
> > >
> > > Aug 17 13:51:10
> > cage
> > Administrator[]:
> > Instrumentation
> > Service EventID:
> > > 1000 Server
> > Administrator
> > starting
> > >
> > > I thought
> > syslog-ng inserts
> > the hostname but by
> > the looks of it it
> > > simply replaces
> > whatever is in the
> > expected field with
> > the hostname it
> > > has just figured
> > out.
> > > As you can see it
> > overwrote the entry
> > "Server".
> > > No biggie in the
> > above case but what
> > if this field
> > contained valuable
> > > information ? I'd
> > loose that.
> > > Any way to squeeze
> > in the hostname so
> > to speak ?
> >
> >
> > what about
> > keep_hostname(yes) ?
> >
> >
> > --
> > Bazsi
> >
> > ______________________________________________________________________________
> > Member info:
> > https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ:
> > http://www.campin.net/syslog-ng/faq.html
> >
> >
> >
> >
> >
> >
> > --
> > stucky
> >
> > ______________________________________________________________________________
> > Member info:
> > https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ:
> > http://www.campin.net/syslog-ng/faq.html
> >
> >
> >
> >
> >
> > --
> >
> > Lance Laursen
> > Demonware Systems Engineer
> >
> >
> > ______________________________________________________________________________
> > Member info:
> > https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ:
> > http://www.campin.net/syslog-ng/faq.html
> >
> >
> >
> >
> >
> > --
> > stucky
> >
> >
> > ____________________________________________
> > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
>
> ______________________________________________________________________________
> Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>
>
>
> --
> stucky
>
>
> ______________________________________________________________________________
> Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>
>
>
> --
> Lance Laursen
> Demonware Systems Engineer
>
>
> ______________________________________________________________________________
> Member info:
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
>
>
>
> --
> stucky
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
--
Bazsi
More information about the syslog-ng
mailing list