[syslog-ng] syslog-ng Digest, Vol 28, Issue 21

Wilson Lai wilsonlai at macausjm.com
Fri Sep 7 11:26:02 CEST 2007


Dear all,
       What happen if the log message is not a standard syslog message?
       Thanks.
Regards,
Wilson Lai
System Engineer
IT Dept., SJM
Office ( : (853)2978585
Mobile ( : (853)66506709
Email +: : wilsonlai at macausjm.com
 
-----Original Message-----
From: syslog-ng-request at lists.balabit.hu 
[mailto:syslog-ng-request at lists.balabit.hu] 
Sent: Friday, August 17, 2007 3:45 PM
To: syslog-ng at lists.balabit.hu
Subject: syslog-ng Digest, Vol 28, Issue 21

Send syslog-ng mailing list submissions to
	syslog-ng at lists.balabit.hu

To subscribe or unsubscribe via the World Wide Web, visit
	https://lists.balabit.hu/mailman/listinfo/syslog-ng
or, via email, send a message with subject or body 'help' to
	syslog-ng-request at lists.balabit.hu

You can reach the person managing the list at
	syslog-ng-owner at lists.balabit.hu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of syslog-ng digest..."


Today's Topics:

   1.  new syslog-ng white paper published (Balazs Scheidler)
   2.  Buffer Overflow : Insufficient buffer space for	retrieving
      STREAMS log message; res='2' ( Nicolas de Marqu? - Fromentin )
   3.  REGEX rewrites on packet body possible? (Eli Stair)
   4. Re:  Buffer Overflow : Insufficient buffer space for
      retrieving STREAMS log message; res='2' (Russell Fulton)
   5. Re:  Logging Third party application logs to Syslog-NG
      (Wilson Lai)
   6. Re:  Buffer Overflow : Insufficient buffer	space	for
      retrieving STREAMS log message; res='2' (Balazs Scheidler)
   7. Re:  REGEX rewrites on packet body possible? (Balazs Scheidler)
   8. Re:  Logging Third party application logs to Syslog-NG
      (Balazs Scheidler)


----------------------------------------------------------------------

Message: 1
Date: Thu, 16 Aug 2007 16:58:31 +0200
From: Balazs Scheidler <bazsi at balabit.hu>
Subject: [syslog-ng] new syslog-ng white paper published
To: syslog-ng at lists.balabit.hu
Message-ID: <1187276311.7565.52.camel at bzorp.balabit>
Content-Type: text/plain

Dear all,

We have published a new, syslog-ng related white paper on our website
titled: "Distributed syslog architectures with syslog-ng"

It basically contains the various deployment scenarios, their pros and
cons we've encountered in our experience.

It does contain information on syslog-ng Premium Edition, but still we
tried to concentrate on the technical content, and some of it equally
applies to syslog-ng OSE.

We'd appreciate feedback at documentation at balabit.com. Thanks.

-- 
Bazsi



------------------------------

Message: 2
Date: Thu, 16 Aug 2007 18:40:03 +0200
From: " Nicolas de Marqu? - Fromentin "	<nicolas.demarque at gmail.com>
Subject: [syslog-ng] Buffer Overflow : Insufficient buffer space for
	retrieving STREAMS log message; res='2'
To: "Syslog-ng users' and developers' mailing list"
	<syslog-ng at lists.balabit.hu>
Message-ID:
	<b778b3cc0708160940y50d0dfb4yc84f86dffa4efc6a at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hello,

During a big flow from a local server to a central server, we have a 
"buffer
overflow" on the local server. After this message, syslog-ng is crash 
and
don't be capable to send any messages.

The local server is a solaris 10 product, the central linux a redhat 
EL4.

Do you know a rule to stop this problem

Best regard,

Nicolas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070816/b5a179d7/attachment-0001.htm 

------------------------------

Message: 3
Date: Thu, 16 Aug 2007 10:07:03 -0700
From: Eli Stair <estair at ilm.com>
Subject: [syslog-ng] REGEX rewrites on packet body possible?
To: syslog-ng at lists.balabit.hu
Message-ID: <46C48437.90103 at ilm.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed


I've got a problem with some network devices that is leading me to need 
to find 
some way to do regex rewriting of portions of the message body of syslog 

messages.

Problem: I've got a bunch of Foundry devices that put their hostname 
followed 
by a comma in the body of the message (and some that do not).  Some of 
these 
look like this:

{
2007-08-16:2007-08-16T09:50:16-07:00 hostname [hostname.local7.notice] 
hostname, Linecard Module 13 temperature 50.0 C degrees is normal
}

# template("$R_ISODATE $HOST [$PROGRAM.$FACILITY.$PRIORITY] $MSG\n")


As you can see from the template, the second hostname reported with the 
comma 
is part of the MSG body.  For reasons of properly searching/indexing 
this data 
I need to strip this out.  I've seen mention of a tool called 
'syslog-mailer' 
that sounds like it would do the job somewhat.  Additionally, I've seen 
blog 
chatter about potentially adding full regex rewrite capability to 
syslog-ng in 
the recent past.  I can't find evidence of either of these methods 
however.

My first try at solving this using an external program showed that when 
passing 
data OUT of syslog-ng to a defined program, only the message body is 
sent and 
before application of a template, the other information is dropped.  
Thus it's 
not possible to do processing of the whole payload externally, re-import 
the 
data via a socket and finish writing because the facility and HOST 
information 
is all gone!

I'm looking at writing a log proxy using Net::Dev::Tools::Syslog in perl 
to 
handle listening, rewriting if necessary, and forwarding full messages 
on to 
syslog-ng after.  I'd just like to know if there are any better 
suggestions, or 
if this has been done before successfully in another way?

Cheers, and thanks for any insight.


/eli


------------------------------

Message: 4
Date: Fri, 17 Aug 2007 09:33:48 +1200
From: Russell Fulton <r.fulton at auckland.ac.nz>
Subject: Re: [syslog-ng] Buffer Overflow : Insufficient buffer space
	for	retrieving STREAMS log message; res='2'
To: Syslog-ng users' and developers' mailing list
	<syslog-ng at lists.balabit.hu>
Message-ID: <46C4C2BC.9040808 at auckland.ac.nz>
Content-Type: text/plain; charset=ISO-8859-1

Hi Nicolas

I suggest you post the actual error messages to the list to give a few
more clues.

Russell


Nicolas de Marqu? - Fromentin wrote:
> Hello,
>  
> During a big flow from a local server to a central server, we have a
> "buffer overflow" on the local server. After this message, syslog-ng
> is crash and don't be capable to send any messages.
>  
> The local server is a solaris 10 product, the central linux a redhat 
EL4.
>  
> Do you know a rule to stop this problem
>  
> Best regard,
>  
> Nicolas
> 
------------------------------------------------------------------------
>
> _______________________________________________
> syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>
>   


------------------------------

Message: 5
Date: Fri, 17 Aug 2007 11:37:36 +0800
From: "Wilson Lai" <wilsonlai at macausjm.com>
Subject: Re: [syslog-ng] Logging Third party application logs to
	Syslog-NG
To: syslog-ng <syslog-ng at lists.balabit.hu>
Message-ID: <H000006e00732964.1187321856.mail.macausjm.com at MHS>
Content-Type: text/plain;	charset="US-ASCII"

Dear Bill,
        If there is a Scalix application which generate the log file in 
"/var/opt/Scalix/ml/s/logs/fatal", where "fatal" is the log file. How 
could
the source be defined to be listened by Syslog-NG?
        Thanks.

Regards,
Wilson Lai
System Engineer
IT Dept., SJM
Office ( : (853)2978585
Mobile ( : (853)66506709
Email +: : wilsonlai at macausjm.com
 
-----Original Message-----
From: syslog-ng-request at lists.balabit.hu 
[mailto:syslog-ng-request at lists.balabit.hu] 
Sent: Thursday, August 16, 2007 6:00 PM
To: syslog-ng at lists.balabit.hu
Subject: syslog-ng Digest, Vol 28, Issue 20

Send syslog-ng mailing list submissions to
	syslog-ng at lists.balabit.hu

To subscribe or unsubscribe via the World Wide Web, visit
	https://lists.balabit.hu/mailman/listinfo/syslog-ng
or, via email, send a message with subject or body 'help' to
	syslog-ng-request at lists.balabit.hu

You can reach the person managing the list at
	syslog-ng-owner at lists.balabit.hu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of syslog-ng digest..."


Today's Topics:

   1.  GLib version problem? (Burns Andrew)
   2. Re:  GLib version problem? (Geller, Sandor (IT))
   3. Re:  GLib version problem? (Valdis.Kletnieks at vt.edu)
   4.  Logging Third party application logs to Syslog-NG in	Linux!
      (Wilson Lai)
   5. Re:  Logging Third party application logs to Syslog-NG in
      Linux! (Bill Nash)
   6.  Troubles with SE-Linux Syslog-ng and ntpd (Martin Voelker)


----------------------------------------------------------------------

Message: 1
Date: Wed, 15 Aug 2007 08:02:38 -0500
From: "Burns Andrew" <aburns at snyderdrug.com>
Subject: [syslog-ng] GLib version problem?
To: <syslog-ng at lists.balabit.hu>
Message-ID:
	<699402D08697B94EAB9429B5083FAA1C042C2290 at exchange_2k.snyderdrug.co
m>
Content-Type: text/plain; charset="us-ascii"

Hello,
 
I'm trying to install syslog-NG onto a Cent OS 5 server, and running
into a problem. The problem appears to be more of a RedHat/Cent problem
(or even Glib problem) rather than Syslog-NG problem, but I thought I'd
check with the community before I harass Red Hat. I have the RPM verion
of glib2.12 installed, however the libraries seemed to be labled 2.0.0.
When I configure syslog-ng, I run across the following error:
    checking pkg-config is at least version 0.9.0... yes
    checking for GLIB... no
    configure: error: Cannot find GLib library version >= 2.2: is
pkg-config in path?
    [root at syslog syslog-ng-2.0.5]#
 
I have the rpm installed:
    [root at syslog syslog-ng-2.0.5]# yum list installed glib2
    Loading "installonlyn" plugin
    Loading "fastestmirror" plugin
    Installed Packages
    glib2.i386                               2.12.3-2.fc6
installed
    [root at syslog syslog-ng-2.0.5]#
 
However the libraries seem to be mislabeled:
    [root at syslog syslog-ng-2.0.5]# locate glib | grep "/lib/libg"
    /lib/libglib-2.0.so.0
    /lib/libglib-2.0.so.0.1200.3
    [root at syslog syslog-ng-2.0.5]#
 
Am I just missing something simple, or is this a problem I should be
trying to speak to Cent/Red Hat about?
 
Thanks!
 
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070815/d033c920/attachment.html 

------------------------------

Message: 2
Date: Wed, 15 Aug 2007 14:13:49 +0100
From: "Geller, Sandor \(IT\)" <Sandor.Geller at morganstanley.com>
Subject: Re: [syslog-ng] GLib version problem?
To: "Syslog-ng users' and developers' mailing list"
	<syslog-ng at lists.balabit.hu>
Message-ID:
	<14F0A35F6E466D48BF11108F4E09E68C07CFDC9F at LNWEXMB58.msad.ms.com>
Content-Type: text/plain;	charset="us-ascii"

Hi,

> Hello,
>  
> I'm trying to install syslog-NG onto a Cent OS 5 server, and 
> running into a problem. The problem appears to be more of a 
> RedHat/Cent problem (or even Glib problem) rather than 
> Syslog-NG problem, but I thought I'd check with the community 
> before I harass Red Hat. I have the RPM verion of glib2.12 
> installed, however the libraries seemed to be labled 2.0.0. 

No, it is actually 2.12.3, the binary compatibility version is
2.0

> When I configure syslog-ng, I run across the following error:
>     checking pkg-config is at least version 0.9.0... yes
>     checking for GLIB... no
>     configure: error: Cannot find GLib library version >= 
> 2.2: is pkg-config in path?
>     [root at syslog syslog-ng-2.0.5]#
>  
> I have the rpm installed:
>     [root at syslog syslog-ng-2.0.5]# yum list installed glib2
>     Loading "installonlyn" plugin
>     Loading "fastestmirror" plugin
>     Installed Packages
>     glib2.i386                               2.12.3-2.fc6     
>       installed
>     [root at syslog syslog-ng-2.0.5]#
>  
> However the libraries seem to be mislabeled:
>     [root at syslog syslog-ng-2.0.5]# locate glib | grep "/lib/libg"
>     /lib/libglib-2.0.so.0
>     /lib/libglib-2.0.so.0.1200.3
>     [root at syslog syslog-ng-2.0.5]#
>  
> Am I just missing something simple, or is this a problem I 
> should be trying to speak to Cent/Red Hat about?

You need the development package too. yum install glib2-devel

Regards,

Sandor
--------------------------------------------------------

NOTICE: If received in error, please destroy and notify sender. Sender 
does not intend to waive confidentiality or privilege. Use of this email 

is prohibited when received in error.


------------------------------

Message: 3
Date: Wed, 15 Aug 2007 11:43:45 -0400
From: Valdis.Kletnieks at vt.edu
Subject: Re: [syslog-ng] GLib version problem?
To: Burns Andrew <aburns at snyderdrug.com>
Cc: Syslog-ng users' and developers' mailing list
	<syslog-ng at lists.balabit.hu>
Message-ID: <5364.1187192625 at turing-police.cc.vt.edu>
Content-Type: text/plain; charset="us-ascii"

On Wed, 15 Aug 2007 08:02:38 CDT, Burns Andrew said:

> check with the community before I harass Red Hat. I have the RPM 
verion
> of glib2.12 installed, however the libraries seemed to be labled 
2.0.0.

Do you have glib2-devel RPM installed?  Not having the -devel will cause
these sort of symptoms.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 226 bytes
Desc: not available
Url : 
http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070815/223d607b/attachment-0001.pgp 

------------------------------

Message: 4
Date: Thu, 16 Aug 2007 12:19:35 +0800
From: "Wilson Lai" <wilsonlai at macausjm.com>
Subject: [syslog-ng] Logging Third party application logs to Syslog-NG
	in	Linux!
To: syslog-ng <syslog-ng at lists.balabit.hu>
Message-ID: <H000006e00726bb4.1187237975.mail.macausjm.com at MHS>
Content-Type: text/plain; charset="us-ascii"

Dear ALL,

       I am now using the Syslog-NG OSE for centralized logging system.
How could I get the third party application (Linux Client) logs

    logging to the Syslog-NG server? These third party application logs
are not managed by the Linux syslog daemon.

       Would there be someone gives some information or suggestion!

       Thanks a lot!!!

 

Regards,

Wilson Lai

System Engineer

IT Dept., SJM

Office ( : (853)2978585

Mobile ( : (853)66506709

Email +: : wilsonlai at macausjm.com

 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070816/42188c0b/attachment-0001.htm 

------------------------------

Message: 5
Date: Wed, 15 Aug 2007 21:16:49 -0700 (MST)
From: Bill Nash <billn at billn.net>
Subject: Re: [syslog-ng] Logging Third party application logs to
	Syslog-NG in Linux!
To: "Syslog-ng users' and developers' mailing list"
	<syslog-ng at lists.balabit.hu>
Message-ID: <Pine.LNX.4.64.0708152115400.23309 at pegasus.billn.net>
Content-Type: TEXT/PLAIN; charset=US-ASCII


If the application itself is capable of generating a log file, you can 
specify that log file as a 'source' within the syslog-ng config. Be sure 


to add that source, once declared, to one of your log directives. It's 
really that easy.

- billn

On Thu, 16 Aug 2007, Wilson Lai wrote:

> Dear ALL,
> 
>        I am now using the Syslog-NG OSE for centralized logging 
system.
> How could I get the third party application (Linux Client) logs
> 
>     logging to the Syslog-NG server? These third party application 
logs
> are not managed by the Linux syslog daemon.
> 
>        Would there be someone gives some information or suggestion!
> 
>        Thanks a lot!!!
> 
>  
> 
> Regards,
> 
> Wilson Lai
> 
> System Engineer
> 
> IT Dept., SJM
> 
> Office ( : (853)2978585
> 
> Mobile ( : (853)66506709
> 
> Email +: : wilsonlai at macausjm.com
> 
>  
> 
> 
> 


------------------------------

Message: 6
Date: Thu, 16 Aug 2007 10:16:54 +0200
From: "Martin Voelker" <martin.voelker at westlotto.com>
Subject: [syslog-ng] Troubles with SE-Linux Syslog-ng and ntpd
To: <syslog-ng at lists.balabit.hu>
Message-ID: <46C424160200008C0000855E at Mclp3_server.wl>
Content-Type: text/plain; charset="ISO-8859-1"

Hi,
I have some troubles using syslog-ng on RHEL4 with SE Linux. There comes 

following message:
audit(1187252020.581:514): avc: denied { write } for pid=2646 
comm="ntpd" name="log" dev=tmpfs ino=14840162 
scontext=root:system_r:ntpd_t tcontext=root:object_r:device_t 
tclass=sock_file

What can I do???

Thanks
Martin


Westdeutsche Lotterie GmbH & Co. OHG | Sitz: M?nster
Registergericht: Amtsgericht M?nster 
Handelsregister: M?nster HRA 4379
Gesch?ftsf?hrer: Dr. Winfried Wortmann
Vorsitzender des Beirates: Ernst Gerlach

Gesellschafter:

Nordwestlotto in Nordrhein-Westfalen GmbH | Sitz: M?nster
Registergericht: Amtsgericht M?nster
Handelsregister: HRB 3840
Gesch?ftsf?hrer: Dr. Winfried Wortmann

NRW.BANK | Sitz: D?sseldorf und M?nster
Rechtsform: Anstalt des ?ffentlichen Rechts
Registergerichte: Amtsgerichte D?sseldorf/M?nster
Handelsregister: D?sseldorf HRA 15277/M?nster HRA 5300



------------------------------

_______________________________________________
syslog-ng maillist  -  syslog-ng at lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng


End of syslog-ng Digest, Vol 28, Issue 20
*****************************************




------------------------------

Message: 6
Date: Fri, 17 Aug 2007 09:38:36 +0200
From: Balazs Scheidler <bazsi at balabit.hu>
Subject: Re: [syslog-ng] Buffer Overflow : Insufficient buffer	space
	for	retrieving STREAMS log message; res='2'
To: Syslog-ng users' and developers' mailing list
	<syslog-ng at lists.balabit.hu>
Message-ID: <1187336316.6771.14.camel at bzorp.balabit>
Content-Type: text/plain

On Fri, 2007-08-17 at 09:33 +1200, Russell Fulton wrote:
> Hi Nicolas
> 
> I suggest you post the actual error messages to the list to give a few
> more clues.

and syslog-ng version. 1.6.4 had a related fix.

-- 
Bazsi



------------------------------

Message: 7
Date: Fri, 17 Aug 2007 09:42:29 +0200
From: Balazs Scheidler <bazsi at balabit.hu>
Subject: Re: [syslog-ng] REGEX rewrites on packet body possible?
To: Syslog-ng users' and developers' mailing list
	<syslog-ng at lists.balabit.hu>
Message-ID: <1187336549.6771.17.camel at bzorp.balabit>
Content-Type: text/plain

On Thu, 2007-08-16 at 10:07 -0700, Eli Stair wrote:
> I've got a problem with some network devices that is leading me to 
need to find 
> some way to do regex rewriting of portions of the message body of 
syslog 
> messages.
> 
> Problem: I've got a bunch of Foundry devices that put their hostname 
followed 
> by a comma in the body of the message (and some that do not).  Some of 
these 
> look like this:
> 
> {
> 2007-08-16:2007-08-16T09:50:16-07:00 hostname [hostname.local7.notice] 

> hostname, Linecard Module 13 temperature 50.0 C degrees is normal
> }
> 
> # template("$R_ISODATE $HOST [$PROGRAM.$FACILITY.$PRIORITY] $MSG\n")
> 
> 
> As you can see from the template, the second hostname reported with 
the comma 
> is part of the MSG body.  For reasons of properly searching/indexing 
this data 
> I need to strip this out.  I've seen mention of a tool called 
'syslog-mailer' 
> that sounds like it would do the job somewhat.  Additionally, I've 
seen blog 
> chatter about potentially adding full regex rewrite capability to 
syslog-ng in 
> the recent past.  I can't find evidence of either of these methods 
however.
> 
> My first try at solving this using an external program showed that 
when passing 
> data OUT of syslog-ng to a defined program, only the message body is 
sent and 
> before application of a template, the other information is dropped.  
Thus it's 
> not possible to do processing of the whole payload externally, 
re-import the 
> data via a socket and finish writing because the facility and HOST 
information 
> is all gone!
> 
> I'm looking at writing a log proxy using Net::Dev::Tools::Syslog in 
perl to 
> handle listening, rewriting if necessary, and forwarding full messages 
on to 
> syslog-ng after.  I'd just like to know if there are any better 
suggestions, or 
> if this has been done before successfully in another way?
> 
> Cheers, and thanks for any insight.

You could do something like this:

f_strip_hostname { match("^[\-a-zA-Z0-0]+,(.*)$") or match("^(.*)$"); };

destination d_out { file("/var/log/messages" template("$R_ISODATE $HOST 
[$PROGRAM.$FACILITY.$PRIORITY] $1\n"); };

The filter will make $1 equal to the message part without a hostname, 
either because 
there was a hostname (first match), or because there wasn't.

Improving the hostname pattern would reduce ambiguity, as the pattern 
will strip 
everything till the first comma.

-- 
Bazsi



------------------------------

Message: 8
Date: Fri, 17 Aug 2007 09:45:09 +0200
From: Balazs Scheidler <bazsi at balabit.hu>
Subject: Re: [syslog-ng] Logging Third party application logs to
	Syslog-NG
To: Syslog-ng users' and developers' mailing list
	<syslog-ng at lists.balabit.hu>
Message-ID: <1187336709.6771.21.camel at bzorp.balabit>
Content-Type: text/plain

On Fri, 2007-08-17 at 11:37 +0800, Wilson Lai wrote:
> Dear Bill,
>         If there is a Scalix application which generate the log file 
in 
> "/var/opt/Scalix/ml/s/logs/fatal", where "fatal" is the log file. How 
> could
> the source be defined to be listened by Syslog-NG?

source s_file { file("/var/opt/Scalix/ml/s/logs/fatal" follow_freq(1)); 
};

This will read the log file, checking every second if there are new 
entries. The current position is remembered accross restarts.

You need a recent syslog-ng version though (2.0.5 is fine)

Read
http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch08s01.html#id2550287 
for more information.

-- 
Bazsi



------------------------------

_______________________________________________
syslog-ng maillist  -  syslog-ng at lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng


End of syslog-ng Digest, Vol 28, Issue 21
*****************************************




More information about the syslog-ng mailing list