- syslog-ng - lists.balabit.hu

Re: [syslog-ng] ArcSight (Again)
by Ivey, Chris 17 Sep '07

17 Sep '07

Concatenate, you are a life saver. I appreciate your insight into this issue, and I will forward your thoughts on the ArcSight/RFC/Cisco Messages over to the ArcSight support folks here and ask that they get with ArcSight and see what they think. I went online and looked at docs, figured out the template deal on my local linux box, then changed their destination to come in with template("$MSG\n"). I think that will give them what they need. Thanks again for your insight on this issue! Chris Ivey Affiliated Computer Services Enterprise Management Integration Services Infrastructure Management Senior Analyst chris.ivey(a)acs-inc.com "I have not failed, I have simply found 10,000 ways which do not work!" -- Thomas Edison "When you find yourself in a hole, the best thing to do is stop digging!" -- Nick Stokes "I reject your reality, and substitute my own!" -- Adam Savage -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Concatenate Sent: Friday, September 14, 2007 10:58 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] ArcSight (Again) OBTW, use a template for that destination. IIRC the $MSG macro sent over to them will be exactly what you need. Just change the destination to a file for testing and see what your template does, easy to test that way. When it's all good change it back. ...... Original Message ....... On Fri, 14 Sep 2007 15:49:02 -0500 "Ivey, Chris" <Chris.ivey(a)acs-inc.com> wrote: > > >OK folks, this has come up (again). Seems that the ArcSight parser is not intelligent enough to handle messages coming from syslog-ng after being forwarded along. So I need some advice on how to handle this issue. First, some background... > >I added our ArcSight server as a syslog-ng target some time ago. The folks who use the ArcSight stuff emailed me and said that the parser for ArcSight could not handle parsing the messages coming from syslog-ng, because of the prepending of the server time to the syslog-ng message. Here is an excerpt from one of the emails from their support folks: > >In looking through all the information, I see that there are lot of parsing issues, all due to what look like malformed syslog messages. > >For example: >Apr 25 22:54:24 x.x.x.x router/router 99228: Apr 25 22:54:23.474 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on GigabitEthernet1/0/1 (not full duplex), with switch FastEthernet0/1 (full duplex). > >This is a raw event from the export. Notice that the second timestamp forward is the actual message, which is CDP, so from a cisco switch or some layer 2 device. > >The actual event from the cisco device should look like as follows, which is what our parser is designed to work with: >Apr 25 22:54:23.474 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on GigabitEthernet1/0/1 (not full duplex), with switch FastEthernet0/1 (full duplex). > >Another excerpt... > >I looked over the information you had uploaded already, and is actually a common issue. When syslog events are forwarded from one syslog server to another syslog server, or pipe, or file, the forwarding syslog server prepends timestamp and other information, which makes the message unusable. > >We require syslog message to adhere to the standard RFC syslog format for the connector to read them, and when forwarding syslog messages that is not the case and we are unable to support that configuration. > >So, the question is what to do about it. I apparently need to send this information on to the ArcSight server without the prepended data (the "Apr 25 22:54:24 x.x.x.x router/router 99228:" portion of the message from the first email excerpt), but I need to keep it in place for EVERY other target I am sending to. Can anyone tell me what are my options here, please? Thanks a LOT in advance!!! > >(Bazsi, please feel free to chime in on this one! LOL) > >Chris Ivey > >Affiliated Computer Services >Enterprise Management Integration Services >Infrastructure Management Senior Analyst > >chris.ivey(a)acs-inc.com > >"I have not failed, I have simply found 10,000 ways which do not work!" -- Thomas Edison >"When you find yourself in a hole, the best thing to do is stop digging!" -- Nick Stokes >"I reject your reality, and substitute my own!" -- Adam Savage > >_______________________________________________ >syslog-ng maillist - syslog-ng(a)lists.balabit.hu >https://lists.balabit.hu/mailman/listinfo/syslog-ng >Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > > _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

Re: [syslog-ng] non standard syslog message!
by Wilson Lai 17 Sep '07

17 Sep '07

Dear Balazs, Could you please give me some information about the script which could convert the non syslog message into syslog standard syslog format? Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com -----Original Message----- From: syslog-ng-request(a)lists.balabit.hu [mailto:syslog-ng-request@lists.balabit.hu] Sent: Friday, September 14, 2007 6:00 PM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 29, Issue 11 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: Compiling on HP/UX 11.11 (Balazs Scheidler) 2. Re: Compiling on HP/UX 11.11 (C Wells) 3. Mysql Syslog Data (John Hala) 4. Re: non standard syslog messgae! (Wilson Lai) 5. Re: Mysql Syslog Data (Paul Robert Marino) 6. Re: Compiling on HP/UX 11.11 (Balazs Scheidler) 7. Re: non standard syslog messgae! (Balazs Scheidler) ---------------------------------------------------------------------- Message: 1 Date: Thu, 13 Sep 2007 19:04:10 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] Compiling on HP/UX 11.11 To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1189703050.15286.20.camel(a)bzorp.balabit> Content-Type: text/plain On Mon, 2007-09-10 at 10:34 -0700, C Wells wrote: > This glib listed below worked fine (thanks), but then > it asked for eventlog version >= 0.2, which I found at > balabit and installed and then it asked for libnet, > for which I found the binary, so finally the configure > worked. Make asked for flex, which I had renamed > because it made configure die. I put flex back and it > passed that point but dies on this > > gcc -g -O2 -Wall -g -o syslog-ng main.o > libsyslog-ng.a -lnsl -lrt -L/usr/local/lib > -lglib-2.0 -lintl -liconv -L/usr/local/lib -levtlog > -lnet > /usr/ccs/bin/ld: Unsatisfied symbols: > linenum (first referenced in > libsyslog-ng.a(cfg-grammar.o)) (data) > lookup_parse_flag (first referenced in > libsyslog-ng.a(cfg-grammar.o)) (code) > strtoll (first referenced in > libsyslog-ng.a(affile.o)) (code) > yylex (first referenced in > libsyslog-ng.a(cfg-grammar.o)) (code) > lex_init (first referenced in > libsyslog-ng.a(cfg.o)) (code) > collect2: ld returned 1 exit status your ld is fine. As it seems cfg-lex.c was regenerated and probably empty, that's why it does not contain the lookup_parse_flag() function. Try removing cfg-lex.c, and rerun make. It should regenerate cfg-lex.c using flex. As I see the configure test did not find libfl.a (or libfl.so) for some reason. Try adding that to your link command line. What's the error message of configure that you get if you have flex installed and not renamed? -- Bazsi ------------------------------ Message: 2 Date: Thu, 13 Sep 2007 11:16:31 -0700 (PDT) From: C Wells <s2audi(a)yahoo.com> Subject: Re: [syslog-ng] Compiling on HP/UX 11.11 To: syslog-ng(a)lists.balabit.hu Message-ID: <663351.40801.qm(a)web60422.mail.yahoo.com> Content-Type: text/plain; charset=iso-8859-1 > What's the error message of configure that you get if you have flex installed and not renamed? 'checking lex output file root... configure: error: cannot find output from flex; giving up' I guess I can try a newer flex maybe, not sure Thanks ________________________________________________________________________ ____________ Tonight's top picks. What will you watch tonight? Preview the hottest shows on Yahoo! TV. http://tv.yahoo.com/ ------------------------------ Message: 3 Date: Thu, 13 Sep 2007 22:24:02 -0400 From: John Hala <john.hala(a)villanova.edu> Subject: [syslog-ng] Mysql Syslog Data To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <294949F5411DFC418D1DEC84BE825AE11DA3CFEA66(a)VUEX2.vuad.villanova.ed u> Content-Type: text/plain; charset="us-ascii" So I have my syslogs going to a mysql database. What are some recommended ways to make this data useful? Here's how I created the table: CREATE TABLE syslog ( host varchar(32) default NULL, facility varchar(10) default NULL, priority varchar(10) default NULL, level varchar(10) default NULL, tag varchar(10) default NULL, date date default NULL, time time default NULL, program varchar(15) default NULL, msg text, seq int(10) unsigned NOT NULL auto_increment, PRIMARY KEY (seq), KEY host (host), KEY seq (seq), KEY program (program), KEY time (time), KEY date (date), KEY priority (priority), KEY facility (facility) ) TYPE=MyISAM; ------------------------------ Message: 4 Date: Fri, 14 Sep 2007 12:03:46 +0800 From: "Wilson Lai" <wilsonlai(a)macausjm.com> Subject: Re: [syslog-ng] non standard syslog messgae! To: syslog-ng <syslog-ng(a)lists.balabit.hu> Message-ID: <H000006e0084a3d0.1189742625.mail.macausjm.com@MHS> Content-Type: text/plain; charset="US-ASCII" Hi, The message is not generated from a Cisco device. It is a third party application log which has the format as follow ; " Error Browser (Service 14) Thu May 10 01:52:15 2007 [OM 0] Pid of logging process: 1029 Last Msg ID : JavaMail.root(a).scalix.x.y.com Last Msg DirectRef: 000a4beace41e153 " " How could I convert it into a standard syslog format? Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com -----Original Message----- From: syslog-ng-request(a)lists.balabit.hu [mailto:syslog-ng-request@lists.balabit.hu] Sent: Thursday, September 13, 2007 6:00 PM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 29, Issue 10 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: syslog-ng Digest, Vol 28, Issue 21 (Balazs Scheidler) ---------------------------------------------------------------------- Message: 1 Date: Wed, 12 Sep 2007 17:06:49 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 28, Issue 21 To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1189609609.7181.4.camel(a)bzorp.balabit> Content-Type: text/plain On Fri, 2007-09-07 at 07:26 -0700, Nate Campi wrote: > On Fri, Sep 07, 2007 at 05:26:02PM +0800, Wilson Lai wrote: > > Dear all, > > What happen if the log message is not a standard syslog message? > > Thanks. > > If a Cisco switch sends a message like this: > 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16 > > ...it'll be written to disk like this: > > Aug 23 03:04:05 switch.company.com 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16 > > syslog servers put in a proper syslog formatted header. > > The behavior is documented here: > > http://www.faqs.org/rfcs/rfc3164.html > > It's not syslog-ng specific behavior. In fact I've added some Cisco date stamp support, so date stamps of some of the Cisco gear are properly recognized. But Cisco is not using consistent timestamps in their different product lines. -- Bazsi ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 29, Issue 10 ***************************************** ------------------------------ Message: 5 Date: Fri, 14 Sep 2007 03:08:47 -0400 From: Paul Robert Marino <prmarino1(a)gmail.com> Subject: Re: [syslog-ng] Mysql Syslog Data To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Message-ID: <1189753731.9CBEB49(a)di12.dngr.org> Content-Type: text/plain; charset="us-ascii"; format="flowed" Have you looked at the phpsyslog-ng project on source forge its quite nice and usefull my only complaint about it is it only works with mysql. ___________________________________________ The average person does a lot of work in the name of laziness! Save youre self the effort by doing it right the first time. Do it with free speech software. ------------------------------ Message: 6 Date: Fri, 14 Sep 2007 10:28:04 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] Compiling on HP/UX 11.11 To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1189758484.7167.3.camel(a)bzorp.balabit> Content-Type: text/plain On Thu, 2007-09-13 at 11:16 -0700, C Wells wrote: > > What's the error message of configure that you get > if you have flex installed and not renamed? > > 'checking lex output file root... configure: error: > cannot find output from flex; giving up' > I guess I can try a newer flex maybe, not sure The config.log file might have more details. The error basically says that the output for flex was not found. Are you sure it is using flex and not the system installed lex? Again, the config.log file has more details. The flex/lex commands generate their output as a fixed named file (usually lex.yy.c) and the makefiles cannot find this file. Try running the flex command line by hand and check whether the file gets generated. -- Bazsi ------------------------------ Message: 7 Date: Fri, 14 Sep 2007 10:29:48 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] non standard syslog messgae! To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1189758588.7167.6.camel(a)bzorp.balabit> Content-Type: text/plain On Fri, 2007-09-14 at 12:03 +0800, Wilson Lai wrote: > Hi, > The message is not generated from a Cisco device. It is a third > party application log which has the format as follow ; > " Error Browser (Service 14) Thu May 10 01:52:15 > 2007 > [OM 0] > Pid of logging process: 1029 > Last Msg ID : JavaMail.root(a).scalix.x.y.com > Last Msg DirectRef: 000a4beace41e153 " " > How could I convert it into a standard syslog format? > Thanks. Is this a log file currently? Syslog-ng would convert this multi-line log message as individual log entries, which is probably not what you want. You can use a script or something that makes this look like syslog and then write it to a named pipe or something and have syslog-ng read that. -- Bazsi ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 29, Issue 11 *****************************************

1 0

Re: [syslog-ng] non standard syslog messgae!
by Wilson Lai 14 Sep '07

14 Sep '07

Hi, The message is not generated from a Cisco device. It is a third party application log which has the format as follow ; " Error Browser (Service 14) Thu May 10 01:52:15 2007 [OM 0] Pid of logging process: 1029 Last Msg ID : JavaMail.root(a).scalix.x.y.com Last Msg DirectRef: 000a4beace41e153 " " How could I convert it into a standard syslog format? Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com -----Original Message----- From: syslog-ng-request(a)lists.balabit.hu [mailto:syslog-ng-request@lists.balabit.hu] Sent: Thursday, September 13, 2007 6:00 PM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 29, Issue 10 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Re: syslog-ng Digest, Vol 28, Issue 21 (Balazs Scheidler) ---------------------------------------------------------------------- Message: 1 Date: Wed, 12 Sep 2007 17:06:49 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 28, Issue 21 To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1189609609.7181.4.camel(a)bzorp.balabit> Content-Type: text/plain On Fri, 2007-09-07 at 07:26 -0700, Nate Campi wrote: > On Fri, Sep 07, 2007 at 05:26:02PM +0800, Wilson Lai wrote: > > Dear all, > > What happen if the log message is not a standard syslog message? > > Thanks. > > If a Cisco switch sends a message like this: > 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16 > > ...it'll be written to disk like this: > > Aug 23 03:04:05 switch.company.com 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16 > > syslog servers put in a proper syslog formatted header. > > The behavior is documented here: > > http://www.faqs.org/rfcs/rfc3164.html > > It's not syslog-ng specific behavior. In fact I've added some Cisco date stamp support, so date stamps of some of the Cisco gear are properly recognized. But Cisco is not using consistent timestamps in their different product lines. -- Bazsi ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 29, Issue 10 *****************************************

2 1

Re: [syslog-ng] Compiling on HP/UX 11.11
by C Wells 14 Sep '07

14 Sep '07

> What's the error message of configure that you get if you have flex installed and not renamed? 'checking lex output file root... configure: error: cannot find output from flex; giving up' I guess I can try a newer flex maybe, not sure Thanks ____________________________________________________________________________________ Tonight's top picks. What will you watch tonight? Preview the hottest shows on Yahoo! TV. http://tv.yahoo.com/

2 1

IP forwarding
by James B Horwath 14 Sep '07

14 Sep '07

Is it possible to forward only certain IP ranges in syslog-ng when it is running in server mode? If I have 10.10.10.0\24 10.10.20.0\24 10.20.10.0\0 and I want to forward 10.20.10.0\0 and 10.10.10.0\24 and nothing else is this possible. Regards, Jim ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

4 3

Re: [syslog-ng] Compiling on HP/UX 11.11
by C Wells 13 Sep '07

13 Sep '07

This glib listed below worked fine (thanks), but then it asked for eventlog version >= 0.2, which I found at balabit and installed and then it asked for libnet, for which I found the binary, so finally the configure worked. Make asked for flex, which I had renamed because it made configure die. I put flex back and it passed that point but dies on this gcc -g -O2 -Wall -g -o syslog-ng main.o libsyslog-ng.a -lnsl -lrt -L/usr/local/lib -lglib-2.0 -lintl -liconv -L/usr/local/lib -levtlog -lnet /usr/ccs/bin/ld: Unsatisfied symbols: linenum (first referenced in libsyslog-ng.a(cfg-grammar.o)) (data) lookup_parse_flag (first referenced in libsyslog-ng.a(cfg-grammar.o)) (code) strtoll (first referenced in libsyslog-ng.a(affile.o)) (code) yylex (first referenced in libsyslog-ng.a(cfg-grammar.o)) (code) lex_init (first referenced in libsyslog-ng.a(cfg.o)) (code) collect2: ld returned 1 exit status Do I need another copy of ld ? Am I closer or further away ? Thanks ============================================== IIRC there's a glib 2.12.x package on the Utah site, which should be fine. http://hpux.connect.org.uk/hppd/hpux/Gtk/Development/glib2-2.12.13/ We have binaries for HP-UX (PA-RISC), but it is a service that we offer for an annual fee. ____________________________________________________________________________________ Need a vacation? Get great deals to amazing places on Yahoo! Travel. http://travel.yahoo.com/

2 1

Re: [syslog-ng] syslog-ng Digest, Vol 28, Issue 21
by Wilson Lai 12 Sep '07

12 Sep '07

Dear all, What happen if the log message is not a standard syslog message? Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com -----Original Message----- From: syslog-ng-request(a)lists.balabit.hu [mailto:syslog-ng-request@lists.balabit.hu] Sent: Friday, August 17, 2007 3:45 PM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 28, Issue 21 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. new syslog-ng white paper published (Balazs Scheidler) 2. Buffer Overflow : Insufficient buffer space for retrieving STREAMS log message; res='2' ( Nicolas de Marqu? - Fromentin ) 3. REGEX rewrites on packet body possible? (Eli Stair) 4. Re: Buffer Overflow : Insufficient buffer space for retrieving STREAMS log message; res='2' (Russell Fulton) 5. Re: Logging Third party application logs to Syslog-NG (Wilson Lai) 6. Re: Buffer Overflow : Insufficient buffer space for retrieving STREAMS log message; res='2' (Balazs Scheidler) 7. Re: REGEX rewrites on packet body possible? (Balazs Scheidler) 8. Re: Logging Third party application logs to Syslog-NG (Balazs Scheidler) ---------------------------------------------------------------------- Message: 1 Date: Thu, 16 Aug 2007 16:58:31 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: [syslog-ng] new syslog-ng white paper published To: syslog-ng(a)lists.balabit.hu Message-ID: <1187276311.7565.52.camel(a)bzorp.balabit> Content-Type: text/plain Dear all, We have published a new, syslog-ng related white paper on our website titled: "Distributed syslog architectures with syslog-ng" It basically contains the various deployment scenarios, their pros and cons we've encountered in our experience. It does contain information on syslog-ng Premium Edition, but still we tried to concentrate on the technical content, and some of it equally applies to syslog-ng OSE. We'd appreciate feedback at documentation(a)balabit.com. Thanks. -- Bazsi ------------------------------ Message: 2 Date: Thu, 16 Aug 2007 18:40:03 +0200 From: " Nicolas de Marqu? - Fromentin " <nicolas.demarque(a)gmail.com> Subject: [syslog-ng] Buffer Overflow : Insufficient buffer space for retrieving STREAMS log message; res='2' To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Message-ID: <b778b3cc0708160940y50d0dfb4yc84f86dffa4efc6a(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" Hello, During a big flow from a local server to a central server, we have a "buffer overflow" on the local server. After this message, syslog-ng is crash and don't be capable to send any messages. The local server is a solaris 10 product, the central linux a redhat EL4. Do you know a rule to stop this problem Best regard, Nicolas

3 2

Frances Wand/Poughkeepsie/IBM is out of the office.
by Frances Wand 10 Sep '07

10 Sep '07

I will be out of the office starting 09/10/2007 and will not return until 09/17/2007. Please contact Suzanne Mahoney/Boulder/IBM@IBMUS for DataPower project questions, and Murugesan Natarajan/India/IBM@IBMIN for any AWStats questions. Please call my cell phone listed in blue page for urgent issues. Thank you!

1 0

Logs from Nauticus
by Delphine D 10 Sep '07

10 Sep '07

Hello, Is there someone here having some knowledge of how to send logs from Nauticus (Sun Secure Application Switch Manager N2120) to syslog-ng ? I receive the logs on the centralized logs server but without any information about the source of the logs (no IP, no hostname). In other words : Sep 7 14:03:29 2007 v0 AAA [0/100e8] [ERROR]: User authentication failure, user: 'test', host: 1.2.3.4, application: httpLogin, method: TACACS(serviceNotAvailable) serviceNotAvailable instead of : Sep 7 14:03:29 2007/nauticus.ourdomain.be v0 AAA [0/100e8] [ERROR]: User authentication failure, user: 'test', host: 1.2.3.4, application: httpLogin, method: TACACS(serviceNotAvailable) serviceNotAvailable Is there a paramater to change in the N2120 ? Thank you very much. _________________________________________________________________ A la recherche d'un ami d'enfance ? Peut-être est-il dans la liste d'amis de vos amis ! http://spaces.live.com/default.aspx?page=Ed01&ss=True

2 4

Re: [syslog-ng] syslog-ng Digest, Vol 29, Issue 7
by Wilson Lai 10 Sep '07

10 Sep '07

Dear Nate, As your words, there will be no solution to standardize the log format! Thanks for your reply! Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com -----Original Message----- From: syslog-ng-request(a)lists.balabit.hu [mailto:syslog-ng-request@lists.balabit.hu] Sent: Saturday, September 08, 2007 6:00 PM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 29, Issue 7 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Logs from Nauticus (Delphine D) 2. Re: Hostname instead of FQDN in logs (Nate Campi) 3. Re: syslog-ng Digest, Vol 28, Issue 21 (Nate Campi) 4. Re: Logs from Nauticus (Nate Campi) 5. Re: Logs from Nauticus (Delphine D) 6. Re: Logs from Nauticus (Nate Campi) 7. Re: How to "drop" a message, effectively skipping further processing and not logging it? (Eli Stair) ---------------------------------------------------------------------- Message: 1 Date: Fri, 07 Sep 2007 15:18:27 +0200 From: "Delphine D" <delphined_1300(a)hotmail.com> Subject: [syslog-ng] Logs from Nauticus To: syslog-ng(a)lists.balabit.hu Message-ID: <BLU106-F41DCA93C3393AA7FA06E78E1C50(a)phx.gbl> Content-Type: text/plain; charset=iso-8859-1; format=flowed Hello, Is there someone here having some knowledge of how to send logs from Nauticus (Sun Secure Application Switch Manager N2120) to syslog-ng ? I receive the logs on the centralized logs server but without any information about the source of the logs (no IP, no hostname). In other words : Sep 7 14:03:29 2007 v0 AAA [0/100e8] [ERROR]: User authentication failure, user: 'test', host: 1.2.3.4, application: httpLogin, method: TACACS(serviceNotAvailable) serviceNotAvailable instead of : Sep 7 14:03:29 2007/nauticus.ourdomain.be v0 AAA [0/100e8] [ERROR]: User authentication failure, user: 'test', host: 1.2.3.4, application: httpLogin, method: TACACS(serviceNotAvailable) serviceNotAvailable Is there a paramater to change in the N2120 ? Thank you very much. _________________________________________________________________ A la recherche d'un ami d'enfance ? Peut-?tre est-il dans la liste d'amis de vos amis ! http://spaces.live.com/default.aspx?page=Ed01&ss=True ------------------------------ Message: 2 Date: Fri, 7 Sep 2007 07:24:03 -0700 From: Nate Campi <nate(a)campin.net> Subject: Re: [syslog-ng] Hostname instead of FQDN in logs To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <20070907142403.GB6166(a)campin.net> Content-Type: text/plain; charset=us-ascii On Fri, Sep 07, 2007 at 10:53:54AM +0200, Delphine D wrote: > > The others also return the hostname and not the FQDN (Ex : 'server2' and not > 'server2.ourdomain.be') but they are using syslog instead of syslog-ng... > That's the only difference... Then it's because this host sends the hostname in the syslog message (syslog-ng always has a full and complete syslog message on the wire), but the boxes using syslogd don't actually send a hostname. A message on your central syslog-ng server from a Linux box running syslogd will be written to disk something like: Sep 7 07:16:20 hostname in.qpopper[7736]: connect from 12.12.12.12 ...but on the wire it looks like this: <13>in.qpopper[7736]: connect from 12.12.12.12 ...and syslog-ng has to put in the rest of the info. This means that syslog-ng on the central box is putting in the FQDN for you. syslog-ng on the client is putting in a full message, including the short hostname, and the central syslog-ng is keeping it. See http://www.campin.net/syslog-ng/syslog.html#missing_parts for more on this. See http://www.campin.net/syslog-ng/faq.html#hostname to figure out the hostname options you want on your central syslog-ng server. Probably "keep_hostname(no)", plus "use_fqdn(yes);" to get the FQDN. HTH, -- Nate Like medieval peasants, computer manufacturers and millions of users are locked in a seemingly eternal lease with their evil landlord, who comes around every two years to collect billions of dollars of taxes in return for mediocre services. --Mark Harris, Electronics Times ------------------------------ Message: 3 Date: Fri, 7 Sep 2007 07:26:12 -0700 From: Nate Campi <nate(a)campin.net> Subject: Re: [syslog-ng] syslog-ng Digest, Vol 28, Issue 21 To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <20070907142612.GC6166(a)campin.net> Content-Type: text/plain; charset=us-ascii On Fri, Sep 07, 2007 at 05:26:02PM +0800, Wilson Lai wrote: > Dear all, > What happen if the log message is not a standard syslog message? > Thanks. If a Cisco switch sends a message like this: 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16 ...it'll be written to disk like this: Aug 23 03:04:05 switch.company.com 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16 syslog servers put in a proper syslog formatted header. The behavior is documented here: http://www.faqs.org/rfcs/rfc3164.html It's not syslog-ng specific behavior. -- Nate "The IBM compatible sector has not yet recognized that 95% of computer usage is devoted to experimenting with different fonts and character styles in documents." - Reiner, Ron ------------------------------ Message: 4 Date: Fri, 7 Sep 2007 07:35:54 -0700 From: Nate Campi <nate(a)campin.net> Subject: Re: [syslog-ng] Logs from Nauticus To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <20070907143554.GD6166(a)campin.net> Content-Type: text/plain; charset=us-ascii On Fri, Sep 07, 2007 at 03:18:27PM +0200, Delphine D wrote: > > I receive the logs on the centralized logs server but without any > information about the source of the logs (no IP, no hostname). > > In other words : > > Sep 7 14:03:29 2007 v0 AAA [0/100e8] [ERROR]: User authentication failure, > user: 'test', host: 1.2.3.4, application: httpLogin, method: > TACACS(serviceNotAvailable) serviceNotAvailable > > instead of : > > Sep 7 14:03:29 2007/nauticus.ourdomain.be v0 AAA [0/100e8] [ERROR]: User > authentication failure, user: 'test', host: 1.2.3.4, application: httpLogin, > method: TACACS(serviceNotAvailable) serviceNotAvailable > > Is there a paramater to change in the N2120 ? Those aren't standard syslog messages, and it's possible that paired with how Solaris sends a header but not a hostname, syslog-ng could be getting confused about this. You should send your "options" part of your syslog-ng.conf, and read http://www.campin.net/syslog-ng/syslog.html to see if it helps you understand what the messages look like on the wire and how syslog-ng makes it's best guesses about what the fields mean. Something similar is the reason for the "bad_hostname" option, but that's for when program names look like hostnames. You have a header section that looks like a hostname, but I'm not sure if you have a keep_hostname(no) that's stripping out your hostname from that weird header section that looks like syslog-ng's "chain_hostnames". So send your options to the list, try setting keep_hostname(yes), or see if you can force a normal syslog format on the client side. What they're sending is wrong in a new way that isn't worked around in syslog-ng (AFAIK). -- Nate "Reader, suppose you were an idiot. And suppose you were a member of Congress. But I repeat myself." - Samuel Clemens ------------------------------ Message: 5 Date: Fri, 07 Sep 2007 16:47:31 +0200 From: "Delphine D" <delphined_1300(a)hotmail.com> Subject: Re: [syslog-ng] Logs from Nauticus To: syslog-ng(a)lists.balabit.hu Message-ID: <BLU106-F26C91524EFE7580500E939E1C50(a)phx.gbl> Content-Type: text/plain; charset=iso-8859-1; format=flowed >On Fri, Sep 07, 2007 at 03:18:27PM +0200, Delphine D wrote: > > > > I receive the logs on the centralized logs server but without any > > information about the source of the logs (no IP, no hostname). > > > > In other words : > > > > Sep 7 14:03:29 2007 v0 AAA [0/100e8] [ERROR]: User authentication >failure, > > user: 'test', host: 1.2.3.4, application: httpLogin, method: > > TACACS(serviceNotAvailable) serviceNotAvailable > > > > instead of : > > > > Sep 7 14:03:29 2007/nauticus.ourdomain.be v0 AAA [0/100e8] [ERROR]: >User > > authentication failure, user: 'test', host: 1.2.3.4, application: >httpLogin, > > method: TACACS(serviceNotAvailable) serviceNotAvailable > > > > Is there a paramater to change in the N2120 ? > >Those aren't standard syslog messages, and it's possible that paired >with how Solaris sends a header but not a hostname, syslog-ng could be >getting confused about this. You should send your "options" part of your >syslog-ng.conf, and read http://www.campin.net/syslog-ng/syslog.html to >see if it helps you understand what the messages look like on the wire >and how syslog-ng makes it's best guesses about what the fields mean. > >Something similar is the reason for the "bad_hostname" option, but >that's for when program names look like hostnames. You have a header >section that looks like a hostname, but I'm not sure if you have a >keep_hostname(no) that's stripping out your hostname from that weird >header section that looks like syslog-ng's "chain_hostnames". > >So send your options to the list, try setting keep_hostname(yes), or see >if you can force a normal syslog format on the client side. What they're >sending is wrong in a new way that isn't worked around in syslog-ng >(AFAIK). Thank you Nate for your help. Here is the syslog-ng.conf from my logs server : options { create_dirs(yes); dir_perm(0705); dir_owner(root); perm(0600); owner(root); sync(0); check_hostname(no); use_fqdn(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(604800); dns_cache_size(400); stats(60); keep_hostname(yes); chain_hostnames(yes); }; I'm not sure that we've the ability to change something in the Nauticus. There is no Syslog or Syslog-ng running on it. There is no configuration files like in "normal" servers (Linux, Solaris,...). There is only a parameters section in the GUI, where you have to configure : - SysLog Host --> IP of the logs server - Syslog Port --> 514 - Filter --> defaultSyslog (by default) - Facility --> local0, local1,.... or local7 But I don't find anything about hostname. The strangest thing is that it was working fine a few weeks ago but it has suddenly stopped working :-( Thanks. _________________________________________________________________ Saviez-vous que Windows Live Messenger est disponible d?s maintenant sur votre GSM ? http://get.live.com/messenger/mobile ------------------------------ Message: 6 Date: Fri, 7 Sep 2007 08:59:05 -0700 From: Nate Campi <nate(a)campin.net> Subject: Re: [syslog-ng] Logs from Nauticus To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <20070907155905.GF6166(a)campin.net> Content-Type: text/plain; charset=us-ascii On Fri, Sep 07, 2007 at 04:47:31PM +0200, Delphine D wrote: > > Thank you Nate for your help. > > Here is the syslog-ng.conf from my logs server : > > options { create_dirs(yes); <snip> > chain_hostnames(yes); > }; Try chain_hostnames(no). No matter what, you're dealing with a bad format that's close enough to a good one to make syslog-ng guess incorrectly. This isn't the fault of syslog-ng, just a result of the imperfect world we live in. > I'm not sure that we've the ability to change something in the Nauticus. > There is no Syslog or Syslog-ng running on it. There is no configuration > files like in "normal" servers (Linux, Solaris,...). > > There is only a parameters section in the GUI, where you have to configure : > > - SysLog Host --> IP of the logs server > - Syslog Port --> 514 > - Filter --> defaultSyslog (by default) > - Facility --> local0, local1,.... or local7 Any other options? Any other Filter options, specifically? I've actually run perl-based syslog proxy code I wrote to work around really crappy syslog formats. The bad_hostname option came as a result of me bitching about the bad formats loudly enough and long enough. :) ISTR that syslog-ng 2.0 now has some rewriting ability. I haven't been paying much attention to that branch, you should probably read up on it. Possible that it's only in the dev branch - like I said I haven't been paying much attention. > But I don't find anything about hostname. > > The strangest thing is that it was working fine a few weeks ago but it has > suddenly stopped working :-( Somebody changed either the syslog-ng server or your client. Maybe not even configs, but version upgrade on the client perhaps. -- Nate An Emacs reference mug is what I want. It would hold ten gallons of coffee. -- Steve VanDevender And, no doubt, have a lid that could only be removed with an obscure finger combination requiring both hands. (Ctrl-Alt-Meta-X gimme-the-damn-coffee) -- William Beegle ------------------------------ Message: 7 Date: Fri, 07 Sep 2007 14:39:26 -0700 From: Eli Stair <estair(a)ilm.com> Subject: Re: [syslog-ng] How to "drop" a message, effectively skipping further processing and not logging it? To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <46E1C50E.9060405(a)ilm.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Geller, Sandor (IT) wrote: > Hello, > > > Is there any way other than this to keep a message from being > > sent to another > > host or a file? If not, is there any way to tell syslog-ng > > NOT to modify the > > state of a file, just to send data TO it? That, or an > > internal "drop" > > stop-processing built-in destination would be extremely useful. > > Simply remove all destinations from the log section and use > flags(final) will do the trick. > Don't I feel daft... I recall trying that a couple years ago, but had errors I don't seem to have resolved before deciding to just write to /dev/null. I must've botched the syntax and thought that you HAD to have a destination in a log line! Thanks for the suggestion. Cheers, /eli ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 29, Issue 7 ****************************************

1 0

How to "drop" a message, effectively skipping further processing and not logging it?
by Eli Stair 07 Sep '07

07 Sep '07

In the cases where there are frequent messages that are known to be trivial and not desired to be seen, I have always dumped them to a "/dev/null" rule like this below on the sender. I would really like to get rid of the modification of /dev/ file permissions, as I find it the opposite of good security to have the syslog daemon modifying files that need to be accessible for many things to work right. This also clashes with things like udev and cfengine, which exist TO properly set perms and maintain files. Is there any way other than this to keep a message from being sent to another host or a file? If not, is there any way to tell syslog-ng NOT to modify the state of a file, just to send data TO it? That, or an internal "drop" stop-processing built-in destination would be extremely useful. Existing things I've tried (/dev/null and fake localhost UDP target)and problems I'm having with them, any suggested workarounds? ( clients running syslog-ng 1.6.5 ) ### conf: destination null { file("/dev/null" owner(root) group(root) perm(0666) dir_perm(0666) create_dirs(no)); }; log { source(src); filter(f_getty_serial_err); destination(null); flags(final); }; # error: 2007-09-06T12:21:33-07:00 hostname [syslog.err] syslog-ng[9467]: Changing permissions on special file /dev/null ### This requires setting the perms correctly, since syslog-ng INSISTS for "security reasons" on modifying/setting the perms of any file it writes to.. and so I write drop rules for the "Changing permissions on special file" errors :) I've tried to work around this by logging to a localhost UDP blackhole that should avoid this, but am having issues with that: ### conf: destination net-blackhole { udp("127.0.0.1" port(666) localip(127.0.0.1)); }; log { source(src); filter(f_getty_serial_err); destination(net-blackhole); flags(final); }; # error: 2007-09-06T13:18:00-07:00 deathstar1401 [syslog.err] syslog-ng[25010]: io.c: do_write: write() failed (errno 111), Connection refused 2007-09-06T13:18:00-07:00 deathstar1401 [syslog.err] syslog-ng[25010]: Connection broken to AF_INET(127.0.0.1:666), reopening in 60 seconds ### pcap: 13:14:59.404524 IP (tos 0x0, ttl 64, id 31643, offset 0, flags [DF], length: 110) 127.0.0.1.51170 > 127.0.0.1.666: [bad udp cksum dfe5!] UDP, length: 82 13:15:59.855390 IP (tos 0x0, ttl 64, id 46755, offset 0, flags [DF], length: 110) 127.0.0.1.51230 > 127.0.0.1.666: [bad udp cksum a8e4!] UDP, length: 82 13:16:59.658501 IP (tos 0x0, ttl 64, id 61705, offset 0, flags [DF], length: 110) 127.0.0.1.51237 > 127.0.0.1.666: [bad udp cksum 9be4!] UDP, length: 82 ### It looks from the error like syslog-ng thinks it is using TCP... since UDP syslog has no connection/handshake at either the IP or application protocol level I don't know why it is having any connection errors. The packets are being successfully sent, and a netcat listen shows that they're perfect... Even an ICMP PORT UNREACHABLE shouldn't have any effect on the sending process. Any thoughts on what's going on here? Thanks, /eli

2 2

Hostname instead of FQDN in logs
by Delphine D 07 Sep '07

07 Sep '07

Hello, We've configured a centralized logs server (Solaris 10) in order to collect and manage all log messages coming from +/- 100 servers (Solaris, Linux,...). For an unknown reason, the logs coming from one of our server (Debian) are coming in the following format on the log server : Sep 7 10:08:37 server1 PAM_unix[6142]: authentication failure; (uid=0) -> delphine for ssh service As you can see, we only receive the hostname (server1) but not the FQDN of this server (server1.ourdomain.be). For all the other servers, we have the FQDN in the logs. Here is an example with server2.ourdomain.be (Debian) : Sep 7 10:11:01 server2.ourdomain.be/server2.ourdomain.be PAM_unix[27542]: authentication failure; (uid=0) -> delphine for ssh service The only difference between server1 and the other ones is that it uses syslog-ng instead of syslog in order to send its logs. Here are the options used in the configuration files. 1° On the log server options { create_dirs(yes); dir_perm(0705); dir_owner(root); perm(0600); owner(root); sync(0); check_hostname(no); use_fqdn(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(604800); dns_cache_size(400); stats(60); keep_hostname(yes); chain_hostnames(yes); }; 2° On server1 options { use_fqdn(yes); use_dns(yes); keep_hostname(yes); chain_hostnames(no); long_hostnames(no); sync(0); }; 3° On server2 we are using syslog instead syslog-ng Any idea ? Thanks. _________________________________________________________________ Grand passioné ? Rassemblez tout ce qui vous intéresse en un seul endroit ! http://get.live.com/live/features

3 3

forwarding to server
by Chris Harner 06 Sep '07

06 Sep '07

Hello everyone! I have a Cacti Server with the Haloe plugin installed. I can get my Windows boxes to talk to the Haloe plugin (syslogs). I have two old servers (running SLES 9 I think) with syslogd installed, reporting in just fine! Here is my issue - I can't figure out how to pipe or forward the data from the box to the syslogs server (Cacti). For those of you who do not know, Cacti is a network monitoring tool/RRDtool front end with a ton of available plugins. Haloe is a centralized syslog server. Thanks! -- Chris Harner [] Server Engineer [] Arnold Transportation Services

5 12

Compiling on HP/UX 11.11
by C Wells 05 Sep '07

05 Sep '07

Is there any difference between trying to compile syslog-ng on HP/UX 11.11 and beating your head against the wall until it bleeds ? I've loaded glib, flex, and many other things it complains about. Even after loading flex, flex can't find what it needs, even after I link the lib it is looking for. So I rename flex so it won't use it, then it chokes on glib, not high enough version. No glib on the Univ of Utah site, so I try to compile and then get lost down that trail of tears. Amazing, who has done it ? May I kiss your feet. Will you accept tithing ? Seriously how can you get this to work on hp/ux (open source version). Thanks ____________________________________________________________________________________ Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games. http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow

2 1

syslog-ng and OpenBSD 4.1
by mailing list 04 Sep '07

04 Sep '07

Anyone has documents and/or information on syslog-ng 2.0.x installation on an OpenBSD 4.1 system? Please forward links, text, docs, etc. I've searched the web and comes up only with OpenBSD 3.5 and older version of syslog-ng (1.5). There's a syslog-ng package for OpenBSD 4.1, but it's 1.6.8. This OpenBSD system will be a syslog-ng client, which will forward the logs to a server (Solaris). The docs at http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html deals with package installation. Is there a doc showing installation from source? or is it simply "make" and "make install" and that's it? Thank you. -- Mike ____________________________________________________________________________________ Sick sense of humor? Visit Yahoo! TV's Comedy with an Edge to see what's on, when. http://tv.yahoo.com/collections/222

1 0

Chroot/Vserver config
by Ed W 04 Sep '07

04 Sep '07

Hi I am setting up a bunch of vserver machines on a single host (basically just a fancy chroot setup) - just looking for advice on a decent setup. 1) Would prefer not to run multiple instances of syslogd in each vserver if possible (no special reason, just seems pointless). 2) Each vserver runs multiple processes, so need a fairly fully featured set of logging for each 3) Want to keep the logs readonly, or basically untouchable in the vservers So I tried a) setting up the host system to run the /vserver/xxx/dev/log as it's source b) using destinations of the form: "/var/log/$HOST/messages", "/var/log/$HOST/mail.log", etc However, this doesn't seem to work because $HOST always shows the same name for every vhost (actually it has the name of the host server) I tried setting up syslog-ng in each vserver to log via a unix-stream to the host and again this has the same issue $HOST is always the same as the host machine What does seem to work is running syslog-ng in each vhost and logging via TCP. OR using duplicating my destination {} and log{} stanzas for every single vserver, ie basically filtering on the source{}. However, this later option seems error prone and boring to maintain, the former option seems to need extra resources and unneccesarily complicated for such a simple setup What I really want is some way that I can set some MACRO in the "source{}" stanza and then use that later on in my destinations{} - is this possible? Any other ways to achieve this? Actually I can't find any way to define custom macros, but this feature would seem to solve a lot of configuration scenarios? Grateful for any pointers Ed W

2 2

Application Log is not in Standard log format?
by Wilson Lai 04 Sep '07

04 Sep '07

Dear ALL, If the third party application logs are not the standard log format as the following error msg: “ ERROR IMAP Server Da(IMAP Server Pr) Sun Jan 14 10:16:50 2007 [OM 24070] Debug message for Lab use : imapMimeGetTextPreview: Could not open container for dref: 0006ee358745d1fc User Name: carrielou / macausjm/CN=Carrie Lou Pid of logging process: 22910 Last Msg Id: 12110767.1167991848536.JavaMail.root(a)scalix-a.macausjm.com Last Msg DirectRef: 0006c6aad146aa20 “ How could I insert this error message into the MSSQL database format? Thanks a lot!!! Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com

1 0

error sending raw frame
by James B Horwath 31 Aug '07

31 Aug '07

I recently upgraded to syslog-ng 2.0.4 running on a Red Hat AS release 4. I have the following on the system: eventlog-0.2.5-1.i386.rpm eventlog-devel-0.2.5-1.i386.rpm libnet-devel-1.1.2.1-6.i386.rpm syslog-ng-2.0.4-1.i386.rpm When I fire up syslog-ng I receive a large number of the following: Aug 30 14:36:07:2007 syslogger1 syslog-ng[31079]: Error sending raw frame; error=\'libnet_write_raw_ipv4(): -1 bytes written (Message too long)\\x0a\' Aug 30 14:36:07:2007 syslogger1 syslog-ng[31079]: Error sending raw frame; error=\'libnet_write_raw_ipv4(): -1 bytes written (Message too long)\\x0a\' Aug 30 14:36:07:2007 syslogger1 syslog-ng[31079]: Error sending raw frame; error=\'libnet_write_raw_ipv4(): -1 bytes written (Message too long)\\x0a\' I am trying to forward packets onto another log product, so I have spoof_source(yes) specified. What is the remedy for this error, am I loosing data? Please advise, Jim ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

2 1

Client forwarding to server
by mailing list 30 Aug '07

30 Aug '07

My syslog-ng is now running. However, I cannot get the client to forward its messages to the server. To be certain, I have disabled my firewall, on both the client and server side, but still get the below errors in /var/adm/messages. The client is logging normally in /var/adm/messages. Aug 29 14:54:33 client1 syslog-ng[18639]: Connection failed; error='Connection refused (146)', time_reopen='10' Aug 29 14:54:43 client1 syslog-ng[18639]: Connection failed; error='Connection refused (146)', time_reopen='10' Aug 29 14:54:53 client1 syslog-ng[18639]: Connection failed; error='Connection refused (146)', time_reopen='10' Aug 29 14:54:59 client1 syslog-ng[18639]: Log statistics; dropped='tcp(AF_INET(w.x.y.z:5140))=0', processed='center(queued)=8844', processed='center(received)=4424', processed='destination(messages)=4420', processed='destination(loghost)=4420', processed='destination(syslog)=4', processed='source(src)=4424' Aug 29 14:55:03 client1 syslog-ng[18639]: Connection failed; error='Connection refused (146)', time_reopen='10' I have "destination loghost { tcp("w.x.y.z" port(5140)); };" in my syslog-ng.conf on the client1. Here's my complete syslog-ng.conf file on the client1 side. options { long_hostnames(off); log_msg_size(8192); sync(1); log_fifo_size(2048); bad_hostname("^(ctld.|cmd|tmd|last)$"); time_reopen(10); }; source src { sun-stream("/dev/log" door("/etc/.syslog_door")); internal(); }; destination syslog { file("/var/log/syslog"); }; destination messages { file("/var/adm/messages"); }; destination loghost { tcp("w.x.y.z" port(5140)); }; filter f_mail { facility(mail); }; filter f_not_mail { not facility(mail); }; log { source(src); filter(f_mail); destination(syslog); }; log { source(src); filter(f_not_mail); destination(messages); }; Any help/pointer is greatly appreciated. Thank you. ____________________________________________________________________________________ Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online. http://smallbusiness.yahoo.com/webhosting

4 6

disk full memory leak
by Alexander Clouter 30 Aug '07

30 Aug '07

Hi, I hope I did not miss the memo but I think I have stumbled on a memory leak; due to my own poor administration :) When a partition fills up and syslog-ng cannot write out log files to, it seems to start soaking up memory. Looking in the output from 'top' you can see the resident memory size ('RES') creeping up and not being released. It seems to be at the rate of logging messages themselves. I'm guessing this bug has not been seen mainly as whilst syslog-ng soaks up memory unattended Linux will kill it once it soaks up all the available RAM; probably people would just assume "ahhhh syslog-ng gracefully died when it could not write out to the disk", I know I would :) It is easy to reproduce, I did it by creating a small partition of about 10MB and letting it fill up (you can hurry it along by dd'ing 8MB's worth of /dev/zero to the partition as a file). Then whilst running top you can see the memory creeping up the moment the partition fills. I found that after a day of "max'ed out partition time" syslog-ng got as high as 512MB of resident memory! I am pretty sure it is not configuration file related but if you would like a copy do ask and I will send it 'offline'. I'm running the regular 2.0.5 tarball from the website with the Debian related package patches applied from 2.0.0-1[1]. If this bug is fixed at least I will be able to lvextend/xfs_growfs a partition to 'gracefully' recover....maybe syslog-ng could flush this data polling regularly to see if it can write the data at a later time? Cheers Alex [1] http://ftp.debian.org/debian/pool/main/s/syslog-ng/syslog-ng_2.0.0-1.diff.gz

1 0

HPUX binary core dump
by Blurry 30 Aug '07

30 Aug '07

How can I figure out why syslog-ng is core dumping ? I have HPUX 11i 11.11 and I downloaded the 11.11 depot and it just says Memory Fault and renders a core dump. I had to load FLEX as a prereq. Thanks

2 3

More syslog-ng woes
by mailing list 29 Aug '07

29 Aug '07

I keep getting the below error and syslog-ng would not starts. Error in configuration, unresolved destination reference; filter='loghost' I have "log { source(loghost); };" in my log entries section, and I thought this would define "loghost" variable? Additionally, if I comment out the below section, then syslog-ng would starts and no error. log { source(src); filter(f_not_mail); filter(f_not_brightmail); destination(loghost); }; Is the above filter important/needed for syslog-ng? I'm thinking the "f_not_brightmail" is something specific if you're using brightmail, but then it would not address the "destination(loghost);" which reference loghost variable. I'm running syslog-ng 2.0.4 on Solaris 10 SPARC, and I'm referencing: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html ...but not seeing anything referencing source(loghost); configuration. I was able to comprehend that syslog-ng.conf breaks into four parts, by referencing http://linux.cudeso.be/linuxdoc/syslog-ng.php but I'm still learning the syntax from the admin's guide above. Thanks. ____________________________________________________________________________________ Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out. http://answers.yahoo.com/dir/?link=list&sid=396545469

1 0

Re: [syslog-ng] Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug
by Marvin.Nipper＠Stream.com 29 Aug '07

29 Aug '07

OK... I'll spare you the more gory details. My environment is all Solaris. A Sparc-based Solaris 8 system (that I'm trying to replace), and an Intel-based Solaris 10 system (running under VMware) that is intended to be the replacement). Although we were doing forwarding in other places, this was the first time that I was able to get some reasonable statistical reporting, between two servers that were initially supposed to be getting the same data (via the forwarding). The short story is that the receiving system "seemed" to be getting a fraction of the packets, and because the environment is slightly more complex (e.g. the VMware piece) I kept ass-u-me-ing that my original problem was not related to syslog-ng, but simply to something else going wrong. When I finally did start capturing packets with snoop, and more importantly, finally moved onto using Wireshark to look at those captures, that's when I realized that all of the forwarded packets WERE, in fact, making it to the second system, but that Solaris was tossing most of them out because their (UDP header) checksum was wrong. When I finally figured that out, I had suspected libnet might be the culprit, and after doing a bit more googling, I finally found someone else complaining about the checksum issue (albeit not in regard to using libnet with syslog-ng), and posting a suggested fix that they claimed solved the problem. Now that I know what I'm looking for, I think that Mike may have addressed the problem in his 1.1.3 Beta, but every time that ever I've tried to compile that version (on several different systems), I get a bazillion compile errors, so I've never used anything other than 1.1.2.1. The short description of the problem is that it's an "odd byte issue". The checksum process is done against 2-byte chunks of data, and if the amount of data being checksum'd is an odd number, then the code was not handling that last byte properly. So... The packets with even-numbered data volumes (in my case, about 1/3 of my forwarded packets) came through just fine, but everything else looks like a corrupted packet (to the receiving OS) and gets tossed in the bin. For your reference, here's where I lucked into the discussion: http://www.securityfocus.com/archive/89/384197/30/90/threaded I had to modify his suggested fix slightly, but this is what is now documented in my own "how to build syslog-ng" documentation: --------------------------------------- >From the libnet package directory, one needs to edit the checksum module, i.e.: vi src/libnet_checksum.c Then, you will need to locate this section of code: libnet_in_cksum(u_int16_t *addr, int len) { int sum; sum = 0; while (len > 1) { sum += *addr++; len -= 2; } if (len == 1) { sum += *(u_int16_t *)addr; } return (sum); } Now, replace that section with the following (and save/quit from the editor): libnet_in_cksum(u_int16_t *addr, int len) { int sum; u_int16_t last_byte; sum = 0; last_byte = 0; while (len > 1) { sum += *addr++; len -= 2; } if (len == 1) { *(u_int8_t*)&last_byte = *(u_int8_t*)addr; sum += last_byte; } return (sum); } -------------------------------------- Obviously, there are only a couple of lines of code modified in there, but for my purposes, it was simpler to just replace the section. I am NOT a C programmer (although I used to program in a lot of other older languages!). I can't speak to whether this is the "best" solution, or not (in terms of efficiency or coding techniques). BUT, bottom line... Once I made this change, it appears that I'm no longer receiving any packet loss at all. The sample captures that I've done, and then loaded into Wireshark, are showing no checksum errors. Whether you use this coding solution, or work up one of your own, the bottom line is that 1.1.2.1 absolutely needs to be modified, in order to make source-spoofed forwarding work as intended. Let me know if you have any questions about the above information. I'll do my best to provide answers. I'd be remiss if I didn't also take this opportunity to say THANKS for all of your syslog-ng efforts. It's a fine product. -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of bazsi(a)balabit.hu Sent: Wednesday, August 29, 2007 7:06 AM To: syslog-ng(a)lists.balabit.hu Subject: Re: [syslog-ng] Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug On Wed, 2007-08-29 at 06:58 -0600, Marvin.Nipper(a)Stream.com wrote: > OK. I've only posted here a couple of times, but I've googled this > list (for help) extensively in the past. Before I waste everyone's > time, I just wanted to find out if I'm the only one who "didn't > already know" that there's a nasty bug in libnet-1.1.2.1, such that > anybody doing source-spoofed forwarding of UDP-based syslog packets > will likely end up losing large volumes of packets in the process (in > my case a 2/3 loss of packets). > > Again. I thought that I had searched the mailing list archives > properly, and that I didn't see a mention of this issue (as I've been > fighting my way through figuring this out, over a multi-week period), > BUT, as this would seem to impact a lot of syslog-ng users, I keep > thinking that I've just missed something, and everyone else already > knows about this. Before I waste time writing a detailed message on > the topic, I figured that I'd ask first. I would appreciate if you could write about this. I don't know about anything related. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

Lost packets; UDP Checksum (chksum) errors; forwarding - source spoofing; libnet bug
by Marvin.Nipper＠Stream.com 29 Aug '07

29 Aug '07

OK. I've only posted here a couple of times, but I've googled this list (for help) extensively in the past. Before I waste everyone's time, I just wanted to find out if I'm the only one who "didn't already know" that there's a nasty bug in libnet-1.1.2.1, such that anybody doing source-spoofed forwarding of UDP-based syslog packets will likely end up losing large volumes of packets in the process (in my case a 2/3 loss of packets). Again. I thought that I had searched the mailing list archives properly, and that I didn't see a mention of this issue (as I've been fighting my way through figuring this out, over a multi-week period), BUT, as this would seem to impact a lot of syslog-ng users, I keep thinking that I've just missed something, and everyone else already knows about this. Before I waste time writing a detailed message on the topic, I figured that I'd ask first. Marvin Nipper Director of Security Stream mailto:marvin.nipper@stream.com PGP Key ID: 0x8EE28551 (DSS/DH) 8C5D 403A D107 0A95 672B B637 BCF1 919A 8EE2 8551

2 1

syslog-ng error
by Javier Terceiro 29 Aug '07

29 Aug '07

Hello list, my problem is the following one, syslog-ng does not write in the corresponding files according to the configuration. The configuration says that all the packages with facility mail, are written in a file, mail.log, whereas those of facility local7 in another file, according to the priority, info in $program.info.log and error in $program.error.log. Being $program the name of the program. In config of local7 I have writes in disc the following thing: template ("$FULLDATE <> $PROGRAM --$FACILITY-- <> $MSGONLY \ n") The problem is that some of the mail message with facility (among others facility's) are written like local7, when they would not have. An example is: 2007 Aug 29 06:25:58 <> cyrus --mail-- <> DBERROR: reading /var/lib/cyrus/db/skipstamp, assuming the worst: No such file or directory 2007 Aug 29 06:25:58 <> cyrus --mail-- <> IOERROR: opening /var/lib/cyrus/mailboxes.db: No such file or directory 2007 Aug 29 05:05:45 <> kernel --kern-- <> drbd0: incompatible states And the config file is: filter f_mail { facility(mail); }; destination df_mail { file("/var/log/mail.log" template("$FULLDATE <> $PROGRAM <> $MSGONLY\n") template_escape(no)); }; log { source(s_udp); filter(f_mail); destination(df_mail); filter f_local0_err { facility(local7) and level(warn) or level(err); }; destination df_local0_err { file("/var/log/$PROGRAM.error.log" template("$FULLDATE <> $PROGRAM --$FACILITY-- <> $MSGONLY\n") template_escape(no)); }; log { source(s_udp); filter(f_local0_err); destination(df_local0_err); }; Some idea of so that this happens? -- A greeting, Javier.

2 1