Re: [syslog-ng] Syslog-ng3 : syslog-ng Digest, Vol 53, Issue 25

30 Sep 2009

      There are no "log" statements in that conf file. That's the problem.

You declare a bunch of message sources, a bunch of filters and a bunch of destinations. You need "log" statements to declare that messages from source A matching filter B should go to destination C.

-----Original Message-----
From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Ashish Shrimali
Sent: 30 September 2009 05:25
To: syslog-ng@lists.balabit.hu
Subject: [syslog-ng] Syslog-ng3 : syslog-ng Digest, Vol 53, Issue 25

Hi,
The syslog-ng conf file is

@version:3.0

options {
        use_dns(persist_only);
        dns_cache_hosts(/etc/hosts);
        #chain_hostnames(yes);
        keep_hostname(yes);
        keep_timestamp(yes);
        };

source all_sources {
        # Solaris used streams to send msgs to Syslogd process, replicate for syslog-ng
        sun-streams("/dev/log" door("/etc/.syslog_door"));
#       sun-streams( door("/etc/.syslog_door"));
        # Messages generated internally by Syslog-ng
        internal();
        # Listens on the specified udp port for incoming messages
        udp(ip(0.0.0.0) port(514));
        };

destination local7apm { file("/var/log/syslog" template("$DATE $HOST $MSGHDR $MSGONLY\n")); };
#destination local7apm { udp("smarts5" port(514) spoof-source(yes)); };
#destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); };
#destination messages { file("/var/adm/messages" template("$DATE $HOST $MSG\n")); };

# Below 5 lines taken from existing configuration

#destination sysmsg { file("/dev/sysmsg" template("$DATE $HOST $MSG\n")); };
#destination operator { usertty("operator" template("$DATE $HOST $MSG\n")); };
destination operator { usertty("operator"); };
#destination all_alerts { usertty("operator" template("$DATE $HOST $MSG\n")); };
destination all_alerts { usertty("operator"); };
#destination all_alerts1 { usertty("root" template("$DATE $HOST $MSG\n")); };
destination all_alerts1 { usertty("root"); };
#destination emergency { usertty("*" template("$DATE $HOST $MSG\n")); };
destination emergency { usertty("*"); };

#

filter local7msgs { facility(local7) and level(info..emerg); };
filter messages { facility(kern, daemon, mail) and level(debug, info, notice, warning, err, crit alert, emerg); };
filter f_sysmsg { facility(kern, auth) and level(notice); };
filter f_all_errors { level(err); };
filter f_operator { facility(kern, daemon) and level(err); };
filter f_all_alerts { level(alert); };
filter f_all_alerts1 { level(alert); };
filter f_emergency { level(emerg); };

Thanks & Regards,
Ashish Shrimali

-----Original Message-----
From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request@lists.balabit.hu
Sent: Wednesday, September 30, 2009 9:52 AM
To: syslog-ng@lists.balabit.hu
Subject: syslog-ng Digest, Vol 53, Issue 25

Send syslog-ng mailing list submissions to
        syslog-ng@lists.balabit.hu

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.balabit.hu/mailman/listinfo/syslog-ng
or, via email, send a message with subject or body 'help' to
        syslog-ng-request@lists.balabit.hu

You can reach the person managing the list at
        syslog-ng-owner@lists.balabit.hu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of syslog-ng digest..."

Today's Topics:

   1. Re:  Colon embedded in messages (:) (chris packham)
   2. Re:  Colon embedded in messages (:) (Evan Rempel)
   3.  Solaris Syslog-ng install lib error : syslog-ng Digest, Vol
      53, Issue 22 (Ashish Shrimali)

----------------------------------------------------------------------

Message: 1
Date: Wed, 30 Sep 2009 09:49:59 +1300
From: "chris packham" <chris.packham@alliedtelesis.co.nz>
Subject: Re: [syslog-ng] Colon embedded in messages (:)
To: "Syslog-ng users' and developers' mailing list"
        <syslog-ng@lists.balabit.hu>
Message-ID: <4AC329C7020000680001703F@gwia.alliedtelesyn.co.nz>
Content-Type: text/plain; charset=US-ASCII
...
...
...
On 9/30/2009 at  9:26 AM, James Kelly <james.kelly@hmsinc.com> wrote:
Thanks a lot. That seems to have worked a lot better but has introduced
another problem. The messages are split up into too many lines and, when
there is a match, the first of N lines are removed, still creating unwanted
messages.
For example, a pretty small insert spans 3 lines... There was a match on the
2nd line, so it is not printed. However, the 1st and 3rd were.
2009-09-29T16:21:16-04:00 blahh-rep2 postgres [10440-1]
postgres-blahh-10.100.10.34(15779)-4690-2009-09-29 16:21:16 EDT-LOG:
statement: insert into
2009-09-29T16:21:16-04:00 blahh-rep2 postgres [10440-3] '1000000000000000');
I think you'll find is a postgres thing. syslog-ng does not split messages up based on size (at least as far as I've seen). I think there is some behaviour to do with carriage returns "\n" embedded in your log data but there are some config options to help with that, search the mailing list/admin guide for that if it turns out to be the case.

------------------------------

Message: 2
Date: Tue, 29 Sep 2009 14:19:46 -0700
From: Evan Rempel <erempel@uvic.ca>
Subject: Re: [syslog-ng] Colon embedded in messages (:)
To: "Syslog-ng users' and developers' mailing list"
        <syslog-ng@lists.balabit.hu>
Message-ID: <4AC279F2.50205@uvic.ca>
Content-Type: text/plain; charset=windows-1252; format=flowed

As far as I know, the messages will always be split on multiple lines by postgresql,
and you will need to match on the process number.

In your example, the 10440 is the process number, and the -1, -2, -3 are the line numers of
the messages that the single process creates. A single connection that runs multiple queries can
create many messages, with ever increasing line number counts.

Evan.

James Kelly wrote:
...
Thanks a lot.  That seems to have worked a lot better but has introduced
another problem.  The messages are split up into too many lines and,
when there is a match, the first of N lines are removed, still creating
unwanted messages.
For example, a pretty small insert spans 3 lines... There was a match on
the 2nd line, so it is not printed.  However, the 1st and 3rd were.
2009-09-29T16:21:16-04:00 blahh-rep2 postgres [10440-1]
postgres-blahh-10.100.10.34(15779)-4690-2009-09-29 16:21:16 EDT-LOG:
statement: insert into
2009-09-29T16:21:16-04:00 blahh-rep2 postgres [10440-3]
'1000000000000000');
I played around with log_msg_size thinking I could control how much is
on each line, but it didn't help.  Am I missing something obvious?  If a
"fragment" of the message is matched, I would want the entire message to
not be logged.
Honestly, I have not spent nearly as much time on this most recent
problem, so if it is an RTFM situation, just let me know.  I did do some
reading and searching on this though and came up empty handed.
Thanks again for this great help,
James
On Tue, Sep 29, 2009 at 3:43 PM, Evan Rempel <erempel@uvic.ca
<mailto:erempel@uvic.ca>> wrote:
PostgeSQL can log to syslog using appropriately formated messages.
    Configure your postgresql.conf file to include the line
log_destination = 'syslog'
and then you will get lines in syslog like;
2009-09-29T12:42:10-07:00 hostname facility.level postgres[20288]:
    [32-1] postgresql statement
then all of the syslog-ng parsing, macros and templates will work
    correctly.
Evan.
James Kelly wrote:
     > Thanks.  This is exactly what is happening.  If, using a template, I
     > just use $PROGRAM, I get a bunch of lines with just "insert".
     >
     > If I use something like the following:
     >
     > template t_postgres_msgs {
     >              template("$ISODATE $HOST $PROGRAM $MSG \n"); };
     >
     > I no longer see the colons.  However, if something is matched by the
     > filter, it still prints a blank message (hence, wasting a ton of
    space
     > and clogging up the logs).  For example, this is what I get
    instead of
     > the matched sql statement:
     >
     > 2009-09-29T10:30:30-04:00 hcdb1-rep2
     >
     > I thought the filter would just "trash" or not record the message at
     > all, not print the host and timestamp for each filtered message.
     >
     > Thanks a lot...  I appreciate this help a lot.
     > James
     >
     > On Tue, Sep 29, 2009 at 10:05 AM, Srinivasan Sreenivasan
     > <srinivasan.srinivasan@sabre.com
    <mailto:srinivasan.srinivasan@sabre.com>
     > <mailto:srinivasan.srinivasan@sabre.com
    <mailto:srinivasan.srinivasan@sabre.com>>> wrote:
     >
     >     I think Syslog-ng thinks insert is a program name. Use a template
     >     with $PROGRAM in it to see if it prints ?insert? to confirm this.
     >
     >     Solution:
     >     Send a program name before your sql statement using a template.
     >
     >
     >
     >     On 9/29/09 8:57 AM, "James Kelly" <james.kelly@hmsinc.com
    <mailto:james.kelly@hmsinc.com>
     >     <mailto:james.kelly@hmsinc.com
    <mailto:james.kelly@hmsinc.com>>> wrote:
     >
     >         Hello,
     >
     >         I am attempting to implement syslog-ng on our servers hosting
     >         postgresql databases.  The general idea is to log "too
    much" at
     >         the database level and then, using syslog filters, reduce
    it to
     >         the information we want to actually hold in the syslog
    and send
     >         to the log server.  So far I am extremely pleased with
    how easy
     >         it is to implement and well documented.  However, I do
    have one
     >         problem that is making it difficult to use.
     >
     >         The problem is for each message that the filter matches,
    it does
     >         not completely drop the message.. rather, it logs the date /
     >         server / and a colon:
     >
     >         For example, this is what I get for a message that is
    matched by
     >         a filter:
     >
     >         "Sep 29 09:43:29 hcdb1-rep2      :"
     >
     >         I notice that even with the unmatched statements, there is a
     >         colon.  For example, in the postgresql log, I see:
     >
     >         insert into "public"."table"(blah,blah,blah) values
     >         (blah,blah,blah);
     >
     >         but the same message once captured from syslog shows the
     >         following in the log and logserver:
     >
     >         Sep 29 09:43:29 hcdb1-rep2      insert: into
     >         "public"."table"(blah,blah,blah) values (blah,blah,blah);
     >
     >         *Note the colon after "insert".*  I can't seem to figure out
     >         where this is coming from or how to avoid it.  It also causes
     >         some filtering problems that I won't go into here so as
    to not
     >         confuse the issues, but safe to say it is also related to
    the colon.
     >
     >         I have spent a lot of time trying to figure this out and
    am at a
     >         dead-end.  Due to the amount of messages I need to filter out
     >         (below is just one of the many filters I need to put in
    and have
     >         tested with the same result), not being able to completely
     >         filter these out is a killer.
     >
     >         I am using the 3.0.4 open-source edition on Ubuntu 8.
     Here is
     >         my config:
     >
     >
    ******************************************************************************************
     >         @version: 3.0
     >
     >         options {
     >         };
     >
     >         ######
     >         # sources
     >         source s_local {
     >         # message generated by Syslog-NG
     >         internal();
     >         # standard Linux log source (this is the default place
    for the
     >         syslog()
     >         # function to send logs to)
     >         unix-stream("/dev/log");
     >         # messages from the kernel
     >         file("/proc/kmsg" program_override("kernel: "));
     >         file("/var/log/postgresql/postgresql-8.3-main.log");
     >         };
     >
     >
     >         ######
     >         # destinations
     >         destination d_messages { file("/var/log/messages"); };
     >
     >         #####
     >         # filters
     >
     >         filter f_inserts2 {
     >                 not match("_health_central" value("MESSAGE"));
     >         };
     >
     >
     >         destination d_logserver { tcp("internal.host.com
    <http://internal.host.com>
     >         <http://internal.host.com> <http://internal.host.com> "); };
     >
     >
     >
     >         log {
     >         source(s_local);
     >         filter(f_inserts2);
     >         destination(d_messages);
     >         #destination(d_logserver);
     >         };
     >
    ******************************************************************************************
     >
     >         Thanks!
     >         James Kelly
     >
     >
    ------------------------------------------------------------------------
     >
    ______________________________________________________________________________
     >         Member info:
    https://lists.balabit.hu/mailman/listinfo/syslog-ng
     >         Documentation:
     >
    http://www.balabit.com/support/documentation/?product=syslog-ng
     >         FAQ: http://www.campin.net/syslog-ng/faq.html
     >
     >
     >
    ______________________________________________________________________________
     >     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
     >     Documentation:
     >     http://www.balabit.com/support/documentation/?product=syslog-ng
     >     FAQ: http://www.campin.net/syslog-ng/faq.html
     >
     >
     >
--
    Evan Rempel                               erempel@uvic.ca
    <mailto:erempel@uvic.ca>
    Senior Programmer Analyst                    250.721.7691
    Unix Services, University Systems, University of Victoria
    ______________________________________________________________________________
    Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
    Documentation:
    http://www.balabit.com/support/documentation/?product=syslog-ng
    FAQ: http://www.campin.net/syslog-ng/faq.html
--
Evan Rempel                               erempel@uvic.ca
Senior Programmer Analyst                    250.721.7691
Unix Services, University Systems, University of Victoria

------------------------------

Message: 3
Date: Wed, 30 Sep 2009 09:51:37 +0530
From: Ashish Shrimali <Ashish_Shrimali@infosys.com>
Subject: [syslog-ng] Solaris Syslog-ng install lib error : syslog-ng
        Digest, Vol 53, Issue 22
To: "syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu>
Message-ID:
        <AE1A7E8803F348438B970CCDCA075C77115F91731A@BLRKECMBX07.ad.infosys.com>

Content-Type: text/plain; charset="us-ascii"

Hi,
You need to install libnet.
Steps are as follows:
1. gunzip and untar the libnet package.
2. cd <libnetdir>
3. ./configure
4. make
5. make install
6. While installing syslog-ng use this command.(with libnet)
./configure -enable-spoof-source -with-libnet=<libnet dir path>/libnet -enable-dynamic-linking

It worked successfully for me.

Thanks & Regards,
Ashish Shrimali

-----Original Message-----
From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request@lists.balabit.hu
Sent: Tuesday, September 29, 2009 7:36 PM
To: syslog-ng@lists.balabit.hu
Subject: syslog-ng Digest, Vol 53, Issue 22

Send syslog-ng mailing list submissions to
        syslog-ng@lists.balabit.hu

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.balabit.hu/mailman/listinfo/syslog-ng
or, via email, send a message with subject or body 'help' to
        syslog-ng-request@lists.balabit.hu

You can reach the person managing the list at
        syslog-ng-owner@lists.balabit.hu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of syslog-ng digest..."

Today's Topics:

   1.  Syslog-ng 3 (Ashish Shrimali)
   2. Re:  Syslog-ng 3 (Fegan, Joe)
   3. Re:  sun solaris syslogng error (Balazs Scheidler)
   4. Re:  syslog-ng client machine IP address in message header
      (Balazs Scheidler)
   5.  Colon embedded in messages (:) (James Kelly)
   6. Re:  Colon embedded in messages (:) (Srinivasan Sreenivasan)

----------------------------------------------------------------------

Message: 1
Date: Tue, 29 Sep 2009 16:41:50 +0530
From: Ashish Shrimali <Ashish_Shrimali@infosys.com>
Subject: [syslog-ng] Syslog-ng 3
To: "syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu>
Message-ID:
        <AE1A7E8803F348438B970CCDCA075C77115F917101@BLRKECMBX07.ad.infosys.com>

Content-Type: text/plain;       charset="us-ascii"

Hi All,

My Syslog-ng is running , which I can see by ps -ef command, but still it is not writing to the log file.

I have specified the log file in the conf dir.

Any inputs would be highly appreciated.

Thanks & Regards,
Ashish Shrimali

-----Original Message-----
From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request@lists.balabit.hu
Sent: Tuesday, September 29, 2009 3:11 PM
To: syslog-ng@lists.balabit.hu
Subject: syslog-ng Digest, Vol 53, Issue 21

Send syslog-ng mailing list submissions to
        syslog-ng@lists.balabit.hu

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.balabit.hu/mailman/listinfo/syslog-ng
or, via email, send a message with subject or body 'help' to
        syslog-ng-request@lists.balabit.hu

You can reach the person managing the list at
        syslog-ng-owner@lists.balabit.hu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of syslog-ng digest..."

Today's Topics:

   1.  Solaris Syslog-ng install lib error (list@net-secured.com)
   2. Re:  Solaris Syslog-ng install lib error (Srinivasan Sreenivasan)
   3.  sun solaris syslogng error (list@net-secured.com)
   4.  syslog-ng client machine IP address in message header
      (Jain, Vaibhav (GE Healthcare))

----------------------------------------------------------------------

Message: 1
Date: Mon, 28 Sep 2009 03:52:33 -0700
From: list@net-secured.com
Subject: [syslog-ng] Solaris Syslog-ng install lib error
To: "syslogng " <syslog-ng@lists.balabit.hu>
Message-ID:
        <20090928035233.9419518bf8837c8c492fb3b5d9b71733.3b4f820868.wbe@email03.secureserver.net>

Content-Type: text/plain; charset="us-ascii"

An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20090928/9ae9e2ac/at...

------------------------------

Message: 2
Date: Mon, 28 Sep 2009 08:00:09 -0500
From: Srinivasan Sreenivasan <srinivasan.srinivasan@sabre.com>
Subject: Re: [syslog-ng] Solaris Syslog-ng install lib error
To: Syslog-ng users' and developers' mailing list
        <syslog-ng@lists.balabit.hu>
Message-ID: <C6E61D89.5EF6%srinivasan.srinivasan@sabre.com>
Content-Type: text/plain; charset="us-ascii"

The PKG_CONFIG_PATH should include directories that have .pc files in them.

On 9/28/09 5:52 AM, "list@net-secured.com" <list@net-secured.com> wrote:
...
Hi
I tryed to install on Sparc v. 10 syslog-ng v3.0.4, runnning:  ./configure
--enable-sun-streams, --enable-sun-door gives this warning:
configure: WARNING: unrecognized options: --enable-sun-streams,
--enable-sun-door
any ideas?
furthermore, I get this error:
checking for GLIB - version >= 2.10.1... no
*** Could not run GLIB test program, checking why...
*** The test program failed to compile or link. See the file config.log for
the
*** exact error that occured. This usually means GLIB is incorrectly
installed.
configure: error: Cannot find GLIB version >= 2.10.1: is pkg-config in path?
===
Adding the path below : produces another error
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/:/usr/lib/pkgconfig/
checking for LIBDBI... Package dbi was not found in the pkg-config search
path.
Perhaps you should add the directory containing `dbi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'dbi' found
no
checking for dbi_initialize in -ldbi... no
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_create in -lpthread... yes
checking for LIBNET... no
checking whether to enable Sun STREAMS support... yes
checking whether to enable Sun door support... yes
checking whether to enable IPv6 support... yes
checking whether to enable SQL support... no
checking whether to enable Linux capability support... no
checking whether to enable PCRE support... yes
checking for pkg-config... (cached) /usr/bin/pkg-config
checking pkg-config is at least version 0.14... yes
checking for GLIB - version >= 2.10.1... yes (version 2.14.1)
checking for EVTLOG... yes
checking for LIBNET... configure: error: libnet-config not found
thanks
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html