- syslog-ng - lists.balabit.hu

logging to PostgreSQL database
by Janez Barbič 06 Jan '11

06 Jan '11

Hi, I am a student of Computer Science at University of Ljubljana. Recently I started working on some project which includes logging into PostgreSQL database. I came across some problem which I can't resolve myself. No matter what I do, syslog-ng refuses to log into local PostgreSQL database. Here is my current configuration: using syslog-ng 3.2 server contents of syslog-ng.conf ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ @version: 3.2 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { time_reap(30); mark_freq(10); keep_hostname(yes); }; ###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel")); }; # source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); }; ######################## # Filters ######################## # Here's come the filter options. With this rules, we can set which # message go where. # messages for eneraptor should have priority level LOG_MAIL and contain keyword "eneraptor" filter f_mail_eneraptor { facility(mail) and match("eneraptor"); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; destination filtered_messages { file("/var/log/messages_filtered"); }; # # SQL logging support # destination d_pgsql { sql(type(pgsql) host("localhost") username("eneraptor") password("eneraptor") database("eneraptordb") table("logs") columns("datetime varchar(16)", "host varchar(32)", "program varchar(8)", "pid varchar(8)", "message varchar(200)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; log { source(s_local); destination(d_messages); # destination(d_pgsql); }; log { source(s_local); filter(f_mail_eneraptor); destination(filtered_messages); destination(d_pgsql); }; ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ using PostgreSQL 8.4 contents of pg_hba.conf: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ... # # Database administrative login by UNIX sockets local all all trust # TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only # local all all trust # IPv4 local connections: # host all all 127.0.0.1/32 md5 # IPv6 local connections: host all all ::1/128 md5 # syslog-ng logger # host eneraptordb eneraptor 127.0.0.1/32 trust ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ As far as I understand this configuration file, any user should have full access to any database on local machine. At this point, I am not concerned about security issues as I plan to address them after I resolve logging problem. postgresql.conf: Below are included only lines I uncommented or changed. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ... listen_addresses = 'localhost' # what IP address(es) to listen on; ... password_encryption = on ... ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ This is the error I get, logged by syslog-ng: *Jan 5 22:00:38 computerName syslog-ng[11080]: Error establishing SQL connection; type='pgsql', host='localhost', port='', username='eneraptor', database='eneraptordb', error='could not connect to server: Connection refused\x0a\x09Is the server running on host "localhost" and accepting\x0a\x09TCP/IP connections on port 0?\x0a'* * * Database has also been created, properly named and has the right user. Message I am trying to log is being sent by test program written in C, line that sends message is (every few seconds): *syslog(LOG_MAIL, "eneraptor test: %d", counter);* >From what I could gather is that syslog-ng probably can't connect to database, but I am running out of options on how to resolve this issue. I would be very grateful if you could help me resolve this issue. Best regards, Janez Barbic

3 4

Re: [syslog-ng] syslog-ng 3.0.8, 3.0.9 not sending logs to centralized log host
by Jarrett Lee 22 Dec '10

22 Dec '10

As I stated before, I ran tcpdump to monitor the network traffic leaving the client system and then attempted to generate some logs, there was no network traffic beyond SSH and some VMware stuff, so I know it wasn't networking, at least not yet, as the syslog-ng client daemon was not even generating network traffic. Anyway, I did figure out the issue. I was editing the wrong conf file, silly me! I was editing /etc/syslog-ng.conf when I should have been editing /opt/syslog-ng/etc/syslog-ng.conf. A coworker showed me this, proving that I had just been staring at the problem for too long to be able to see the answer ;) Thank you for your help guys! It helped us get the gears in our heads moving again! On Dec 21, 2010, at 17:09, Alan McKinnon wrote: > Your config appears correct - I use a similar setup on a wide variety of > logging clients using all sorts of versions of sysloggers. > > So I imagine your problem is more network and less syslog. You don't mention > basic networking tests you might have done, such as can you ping the log host > and can you telnet to the port and get a sensible result? I'd like to > establish that packets can at least go from client to server. And don't forget > to check silly things /etc/hosts - I once had a server listed there with > address 127.0.0.1..... that took an awfully long time to find. [I see your > config implies you send to an ip not a hostname. But still, basic checks are > good.] > > > > Apparently, though unproven, at 00:03 on Wednesday 22 December 2010, Jarrett > Lee did opine thusly: > >> This is the client side that I'm having issues with, not the syslog server, >> or loghost, side. Does it really need network configuration information in >> the source statement? I thought that was on the server side to show it >> which interface/port to listen on for clients. >> >> On Dec 21, 2010, at 16:59, Clayton Dukes wrote: >> >> Looks like you need to define UDP or TCP (or both) in your src statement. >> >> Here's a short (hopefully helpful) link to a video for syslog-ng >> configuration: http://www.logzilla.info/SearchResults.asp?Cat=49 >> >> <http://www.logzilla.info/SearchResults.asp?Cat=49>Full disclosure, >> LogZilla is my log analysis software, but hopefully the video helps. >> >> >> ______________________________________________________________ >> >> Clayton Dukes >> ______________________________________________________________ >> >> >> On Tue, Dec 21, 2010 at 4:43 PM, Jarrett Lee >> <jarrett.lee(a)oversightsystems.com<mailto:jarrett.lee@oversightsystems.com> >>> wrote: I have syslog-ng 3.0.9 (also tried 3.0.8) on a CentOS 5.5 system, >> firewall (iptables) turned off, and SELinux disabled. For some reason it >> refuses to send logs to my log host, though it will put them in my >> messages file. I've even broken out tcpdump to monitor the port while >> generating logs to see if I can see any network traffic generated, but >> it's crickets on the wire. >> >> Anybody have this problem? Is there something I'm missing, perhaps I've >> been looking at it for too long and need fresh eyes? I've had this working >> before on other platforms, Solaris and other distros of Linux, but this >> time it's kicking my butt... >> >> Here's my syslog-ng.conf (with IP and port redacted): >> #### BEGIN syslog-ng.conf #### >> @version: 3.0 >> >> options { >> }; >> >> source src { >> internal(); >> unix-stream("/dev/log"); >> file("/proc/kmsg" program_override("kernel: ")); >> }; >> >> destination local { >> file("/var/log/messages"); >> }; >> destination loghost { >> tcp("IPADDR" port(PORT)); >> }; >> >> log { >> source(src); >> destination(local); >> }; >> log { >> source(src); >> destination(loghost); >> }; >> #### END syslog-ng.conf #### >> >> >> Thanks, >> Jarrett >> >> Jarrett Lee, UNIX Administrator >> OVERSIGHT SYSTEMS | >> www.oversightsystems.com<http://www.oversightsystems.com/> >> __________________________________________________________________________ >> ____ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: >> http://www.campin.net/syslog-ng/faq.html >> >> >> ___________________________________________________________________________ >> ___ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: >> http://www.campin.net/syslog-ng/faq.html > > -- > alan dot mckinnon at gmail dot com

3 3

syslog-ng 3.0.8, 3.0.9 not sending logs to centralized log host
by Jarrett Lee 21 Dec '10

21 Dec '10

I have syslog-ng 3.0.9 (also tried 3.0.8) on a CentOS 5.5 system, firewall (iptables) turned off, and SELinux disabled. For some reason it refuses to send logs to my log host, though it will put them in my messages file. I've even broken out tcpdump to monitor the port while generating logs to see if I can see any network traffic generated, but it's crickets on the wire. Anybody have this problem? Is there something I'm missing, perhaps I've been looking at it for too long and need fresh eyes? I've had this working before on other platforms, Solaris and other distros of Linux, but this time it's kicking my butt... Here's my syslog-ng.conf (with IP and port redacted): #### BEGIN syslog-ng.conf #### @version: 3.0 options { }; source src { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; destination local { file("/var/log/messages"); }; destination loghost { tcp("IPADDR" port(PORT)); }; log { source(src); destination(local); }; log { source(src); destination(loghost); }; #### END syslog-ng.conf #### Thanks, Jarrett Jarrett Lee, UNIX Administrator OVERSIGHT SYSTEMS | www.oversightsystems.com

2 3

(no subject)
by carl andreasson 19 Dec '10

19 Dec '10

http://www.cookie.purespace.de/rew.html

2 1

Re: [syslog-ng] Newer libtool and autoconf versions?
by Matthew Hall 18 Dec '10

18 Dec '10

They are related only because my issue happened in exactly the same line and I thought it might be nice to fix both at the same time. ;-) Matthew. Sent from my mobile device Corinna Vinschen <vinschen(a)redhat.com> wrote: >On Dec 17 11:00, Matthew Hall wrote: >> On Fri, Dec 17, 2010 at 02:07:51PM +0100, Corinna Vinschen wrote: >> > Here's how you avoid the warning: >> > >> > diff --git a/configure.in b/configure.in >> > index ea671be..0e534f8 100644 >> > --- a/configure.in >> > +++ b/configure.in >> > @@ -156,7 +156,7 @@ AC_PROG_YACC >> > AM_PROG_LEX >> > AC_PROG_MAKE_SET >> > PKG_PROG_PKG_CONFIG >> > -LT_INIT([disable-static]) >> > +LT_INIT([dlopen disable-static]) >> > >> > dnl *************************************************************************** >> > dnl Validate yacc >> > diff --git a/modules/dbparser/Makefile.am b/modules/dbparser/Makefile.am >> > index 2a08c29..76a6cef 100644 >> > --- a/modules/dbparser/Makefile.am >> > +++ b/modules/dbparser/Makefile.am >> > @@ -26,7 +26,7 @@ libdbparser_la_LDFLAGS = -avoid-version -module >> > bin_PROGRAMS = pdbtool >> > pdbtool_SOURCES = pdbtool.c patternize.c patternize.h >> > pdbtool_CPPFLAGS = $(AM_CPPFLAGS) @OPENSSL_CFLAGS@ >> > -pdbtool_LDADD = ../../lib/libsyslog-ng.la libdbparser.la @OPENSSL_LIBS@ @DEPS_LIBS@ >> > +pdbtool_LDADD = ../../lib/libsyslog-ng.la -dlpreopen libdbparser.la @OPENSSL_LIBS@ @DEPS_LIBS@ >> > >> > BUILT_SOURCES = dbparser-grammar.y dbparser-grammar.c dbparser-grammar.h >> > EXTRA_DIST = $(BUILT_SOURCES) radix-find.c dbparser-grammar.ym >> >> Hi Corinna, >> >> Funny you should mention this one. >> >> I had a similar problem here where listing the OpenSSL ahead of listing >> the DEPS_LIBS caused me to pick up the system OpenSSL instead of the >> more updated OpenSSL in the install prefix. >> >> So it seems I'm not the only one who ran into it. >> >> For what it's worth on my end I worked around it using LIBS env var >> which gets subbed into the linker args ahead of where LDFLAGS does so >> your stuff gets put earlier in the search path. > >Hi Matthew, > >I'm sorry, but I don't see how these two items are related. The problem >the above patch is trying to fix is to avoid a warning when linking >against a dlopen module. > > >Corinna > >-- >Corinna Vinschen >Cygwin Project Co-Leader >Red Hat >______________________________________________________________________________ >Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >FAQ: http://www.campin.net/syslog-ng/faq.html >

1 0

Help with pattern db correlation
by Martin Holste 18 Dec '10

18 Dec '10

I'm trying to build a correlation message for Ironport similar to the Postfix example that was referred to in the discussions. Here's what I have so far, but it's not quite working: <ruleset> <rules> <rule class="10" id="10" context-id="ironport-mid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: Start MID @NUMBER:mid:@ ICID @NUMBER:icid:@</pattern> </patterns> <values> <value name="icid">$icid</value> </values> <examples> <example> <test_message program="ironport_mail_logs">Info: Start MID 144753300 ICID 696117306</test_message> </example> </examples> </rule> <rule class="10" id="10" context-id="ironport-mid" context-timeout="10" context-scope="program"> <patterns> <pattern>Info: Message finished MID @NUMBER:mid:@ done</pattern> </patterns> <actions> <action> <message> <values> <value name="MESSAGE">IronPort message complete: icid: $icid, mid: $mid</value> </values> </message> </action> </actions> <examples> <example> <test_message program="ironport_mail_logs">Info: Message finished MID 144753300 done</test_message> </example> </examples> </rule> </rules> </ruleset> I'm getting the triggered action, but the icid is null while the mid is filled in. What am I missing?

3 4

load failures in afsocket and afsql
by Matthew Hall 17 Dec '10

17 Dec '10

I am getting the following on load: Error opening plugin module; module='afsocket', error='/home/y/lib64/syslog-ng/libafsocket.so: undefined symbol: libnet_build_ipv4' Error opening plugin module; module='afsql', error='/home/y/lib64/syslog-ng/libafsql.so: undefined symbol: dbi_result_free' Error opening plugin module; module='afsocket', error='/home/y/lib64/syslog-ng/libafsocket.so: undefined symbol: libnet_build_ipv4' The rpath looks OK: megahall@logproxy2:~$ readelf -a /home/y/lib64/syslog-ng/libafsocket.so | fgrep -i rpath 0x000000000000000f (RPATH) Library rpath: [/home/y/lib64] megahall@logproxy2:~$ readelf -a /home/y/lib64/syslog-ng/libafsql.so | fgrep -i rpath 0x000000000000000f (RPATH) Library rpath: [/home/y/lib64] megahall@logproxy2:~$ megahall@logproxy2:~$ ldd /home/y/lib64/syslog-ng/libafs* | fgrep -i '(dbi|net)' megahall@logproxy2:~$ The libraries are in a reasonable location: /home/y/lib64/libdbi.so.1.0.0 /home/y/lib64/libnet.so.1.5.0 /home/y/lib64/dbd/libdbdsqlite3.so /home/y/lib64/dbd/libdbdmysql.so /home/y/lib64/libnet.so.1 /home/y/lib64/libdbi.so.1 /home/y/lib64/libdbi.so Reading through the glib docs for glib modules, it seems like the .la files are maybe not containing the right library dependencies, or something like this. However adding the library directories using LD_LIBRARY_PATH as a temporary test does not help. Because this step fails, it's not possible to use tcp, udp, or any of the other important drivers you need to collect logs. I could really use some advice on this one! Matthew.

5 25

Help with db_parser()
by Giovanni Mancuso 17 Dec '10

17 Dec '10

Hi, I configure my Java application to log with log4j in syslog server, and i configure my syslog-ng to store data in mysql database, but this doen't work. My syslog-ng configuration is: @version: 3.0 options { chain_hostnames(no); stats_freq(43200); }; source src { unix-stream("/dev/log" max-connections(256)); internal(); udp(ip("127.0.0.1") port(514)); file("/proc/kmsg"); }; destination mcs { file("/var/log/mw-collaboration/mw-collaboration-loginfile.log"); }; destination mcs_sql { sql( type(mysql) host("localhost") username("syslogng") password("syslogng") database("mcslogin") table("mcslogin") columns("date varchar(32)","loginuser varchar(32) ","ipsource varchar(32)") values("${S_YEAR}-${S_MONTH}-${S_DAY} ${S_HOUR}:${S_MIN}:${S_SEC}","${LOGIN_USER}","${IP_SOURCE}") ); }; filter f_mcs { facility(local2); }; parser p_mcs { db_parser(file("/etc/syslog-ng/patterndb.d/mcs.xml")); }; log { source(src); filter(f_mcs); parser(p_mcs); destination(mcs_sql); destination(mcs); }; I created also a db_parser file that is: <patterndb version='1' pub_date='2010-12-14'> <program name='mcs'> <pattern>mcs</pattern> <rule id='mcs' class='system'> <pattern>###############Accesso dell'utente @STRING:LOGIN_USER@ da ip @IPv4:IP_SOURCE@</pattern> </rule> </program> </patterndb> With this configuration, i have that in "mcs" destination it writes all information: Dec 16 11:55:44 localhost mcs[123] ###############Accesso dell'utente xxxx(a)xxx.mailware.it da ip 111.222.333.444 Dec 16 12:53:23 localhost mcs[123] ###############Accesso dell'utente xxxx.1(a)xxx.mailware.it da ip 111.222.333.444 Dec 16 14:07:40 localhost mcs[123] ###############Accesso dell'utente xxxx.1(a)xxx.mailware.it da ip 111.222.333.444 but in "mcs_sql" destination, it writes only date: mysql> desc mcslogin; +-----------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------+-------------+------+-----+---------+-------+ | date | varchar(32) | YES | MUL | NULL | | | loginuser | varchar(32) | YES | | NULL | | | ipsource | varchar(32) | YES | | NULL | | +-----------+-------------+------+-----+---------+-------+ 3 rows in set (0.00 sec) mysql> select * from mcslogin limit 10 -> ; +---------------------+-----------+----------+ | date | loginuser | ipsource | +---------------------+-----------+----------+ | 2010-12-15 11:02:16 | | | | 2010-12-15 11:11:09 | | | | 2010-12-15 17:53:01 | | | | 2010-12-15 18:11:55 | | | | 2010-12-15 18:12:54 | | | | 2010-12-15 18:35:07 | | | | 2010-12-16 11:55:36 | | | | 2010-12-16 11:55:44 | | | | 2010-12-16 11:55:44 | | | | 2010-12-16 12:53:23 | | | +---------------------+-----------+----------+ 10 rows in set (0.00 sec) Can you help me? Thanks

3 6

memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347
by Mukarram Syed 17 Dec '10

17 Dec '10

Hi , I am getting the following error when starting syslog-ng.conf file which has more than 5000 lines in it and getting the following error: # service syslog-ng start memory exhausted in /opt/syslog-ng/etc/syslog-ng.conf at line 5347. syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng [FAILED] When I truncate the file to < 5000 lines it starts up fine. This is running version syslog-ng-3.0.8-1.rhel5.x86_64 Mailing list does not have any clues to it. Anything in the syslog-ng.conf file that needs to be done to accept more lines? Any ideas would be appreciated. Thanks # mukarram Mukarram Syed

6 23

Patch for launch path bug in syslog-ng.h
by Matthew Hall 17 Dec '10

17 Dec '10

Hello, When compiling syslog-ng 3.2.1 into my environment I discovered a bug in how this configure option is applied: --libexecdir=/home/y/libexec64 When this option is entered, this is generated in lib/syslog-ng.h: #if ENABLE_ENV_WRAPPER #define PATH_SYSLOGNG PATH_PREFIX "/libexec/syslog-ng" #endif As you can see this is a hard coded libexec path which does not match what my environment expects. Unfortunately it appears that the libexec variable never makes it into a C macro so I'm not totally clear on how to make a permanent patch for this one. I will work around it with something like this temporary patch for now. --- /home/megahall/syslog-ng.h 2010-12-13 15:54:56.000000000 -0800 +++ syslog-ng-3.2.1/lib/syslog-ng.h 2010-12-13 16:06:22.000000000 -0800 @@ -44,7 +44,7 @@ #define PATH_PIDFILE PATH_PIDFILEDIR "/syslog-ng.pid" #define PATH_CONTROL_SOCKET PATH_PIDFILEDIR "/syslog-ng.ctl" #if ENABLE_ENV_WRAPPER -#define PATH_SYSLOGNG PATH_PREFIX "/libexec/syslog-ng" +#define PATH_SYSLOGNG PATH_PREFIX "/libexec64/syslog-ng" #endif #define PATH_PERSIST_CONFIG PATH_LOCALSTATEDIR "/syslog-ng.persist" Another issue. Because of this bug syslog-ng fails to launch with return code 127. But it prints nothing to the console explaining why it failed: execve("/home/y/libexec/syslog-ng", ["/home/y/sbin64/syslog-ng"], [/* 21 vars */]) = -1 ENOENT (No such file or directory) exit_group(127) = ? I think some message should be added in syslog-ng/wrapper.c. This is an uncompiled example but should be very close. --- /home/megahall/wrapper.c 2010-12-13 16:15:29.000000000 -0800 +++ syslog-ng-3.2.1/syslog-ng/wrapper.c 2010-12-13 16:14:21.000000000 -0800 @@ -45,6 +45,7 @@ putenv(ldlibpath); } #endif + fprintf(stderr, "syslog-ng failed to launch because of XYZ\n"); execv(PATH_SYSLOGNG, argv); return 127; } Thanks for maintaining syslog-ng. I hope this helps keep the product great! Regards, Matthew Hall.

2 2