- syslog-ng - lists.balabit.hu

mojology: syslog-ng and mongodb meet the web
by Gergely Nagy 09 Jan '11

09 Jan '11

Hi! During the past couple of nights, I was busy coming up with something that would showcase the power of syslog-ng combined with the mongodb destination, and that is how mojology was born: http://mojology.madhouse-project.org/ source available @ https://github.com/algernon/mojology To sum it up: it's a web based log browser, a reasonably simple one: one can't search, or do advanced filtering (yet!), only a few basic stuff. The interesting part comes from the fact that the logs it displays are stored in mongodb, a document store, and as such, the structure of them is very, very flexible. Although, the application does make a few assumptions (see the about page or the readme in the sources), it still retains a lot of flexibility. Without going into the gory details (that's what the source is for ;), let me show you a few interesting pages! Examples -------- * http://mojology.madhouse-project.org/log/4d28cd01f310ef4f00000024 This one was originally made with the following command: logger -i -t hi 'Hello World! This concludes our demo session." Then I went into the MongoDB shell and added a few other keys below the "dyn" sub-document, alongside the default "classifier" (courtesy of patterndb). As you can see, no matter how deep one nests, it still displays properly. It's also easy to query, even at such depths. * http://mojology.madhouse-project.org/log/4d28cba4f310ef4f0000001e Logging in from one computer to my desktop, using password authentication. This entry was made fully automatically. Other notes ----------- The database behind the demo application is fairly small, only 39 messages stored, but hey, it's a demo! The program itself is a little over a hundred lines of python + templates; built upon Flask & PyMongo. For the best experience, a recent browser is strongly advised: Chromium 6+, Firefox 4+ for best results; Firefox 3 works very well too, but some of the tiny cosmetic features do not; the site displays in IE7+ too, and can be used, but the experience is rather disappointing. I hope this little weekend hack helps demonstrate how flexible mongodb can be for logging, and how well syslog-ng can make use of that flexibility. -- |8]

3 5

[Bug 78] running multiple syslog-ng instances precludes STATS socket use
by bugzilla＠bugzilla.balabit.com 09 Jan '11

09 Jan '11

https://bugzilla.balabit.com/show_bug.cgi?id=78 Balazs Scheidler <bazsi(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|NEW |RESOLVED --- Comment #1 from Balazs Scheidler <bazsi(a)balabit.hu> 2011-01-09 08:59:10 --- syslog-ng 3.1 & 3.2 has a command line option to control the name of the control socket. so I'm closing this ticket. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 64] misleading error message for missing destination
by bugzilla＠bugzilla.balabit.com 09 Jan '11

09 Jan '11

https://bugzilla.balabit.com/show_bug.cgi?id=64 Balazs Scheidler <bazsi(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution| |FIXED Status|NEW |RESOLVED --- Comment #1 from Balazs Scheidler <bazsi(a)balabit.hu> 2011-01-09 08:57:33 --- this is already integrated, I've checked 3.1.x & 3.2.x both have the change. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 56] default config file if syslog-ng was compiled from source
by bugzilla＠bugzilla.balabit.com 09 Jan '11

09 Jan '11

https://bugzilla.balabit.com/show_bug.cgi?id=56 Balazs Scheidler <bazsi(a)balabit.hu> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |3.2.1 CC| |bazsi(a)balabit.hu Resolution| |FIXED Status|NEW |RESOLVED --- Comment #1 from Balazs Scheidler <bazsi(a)balabit.hu> 2011-01-09 08:54:03 --- A default configuration file is supplied from 3.2 onwards as the system dependent parts of the config file are taken care of the system() source. Also this configuration file is installed by "make install". So I'm closing this ticket with target milestone set to 3.2.1. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

Re: [syslog-ng] logging to PostgreSQL database
by Sean McAfee 06 Jan '11

06 Jan '11

On 01/05/11 16:24, Janez Barbič wrote: > using PostgreSQL 8.4 > contents of pg_hba.conf: > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > ... > # > # Database administrative login by UNIX sockets > local all all trust > > # TYPE DATABASE USER CIDR-ADDRESS METHOD > > # "local" is for Unix domain socket connections only > # local all all trust > # IPv4 local connections: > # host all all 127.0.0.1/32 md5 > # IPv6 local connections: > host all all ::1/128 md5 > # syslog-ng logger > # host eneraptordb eneraptor 127.0.0.1/32 > trust > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > As far as I understand this configuration file, any user should have full access > to any database on local machine. At this point, I am not concerned about > security issues as I plan to address them after I resolve logging problem. In this context, "local" refers to the Unix socket (typically in /tmp), not a TCP/IP socket. This is how the command-line psql connects to the host (unless otherwise specified). You should be able to confirm that it's pg_hba-related by looking in the Postgres log file. I don't know if syslog-ng would take "/tmp" as an address, but you should be able to fix it in pg_hba.conf by either: 1. Uncommenting the "eneraptordb" line 2. Uncommenting the line below "IPv4 local connections" -- Sean McAfee

1 0

logging to PostgreSQL database
by Janez Barbič 06 Jan '11

06 Jan '11

Hi, I am a student of Computer Science at University of Ljubljana. Recently I started working on some project which includes logging into PostgreSQL database. I came across some problem which I can't resolve myself. No matter what I do, syslog-ng refuses to log into local PostgreSQL database. Here is my current configuration: using syslog-ng 3.2 server contents of syslog-ng.conf ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ @version: 3.2 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { time_reap(30); mark_freq(10); keep_hostname(yes); }; ###### # sources source s_local { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" program_override("kernel")); }; # source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); }; ######################## # Filters ######################## # Here's come the filter options. With this rules, we can set which # message go where. # messages for eneraptor should have priority level LOG_MAIL and contain keyword "eneraptor" filter f_mail_eneraptor { facility(mail) and match("eneraptor"); }; ###### # destinations destination d_messages { file("/var/log/messages"); }; destination filtered_messages { file("/var/log/messages_filtered"); }; # # SQL logging support # destination d_pgsql { sql(type(pgsql) host("localhost") username("eneraptor") password("eneraptor") database("eneraptordb") table("logs") columns("datetime varchar(16)", "host varchar(32)", "program varchar(8)", "pid varchar(8)", "message varchar(200)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSG") indexes("datetime", "host", "program", "pid", "message")); }; log { source(s_local); destination(d_messages); # destination(d_pgsql); }; log { source(s_local); filter(f_mail_eneraptor); destination(filtered_messages); destination(d_pgsql); }; ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ using PostgreSQL 8.4 contents of pg_hba.conf: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ... # # Database administrative login by UNIX sockets local all all trust # TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only # local all all trust # IPv4 local connections: # host all all 127.0.0.1/32 md5 # IPv6 local connections: host all all ::1/128 md5 # syslog-ng logger # host eneraptordb eneraptor 127.0.0.1/32 trust ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ As far as I understand this configuration file, any user should have full access to any database on local machine. At this point, I am not concerned about security issues as I plan to address them after I resolve logging problem. postgresql.conf: Below are included only lines I uncommented or changed. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ... listen_addresses = 'localhost' # what IP address(es) to listen on; ... password_encryption = on ... ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ This is the error I get, logged by syslog-ng: *Jan 5 22:00:38 computerName syslog-ng[11080]: Error establishing SQL connection; type='pgsql', host='localhost', port='', username='eneraptor', database='eneraptordb', error='could not connect to server: Connection refused\x0a\x09Is the server running on host "localhost" and accepting\x0a\x09TCP/IP connections on port 0?\x0a'* * * Database has also been created, properly named and has the right user. Message I am trying to log is being sent by test program written in C, line that sends message is (every few seconds): *syslog(LOG_MAIL, "eneraptor test: %d", counter);* >From what I could gather is that syslog-ng probably can't connect to database, but I am running out of options on how to resolve this issue. I would be very grateful if you could help me resolve this issue. Best regards, Janez Barbic

3 4

Re: [syslog-ng] syslog-ng 3.0.8, 3.0.9 not sending logs to centralized log host
by Jarrett Lee 22 Dec '10

22 Dec '10

As I stated before, I ran tcpdump to monitor the network traffic leaving the client system and then attempted to generate some logs, there was no network traffic beyond SSH and some VMware stuff, so I know it wasn't networking, at least not yet, as the syslog-ng client daemon was not even generating network traffic. Anyway, I did figure out the issue. I was editing the wrong conf file, silly me! I was editing /etc/syslog-ng.conf when I should have been editing /opt/syslog-ng/etc/syslog-ng.conf. A coworker showed me this, proving that I had just been staring at the problem for too long to be able to see the answer ;) Thank you for your help guys! It helped us get the gears in our heads moving again! On Dec 21, 2010, at 17:09, Alan McKinnon wrote: > Your config appears correct - I use a similar setup on a wide variety of > logging clients using all sorts of versions of sysloggers. > > So I imagine your problem is more network and less syslog. You don't mention > basic networking tests you might have done, such as can you ping the log host > and can you telnet to the port and get a sensible result? I'd like to > establish that packets can at least go from client to server. And don't forget > to check silly things /etc/hosts - I once had a server listed there with > address 127.0.0.1..... that took an awfully long time to find. [I see your > config implies you send to an ip not a hostname. But still, basic checks are > good.] > > > > Apparently, though unproven, at 00:03 on Wednesday 22 December 2010, Jarrett > Lee did opine thusly: > >> This is the client side that I'm having issues with, not the syslog server, >> or loghost, side. Does it really need network configuration information in >> the source statement? I thought that was on the server side to show it >> which interface/port to listen on for clients. >> >> On Dec 21, 2010, at 16:59, Clayton Dukes wrote: >> >> Looks like you need to define UDP or TCP (or both) in your src statement. >> >> Here's a short (hopefully helpful) link to a video for syslog-ng >> configuration: http://www.logzilla.info/SearchResults.asp?Cat=49 >> >> <http://www.logzilla.info/SearchResults.asp?Cat=49>Full disclosure, >> LogZilla is my log analysis software, but hopefully the video helps. >> >> >> ______________________________________________________________ >> >> Clayton Dukes >> ______________________________________________________________ >> >> >> On Tue, Dec 21, 2010 at 4:43 PM, Jarrett Lee >> <jarrett.lee(a)oversightsystems.com<mailto:jarrett.lee@oversightsystems.com> >>> wrote: I have syslog-ng 3.0.9 (also tried 3.0.8) on a CentOS 5.5 system, >> firewall (iptables) turned off, and SELinux disabled. For some reason it >> refuses to send logs to my log host, though it will put them in my >> messages file. I've even broken out tcpdump to monitor the port while >> generating logs to see if I can see any network traffic generated, but >> it's crickets on the wire. >> >> Anybody have this problem? Is there something I'm missing, perhaps I've >> been looking at it for too long and need fresh eyes? I've had this working >> before on other platforms, Solaris and other distros of Linux, but this >> time it's kicking my butt... >> >> Here's my syslog-ng.conf (with IP and port redacted): >> #### BEGIN syslog-ng.conf #### >> @version: 3.0 >> >> options { >> }; >> >> source src { >> internal(); >> unix-stream("/dev/log"); >> file("/proc/kmsg" program_override("kernel: ")); >> }; >> >> destination local { >> file("/var/log/messages"); >> }; >> destination loghost { >> tcp("IPADDR" port(PORT)); >> }; >> >> log { >> source(src); >> destination(local); >> }; >> log { >> source(src); >> destination(loghost); >> }; >> #### END syslog-ng.conf #### >> >> >> Thanks, >> Jarrett >> >> Jarrett Lee, UNIX Administrator >> OVERSIGHT SYSTEMS | >> www.oversightsystems.com<http://www.oversightsystems.com/> >> __________________________________________________________________________ >> ____ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: >> http://www.campin.net/syslog-ng/faq.html >> >> >> ___________________________________________________________________________ >> ___ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: >> http://www.campin.net/syslog-ng/faq.html > > -- > alan dot mckinnon at gmail dot com

3 3

syslog-ng 3.0.8, 3.0.9 not sending logs to centralized log host
by Jarrett Lee 21 Dec '10

21 Dec '10

I have syslog-ng 3.0.9 (also tried 3.0.8) on a CentOS 5.5 system, firewall (iptables) turned off, and SELinux disabled. For some reason it refuses to send logs to my log host, though it will put them in my messages file. I've even broken out tcpdump to monitor the port while generating logs to see if I can see any network traffic generated, but it's crickets on the wire. Anybody have this problem? Is there something I'm missing, perhaps I've been looking at it for too long and need fresh eyes? I've had this working before on other platforms, Solaris and other distros of Linux, but this time it's kicking my butt... Here's my syslog-ng.conf (with IP and port redacted): #### BEGIN syslog-ng.conf #### @version: 3.0 options { }; source src { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; destination local { file("/var/log/messages"); }; destination loghost { tcp("IPADDR" port(PORT)); }; log { source(src); destination(local); }; log { source(src); destination(loghost); }; #### END syslog-ng.conf #### Thanks, Jarrett Jarrett Lee, UNIX Administrator OVERSIGHT SYSTEMS | www.oversightsystems.com

2 3

(no subject)
by carl andreasson 19 Dec '10

19 Dec '10

http://www.cookie.purespace.de/rew.html

2 1

Re: [syslog-ng] Newer libtool and autoconf versions?
by Matthew Hall 18 Dec '10

18 Dec '10

They are related only because my issue happened in exactly the same line and I thought it might be nice to fix both at the same time. ;-) Matthew. Sent from my mobile device Corinna Vinschen <vinschen(a)redhat.com> wrote: >On Dec 17 11:00, Matthew Hall wrote: >> On Fri, Dec 17, 2010 at 02:07:51PM +0100, Corinna Vinschen wrote: >> > Here's how you avoid the warning: >> > >> > diff --git a/configure.in b/configure.in >> > index ea671be..0e534f8 100644 >> > --- a/configure.in >> > +++ b/configure.in >> > @@ -156,7 +156,7 @@ AC_PROG_YACC >> > AM_PROG_LEX >> > AC_PROG_MAKE_SET >> > PKG_PROG_PKG_CONFIG >> > -LT_INIT([disable-static]) >> > +LT_INIT([dlopen disable-static]) >> > >> > dnl *************************************************************************** >> > dnl Validate yacc >> > diff --git a/modules/dbparser/Makefile.am b/modules/dbparser/Makefile.am >> > index 2a08c29..76a6cef 100644 >> > --- a/modules/dbparser/Makefile.am >> > +++ b/modules/dbparser/Makefile.am >> > @@ -26,7 +26,7 @@ libdbparser_la_LDFLAGS = -avoid-version -module >> > bin_PROGRAMS = pdbtool >> > pdbtool_SOURCES = pdbtool.c patternize.c patternize.h >> > pdbtool_CPPFLAGS = $(AM_CPPFLAGS) @OPENSSL_CFLAGS@ >> > -pdbtool_LDADD = ../../lib/libsyslog-ng.la libdbparser.la @OPENSSL_LIBS@ @DEPS_LIBS@ >> > +pdbtool_LDADD = ../../lib/libsyslog-ng.la -dlpreopen libdbparser.la @OPENSSL_LIBS@ @DEPS_LIBS@ >> > >> > BUILT_SOURCES = dbparser-grammar.y dbparser-grammar.c dbparser-grammar.h >> > EXTRA_DIST = $(BUILT_SOURCES) radix-find.c dbparser-grammar.ym >> >> Hi Corinna, >> >> Funny you should mention this one. >> >> I had a similar problem here where listing the OpenSSL ahead of listing >> the DEPS_LIBS caused me to pick up the system OpenSSL instead of the >> more updated OpenSSL in the install prefix. >> >> So it seems I'm not the only one who ran into it. >> >> For what it's worth on my end I worked around it using LIBS env var >> which gets subbed into the linker args ahead of where LDFLAGS does so >> your stuff gets put earlier in the search path. > >Hi Matthew, > >I'm sorry, but I don't see how these two items are related. The problem >the above patch is trying to fix is to avoid a warning when linking >against a dlopen module. > > >Corinna > >-- >Corinna Vinschen >Cygwin Project Co-Leader >Red Hat >______________________________________________________________________________ >Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >FAQ: http://www.campin.net/syslog-ng/faq.html >

1 0