January 2005 - syslog-ng - lists.balabit.hu

syslog-ng development release 1.9.2
by Balazs Scheidler 05 Jan '05

05 Jan '05

Hi, It is my pleasure to announce the availability of the latest development release for syslog-ng, and a related library called eventlog. This release is actually usable, and although it is still not recommended to be used on production systems, running it on a spare system with a real log feed would be very useful indeed, to see how it behaves with various real life configurations. The major differences between syslog-ng 1.9.2 and the releases in the 1.6.x branch are outlined in the NEWS file attached to the end of this message. A couple of NEWS News for the 1.9.2 release Mon, 03 Jan 2005 22:03:02 +0100 Well, it's been a long time since the last syslog-ng development release simply because I was involved in too many things in parallel. Good news that the release is actually quite usable, I've been running it on my laptop for a week now and I think it is now time to gather some real life experience. * Fixed a couple of memory leaks. * Fixed SIGHUP support. * Fixed program startup to make sure all important error information is logged. * Fixed program destinations by adding setsid() call after forking. * Fixed flow controlled log path support. * Added the possibility to use braces around macro references (like in '${MSG}\n') * Added macros missing from this branch added to 1.6.x since the last release (PRI and MSGONLY). News for the 1.9.1 release Wed, 22 Jan 2003 12:13:44 +0100 This release is still not intended for production use. It has a good number of new features and a couple of them may not be stable enough. New features: * template declarations, the possibility to define a template once and refer to it * flow controlled log paths work, and they are also configurable * supports RFC3339 timestamps with second fractions and time zones, time zone conversion & source time zone specification * MARK messages are generated * keep alive for connections as well as listeners * the ability to follow log files as they are written to * the possibility to mark parts of a message using standard '(' and ')' regexp operators and refer to matches within templates (either filenames or content) This program is still ALPHA. News for the 1.9.0 release Thu, 9 May 2002 20:01:11 +0200 This release is the first release of the new reimplementation of syslog-ng. I moved to using glib instead of libol, as glib is more mature and provides several nice & easy to use features. Here's a bunch of nice things about 1.9.0: * supports flow controlled log paths (still not possible to configure it, but the code exists in the core) * better support for macros (macro support is moved out of affile, thus any other drivers may use it) * more robust detection of closed destination sockets (a closed channel is immediately detected, not at the first write) This avoids losing a log message when the connection is broken. * supports respawning destination programs (should be used with care to avoid DoS situations). This makes it easier to integrate third party extensions to syslog-ng. (SQL backends, on-line log analysis etc) * performance improvements, instead of using linear lookup in linked lists, I use hashtables everywhere. * the ability to define unparsed sources, makes it easier to integrate things like Apache logs * many internal reorganizations to make it easier to implement things like hashed log files, SSL protected TCP sessions and so on. Things still missing: * there are some new features with no configuration interface * internal messages always go to /dev/stderr * Sun's STREAMS is not yet supported 1.9.0 is currently in an ALPHA state, it's been tested only on my development computer, and not real-life traffic. -- Bazsi

2 3

MYSQL pipe
by Robert.COward＠deploymenthealth.osd.mil 03 Jan '05

03 Jan '05

I have syslo-ng running fine on OpenBSD 3.6. Logging works properly to file. I also have MYSQL up and running. I can get a named pipe open and get MYSQL to pickup the log entries with no problem. However I am at a lost as to how to start this process at boot up or automatically. If I call the following script, which works from the command line, from either /etc/rc or /etc/rc.local the mysql client is never kicked of, and /etc/rc hangs. If I put the same script into a cron job, cron hangs. Script: #!/bin/sh ps=`ps -ax | grep "\--password=x" | awk '{ print $9 }'` if [[ $ps != "" ]]; then echo "syslog-ng2mysql is alive." else if [ ! -p /tmp/mysql.pipe ]; then mkfifo /tmp/mysql.pipe kill -HUP `cat /var/run/syslog-ng.pid` fi mysql -u root --password=password syslog < /tmp/mysql.pipe fi I really do not know what is wrong. Can anyone provide examples on their scripts for starting syslog-ng to named pipe to mysql? --- 3 January 2005 "Hackers make toys. Crackers break them." (Peter Seebach) --- Robert V. Coward UNIX System Administrator Computer Science Corporation

2 1

which os for a central logserver?
by Tamas MEZEI 03 Jan '05

03 Jan '05

Hi, I was just wondering in these days of the big'n'mighty systems if there's a preference for the OS (more exactly Gentoo Linux or FreeBSD) of a central logserver using syslog-ng. Are there any OS-specific problems or specialities that I should be aware of if I'm using this or that OS? Or are there any syslog-ng specific things depending on the host OS? If anybody has some experience using those OSes as central logservers, would he say some words about the positive-negative sides (in private, maybe not everybody's interested in this chit-chat). The machine would receive logs from nearly 20 servers (mostly Linux and 3 Windows machines), and has the job to analyze the stuff periodically. Thanks in advance, Tamas

2 2

Reliability of various mysql logging methods
by Ed Walker 02 Jan '05

02 Jan '05

We're using syslog-ng, logging about 2million lines/day from a number of email filtering servers. We log to a new file each day. Right now we've got a perl script that runs just after midnight, and parses each file and dumps the info into a mysql database. For each domain, we have a separate database, for each month, a separate table. We're interested in moving to direct SQL injection, but are concerned about the possibility of losing logging information if the sql server goes down, or the pipe or injection script die. What's the most reliable way to do this? Named pipe? Socket? Anybody willing to share their configurations and/or scripts? Thanks- Ed

2 1

Binaries or working Makefile for sqlsyslogd ?
by Daniel Baker 02 Jan '05

02 Jan '05

Greetings. I have been looking for a working Makefile or binaries for an syslog-ng to sql source I have found at: http://www.frasunek.com/sources/security/sqlsyslogd/ Does anyone here have any good links for me? I would be most appreciative. Dan.

1 0

Playing "catchup"...
by Ed Walker 02 Jan '05

02 Jan '05

Thought of this just after sending that last message.... In the event that SQL injection dies on the central loghost, we've thought of keeping a copy in a file as well. And if, for some reason, syslog-ng on the central server dies, or connectivity is lost, we're already keeping a copy of the logging data on each remote server. So, what's the best way to play "catch up" when something dies, and making it as easy as possible to import the missed data, without accidentally introducing duplication? Any ideas? Thanks- Ed

2 1