January 2013 - syslog-ng - lists.balabit.hu

getting syslog stats without syslog-ng-ctl?
by Daniel Neubacher 08 Jan '13

08 Jan '13

Hello there, we are plotting and monitoring lot of data and for this I'm using a little bash script which does sum up the output of syslog-ng-ctl. But this is kind of slow with 17k lines to parse :P Is there a way to get the stats directly without syslog-ng-ctl? My goal is to write a ruby script which only parse the stats i'm needing. I'm no developer so please answer admin friendly :)

3 4

Problem parsing apache logs
by Luis Fernando Alen 07 Jan '13

07 Jan '13

Hello, list. I've been trying to parse my apache logs in order to separate log messages into different files based on a column provided by the parser, which in my case is $APACHE.SERVER_NAME. It's pretty much the same as the example available on the Administrator Guide, in section 3.8, p 60. Looks like I'm close to get it working but there are two problems which I could not solve: 1- The first field of the log (APACHE.CLIENT_IP) vanishes when it reaches its final destination and for that reason APACHE.SERVER_NAME, which is the last column in the parser is empty. That messes up the file creation on the server. Not mentioning the important log data loss. 2- Syslog-ng server only logs for a few seconds and suddenly stops. Whenever the client is reloaded, it starts logging for a few seconds again. Nothing useful for debugging shows up at the server or client internal logs. Hope you guys can lend me a hand here. Please let me know if you need any other information. Here are the meaningful config sections on both the server and the client. (I think it's worth mentioning that the sources, destinations and the parser below are not used by any other log paths.) ########### SERVER options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); owner("root"); group("adm"); perm(0640); }; source s_remote_apache { tcp(ip(0.0.0.0) port(51480)); }; destination d_apache { file("/var/log/httpd/${APACHE.SERVER_NAME:-nouser}.log" template("MSGONLY\n")); }; parser p_apache { csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME", "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS", "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT", "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME") flags(escape-double-char,strip-whitespace) delimiters(" ") quote-pairs('""[]') ); }; log { source(s_remote_apache); parser(p_apache); destination(d_apache); }; ############ CLIENT options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_apache { file ("/var/log/httpd/access_log" flags(no-parse)); }; destination d_apache_remote { tcp("myhost.example.com" port(51480)); }; log { source(s_apache); destination(d_apache_remote); }; Luis Alen www.izap.com.br Ligue com tarifa local de todo o Brasil 4020.3000

2 1

syslog-ng 3.4.0rc1 has been released
by devel＠balabit.hu 07 Jan '13

07 Jan '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0rc1 SUMMARY : new stable release candidate DATE : Jan 7, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new release candidate of syslog-ng Open Source Edition (3.4.0rc1) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.0rc1 Mon, 7 Jan 2013 16:00:00 +0100 Features ======== * Make it possible to link crypto stuff directly into libsyslog-ng, using the new --with-embedded-crypto option to configure. Bugfixes ======== * DNS resolving was made thread-safe, by either using getnameinfo() on platforms where it is available, or protecting the lookups with a mutex. * Restored support for systems that lack GLOB_NOMAGIC. * The syslog() destination now correctly enforces IPv4, so using udp6() followed by syslog() in the same source block will not result in syslog() trying to use IPv6. To use the syslog protocol over IPv6, use the new network() statement. * On AIX, load modules from "a" archives instead of "so" ones. * The config version checking mechanism was patched, which fixes a crash in certain cases when running under systemd. * The GeoIP module can now be built even on systems that do not have a pkg-config file for the geoip library, by setting the GEOIP_LIBS environment variable prior to running configure. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balint Kovacs <blint(a)blint.hu> Balazs Scheidler <bazsi(a)balabit.hu> Ben Lentz <blentz(a)cswg.com> Brian Kroth <bpkroth(a)gmail.com> Gergely Nagy <algernon(a)balabit.hu> Jan Schaumann <jschauma(a)netmeister.org> Peter Czanik <czanik(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> Viktor Juhasz <jviktor(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

sql-driver and "ON duplicate KEY UPDATE"
by system＠ra-schaal.de 02 Jan '13

02 Jan '13

Hi all, is it possible to perform an "ON duplicate KEY UPDATE" with the sql-driver for a mysql-database? At the moment I use destination d_sql { program("/usr/bin/mysql -uUSER -pPASSWORD DATABASE" template("INSERT INTO TABLE VALUES (inet_aton('${.apache.client_ip}') ,${.apache.content_length},CURRENT_TIMESTAMP,1 ) ON duplicate KEY UPDATE bytes=bytes+${.apache.content_length}, hits=hits+1;\n") template-escape(yes)); }; without any problems. By using this destination, syslog-ng can´t use the sql-driver. I`m not sure if there is any difference between sql-driver and the destination above. regards Florian

2 1