- syslog-ng - lists.balabit.hu

syslog-ng 3.3.7 DNS resolving Problem
by Daniel Neubacher 09 Jan '13

09 Jan '13

Hello there, I've got a little trouble with the DNS resolving of syslog-ng. Last week I patched my syslog installation with the threaded dns bugfix (https://bugzilla.balabit.com/show_bug.cgi?id=212) and it seems like most of my problems are gone but one is still remaining. Many times a day messages are sorted into a folder with the DNS name of my syslog-ng server instead of the real host where the log is coming from. The log line still has the right host in the text and most of the time it is working but I could not find any way to reproduce the problem on demand yet. For debugging I've disabled any logging for the server itself but it still happens. My destinations are configured like this: destination d_syslog { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$FULLHOST_FROM/$PROGRAM" template(t_plain)); }; And my dns options: use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(300); dns_cache_expire_failed(10); I've tried disabling the syslog-ng cache,installing a local caching bind and after that a nscd but with no success. With 750 servers sending 30k-40k logs per second the dns querys are too expensive and I need the internal syslog-ng caching. With local bind caching the logs per second are dropping down to 2500. Does anybody has an idea to fix this? -- Daniel Neubacher, Network Administrator daniel.neubacher(a)xing.com<mailto:daniel.neubacher@xing.com> XING AG Gaensemarkt 43, 20354 Hamburg, Germany Tel. +49 40 419131-28, Fax +49 40 419131-11 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.

2 9

loggen read-file tokens?
by Clayton Dukes 08 Jan '13

08 Jan '13

What is the expected file format used for loggen? i.e: which tokens should each line consist of? >From the docs: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… I'd like to pump a bunch of messages (other than PADPADPAD) through loggen to generate tests. But it doesn't say anything about what format to use. I've tried using just a text file with a message per line, but I get Invalid line X for some messages an not others. Am I just misunderstanding what this command line option is for? Sample Messages: 2012-11-27 08:48:29 16.22.1.42 %SYS-3-CPUHOG: the CPU is hogged 2012-11-27 08:48:29 16.22.2.42 %IR-6-INSCARD: Card inserted in slot 0, interfaces administratively shut down 2012-11-27 08:48:29 16.22.2.42 %OIR-6-REMCARD: Card removed from slot 0, interfaces disabled So it seem I should be able to use: loggen --size 300 --rate 6000 --interval 600 192.168.28.29 514 -R ./myfile.txt but then only some lines work and some don't. Sometimes the host comes in on the receiving server as an invalid IP. Is there a way to specify tokens, like f1=datetime -f2=host -f3=PRI -f4=msg. Would really be nice if we could also specify the delimiter between them, like: f1=datetime -f2=host -f3=PRI -f4=msg --delim=\t ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 2

AUTO: James B Horwath is out of the office (returning 01/09/2013)
by James B Horwath 08 Jan '13

08 Jan '13

I am out of the office until 01/09/2013. Note: This is an automated response to your message "syslog-ng Digest, Vol 93, Issue 7" sent on 1/8/2013 15:43:58. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

2 1

getting syslog stats without syslog-ng-ctl?
by Daniel Neubacher 08 Jan '13

08 Jan '13

Hello there, we are plotting and monitoring lot of data and for this I'm using a little bash script which does sum up the output of syslog-ng-ctl. But this is kind of slow with 17k lines to parse :P Is there a way to get the stats directly without syslog-ng-ctl? My goal is to write a ruby script which only parse the stats i'm needing. I'm no developer so please answer admin friendly :)

3 4

Problem parsing apache logs
by Luis Fernando Alen 07 Jan '13

07 Jan '13

Hello, list. I've been trying to parse my apache logs in order to separate log messages into different files based on a column provided by the parser, which in my case is $APACHE.SERVER_NAME. It's pretty much the same as the example available on the Administrator Guide, in section 3.8, p 60. Looks like I'm close to get it working but there are two problems which I could not solve: 1- The first field of the log (APACHE.CLIENT_IP) vanishes when it reaches its final destination and for that reason APACHE.SERVER_NAME, which is the last column in the parser is empty. That messes up the file creation on the server. Not mentioning the important log data loss. 2- Syslog-ng server only logs for a few seconds and suddenly stops. Whenever the client is reloaded, it starts logging for a few seconds again. Nothing useful for debugging shows up at the server or client internal logs. Hope you guys can lend me a hand here. Please let me know if you need any other information. Here are the meaningful config sections on both the server and the client. (I think it's worth mentioning that the sources, destinations and the parser below are not used by any other log paths.) ########### SERVER options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); owner("root"); group("adm"); perm(0640); }; source s_remote_apache { tcp(ip(0.0.0.0) port(51480)); }; destination d_apache { file("/var/log/httpd/${APACHE.SERVER_NAME:-nouser}.log" template("MSGONLY\n")); }; parser p_apache { csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME", "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS", "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT", "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME") flags(escape-double-char,strip-whitespace) delimiters(" ") quote-pairs('""[]') ); }; log { source(s_remote_apache); parser(p_apache); destination(d_apache); }; ############ CLIENT options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_apache { file ("/var/log/httpd/access_log" flags(no-parse)); }; destination d_apache_remote { tcp("myhost.example.com" port(51480)); }; log { source(s_apache); destination(d_apache_remote); }; Luis Alen www.izap.com.br Ligue com tarifa local de todo o Brasil 4020.3000

2 1

syslog-ng 3.4.0rc1 has been released
by devel＠balabit.hu 07 Jan '13

07 Jan '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0rc1 SUMMARY : new stable release candidate DATE : Jan 7, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new release candidate of syslog-ng Open Source Edition (3.4.0rc1) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.0rc1 Mon, 7 Jan 2013 16:00:00 +0100 Features ======== * Make it possible to link crypto stuff directly into libsyslog-ng, using the new --with-embedded-crypto option to configure. Bugfixes ======== * DNS resolving was made thread-safe, by either using getnameinfo() on platforms where it is available, or protecting the lookups with a mutex. * Restored support for systems that lack GLOB_NOMAGIC. * The syslog() destination now correctly enforces IPv4, so using udp6() followed by syslog() in the same source block will not result in syslog() trying to use IPv6. To use the syslog protocol over IPv6, use the new network() statement. * On AIX, load modules from "a" archives instead of "so" ones. * The config version checking mechanism was patched, which fixes a crash in certain cases when running under systemd. * The GeoIP module can now be built even on systems that do not have a pkg-config file for the geoip library, by setting the GEOIP_LIBS environment variable prior to running configure. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balint Kovacs <blint(a)blint.hu> Balazs Scheidler <bazsi(a)balabit.hu> Ben Lentz <blentz(a)cswg.com> Brian Kroth <bpkroth(a)gmail.com> Gergely Nagy <algernon(a)balabit.hu> Jan Schaumann <jschauma(a)netmeister.org> Peter Czanik <czanik(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> Viktor Juhasz <jviktor(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

sql-driver and "ON duplicate KEY UPDATE"
by system＠ra-schaal.de 02 Jan '13

02 Jan '13

Hi all, is it possible to perform an "ON duplicate KEY UPDATE" with the sql-driver for a mysql-database? At the moment I use destination d_sql { program("/usr/bin/mysql -uUSER -pPASSWORD DATABASE" template("INSERT INTO TABLE VALUES (inet_aton('${.apache.client_ip}') ,${.apache.content_length},CURRENT_TIMESTAMP,1 ) ON duplicate KEY UPDATE bytes=bytes+${.apache.content_length}, hits=hits+1;\n") template-escape(yes)); }; without any problems. By using this destination, syslog-ng can´t use the sql-driver. I`m not sure if there is any difference between sql-driver and the destination above. regards Florian

2 1

Statistics summary
by Anton Koldaev 30 Dec '12

30 Dec '12

Is there a nice way to get total numbers of processed/stored/dropped messages in syslog-ng v3.3? >From docs: > The *center(received)* entry shows the total number of messages received from every configured sources. But in my stats center() shows zero almost for every host. I see zero in some other examples in docs too. -- Best regards, Koldaev Anton

3 7

SYSLOG-NG 3.4 exiting without any error print
by Deepak Gaur 28 Dec '12

28 Dec '12

Hello, I have cross-compiled syslog-ng for ARM target but it is exiting without showing any error code (on target). I am attaching strace output (syslog.txt). If anyone can help me out # syslog-ng -V syslog-ng 3.4.0alpha3 Installer-Version: 3.4.0alpha3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#master#709ab1252920063e6d4d0d610ad5c1ee5ebdfb75 Compile-Date: Nov 27 2012 14:59:53 Available-Modules: syslogformat,afsocket-notls,afuser,basicfuncs,confgen,dbparser,tfuuid,afmongodb,afsocket-tls,afprog,afsocket,affile,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: on ATTACHMENT syslog.txt

2 2

Re: [syslog-ng] syslog-ng 3.4.0beta1 has been released
by Evan Rempel 27 Dec '12

27 Dec '12

I thought that 3.4 was going to include - junctions so that results of multiple parsers could be combined - the ability to run a parser against a user defined macro rather than always parsing the complete MESSAGE marcro. Did these features fail to make it into 3.4 or is the documentation just missing these items? Thanks, -- Evan Rempel Senior Systems Administrator, Data Centre Services University of Victoria 250.721.7691

4 6