- syslog-ng - lists.balabit.hu

Flow-control
by Anton Koldaev 16 Jan '13

16 Jan '13

Could you please explain the following statement: > If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination. What does that mean - "stops reading messages from the source"? My applications is still sending messages to this souce so where will all the logs at that moment? Where will it start reading the source? From the same point it stopped or not? -- Best regards, Koldaev Anton

2 6

Stats levels
by Anton Koldaev 16 Jan '13

16 Jan '13

I'm using stats_level(0) on my syslog-ng clients and I'm able to see processed, stored and dropped messages. But there are only processed messages counters at my central syslog-ng server when using stats_level(0). Setting stats_level to 1 gives me dropped and stored counters but the total number of counters on my central syslog-ng server at stats_level(1) is about 70 000 and it takes too long to get it. Any advices? -- Best regards, Koldaev Anton

1 0

"make check" in a restricted environment
by Peter Czanik 15 Jan '13

15 Jan '13

Hello, I'm building my packages mainly in the openSUSE build service (OBS: http://build.opensuse.org/) My problem is, that while "make check" runs without problems on my local machine, it fails in the OBS. You can check where it fails (most likely hiding other failures) at https://build.opensuse.org/package/live_build_log?arch=x86_64&package=syslo… The OBS builds packages in a very restricted virtualized environment, with no networking, etc. to make sure, that even harmful sources can't manipulate the build infrastructure. The above problem also seems to be related. Could you check the tests and make sure, that no name resolution, etc. is needed to pass the tests? Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

syslog-ng OSE configurator
by András Mitzki 15 Jan '13

15 Jan '13

Hello, My name is Andras Mitzki, I work for BalaBit as a syslog-ng test engineer. I started to develop a syslog-ng OSE configurator web application. My goal with this web app is to generate a syntactically correct syslog-ng OSE config with only a few clicks. Please try it, and let me know if something is not clear, or you have found a bug. Of course any ideas are welcome. syslog-ng OSE configurator app: http://mitzkia.github.com/syslog-ng-ose-configurator syslog-ng OSE configurator source: https://github.com/mitzkia/syslog-ng-ose-configurator I have also written a short blog post about it: http://micek.blogs.balabit.com/en/2013/01/syslog-ng-ose-configurator/ Thanks for trying it.

7 15

syslog-ng 3.3.7 DNS resolving Problem
by Daniel Neubacher 09 Jan '13

09 Jan '13

Hello there, I've got a little trouble with the DNS resolving of syslog-ng. Last week I patched my syslog installation with the threaded dns bugfix (https://bugzilla.balabit.com/show_bug.cgi?id=212) and it seems like most of my problems are gone but one is still remaining. Many times a day messages are sorted into a folder with the DNS name of my syslog-ng server instead of the real host where the log is coming from. The log line still has the right host in the text and most of the time it is working but I could not find any way to reproduce the problem on demand yet. For debugging I've disabled any logging for the server itself but it still happens. My destinations are configured like this: destination d_syslog { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$FULLHOST_FROM/$PROGRAM" template(t_plain)); }; And my dns options: use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(300); dns_cache_expire_failed(10); I've tried disabling the syslog-ng cache,installing a local caching bind and after that a nscd but with no success. With 750 servers sending 30k-40k logs per second the dns querys are too expensive and I need the internal syslog-ng caching. With local bind caching the logs per second are dropping down to 2500. Does anybody has an idea to fix this? -- Daniel Neubacher, Network Administrator daniel.neubacher(a)xing.com<mailto:daniel.neubacher@xing.com> XING AG Gaensemarkt 43, 20354 Hamburg, Germany Tel. +49 40 419131-28, Fax +49 40 419131-11 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.

2 9

loggen read-file tokens?
by Clayton Dukes 08 Jan '13

08 Jan '13

What is the expected file format used for loggen? i.e: which tokens should each line consist of? >From the docs: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… I'd like to pump a bunch of messages (other than PADPADPAD) through loggen to generate tests. But it doesn't say anything about what format to use. I've tried using just a text file with a message per line, but I get Invalid line X for some messages an not others. Am I just misunderstanding what this command line option is for? Sample Messages: 2012-11-27 08:48:29 16.22.1.42 %SYS-3-CPUHOG: the CPU is hogged 2012-11-27 08:48:29 16.22.2.42 %IR-6-INSCARD: Card inserted in slot 0, interfaces administratively shut down 2012-11-27 08:48:29 16.22.2.42 %OIR-6-REMCARD: Card removed from slot 0, interfaces disabled So it seem I should be able to use: loggen --size 300 --rate 6000 --interval 600 192.168.28.29 514 -R ./myfile.txt but then only some lines work and some don't. Sometimes the host comes in on the receiving server as an invalid IP. Is there a way to specify tokens, like f1=datetime -f2=host -f3=PRI -f4=msg. Would really be nice if we could also specify the delimiter between them, like: f1=datetime -f2=host -f3=PRI -f4=msg --delim=\t ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 2

AUTO: James B Horwath is out of the office (returning 01/09/2013)
by James B Horwath 08 Jan '13

08 Jan '13

I am out of the office until 01/09/2013. Note: This is an automated response to your message "syslog-ng Digest, Vol 93, Issue 7" sent on 1/8/2013 15:43:58. This is the only notification you will receive while this person is away. ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

2 1

getting syslog stats without syslog-ng-ctl?
by Daniel Neubacher 08 Jan '13

08 Jan '13

Hello there, we are plotting and monitoring lot of data and for this I'm using a little bash script which does sum up the output of syslog-ng-ctl. But this is kind of slow with 17k lines to parse :P Is there a way to get the stats directly without syslog-ng-ctl? My goal is to write a ruby script which only parse the stats i'm needing. I'm no developer so please answer admin friendly :)

3 4

Problem parsing apache logs
by Luis Fernando Alen 07 Jan '13

07 Jan '13

Hello, list. I've been trying to parse my apache logs in order to separate log messages into different files based on a column provided by the parser, which in my case is $APACHE.SERVER_NAME. It's pretty much the same as the example available on the Administrator Guide, in section 3.8, p 60. Looks like I'm close to get it working but there are two problems which I could not solve: 1- The first field of the log (APACHE.CLIENT_IP) vanishes when it reaches its final destination and for that reason APACHE.SERVER_NAME, which is the last column in the parser is empty. That messes up the file creation on the server. Not mentioning the important log data loss. 2- Syslog-ng server only logs for a few seconds and suddenly stops. Whenever the client is reloaded, it starts logging for a few seconds again. Nothing useful for debugging shows up at the server or client internal logs. Hope you guys can lend me a hand here. Please let me know if you need any other information. Here are the meaningful config sections on both the server and the client. (I think it's worth mentioning that the sources, destinations and the parser below are not used by any other log paths.) ########### SERVER options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); owner("root"); group("adm"); perm(0640); }; source s_remote_apache { tcp(ip(0.0.0.0) port(51480)); }; destination d_apache { file("/var/log/httpd/${APACHE.SERVER_NAME:-nouser}.log" template("MSGONLY\n")); }; parser p_apache { csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME", "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS", "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT", "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME") flags(escape-double-char,strip-whitespace) delimiters(" ") quote-pairs('""[]') ); }; log { source(s_remote_apache); parser(p_apache); destination(d_apache); }; ############ CLIENT options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_apache { file ("/var/log/httpd/access_log" flags(no-parse)); }; destination d_apache_remote { tcp("myhost.example.com" port(51480)); }; log { source(s_apache); destination(d_apache_remote); }; Luis Alen www.izap.com.br Ligue com tarifa local de todo o Brasil 4020.3000

2 1

syslog-ng 3.4.0rc1 has been released
by devel＠balabit.hu 07 Jan '13

07 Jan '13

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.4.0rc1 SUMMARY : new stable release candidate DATE : Jan 7, 2013 ------------------------------------------------------------------------------ DESCRIPTION: A new release candidate of syslog-ng Open Source Edition (3.4.0rc1) has been released. For latest fixes in the 3.4.x feature branch you are recommended to upgrade to this version. CHANGES: 3.4.0rc1 Mon, 7 Jan 2013 16:00:00 +0100 Features ======== * Make it possible to link crypto stuff directly into libsyslog-ng, using the new --with-embedded-crypto option to configure. Bugfixes ======== * DNS resolving was made thread-safe, by either using getnameinfo() on platforms where it is available, or protecting the lookups with a mutex. * Restored support for systems that lack GLOB_NOMAGIC. * The syslog() destination now correctly enforces IPv4, so using udp6() followed by syslog() in the same source block will not result in syslog() trying to use IPv6. To use the syslog protocol over IPv6, use the new network() statement. * On AIX, load modules from "a" archives instead of "so" ones. * The config version checking mechanism was patched, which fixes a crash in certain cases when running under systemd. * The GeoIP module can now be built even on systems that do not have a pkg-config file for the geoip library, by setting the GEOIP_LIBS environment variable prior to running configure. Credits: ======== syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. These people have helped in this release: Balint Kovacs <blint(a)blint.hu> Balazs Scheidler <bazsi(a)balabit.hu> Ben Lentz <blentz(a)cswg.com> Brian Kroth <bpkroth(a)gmail.com> Gergely Nagy <algernon(a)balabit.hu> Jan Schaumann <jschauma(a)netmeister.org> Peter Czanik <czanik(a)balabit.hu> Tamas Pal <folti(a)balabit.hu> Viktor Juhasz <jviktor(a)balabit.hu> DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0