January 2012 - syslog-ng - lists.balabit.hu

syslog-ng Insider - January 2012
by Peter Czanik 12 Jan '12

12 Jan '12

Dear syslog-ng users, This is the 10th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> POLL We would like to know, which features of syslog-ng do you regard as most important. Please check our survey at http://balabit.com/poll/syslog-ng-features FEATURED NEWS syslog-ng 3.3.4 is being prepared --------------------------------- The release of 3.3.3 brought many new users to the latest syslog-ng version, which helped to uncover some more bugs in advanced configurations. Until a new release arrives, please check the git tree ( git://git.balabit.hu/bazsi/syslog-ng-3.3 ) and the mailing list archives, if your problem is addressed by a patch. If working directly from git is problematic in your environment, a daily shapshot of 3.3 development is available at http://packages.madhouse-project.org/syslog-ng/3.3/syslog-ng-3.3-HEAD.tar.gz Meet us at FOSDEM 2012 in Brussels! ----------------------------------- There will be one or two presentations held by BalaBit guys this year. As schedule and rooms are not yet final, please check http://www.fosdem.org/ if you want to meet us or drop me an e-mail! Junction -------- Version 3.4 of syslog-ng is under intensive development. One of the new features, the “junction”, which makes syslog-ng.conf even more flexible, was just introduced on the syslog-ng mailing list: https://lists.balabit.hu/pipermail/syslog-ng/2012-January/018074.html OTHER SHORT NEWS * syslog-ng web GUI blog updated: http://czanik.blogs.balabit.com/2011/12/graphical-user-interfaces-for-use-w… WHITE PAPERS * SIEM solutions are the core of enterprise security. But they are not much use without data integrity and high performance centralized log collection. Learn how to enhance your SIEM with syslog-ng from our latest white paper at http://www.balabit.com/support/documentation/uc_opt_siem_en_web03.pdf ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

documentation missing sql options
by Patrick Hemmer 10 Jan '12

10 Jan '12

The documentation (at least http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid…>) is missing a couple of options for the sql destination driver * session-statements * retries Was just going through and noticed cause I need to use the session-statements one. <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid…>

1 1

set tag in log statement
by Patrick Hemmer 10 Jan '12

10 Jan '12

Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s. I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`. Such a config would look like filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); tags("tag1"); }; filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); tags("tag1"); }; filter f_c { not tags("tag1"); }; log { source(s_local); filter(f_c); destination (d_c); };

4 8

Escape slashes etc in macro variables ($PROGRAM)
by Göran Sandahl 10 Jan '12

10 Jan '12

Howdy, Is there a option for escaping macro variables such as $PROGRAM? I've seen a couple of discussions about this on the list but they never seem to come to a conclusion. Case in point: I'm creating an output file macro like this: file("/var/log/$SOURCEIP.$PROGRAM.log" ... and Syslog-ng failes to write the file whenever $PROGRAM looks similar to "postfix/quemgr" since its then trying to "create" the directory postfix due to the slash. Of course, thats not what I want. Can this be circumvented somehow? Cheers! Goran

5 5

Re: [syslog-ng] Syslog-ng 3.2 connection timeout with firewall
by André Larose 10 Jan '12

10 Jan '12

Hi, I found the timeout issue. It turns out that the keep_alive is not generated by Syslog-NG but by the OS. I adjusted the following values: tcp_keepalive_time tcp_keepalive_intvl tcp_keepalive_probes First in the live Kernel with the command : sysctl -w net.ipv4.tcp_keepalive_time=180 net.ipv4.tcp_keepalive_intvl=180 net.ipv4.tcp_keepalive_probes=9 Then I made it permanent with adding to /etc/sysctl.conf: cat >>/etc/sysctl.conf <<EOF # # Keepalive parameters for the syslog-ng # net.ipv4.tcp_keepalive_time = 180 net.ipv4.tcp_keepalive_intvl = 180 net.ipv4.tcp_keepalive_probes = 9 EOF You can verify your work is done with the following command: sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes net.ipv4.tcp_keepalive_time = 180 net.ipv4.tcp_keepalive_intvl = 180 net.ipv4.tcp_keepalive_probes = 9 I found the information from this URL: http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html Maybe it could be mentioned in the documentation that the so_keep-alive option can be tweaked at the kernel level. Regards, ------------------------------ Message: 8 Date: Tue, 3 Jan 2012 17:46:06 -0500 From: Andr? Larose <andre.larose(a)telus.com<mailto:andre.larose@telus.com>> Subject: [syslog-ng] Syslog-ng 3.2 connection timeout with firewall To: "syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>" <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Message-ID: <BDB004F99074254BBCEE3AB94F31FDF5435439F706(a)WP40066.corp.ads<mailto:BDB004F99074254BBCEE3AB94F31FDF5435439F706@WP40066.corp.ads>> Content-Type: text/plain; charset="iso-8859-1" Hi, I have two syslog-ng 3.2, one client and one server. The two are separated by a firewall and a load balancer. I noticed that after some inactivity I was not able to receive logs from my client. So I started some tcpdump on both servers to check the traffic. From what I see the firewall will close the connection after some time, so when the client sends traffic it gets dropped. I added the keep-alive(yes) and so_keep-alive(yes) and the mark_freq(60) to the configs. But I still do not see keepalive packets with tcpdump. Am I missing other parameters to have "keepalive" traffic sent ? Thank you in advance.

1 0

program destination and init??
by Matt Zagrabelny 09 Jan '12

09 Jan '12

Hi, I've got a question regarding the program destination and how/when it spawns the program. I've got a few program destination directives in my syslog-ng config. When I start the syslog-ng daemon, the program's (of the program destinations) parents are the syslog-ng daemon. After some amount of time (perhaps 20 or 30 hours) I get more programs spawned. Some of the programs have parents that are 'init' (PID = 1). One of my programs is 'vpndb' - see the grep output for an example: $ ps -ef | grep vpndb root 610 1 41 Nov23 ? 2-01:29:55 perl /usr/bin/vpndb root 4036 605 0 06:44 ? 00:00:00 /bin/sh -c /usr/bin/vpndb root 4039 4036 0 06:44 ? 00:00:00 perl /usr/bin/vpndb root 5231 1 26 Nov24 ? 1-02:09:12 perl /usr/bin/vpndb root 5322 1 1 Nov27 ? 00:21:26 perl /usr/bin/vpndb root 5658 1381 0 08:33 pts/4 00:00:00 grep vpndb root 16167 1 12 Nov26 ? 06:19:10 perl /usr/bin/vpndb root 26542 1 19 Nov25 ? 14:12:19 perl /usr/bin/vpndb Above there are 5 vpndb programs with a parent of PID = 1. One of the vpndb's belongs to syslog-ng. Two of my other program destinations are "ssh" commands. They exhibit the same symptoms (multiple programs running with parents of PID = 1). Hence I don't think this problem is related to the specific program being executed. Does anyone have any idea why I am getting multiple programs spawned? (I presume that syslog-ng is spawning these programs.) I do have cron jobs that will 'reload' the syslog-ng daemon. Thanks for any hints. -matt zagrabelny

5 11

[PATCH] logrewrite: Correctly initialize the conditional rules.
by Gergely Nagy 09 Jan '12

09 Jan '12

In case where the condition of a rewrite used a filter reference instead of inlining the filter itself, the filter expression was never initialized. This resulted in the condition always evaluating to not-match, making the following construct completely useless: rewrite("new-value", value("MESSAGE"), condition(filter(f_my_filter))) The fix is to introduce a new function, log_rewrite_init_method(), which will initialize the condition filter if it exists, and has an init method. Then we bind this method to LogPipe's init callback during log_rewrite_init(). This way, when initializing the rewrite pipe, the condition will get initialised too. Reported-by: Thomas Wollner <tw(a)wollner-net.de> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logrewrite.c | 12 ++++++++++++ 1 files changed, 12 insertions(+), 0 deletions(-) diff --git a/lib/logrewrite.c b/lib/logrewrite.c index a578439..3ff245a 100644 --- a/lib/logrewrite.c +++ b/lib/logrewrite.c @@ -75,6 +75,17 @@ log_rewrite_free_method(LogPipe *s) log_process_pipe_free_method(s); } +static gboolean +log_rewrite_init_method(LogPipe *s) +{ + LogRewrite *self = (LogRewrite *) s; + + if (self->condition && self->condition->init) + self->condition->init(self->condition, log_pipe_get_config(s)); + + return TRUE; +} + static void log_rewrite_init(LogRewrite *self) { @@ -83,6 +94,7 @@ log_rewrite_init(LogRewrite *self) self->super.super.flags |= PIF_CLONE; self->super.super.free_fn = log_rewrite_free_method; self->super.super.queue = log_rewrite_queue; + self->super.super.init = log_rewrite_init_method; self->value_handle = LM_V_MESSAGE; } -- 1.7.7.3

2 1

Error while starting Syslog-ng
by Kareep Navas 03 Jan '12

03 Jan '12

Error creating AF_INET socket (Error 0) Error initializing configuration, exiting.

2 1

What is the latest version of Syslog-ng?
by Kareep Navas 03 Jan '12

03 Jan '12

Hi All, We are currently using syslog-ng 1.4.14 and i would like to use & know the latest version of this syslog-ng? What is the purpose of the latest version? How to download the latest version? What are all the dependencies? Please direct me.. Thank you,

2 1

missing IP information at the message part
by Oğuz Yarımtepe 01 Jan '12

01 Jan '12

I have syslog-ng and logzilla 2.9.9g installed on a Debian squeeze machine. I can see the logs at the web interface. I configured syslog-ng to collect apache error and access logs. The problem is that i don't see IP information at the message part, on web interface. For ex, one of the machine that is sending apache logs is "www" machine. The syslog-ng configuration is as below: # All messages send to a remote site # #log { source(s_src); destination(d_net); }; #manual edit options { log_msg_size(8192); }; ####### # sources # ####### source s_apache_access { # unix-stream("/var/log/httpd/apache_log.socket" # max-connections(512) # keep-alive(yes)); file ("/var/log/apache2/access.log" flags(no-parse)); }; source s_apache_error { # unix-stream("/var/log/httpd/apache_log.socket" # max-connections(512) # keep-alive(yes)); file ("/var/log/apache2/error.log" flags(no-parse)); }; ################ # destinations # ################ destination d_custom_access { file("/var/log/apache2/custom_access.log"); }; destination d_custom_error { file("/var/log/apache2/custom_error.log"); }; destination d_apache_tls { tcp("192.168.1.145" port(6514) tls(ca_dir("/etc/syslog-ng/etc/ca.d") key_file("/etc/syslog-ng/etc/key.d/debian4key.pem") cert_file("/etc/syslog-ng/etc/cert.d/debian4cert.pem"))); }; ########### # filters # ########### filter f_apache_access { message("GET|POST"); }; filter f_apache_error { message("error"); }; log { source(s_apache_access); # filter(f_apache_access); destination(d_apache_tls); }; log { source(s_apache_error); # filter(f_apache_access); destination(d_apache_tls); }; and # tail -n1 /var/log/apache2/access.log 78.185.240.170 - - [01/Jan/2012:13:35:06 +0200] "GET /images/rand/randpic11.jpg HTTP/1.1" 200 45670 "http://www.comu.edu.tr/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7" www:/home/oguz# For the www machine i don't see the 78.185.240.170 information at the web ui, but the rest of the line starting with -- Thogh another machine, "reverse", is sending apache logs with and additional field and i can see its log at the web ui without any missing information. The reverse has the same configuration except from the key information. # tail -n1 /var/log/apache2/other_vhosts_access.log www.beeseurope.eu:80 178.154.160.29 - - [01/Jan/2012:13:36:46 +0200] "GET /mod/forum/discuss.php?d=55&parent=88 HTTP/1.1" 200 34557 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" At the web interface, I can see the IP information also at the message part. My syslog-ng configuration at the server is as below: options { long_hostnames(off); # doesn't actually help on Solaris, log(3) truncates at 1024 chars log_msg_size(8192); # buffer just a little for performance flush_lines(16384); # memory is cheap, buffer messages unable to write (like to loghost) log_fifo_size(16384); # Hosts we don't want syslog from #bad_hostname("^(ctld.|cmd|tmd|last)$"); # The time to wait before a dead connection is reestablished (seconds) time_reopen(10); #Use DNS so that our good names are used, not hostnames use_dns(yes); dns_cache(yes); #Use the whole DNS name use_fqdn(yes); keep_hostname(yes); chain_hostnames(no); #Read permission for everyone perm(0644); # The default action of syslog-ng 1.6.0 is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # # how many messages syslog-ng missed (0). # stats(43200); log_msg_size(8192); }; source s_apache { tcp(ip(0.0.0.0) port(6514) tls( key_file("/etc/syslog-ng/etc/key.d/debian1key.pem") cert_file("/etc/syslog-ng/etc/cert.d/debian1cert.pem") ca_dir("/etc/syslog-ng/etc/ca.d"))); }; filter f_apache_access { message("GET|POST"); }; filter f_apache_error { message("error"); }; # Create destination to LogZilla destination d_logzilla { program("/opt/logzilla-2.9.9g/scripts/db_insert.pl" template("$HOST\t$FACILITY\t$PRIORITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n") #template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n") ); }; # test purposes destination d_apache_test_access { file("/var/log/apache2/hosts/$HOST/$YEAR/$MONTH/$DAY/access.log" create_dirs(yes)); }; destination d_apache_test_error { file("/var/log/apache2/hosts/$HOST/$YEAR/$MONTH/$DAY/error.log" create_dirs(yes)); }; # Tell syslog-ng to log to our new destination log { source(s_apache); destination(d_logzilla); }; log { source(s_apache); filter(f_apache_error); destination(d_apache_test_error); }; log { source(s_apache); filter(f_apache_access); destination(d_apache_test_access); }; Any ides about how i can fix the missing IP information at the web ui of logzilla?

1 0