- syslog-ng - lists.balabit.hu

High availabilty configuration
by Ramesh Basukala 11 Mar '14

11 Mar '14

Hi, I am running open source version of syslog-ng server, currently I have only one server. I would like to add another server and configure high availability, such that log data will still be available in case my primary server dies. Looking at the documentation, syslog-ng itself does not support high availability configuration and has to be done at Operating System level. I need help setting up high availability, please point me to any resource or documentation to start with. Thanks for the help. -RB

6 7

Re: [syslog-ng] s High availabilty configuration
by Terry Slattery 10 Mar '14

10 Mar '14

My recommendation to customers who need high reliability is to run two instances, each in a separate data center. Configure all gear to send to both servers. If syslog forwarding to other consumers is needed, manually configure it on one of the two servers. Upon a failure, confirmed by a human, the servers are reconfigured for forwarding. Many syslog consumers can't handle more than one feed, thus the need to reconfigure upon a failure. This isn't optimum for a site that wants automatic operation. However, building a system that can properly determine and avoid split-brain failures (where the two systems can't talk with one another, but the network gear can send to them) is an "interesting challenge." It would be easy to build something to generate an alert when the two systems can't talk with one another and make this alert the highest priority. Humans can then step in to diagnose the problem and take the appropriate action. Meanwhile, both systems are logging whatever they receive, which may be the whole feed or a part of a feed, if the network becomes partitioned. -tcs On 3/8/14 2:57 AM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Fri, 07 Mar 2014 15:50:43 -0500 > From: Ramesh Basukala<basukalaramesh(a)gmail.com> > Subject: [syslog-ng] High availabilty configuration > To:syslog-ng@lists.balabit.hu > Message-ID:<531A3123.5080007(a)gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi, > > I am running open source version of syslog-ng server, currently I have > only one server. I would like to add another server and configure high > availability, such that log data will still be available in case my > primary server dies. > Looking at the documentation, syslog-ng itself does not support high > availability configuration and has to be done at Operating System level. > > I need help setting up high availability, please point me to any > resource or documentation to start with. > > Thanks for the help. > -RB -- Terry Slattery CCIE# 1026

2 1

Program() in destination driver not working for Macros defined in CSV-PARSER
by Justin B 10 Mar '14

10 Mar '14

Hello On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); }; I want to launch a script whenever the UDP messages are in.So I defined the destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; script is working fine with other destination drivers. Please help -- Kale

3 7

Syslog-ng & Distributed File Systems??
by Jiann-Ming Su 08 Mar '14

08 Mar '14

Is there a version of syslog-ng that can properly use distributed filesystems like GlusterFS? For example, I have two nodes each running syslog-ng but also sharing a GlusterFS filesystem. Can those independent syslog-ng processes running on different servers write to the same file on the GlusterFS?

7 20

syslog-ng OSE 3.5.1 is not accepting global options
by Tamayo, Andres 07 Mar '14

07 Mar '14

Hello, I am experiencing problems with syslog-ng OSE 3.5.1 reading the configuration file and accepting its global options. I had problems with 2 global options specified in my syslog-ng.conf file: @version: 3.5 @include "scl.conf" options { flush_lines(0); time_reopen(10); log_fifo_size(1000); use_dns(no); use_fqdn(yes); keep_hostname(no); keep_timestamp(yes); chain_hostnames(no); perm(0640); owner("root"); group("root"); create_dirs(yes); dir_perm(0750); dir_owner("root"); dir_group("root"); }; source s_cache { file("/path/to/some/file.txt", flags(no-parse)); }; destination d_network { syslog("192.168.1.10" port(1901) transport("tcp")); }; log { source(s_cache); destination(d_network); }; The time_reopen and log_fifo_size seem to be forced to zero if I try to specify alternate values for them as shown above. If I do not include them in the configuration file, syslog-ng uses their default values time_reopen=60 and log_fifo_size=10000. Does anybody know what may be causing this problem? The log_fifo_size default value is OK but I need a faster time_reopen window and I cannot find a way to specify it. If I do, syslog-ng forces it to time_reopen=0. I can tell because I get message like this when it is trying to reconnect: Syslog connection failed; fd='8', server='AF_INET(192.168.1.10:1901)', error='Connection refused (146)', time_reopen='0' If I remove the time_reopen(10) global option from the syslog-ng.conf file, then I get the following message when syslog-ng it is trying to reconnect: Syslog connection failed; fd='8', server='AF_INET(192.168.1.10:1901)', error='Connection refused (146)', time_reopen='60' Please let me know if you have any questions for me. Thank you for your help.

2 5

[Bug 272] New: When syslog-ng server is not available, all syslog-ng clients are frozen
by bugzilla＠bugzilla.balabit.com 07 Mar '14

07 Mar '14

https://bugzilla.balabit.com/show_bug.cgi?id=272 Summary: When syslog-ng server is not available, all syslog-ng clients are frozen Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: critical Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: carl.chenet-ext(a)cloudwatt.com Type of the Report: bug Estimated Hours: 0.0 Os: Ubuntu Precise Syslog-ng version: 3.4.7-1~mhp1.sysvinit Hi, When the syslog-ng server is not available, all syslog-ng clients are frozen. We have to restart syslog-ng by mco because no other access work (ssh, direct console...). I'm including your syslog-ng configuration files. Feel free to ask me whenever information you need. Regards, Carl Chenet -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

Problems creating syslog-ng.persit- file
by Tamayo, Andres 07 Mar '14

07 Mar '14

Hello, I am having problems starting up syslog-ng when I specify the location and name for its persist file. Here is the way I am trying to start it: syslog-ng --cfgfile=/etc/syslog-ng.conf --pidfile=/mnt/config/syslog-ng.pid --persist-file=/mnt/config/syslog-ng.persist When I use this command, syslog-ng starts and creates the /mnt/config/syslog-ng.persist file but it adds a '-' (dash) character to the end of the filename, like this: /mnt/config/syslog-ng.persist- Then it executes right to the point where it is about to parse the configuration file (I can see this in the debug logs) and stops. The command return value is 2. I have seen syslog-ng return 0 on success and 1 on failure but I do not know what is the meaning of 2 as the return value. I tried the same filename but a different location (/tmp) and it worked. The filename did not have the appended '-' and syslog-ng returned 0. The location I am trying to use is a flash memory mount that has JFFS2 file system. I wonder if this is part of the problem. I tried to change the name of the persist file to something different like: --persist-file=/mnt/config/something.else and the behavior is the same. The file is created like this: /mnt/config/something.else- and syslog-ng returns 2. Does anybody know why the '-' is appended to the filename or what is the meaning of a 2 return value? Thank you for your help.

3 4

$facility variable not working in filename - Solaris OS
by Feroz Basir 07 Mar '14

07 Mar '14

Hi all, Need some help regarding filename that use $facility variable. For some reason the facility variable was set to 'a' during file naming convention. I don't recall to have 'a' as a facility on remote server. What could be the issue here? This only happen on Solaris syslog-ng server. Others OS seem ok. Config sample: destination d_remoteall {file ("/syslog/$sourceip@$facility@$r_year:$r_month:$r_day:" create_dirs(yes) perm(0644));}; Sample output: 10.10.10.1@a@2014:03:05.gz I'm running syslog-ng version 3.0.2. Thanks. Regards, Feroz Basir

4 8

syslog-ng stats and performance
by Jesse Bowling 07 Mar '14

07 Mar '14

Hello, I'm running into an issue where we're fairly certain that we're dropping log messages somewhere along this path: device -> network -> VMware -> RHEL host -> syslog-ng What I'd like to understand better is what statistics I can gather from syslog-ng itself to help show or rule out drops in the software. I'm using the following general config: syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jan 15 2012 19:47:30 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off options { flush_lines (100); time_reopen (2); log_iw_size(100); log_fifo_size (65536); log_msg_size(8192); long_hostnames (off); use_dns(yes); use_fqdn(yes); keep_hostname (no); stats_freq(3600); stats_level(1); dns_cache(yes); keep_timestamp(no); }; When I looked at "syslog-ng-ctl stats" I see these "types" dropped processed stamp stored However I only see "dropped" counters for tcp destinations, and not for any of the sources or local destinations. Does "dropped" only make sense in the remote destination case? Is there anything I can turn on/examine to tune my syslog-ng performance and verify whether I have drops occurring within syslog-ng? For the RHEL host portion I've tried watching netstat -su and netstat -st but the error counters for those do not seem to indicate that the level of issue we're seeing lies there. The processor for syslog-ng is busy, but averages 75%... Is it foolish to expect VMware to keep up with the level of logs we're taking in? Might virtualization be hiding drops from me? Any help appreciated... Cheers, Jesse -- Jesse Bowling

3 3

TLS-Syslog not working
by stefan.zahnd＠id.unibe.ch 07 Mar '14

07 Mar '14

Hi I hope someone can help me! The syslog-ng in our environment sends syslog messages using tls to our SIEM (Qradar). The following is the configuration of the syslog-ng (ip changed): # First, set some global options. options { chain_hostnames(off); flush_lines(0); use_dns(yes); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; destination d_qradar_tls {tcp(„1.2.3.4" port(6514) tls( peer-verify(required-untrusted) ca_dir("/opt/syslog-ng/etc/syslog-ng/ca.d")) ); }; destination d_qradar_local { file("/tmp/qradar_local"); }; source s_testlog { file("/tmp/testlog" flags(no-parse)); }; log { source(s_testlog); destination(d_qradar_local); destination(d_qradar_tls); }; When I insert a message into the testlog it is parsed and written into the local destination „d_qradar_local“ but not sent to the remote destination. Syslog-NG in debugging mode (syslog-ng –Fevdt) shows the following: … Syslog connection established; fd='7', server=‚AF_INET(1.2.3.4:6514)', local='AF_INET(0.0.0.0:0)' Incoming log entry; line='test' Initializing destination file writer; template='/tmp/qradar_local', filename='/tmp/qradar_local' Destination timed out, reaping; template='/tmp/qradar_local', filename='/tmp/qradar_local' Closing log transport fd; fd=’15' Using tcpdump to check if some packets are sent to Qradar reveals that only the first insertion of a message into the testlog after a restart of syslog-ng leads to a packet sent to Qradar. Every other insertion has no effect on the remote destination but is always inserted into the local destination (file). Also during the start of syslog-ng two packets are sent to qradar. I’ve also opened a ticket at IBM and awating response. Thank you very much in advance for any help on this! Kind regards, Stefan -- University of Bern IT Services Department

2 1