- syslog-ng - lists.balabit.hu

GSoC Introduction
by Tibor Benke 11 Mar '14

11 Mar '14

Dear All, My name is Tibor Benke, I'm a first year MSc student at Budapest University of Technology and Economics, where I study Computer Science. I heard about GSoC and the most interesting organization was the syslog-ng project. I've professional experience in C/C++, Java, Python and worked with git. Since my childhood I always wanted to be an engineer. After graduating I got a certificate of "IT network installer and operator". Then I went to university, where I specialize in Networks and services, but I concentrate on designing and writing networked applications. My BSc thesis was to write a web based network management application in Python. During the academic years I participated in some competitions. What I'm most proud of is the national EuroSkills 2013 qualifying rounds, where my team earned the sixth place in which I was the Linux specialist. I also have work experience: one summer I patched a network driver in the Linux kernel, then did some research in the field of Hadoop and Big Data. In 2013 I contributed to some open source projects, like Log4j2, Logback and syslog-ng. Along with my studies I lead a group of 60 volunteer students, who operate the network and server of my faculty's dormitory, write games, mobile- and other applications. The project I'm most interested in is "Integration with config management systems". Since I wrote my thesis in Python, so the Salt part shouldn't be so hard. I like to learn new languages, so I'm looking forward to enhance my knowledge by learning Ruby. Regards, Tibor Benke

2 1

[GSOC 2014] Interested in working on "XMPP Destination"
by Shonali Balakrishna 11 Mar '14

11 Mar '14

Hi all, I am Shonali Balakrishna from Bangalore, India. I am going to be starting graduate school(MSc.) at TU Delft this fall and wish to participate in Google Summer of Code 2014 as a part of Syslog-ng. I am an Electronics and Telecommunications engineering graduate(2008-2012) from PESIT, Bangalore and I'm currently working in Oracle India, on automation/scripting for refresh(cloning) tasks in their Cloud Division. I am proficient in programming/scripting in C, Java and Perl. I have hands on experience with working on XML and XMPP through an automation project at work. I have user level familiarity with syslog-ng. Most importantly, I enjoy learning and am always willing to learning something new. I am interested in working on "XMPP Destination" and intend to apply for this project through GSOC 2014. I have been familiarizing myself with Syslog-ng source code in the last couple of weeks. I am working on a project proposal for the same and would like to run it by you for feedback when I'm done. I would be extremely grateful for any pointers or leads to resources you could provide me with, that would help me with this project. Thank you for your time. Best Regards, -- Shonali Balakrishna

2 1

GSoC - ZMQ transport
by Laci Mészáros 11 Mar '14

11 Mar '14

Hello, I am Laszlo Meszaros a student at Szechenyi Istvan University, studying Software Engineering.I have read the GSoC idea list and "ZMQ transport" idea. I think it is a good idea either it is source, destination, transport or all of them. There are use cases when people can use that to make their everyday job easier. (e.g.: salt monitoring, to build scaled systems, or in case of zmq transport use multiple syslog-ng on different nodes)I hope I have most of the skills that is necessary for that, but not all of them. I would like to gather the missing ones during GSoC. (e.g.: I know syslog-ng code base on a basic level, but I would like to get deeper knowledge.) I have programming knowledge in python, C and very minimal in unix shell which I think are enough to complete the project during the summer. So I would like to participate to GSoC as student who undertake to solve ZMQ transport idea. Best Regards, Laci

2 1

GSOC 2014 - ZMQ transport
by Sanidhya Kashyap 11 Mar '14

11 Mar '14

Hello everyone, I am a prospective GSOC applicant and I am quite interested in working on the project idea titled - 'ZMQ transport'. Since, neither was Viktor online nor was Gergely available while I was online, so I thought I should sent a mail on the mailing list. A bit of introduction about me. I am a Masters student studying at IIIT Hyderabad, where I have worked on live migration of virtual machines. There, I have also used ZMQ for one of my sub-projects where I was required to do some distributed data transfer across multiple physical machines. Hoping to hear from you people soon. Thanks, Sanidhya

2 1

Re: [syslog-ng] High availabilty configuration
by Terry Slattery 11 Mar '14

11 Mar '14

Ramesh, My customers are network folks, not sysadmins. Network folks are good at networking and network troubleshooting, but not great at installing and maintaining things like DRBD. So DRBD is not an option. I don't see how you could use rsync with the log folder to make a single data set. The folders/directories become different in a split-brain situation and rsync operates on files and directories. Consider the split-brain case where two syslog-ng servers can't see each other and some network devices can still send to one of the two serves. The syslog-ng data now becomes different, with some devices able to log to one syslog-ng server, but not the other. If you're forwarding syslog data to other consumers (alerting systems, security systems, etc), only one server will typically be able to reach some of these systems. What happens depends on your architecture and the devices involved. This is what makes automatic handling difficult. Failure of the syslog-ng server itself is easy to handle; it is a network failure that creates the split-brain situation that's difficult. -tcs On 3/11/14 2:15 AM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Mon, 10 Mar 2014 14:10:24 -0400 > From: Ramesh Basukala<basukalaramesh(a)gmail.com> > Subject: Re: [syslog-ng] s High availabilty configuration > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Message-ID: > <CABJn1Wj3TEESuNsE1CcTvHKCBEKAhwOmV6gr6y8ibzR+w+U+MQ(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Terry, > > Thanks for your suggestion. The reason I am trying to configure high > availability with heartbeat/DRBD is data consistency. I prefer no to have 2 > different sets of data in case one of the syslog server went offline. Do > you think we might be able to rsync the log folder once in a while to make > data consistent? > > Thanks, > RB -- Terry Slattery CCIE# 1026

1 0

High availabilty configuration
by Ramesh Basukala 11 Mar '14

11 Mar '14

Hi, I am running open source version of syslog-ng server, currently I have only one server. I would like to add another server and configure high availability, such that log data will still be available in case my primary server dies. Looking at the documentation, syslog-ng itself does not support high availability configuration and has to be done at Operating System level. I need help setting up high availability, please point me to any resource or documentation to start with. Thanks for the help. -RB

6 7

Re: [syslog-ng] s High availabilty configuration
by Terry Slattery 10 Mar '14

10 Mar '14

My recommendation to customers who need high reliability is to run two instances, each in a separate data center. Configure all gear to send to both servers. If syslog forwarding to other consumers is needed, manually configure it on one of the two servers. Upon a failure, confirmed by a human, the servers are reconfigured for forwarding. Many syslog consumers can't handle more than one feed, thus the need to reconfigure upon a failure. This isn't optimum for a site that wants automatic operation. However, building a system that can properly determine and avoid split-brain failures (where the two systems can't talk with one another, but the network gear can send to them) is an "interesting challenge." It would be easy to build something to generate an alert when the two systems can't talk with one another and make this alert the highest priority. Humans can then step in to diagnose the problem and take the appropriate action. Meanwhile, both systems are logging whatever they receive, which may be the whole feed or a part of a feed, if the network becomes partitioned. -tcs On 3/8/14 2:57 AM, syslog-ng-request(a)lists.balabit.hu wrote: > Date: Fri, 07 Mar 2014 15:50:43 -0500 > From: Ramesh Basukala<basukalaramesh(a)gmail.com> > Subject: [syslog-ng] High availabilty configuration > To:syslog-ng@lists.balabit.hu > Message-ID:<531A3123.5080007(a)gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi, > > I am running open source version of syslog-ng server, currently I have > only one server. I would like to add another server and configure high > availability, such that log data will still be available in case my > primary server dies. > Looking at the documentation, syslog-ng itself does not support high > availability configuration and has to be done at Operating System level. > > I need help setting up high availability, please point me to any > resource or documentation to start with. > > Thanks for the help. > -RB -- Terry Slattery CCIE# 1026

2 1

Program() in destination driver not working for Macros defined in CSV-PARSER
by Justin B 10 Mar '14

10 Mar '14

Hello On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); }; I want to launch a script whenever the UDP messages are in.So I defined the destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; script is working fine with other destination drivers. Please help -- Kale

3 7

Syslog-ng & Distributed File Systems??
by Jiann-Ming Su 09 Mar '14

09 Mar '14

Is there a version of syslog-ng that can properly use distributed filesystems like GlusterFS? For example, I have two nodes each running syslog-ng but also sharing a GlusterFS filesystem. Can those independent syslog-ng processes running on different servers write to the same file on the GlusterFS?

7 20

syslog-ng OSE 3.5.1 is not accepting global options
by Tamayo, Andres 07 Mar '14

07 Mar '14

Hello, I am experiencing problems with syslog-ng OSE 3.5.1 reading the configuration file and accepting its global options. I had problems with 2 global options specified in my syslog-ng.conf file: @version: 3.5 @include "scl.conf" options { flush_lines(0); time_reopen(10); log_fifo_size(1000); use_dns(no); use_fqdn(yes); keep_hostname(no); keep_timestamp(yes); chain_hostnames(no); perm(0640); owner("root"); group("root"); create_dirs(yes); dir_perm(0750); dir_owner("root"); dir_group("root"); }; source s_cache { file("/path/to/some/file.txt", flags(no-parse)); }; destination d_network { syslog("192.168.1.10" port(1901) transport("tcp")); }; log { source(s_cache); destination(d_network); }; The time_reopen and log_fifo_size seem to be forced to zero if I try to specify alternate values for them as shown above. If I do not include them in the configuration file, syslog-ng uses their default values time_reopen=60 and log_fifo_size=10000. Does anybody know what may be causing this problem? The log_fifo_size default value is OK but I need a faster time_reopen window and I cannot find a way to specify it. If I do, syslog-ng forces it to time_reopen=0. I can tell because I get message like this when it is trying to reconnect: Syslog connection failed; fd='8', server='AF_INET(192.168.1.10:1901)', error='Connection refused (146)', time_reopen='0' If I remove the time_reopen(10) global option from the syslog-ng.conf file, then I get the following message when syslog-ng it is trying to reconnect: Syslog connection failed; fd='8', server='AF_INET(192.168.1.10:1901)', error='Connection refused (146)', time_reopen='60' Please let me know if you have any questions for me. Thank you for your help.

2 5