- syslog-ng - lists.balabit.hu

syslog-ng 3.0.9 has been released
by devel＠balabit.hu 26 Nov '10

26 Nov '10

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.0.9 SUMMARY : new stable release DATE : Nov 26, 2010 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.0.9) has been released. For latest fixes in the 3.0.x branch you are recommended to upgrade to this version. CHANGES: 3.0.9 Thu, 18 Nov 2010 06:43:49 +0100 Bugfixes: * csv-parser() defaults to "escape-none" mode even if it is not specified. * Fixed a possible CPU spinning in case mark_freq() is changed from a non-zero value to zero and syslog-ng is reloaded. * Fixed a flow control problem for the internal() source possibly causing an assertion to fail, e.g. syslog-ng to crash voluntarily. * Make it possible to use a literal '$' character by using two of them. * Make it possible to use binary characters within strings in configuration files by using a C-like \xHH, \oOOO syntax. * Attempt to flush the messages sitting in output queues a bit harder at shutdown. * Handle ESTALE errors in the file source for NFS file systems, which might cause a spin on the CPU otherwise. * Fixed a possible use-after-free problem potentially causing crashes or misbehaviour. No concrete errors were reported though. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…

1 0

non syslog files
by maillists0＠gmail.com 22 Nov '10

22 Nov '10

I have some homegrown application logs that are not in syslog format. I want to have syslog-ng tail those files and forward them to a syslog server (well, really a Splunk server.) In this case, does syslog-ng care what format the file is in? If so, is there a way around it?

2 1

use_fqdn()
by Brian A. Seklecki 22 Nov '10

22 Nov '10

Quick question: How does use_fqdn() interact with the various HOST, HOST_FROM, FULLHOST, FULLHOST_FROM? My understanding is that $FULLHOST and $FULLHOST_FROM are the only ones that will include the FQDN. Does use_fqdn() override that? Is the ideal config to have use_fqdn(yes) + use_dns(yes) + keep_hostname(no) and set $FULLHOST_FROM or $FULLHOST everywhere you want a FQDN so long as DNS is there? That doesnt seem to be working for me, use_fqdn(no) just kills all FQDNs everywhere regardless of $HOST* variable. I'm running the latest 2.x because I suck so much; maybe this is fixed in 3.x Thanks, ~BAS

2 3

Syslog-ng error
by Rakesh Pillai 22 Nov '10

22 Nov '10

Hi, I am trying to compile syslog-ng-2.1.3 using the --enable-static-linking option in the cnfigure part. But, while trying to do the make , I get the below error. /usr/lib/gcc/s390x-redhat-linux/4.1.2/../../../../lib64/libwrap.a(options.o): In function `group_option': (.text+0x85a): warning: Using 'endgrent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking libsyslog-ng.a(misc.o): In function `resolve_user': /tmp/TEST/WORK/syslog-ng-2.1.3/src/misc.c:258: warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/local/lib/libglib-2.0.a(gutils.o): In function `g_get_any_init_do': /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1645: warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1644: warning: Using 'setpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1646: warning: Using 'endpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1588: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1591: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking libsyslog-ng.a(afinet.o): In function `afinet_resolve_name': /tmp/TEST/WORK/syslog-ng-2.1.3/src/afinet.c:103: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking libsyslog-ng.a(misc.o): In function `resolve_hostname': /tmp/TEST/WORK/syslog-ng-2.1.3/src/misc.c:166: warning: Using 'gethostbyaddr' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking Can you please help me out here. Regards, Rakesh. CSC • This is a PRIVATE message. If you are not the intended recipient, please delete without copying and kindly advise us by e-mail of the mistake in delivery. NOTE: Regardless of content, this e-mail shall not operate to bind CSC to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose • Computer Sciences Corporation India Pvt Ltd • 7th, Floor, Block 1B, DLF IT Park, Sivaji Garden, Nandambakkam Post, Ramapuram, Chennai-600 089.

2 3

log sample collecting project
by Peter Czanik 19 Nov '10

19 Nov '10

Hello, As part of the patterndb project, we plan to start a log sample collecting project. At http://czanik.blogs.balabit.com/2010/11/log-sample-collecting-project/ you can read a document, which describes it. It has three main parts: 1. background / what is it good for 2. methods 3. technical requirements It still has some "FIXME" parts in it, but already enough to get started. Please let us know what you think about it, if you have any questions, miss any information, etc.! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

Re: [syslog-ng] Better performance between udp, unix-stream or pipe ?
by Matthew Hall 18 Nov '10

18 Nov '10

As a connectionless protocol UDP cannot always detect existing listeners on all platforms. However if you think about it it's clear why this wouldn't work. How would it know which messages should be parse or noparse? It can't read your mind! ;) Matthew. Sent from my mobile device "Yann I." <yann.frm(a)gmail.com> wrote: >> >> Does this work? >> >> source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; >> source s_udp { udp(port(514)); }; >> >> (...) >> >> >I tried that solution few days ago and it didn't work. Two processes >"syslog-ng" listened on port UDP/514 : the messages are not handled >correctly. They would be handled by "s_udp_not_parsed" or "s_udp". > >nb : it's strange that two processes can listen on the same port (for the >protocol UDP) isn't it ? This is the same behaviour with "netcat". I tried >with netcat (example : nc -l -u 1234) and I have two processes which listen >the port UDP/1234. Messages send by "nc" on the port 1234 are received by >the last "netcat" which has been started. > > >For passing yourself back the reparsed message I would recommend >> unix-dgram. AF_UNIX is usually better than pipes for me at least. >> > >Thank you ! I'll use AF_UNIX. > >Regards, > >Yann I. > > >2010/11/16 Matthew Hall <mhall(a)mhcomputing.net> > >> For passing yourself back the reparsed message I would recommend >> unix-dgram. AF_UNIX is usually better than pipes for me at least. >> >> Matthew. >> >> On Tue, Nov 16, 2010 at 12:14:53PM -0600, Martin Holste wrote: >> > Does this work? >> > >> > source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; >> > source s_udp { udp(port(514)); }; >> > >> > log { >> > source(s_udp); >> > parser(db-parser()); >> > destination(d_parsed); >> > }; >> > log { >> > source(s_udp_not_parsed); >> > destination(d_not_parsed); >> > }; >> > >> > Otherwise, try reassembling a no-parse like message with a different >> > output template. >> > >> > On Tue, Nov 16, 2010 at 11:13 AM, Yann I. <yann.frm(a)gmail.com> wrote: >> > > Well I'm not sure because of the flag I used for the UDP source which >> is set >> > > to "no-parse". >> > > >> > > Here is my problem. From the UDP source, I may receive logs which are >> not >> > > "syslog compliance". So I'm using the flag 'no-parse' then I rewrite >> the >> > > message. After that rewrite, I forward the new message to the same >> syslog-ng >> > > server. >> > > Then... I can apply filter, parser, etc on that new message which is >> now >> > > "syslog compliance" :-) >> > > >> > > So, I think I can't use log statement. I need to use that mecanism... >> > > There might be another solution but this one seems to be a good >> solution. >> > > >> > > >> > > 2010/11/16 Martin Holste <mcholste(a)gmail.com> >> > >> >> > >> Ok, then this should be accomplished with a standard log statement >> > >> like you've already begun to write. What do your destinations look >> > >> like? >> > >> >> > >> On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm(a)gmail.com> wrote: >> > >> > In fact, this is the same process... There is only one process. >> > >> > >> > >> > >> > >> > 2010/11/16 Martin Holste <mcholste(a)gmail.com> >> > >> >> >> > >> >> Why do you need separate syslog-ng processes running? >> > >> >> >> > >> >> On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm(a)gmail.com> >> wrote: >> > >> >> > Hi ! >> > >> >> > >> > >> >> > I have a question about the use of udp, unix-stream or pipe. I >> would >> > >> >> > like to >> > >> >> > forward a syslog message to the same syslog server like this : >> > >> >> > >> > >> >> > | log { >> > >> >> > | source (s_r_udp); (<-- listen on UDP/514) >> > >> >> > | >> > >> >> > | filter (....); >> > >> >> > | filter (....); >> > >> >> > | parser (...); >> > >> >> > | >> > >> >> > | destination (d_local_syslog); (<-- send the message to a >> local >> > >> >> > syslog >> > >> >> > by using unix-stream, udp or pipe mecanism) >> > >> >> > | }; >> > >> >> > >> > >> >> > (...) >> > >> >> > >> > >> >> > | log { >> > >> >> > | source (s_local_syslog); (<--- here I receive the >> messages >> > >> >> > sent >> > >> >> > by >> > >> >> > the "d_syslog_loop") >> > >> >> > | >> > >> >> > | filter (...); >> > >> >> > | filter (...); >> > >> >> > | parser (...); >> > >> >> > | >> > >> >> > | destination (d_remote_syslog); >> > >> >> > >> > >> >> > I'm looking for the better way to send syslog message to the same >> > >> >> > syslog >> > >> >> > server : which mecanism provides the better performances : pipe, >> udp >> > >> >> > (by >> > >> >> > using network) or unix-stream ? >> > >> >> > Maybe the "pipe" is the better solution ?... >> > >> >> > >> > >> >> > I'm using the syslog-ng OSE 3.1.2 on CentOS. >> > >> >> > >> > >> >> > Regards, >> > >> >> > >> > >> >> > Yann I. >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> > >> ______________________________________________________________________________ >> > >> >> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > >> >> > Documentation: >> > >> >> > http://www.balabit.com/support/documentation/?product=syslog-ng >> > >> >> > FAQ: http://www.campin.net/syslog-ng/faq.html >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> >> > >> >> >> > >> >> >> ______________________________________________________________________________ >> > >> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > >> >> Documentation: >> > >> >> http://www.balabit.com/support/documentation/?product=syslog-ng >> > >> >> FAQ: http://www.campin.net/syslog-ng/faq.html >> > >> >> >> > >> > >> > >> > >> > >> >> > >> >> ______________________________________________________________________________ >> > >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > >> Documentation: >> > >> http://www.balabit.com/support/documentation/?product=syslog-ng >> > >> FAQ: http://www.campin.net/syslog-ng/faq.html >> > >> >> > > >> > > >> > >> ______________________________________________________________________________ >> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> > FAQ: http://www.campin.net/syslog-ng/faq.html >> > >>

2 3

syslog-ng Open Source Edition 3.2 Administrator Guide – First Edition
by Fekete Robert 18 Nov '10

18 Nov '10

Hi, The administrator guide for version 3.2 of syslog-ng OSE is available at the BalaBit Documentation Page at http://www.balabit.com/support/documentation/?product_line=46&product_t…. Actually, it is mostly the same as the latest draft, except that I incorporated Bazsi's comments and corrected some minor typos and errors. The main new features of syslog-ng OSE 3.2 are listed in the 'Summary of changes' section of the guide (http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gui…) As always, comments and other feedback is greatly appreciated. Regards, Robert

1 0

Syslog-ng not receiving messages
by keshava Veerabhadraiah 18 Nov '10

18 Nov '10

Hi I am new to syslog-ng and I have gone through other post to see if I can get a resolution to my problem. Syslog is not writing to the destination file any messages received on udp() or tcp(). I have made sure that syslog server is receiving the syslog messages as seen from the tcpdump 15:09:55.422423 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.warning, length: 153 15:09:55.434638 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 184 15:09:55.470383 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 176 15:09:55.473519 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 190 15:09:55.493361 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 180 15:09:55.493748 IP aspsyslog.sungardebs.com.ssh > nim.sungardebs.com.42703: P 128608:129696(1088) ack 289 win 461 <nop,nop,timestamp 88706531 1310848493> 15:09:55.495519 IP 10.140.141.9.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 188 15:09:55.495548 IP 10.140.141.9.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.debug, length: 90 15:09:55.495556 IP 10.140.141.9.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.debug, length: 85 15:09:55.521115 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.debug, length: 87 15:09:55.521188 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 188 15:09:55.522041 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 175 15:09:55.522212 IP 10.140.141.7.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 164 Here is how my syslog-ng config looks. @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { chain_hostnames(no); create_dirs (no); dir_perm(0755); dns_cache(no); keep_hostname(yes); log_fifo_size(2048); log_msg_size(1024); log_iw_size (500); long_hostnames(on); perm(0644); stats_freq(3600); flush_lines(100); time_reopen (10); use_dns(no); use_fqdn(yes); # max_connections(100); }; source s_all { udp(so_rcvbuf(2048576)); tcp(); unix-stream("/dev/log"); internal(); file("/proc/kmsg"); }; destination d_file_normal {file("/var/log/messages_syslog-ng.log"); }; log { source(s_all); destination (d_file_normal); }; Any help would be greatly appreciated. Thanks

7 16

Better performance between udp, unix-stream or pipe ?
by Yann I. 17 Nov '10

17 Nov '10

Hi ! I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this : | log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | }; (...) | log { | source (s_local_syslog); (<--- here I receive the messages sent by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog); I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?... I'm using the syslog-ng OSE 3.1.2 on CentOS. Regards, Yann I.

3 7

patterndb ESTRING delimiter char
by Martin Holste 16 Nov '10

16 Nov '10

I need to find a valid char to use as the ESTRING delimiter that won't appear in a URI. I had been using comma, which usually works, but lots of sites have commas in the URI which messes up the parsing of these logs. Tab is outlawed as per the documentation or I'd have used that. I've seen pipe used in URI's, so that's no better. Keep in mind that what I'm parsing has been URI unescaped. Is there a way to use a null byte or something? Thanks, Martin

2 4