November 2010 - syslog-ng - lists.balabit.hu

Syslog-ng error
by Rakesh Pillai 22 Nov '10

22 Nov '10

Hi, I am trying to compile syslog-ng-2.1.3 using the --enable-static-linking option in the cnfigure part. But, while trying to do the make , I get the below error. /usr/lib/gcc/s390x-redhat-linux/4.1.2/../../../../lib64/libwrap.a(options.o): In function `group_option': (.text+0x85a): warning: Using 'endgrent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking libsyslog-ng.a(misc.o): In function `resolve_user': /tmp/TEST/WORK/syslog-ng-2.1.3/src/misc.c:258: warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/local/lib/libglib-2.0.a(gutils.o): In function `g_get_any_init_do': /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1645: warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1644: warning: Using 'setpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1646: warning: Using 'endpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1588: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/TEST/WORK/glib-2.16.5/glib/gutils.c:1591: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking libsyslog-ng.a(afinet.o): In function `afinet_resolve_name': /tmp/TEST/WORK/syslog-ng-2.1.3/src/afinet.c:103: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking libsyslog-ng.a(misc.o): In function `resolve_hostname': /tmp/TEST/WORK/syslog-ng-2.1.3/src/misc.c:166: warning: Using 'gethostbyaddr' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking Can you please help me out here. Regards, Rakesh. CSC • This is a PRIVATE message. If you are not the intended recipient, please delete without copying and kindly advise us by e-mail of the mistake in delivery. NOTE: Regardless of content, this e-mail shall not operate to bind CSC to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose • Computer Sciences Corporation India Pvt Ltd • 7th, Floor, Block 1B, DLF IT Park, Sivaji Garden, Nandambakkam Post, Ramapuram, Chennai-600 089.

2 3

log sample collecting project
by Peter Czanik 19 Nov '10

19 Nov '10

Hello, As part of the patterndb project, we plan to start a log sample collecting project. At http://czanik.blogs.balabit.com/2010/11/log-sample-collecting-project/ you can read a document, which describes it. It has three main parts: 1. background / what is it good for 2. methods 3. technical requirements It still has some "FIXME" parts in it, but already enough to get started. Please let us know what you think about it, if you have any questions, miss any information, etc.! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 2

Re: [syslog-ng] Better performance between udp, unix-stream or pipe ?
by Matthew Hall 18 Nov '10

18 Nov '10

As a connectionless protocol UDP cannot always detect existing listeners on all platforms. However if you think about it it's clear why this wouldn't work. How would it know which messages should be parse or noparse? It can't read your mind! ;) Matthew. Sent from my mobile device "Yann I." <yann.frm(a)gmail.com> wrote: >> >> Does this work? >> >> source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; >> source s_udp { udp(port(514)); }; >> >> (...) >> >> >I tried that solution few days ago and it didn't work. Two processes >"syslog-ng" listened on port UDP/514 : the messages are not handled >correctly. They would be handled by "s_udp_not_parsed" or "s_udp". > >nb : it's strange that two processes can listen on the same port (for the >protocol UDP) isn't it ? This is the same behaviour with "netcat". I tried >with netcat (example : nc -l -u 1234) and I have two processes which listen >the port UDP/1234. Messages send by "nc" on the port 1234 are received by >the last "netcat" which has been started. > > >For passing yourself back the reparsed message I would recommend >> unix-dgram. AF_UNIX is usually better than pipes for me at least. >> > >Thank you ! I'll use AF_UNIX. > >Regards, > >Yann I. > > >2010/11/16 Matthew Hall <mhall(a)mhcomputing.net> > >> For passing yourself back the reparsed message I would recommend >> unix-dgram. AF_UNIX is usually better than pipes for me at least. >> >> Matthew. >> >> On Tue, Nov 16, 2010 at 12:14:53PM -0600, Martin Holste wrote: >> > Does this work? >> > >> > source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; >> > source s_udp { udp(port(514)); }; >> > >> > log { >> > source(s_udp); >> > parser(db-parser()); >> > destination(d_parsed); >> > }; >> > log { >> > source(s_udp_not_parsed); >> > destination(d_not_parsed); >> > }; >> > >> > Otherwise, try reassembling a no-parse like message with a different >> > output template. >> > >> > On Tue, Nov 16, 2010 at 11:13 AM, Yann I. <yann.frm(a)gmail.com> wrote: >> > > Well I'm not sure because of the flag I used for the UDP source which >> is set >> > > to "no-parse". >> > > >> > > Here is my problem. From the UDP source, I may receive logs which are >> not >> > > "syslog compliance". So I'm using the flag 'no-parse' then I rewrite >> the >> > > message. After that rewrite, I forward the new message to the same >> syslog-ng >> > > server. >> > > Then... I can apply filter, parser, etc on that new message which is >> now >> > > "syslog compliance" :-) >> > > >> > > So, I think I can't use log statement. I need to use that mecanism... >> > > There might be another solution but this one seems to be a good >> solution. >> > > >> > > >> > > 2010/11/16 Martin Holste <mcholste(a)gmail.com> >> > >> >> > >> Ok, then this should be accomplished with a standard log statement >> > >> like you've already begun to write. What do your destinations look >> > >> like? >> > >> >> > >> On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm(a)gmail.com> wrote: >> > >> > In fact, this is the same process... There is only one process. >> > >> > >> > >> > >> > >> > 2010/11/16 Martin Holste <mcholste(a)gmail.com> >> > >> >> >> > >> >> Why do you need separate syslog-ng processes running? >> > >> >> >> > >> >> On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm(a)gmail.com> >> wrote: >> > >> >> > Hi ! >> > >> >> > >> > >> >> > I have a question about the use of udp, unix-stream or pipe. I >> would >> > >> >> > like to >> > >> >> > forward a syslog message to the same syslog server like this : >> > >> >> > >> > >> >> > | log { >> > >> >> > | source (s_r_udp); (<-- listen on UDP/514) >> > >> >> > | >> > >> >> > | filter (....); >> > >> >> > | filter (....); >> > >> >> > | parser (...); >> > >> >> > | >> > >> >> > | destination (d_local_syslog); (<-- send the message to a >> local >> > >> >> > syslog >> > >> >> > by using unix-stream, udp or pipe mecanism) >> > >> >> > | }; >> > >> >> > >> > >> >> > (...) >> > >> >> > >> > >> >> > | log { >> > >> >> > | source (s_local_syslog); (<--- here I receive the >> messages >> > >> >> > sent >> > >> >> > by >> > >> >> > the "d_syslog_loop") >> > >> >> > | >> > >> >> > | filter (...); >> > >> >> > | filter (...); >> > >> >> > | parser (...); >> > >> >> > | >> > >> >> > | destination (d_remote_syslog); >> > >> >> > >> > >> >> > I'm looking for the better way to send syslog message to the same >> > >> >> > syslog >> > >> >> > server : which mecanism provides the better performances : pipe, >> udp >> > >> >> > (by >> > >> >> > using network) or unix-stream ? >> > >> >> > Maybe the "pipe" is the better solution ?... >> > >> >> > >> > >> >> > I'm using the syslog-ng OSE 3.1.2 on CentOS. >> > >> >> > >> > >> >> > Regards, >> > >> >> > >> > >> >> > Yann I. >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> > >> ______________________________________________________________________________ >> > >> >> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > >> >> > Documentation: >> > >> >> > http://www.balabit.com/support/documentation/?product=syslog-ng >> > >> >> > FAQ: http://www.campin.net/syslog-ng/faq.html >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> >> > >> >> >> > >> >> >> ______________________________________________________________________________ >> > >> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > >> >> Documentation: >> > >> >> http://www.balabit.com/support/documentation/?product=syslog-ng >> > >> >> FAQ: http://www.campin.net/syslog-ng/faq.html >> > >> >> >> > >> > >> > >> > >> > >> >> > >> >> ______________________________________________________________________________ >> > >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > >> Documentation: >> > >> http://www.balabit.com/support/documentation/?product=syslog-ng >> > >> FAQ: http://www.campin.net/syslog-ng/faq.html >> > >> >> > > >> > > >> > >> ______________________________________________________________________________ >> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> > Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> > FAQ: http://www.campin.net/syslog-ng/faq.html >> > >>

2 3

syslog-ng Open Source Edition 3.2 Administrator Guide – First Edition
by Fekete Robert 18 Nov '10

18 Nov '10

Hi, The administrator guide for version 3.2 of syslog-ng OSE is available at the BalaBit Documentation Page at http://www.balabit.com/support/documentation/?product_line=46&product_t…. Actually, it is mostly the same as the latest draft, except that I incorporated Bazsi's comments and corrected some minor typos and errors. The main new features of syslog-ng OSE 3.2 are listed in the 'Summary of changes' section of the guide (http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gui…) As always, comments and other feedback is greatly appreciated. Regards, Robert

1 0

Syslog-ng not receiving messages
by keshava Veerabhadraiah 18 Nov '10

18 Nov '10

Hi I am new to syslog-ng and I have gone through other post to see if I can get a resolution to my problem. Syslog is not writing to the destination file any messages received on udp() or tcp(). I have made sure that syslog server is receiving the syslog messages as seen from the tcpdump 15:09:55.422423 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.warning, length: 153 15:09:55.434638 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 184 15:09:55.470383 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 176 15:09:55.473519 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 190 15:09:55.493361 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 180 15:09:55.493748 IP aspsyslog.sungardebs.com.ssh > nim.sungardebs.com.42703: P 128608:129696(1088) ack 289 win 461 <nop,nop,timestamp 88706531 1310848493> 15:09:55.495519 IP 10.140.141.9.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 188 15:09:55.495548 IP 10.140.141.9.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.debug, length: 90 15:09:55.495556 IP 10.140.141.9.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.debug, length: 85 15:09:55.521115 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.debug, length: 87 15:09:55.521188 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 188 15:09:55.522041 IP 10.140.141.6.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 175 15:09:55.522212 IP 10.140.141.7.syslog > aspsyslog.sungardebs.com.syslog: SYSLOG local4.info, length: 164 Here is how my syslog-ng config looks. @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { chain_hostnames(no); create_dirs (no); dir_perm(0755); dns_cache(no); keep_hostname(yes); log_fifo_size(2048); log_msg_size(1024); log_iw_size (500); long_hostnames(on); perm(0644); stats_freq(3600); flush_lines(100); time_reopen (10); use_dns(no); use_fqdn(yes); # max_connections(100); }; source s_all { udp(so_rcvbuf(2048576)); tcp(); unix-stream("/dev/log"); internal(); file("/proc/kmsg"); }; destination d_file_normal {file("/var/log/messages_syslog-ng.log"); }; log { source(s_all); destination (d_file_normal); }; Any help would be greatly appreciated. Thanks

7 16

Better performance between udp, unix-stream or pipe ?
by Yann I. 17 Nov '10

17 Nov '10

Hi ! I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this : | log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | }; (...) | log { | source (s_local_syslog); (<--- here I receive the messages sent by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog); I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?... I'm using the syslog-ng OSE 3.1.2 on CentOS. Regards, Yann I.

3 7

patterndb ESTRING delimiter char
by Martin Holste 16 Nov '10

16 Nov '10

I need to find a valid char to use as the ESTRING delimiter that won't appear in a URI. I had been using comma, which usually works, but lots of sites have commas in the URI which messes up the parsing of these logs. Tab is outlawed as per the documentation or I'd have used that. I've seen pipe used in URI's, so that's no better. Keep in mind that what I'm parsing has been URI unescaped. Is there a way to use a null byte or something? Thanks, Martin

2 4

patterndb and syslog from cisco
by Yann Forum 16 Nov '10

16 Nov '10

Hello, I'm writing patterndb.xml files to filter syslog messages from servers and CISCO routers. Currently, CISCO sends syslog with that format: Nov 3 15:36:02 srv01.dom.test 36779: .Nov 3 14:50:30.403: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user01] [Source: 10.0.0.1] [localport: 22] at 15:50:30 CET Wed Nov 3 2010 Nov 3 15:39:02 srv01.dom.test 36780: .Nov 3 14:53:30.255: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user01] [Source: 10.0.0.1] [localport: 22] at 15:53:30 CET Wed Nov 3 2010 Nov 3 15:42:01 srv01.dom.test 36781: .Nov 3 14:56:30.378: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user01] [Source: 10.0.0.1] [localport: 22] at 15:56:30 CET Wed Nov 3 2010 The problem comes from the program name which changes for each message: 36779, 36780, 36781, etc. For this reason, I can't use patterndb mechanism. How may I solve my problem? I think it's not allowed to change the program name with the "rewrite" rule. I have the same problem with switches from Alcatel... Regards, Yann I.

5 5

Configuration tuning for reliability
by Ben Tisdall 16 Nov '10

16 Nov '10

Dear list, Apologies for the long winded post but I'd really appreciate your comments on: 1) A configuration that did not perform adequately 2) My understanding of the relevant tuneables Number of clients: 40 Log sources per client: 35. 34 of these are pure file source driver, system logs use the standard redhat config. Dest: single loghost over tcp/ip, loghost uses mysql driver Due to other pressures I made no attempt to tune parameters that influence reliability before the test and performance was commensurately poor (about 40% of entries discarded at the client, confirmed by packet capture on the loghost). Client global opts: * log_msg_size (24576) * log_fifo_size (1000) * log_fetch_limit (10) * flush_lines(0) Client logging options: * each source does a program-override for loghost filtering, no other processing * flags(flow-control) NOT SET Loghost global opts * log_msg_size(32768) * log_fifo_size(1000) Loghost logging opts: * flags(flow-control) - NOT SET * streams undergoes a rewrite pass a simple filter (on program name), rewrite, parse, then mysql dest - the loghost load has always been negligable. >From what I NOW understand I need to do something like this: Client: log_iw_size >= SOURCES_PER_CLIENT * log_fetch_limit eg 35 * 10 = 350 log_fifo_size >= SOURCES_PER_CLIENT * log_fetch_limit eg 35 * 10 = 350 AND log_fifo_size >= SOURCES_PER_CLIENT * log_iw_size eg 35 * 350 = 12250 So it appears to me that setting log_fifo_size to > 12250 would be correct. Loghost Less idea about this, do I need: log_iw_size >= NUMBER_OF_CLIENTS * log_fetch_limit ( * SOURCES_PER_CLIENT ? ) eg 40 * 10 * 35 = 14000 And log_fifo_size >= log_iw_size ? Is flow control important between the network source and the mysql dests? Further information: * Some of the larger logfiles output ~ 4 lines/sec * Approx 4G aggregate logs generated over 14 hours by the 40 hosts * Reliability is more important than speed - these logs are not analysed in real time. That said, the client should have to spend hours completing the log transfer once its workload has been processed.. Again, apologies for the length of the post and many thanks in advance for any help. Ben Tisdall PhotoBox

4 6

syslog-ng OSE 3.2 Administrator Guide - beta
by Fekete Robert 14 Nov '10

14 Nov '10

Hi, I have released a beta version of the new administrator guide. I have updated it to cover (hopefully) every important change and feature in 3.2beta, but it still needs a review. So if something isn't working as written, is missing, or contains errors, please let me know. You can find a summary of new sections here: http://robert.blogs.balabit.com/2010/11/syslog-ng-open-source-edition-3-2-a… Regards, Robert

2 1