- syslog-ng - lists.balabit.hu

patterndb ESTRING delimiter char
by Martin Holste 16 Nov '10

16 Nov '10

I need to find a valid char to use as the ESTRING delimiter that won't appear in a URI. I had been using comma, which usually works, but lots of sites have commas in the URI which messes up the parsing of these logs. Tab is outlawed as per the documentation or I'd have used that. I've seen pipe used in URI's, so that's no better. Keep in mind that what I'm parsing has been URI unescaped. Is there a way to use a null byte or something? Thanks, Martin

2 4

patterndb and syslog from cisco
by Yann Forum 16 Nov '10

16 Nov '10

Hello, I'm writing patterndb.xml files to filter syslog messages from servers and CISCO routers. Currently, CISCO sends syslog with that format: Nov 3 15:36:02 srv01.dom.test 36779: .Nov 3 14:50:30.403: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user01] [Source: 10.0.0.1] [localport: 22] at 15:50:30 CET Wed Nov 3 2010 Nov 3 15:39:02 srv01.dom.test 36780: .Nov 3 14:53:30.255: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user01] [Source: 10.0.0.1] [localport: 22] at 15:53:30 CET Wed Nov 3 2010 Nov 3 15:42:01 srv01.dom.test 36781: .Nov 3 14:56:30.378: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user01] [Source: 10.0.0.1] [localport: 22] at 15:56:30 CET Wed Nov 3 2010 The problem comes from the program name which changes for each message: 36779, 36780, 36781, etc. For this reason, I can't use patterndb mechanism. How may I solve my problem? I think it's not allowed to change the program name with the "rewrite" rule. I have the same problem with switches from Alcatel... Regards, Yann I.

5 5

Configuration tuning for reliability
by Ben Tisdall 16 Nov '10

16 Nov '10

Dear list, Apologies for the long winded post but I'd really appreciate your comments on: 1) A configuration that did not perform adequately 2) My understanding of the relevant tuneables Number of clients: 40 Log sources per client: 35. 34 of these are pure file source driver, system logs use the standard redhat config. Dest: single loghost over tcp/ip, loghost uses mysql driver Due to other pressures I made no attempt to tune parameters that influence reliability before the test and performance was commensurately poor (about 40% of entries discarded at the client, confirmed by packet capture on the loghost). Client global opts: * log_msg_size (24576) * log_fifo_size (1000) * log_fetch_limit (10) * flush_lines(0) Client logging options: * each source does a program-override for loghost filtering, no other processing * flags(flow-control) NOT SET Loghost global opts * log_msg_size(32768) * log_fifo_size(1000) Loghost logging opts: * flags(flow-control) - NOT SET * streams undergoes a rewrite pass a simple filter (on program name), rewrite, parse, then mysql dest - the loghost load has always been negligable. >From what I NOW understand I need to do something like this: Client: log_iw_size >= SOURCES_PER_CLIENT * log_fetch_limit eg 35 * 10 = 350 log_fifo_size >= SOURCES_PER_CLIENT * log_fetch_limit eg 35 * 10 = 350 AND log_fifo_size >= SOURCES_PER_CLIENT * log_iw_size eg 35 * 350 = 12250 So it appears to me that setting log_fifo_size to > 12250 would be correct. Loghost Less idea about this, do I need: log_iw_size >= NUMBER_OF_CLIENTS * log_fetch_limit ( * SOURCES_PER_CLIENT ? ) eg 40 * 10 * 35 = 14000 And log_fifo_size >= log_iw_size ? Is flow control important between the network source and the mysql dests? Further information: * Some of the larger logfiles output ~ 4 lines/sec * Approx 4G aggregate logs generated over 14 hours by the 40 hosts * Reliability is more important than speed - these logs are not analysed in real time. That said, the client should have to spend hours completing the log transfer once its workload has been processed.. Again, apologies for the length of the post and many thanks in advance for any help. Ben Tisdall PhotoBox

4 6

syslog-ng OSE 3.2 Administrator Guide - beta
by Fekete Robert 14 Nov '10

14 Nov '10

Hi, I have released a beta version of the new administrator guide. I have updated it to cover (hopefully) every important change and feature in 3.2beta, but it still needs a review. So if something isn't working as written, is missing, or contains errors, please let me know. You can find a summary of new sections here: http://robert.blogs.balabit.com/2010/11/syslog-ng-open-source-edition-3-2-a… Regards, Robert

2 1

dont-create-tables flag in 3.2 admin documentation
by Andreas Maechler 14 Nov '10

14 Nov '10

Hi all First of all, thank you for another nice syslog-ng documentation. One comment about the new 3.2 guide: Shouldn't the sql() option 'dont-create-tables' be listed as part of the flag() option instead of being listed as a separate option? - Andreas

2 1

Problems using the syslog-protocol flag
by Andreas Maechler 14 Nov '10

14 Nov '10

Hi all Running syslog-ng 3.2beta1 on FreeBSD, I'm trying to use the 'syslog-protocol' flag in a unix-dgram source: source s_local { unix-dgram("/var/run/log" flags(syslog-protocol)); unix-dgram("/var/run/logpriv" perm(0600)); file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse)); internal(); }; That option seems to be ignored though. If I force the option by setting it manually in afsocket_sd_init_instance(), afsocket.c, all works well and incoming messages get parsed according to IETF. Am I missing something or is this a bug? Thanks in advance. Andreas

2 1

CSV parser and empty fields
by Bill Anderson 14 Nov '10

14 Nov '10

I'm using a tab separated format from apache for access logs. My last two fields are referrer and user-agent. Obviously sometimes there is no referrer. Unfortunately when there isn't one apache only logs an empty string instead of the more common "-". This isn't a problem in scripts that parse the resulting logfile as they see the resulting empty field when I log $MSG. However, I just started a new log file that uses the csv-parser w/tab as delimiter and when the referrer field is empty, APACHE.USERAGENT (the last field) gets rolled into APACHE.REFERRER, the second to last field. As a result the template for this page (which uses APACHE.REFERRER) isn't reliable. When REFERRER is empty I want it to be empty (or something I can specify, like a default) not he next field in the parser definition. I've look at the manual and don't see anything about handling empty fields. How do I get syslog-ng/csv-parser to log the empty field instead of moving to the next one? Cheers, Bill

4 10

Not able to recive all syslog messages
by add gy 13 Nov '10

13 Nov '10

Hi, I have installed syslog-ng on RHEL server as log host server for 250 router , switches but some time i recive messages some times i dont , when i check it on other syslog server on windows i revice syslog messages with any problem. Request to please review syslog-ng config file and give some suggetion. config file ------------------------------------------------------------------------------------------------------------------------- # configuration file for syslog-ng, customized for remote logging # source s_internal { internal(); }; destination d_syslognglog { file("/var/log/syslog-ng.log"); }; log { source(s_internal); destination(d_syslognglog); }; # Local sources, filters and destinations are commented out # If you want to replace sysklogd simply uncomment the following # parts and disable sysklogd # # Local sources # #source s_local { # unix-dgram("/dev/log"); # file("/proc/kmsg" log_prefix("kernel:")); #}; # # Local filters # #filter f_messages { level(info..emerg); }; #filter f_secure { facility(authpriv); }; #filter f_mail { facility(mail); }; #filter f_cron { facility(cron); }; #filter f_emerg { level(emerg); }; #filter f_spooler { level(crit..emerg) and facility(uucp, news); }; #filter f_local7 { facility(local7); }; # # Local destinations # #destination d_messages { file("/var/log/messages"); }; #destination d_secure { file("/var/log/secure"); }; #destination d_maillog { file("/var/log/maillog"); }; #destination d_cron { file("/var/log/cron"); }; #destination d_console { usertty("root"); }; #destination d_spooler { file("/var/log/spooler"); }; #destination d_bootlog { file("/var/log/boot.log"); }; # # Local logs - order DOES matter ! # #log { source(s_local); filter(f_emerg); destination(d_console); }; #log { source(s_local); filter(f_secure); destination(d_secure); flags(final); }; #log { source(s_local); filter(f_mail); destination(d_maillog); flags(final); }; #log { source(s_local); filter(f_cron); destination(d_cron); flags(final); }; #log { source(s_local); filter(f_spooler); destination(d_spooler); }; #log { source(s_local); filter(f_local7); destination(d_bootlog); }; #log { source(s_local); filter(f_messages); destination(d_messages); }; # Remote logging source s_remote { tcp(ip(0.0.0.0) port(514)); udp(ip(0.0.0.0) port(514)); }; destination d_separatedbyhosts { file("/var/log/syslog-ng/$HOST/messages" owner("root") group("root") perm(0655) dir_perm(0744) create_dirs(yes)); }; log { source(s_remote); destination(d_separatedbyhosts); }; ------------------------------------------------------------------------------------------------------------------------ Regards, Add

4 14

Source udp driver and flags
by Yann I. 10 Nov '10

10 Nov '10

Hello List, I'm using the syslog-ng OSE version 3.1.2. Currently, I have a syslog-ng server which collects events from several devices : routers (Cisco), switches (Alcatel), other devices which send events to the syslog server by using the port 514 with the protocol UDP. For the router and the switch, I need to use the flag "no-parse" because I have to parse the content of the message by using "patterndb.xml". But... for the others devices I don't want to use the flag "no-parse" because the events respect the syslog format. I think it's not possible to configure syslog-ng like this : source s_udp_01 { udp (flags(no-parse); }; source s_udp_02 { udp (); }; log { source (s_udp_01); (...) }; log { source (s_udp_02); (...) }; The workaround could be the modification of udp port destination for the devices but it's not possible for the switch Alcatel. For the router Cisco, I don't know if it's possible or not... Someone would have an idea for that "problem" ? Regards, Yann I.

3 4

clearing log files
by vijay 10 Nov '10

10 Nov '10

hi all, In my system /var/log all syslog files are accumulating.Is there any way to delete files automatically. vijay

4 5