- syslog-ng - lists.balabit.hu

URL for open source and premium version comparison
by Clayton Dukes 06 Apr '10

06 Apr '10

Hey folks, Someone from work asked me why they should pay for the Premium edition as opposed to using the OSE. I thought I would just point him to a URL the compares the two, but can't seem to locate it. Is it out there? -- ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 2

syslog-ng 3.0.4 not adjusting for daylight savings time
by Chris Fabri 05 Apr '10

05 Apr '10

I have a red hat 3 system running 3.0.4 It did not successfully adjust for daylight savings time. Everything else on the system is fine - syslog is showing the correct time stamps, the system reports the correct time. Restart of syslog-ng, and shutting down both syslog and syslog-ng did not help. I didn't see anything in the lists addressing this, here are details of my syslog-ng: [fabric@netlog dhcp]$ sudo /usr/local/sbin/syslog-ng -V syslog-ng 3.0.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#1b5d618e301ad94aa20e692ffba16469dece8d10 Compile-Date: Aug 24 2009 16:54:31 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: off I didn't see anything in the man page for adjusting for time. I have keep_timestamp(no) configured in my conf file. The systems sending the syslog files time is correctly adjusted for daylight savings. Is this a bug in this particular version, or am I just missing the right flag or something? chris

3 13

try tcp destination but fail quickly if unavailable
by Jan Schaumann 05 Apr '10

05 Apr '10

Hello, I have a tcp destination that, unfortunately, may occasionally become unavailable. When that happens, I believe that the system will try to retransmit the messages and/or re-establish the connection (time_reopen), but either way, messages will accumulate. Is there a way to tell syslog-ng "try this destination, but if it can't be delivered within <time>, drop the message / skip the destination" ? -Jan

2 1

Change the facility or the priority of a syslog message ?
by Ilas, Yann 05 Apr '10

05 Apr '10

Hello, I'm using the syslog-ng application version 3.1.0. I would like to change the severity and/or the facility of an event. Here is the configuration for this test : template t_format { template ("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC;$HOST;$FACILITY;$PRIORITY;$PROGRAM;$MSG\n"); }; source s_local_test { unix-stream("/dev/log"); }; destination d_test { file ( "/tmp/test" template (t_format) ); }; rewrite r_rewrite_set { set ( "my_program" value("PROGRAM") ); }; log { source (s_local_test); rewrite (r_rewrite_set); destination (d_test); }; I generated a message by using "logger" : # logger -i -p local0.info "Test Message : ABCDEFGHIJKLMNOPQRST ## $(date) ##" => /tmp/test: 2010-04-01 15:12:14;pc-dev.dom;local0;info;MY_PROGRAM;Test Message : ABCDEFGHIJKLMNOPQRST ## jeu avr 1 15:12:14 CEST 2010 ## Ok, I can change the PROGRAM name... :-) ...So I changed the rule "r_rewrite_set" to modify the value to "facility" : rewrite r_rewrite_set { set ( "local4" value("FACILITY") ); }; And when I restarted the syslog-ng, I had the following errors : # /etc/init.d/syslog-ng restart Macros are read-only, they cannot be changed in rewrite rules, falling back to MESSAGE instead; macro='FACILITY' Restarting syslog-ng: Stopping syslog-ng: [ OK ] Starting syslog-ng: Macros are read-only, they cannot be changed in rewrite rules, falling back to MESSAGE instead; macro='FACILITY' [ OK ] I have the same error when I changed the "PRIORITY" of the rule "r_rewrite_set" : > Macros are read-only, they cannot be changed in rewrite rules, falling back to MESSAGE instead; macro='PRIORITY' I tried to send a message by using "logger" but as it written in the above error, the MESSAGE was changed... :-( My questions are : 1. Which macros are read-only ? PRIORITY, FACILITY, another ? 2. Is there a way to change the facility or the priority of an event ? 3. If not, why can't I change the facility or the priority ? Regards, Yann

2 1

Problem using tags with syslog-ng 3.1
by Hoenig, Stefan, VF-Group 04 Apr '10

04 Apr '10

Hi all, I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the source system and want to separate them again on the central logging server. The client configuration looks like this: ------------------------------------------------------------------------ ---------------------------- source s_app { file("/var/log/log1.log"); file("/opt//log/log2.log" tags("log2")); file("/opt/log/log3.log" tags("log3")); }; options { }; destination d_app { tcp("logrelay01.domain.com" port(13074)); }; log { source(s_app); destination(d_app); }; ------------------------------------------------------------------------ ---------------------------- The log relay does nothing than forward the messages to the central logging server with the following config: ------------------------------------------------------------------------ ---------------------------- options { time_sleep(20); log_fifo_size(1000); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); }; source s_remote { tcp(ip("0.0.0.0") port(13074)); }; destination remote_tcp { tcp("centrallog01.domain.com" port(13074)); }; log { source(s_remote); destination(remote_tcp); }; ------------------------------------------------------------------------ ---------------------------- On the central logging server I use filters to separate the logfiles again: ------------------------------------------------------------------------ ---------------------------- @version: 3.0 include "/opt/config/syslogng-inc.conf"; options { time_sleep(20); dns_cache(2000); dns_cache_expire(87600); keep_hostname(yes); create_dirs(yes); }; source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); }; ============================================ This is the confoguration in /opt/config/syslogng-inc.conf # Filter filter f_log1 { host("web00(09|10)"); }; filter f_log2 { host("web00(09|10)") and tags("log2"); }; filter f_log3 { host("web00(09|10)") and tags("log3"); }; #Configuration for Destinations destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); }; destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); }; destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); }; # Logfile log1 log { source(s_remote); filter(f_log1); destination(d_log1); }; # Logfile log2 log { source(s_remote); filter(f_log2); destination(d_log2); }; # Logfile log3 log { source(s_remote); filter(f_log3); destination(d_log3); }; ------------------------------------------------------------------------ ---------------------------- Does anybody have an idea, why it does not work as expected. Thanks for any suggestion and/or idea. Best regards Stefan

6 10

Tailing tomcat catalina.out
by BoOnSiOnG 30 Mar '10

30 Mar '10

Hello all, Good day, I have configure syslog-ng to checks catalina.out every second, but syslog-ng will append date time and machine name to the log, I would like to know is it possible to configure syslog-ng do not alter the original log? my config, source s_tomcat_catalina { file("/srv/tomcat/logs/catalina.out" default-facility(local3) default-priority(info)); }; log output, Mar 29 21:43:38 forward Mar: 29, 2010 9:43:37 PM org.apache.coyote.http11.Http11Protocol start Mar 29 21:43:38 forward INFO: Starting Coyote HTTP/1.1 on http-8080 Mar 29 21:43:38 forward Mar: 29, 2010 9:43:37 PM org.apache.jk.common.ChannelSocket init Mar 29 21:43:38 forward INFO: JK: ajp13 listening on /0.0.0.0:8009 Mar 29 21:43:38 forward Mar: 29, 2010 9:43:37 PM org.apache.jk.server.JkMain start Mar 29 21:43:38 forward INFO: Jk running ID=0 time=0/14 config=null Mar 29 21:43:38 forward Mar: 29, 2010 9:43:37 PM org.apache.catalina.startup.Catalina start Mar 29 21:43:38 forward INFO: Server startup in 428 ms Thanks. :)

3 5

queue management
by Eric Snow 29 Mar '10

29 Mar '10

Hi, I am looking to manage my output buffers. I have one destination that sometimes has trouble and I want to be able to manage the messages that got buffered while I fix my flaky destination. Is there any tool I can use to manage those buffered messages? If not, isn't there any exception handling that I can apply to a log such that errors or other failures for that log can be redirected to another destination? I want to avoid clogging up my output queue and especially to avoid losing any messages. Thanks! -eric This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you.

2 4

3.0.5 & Cisco TCP problems
by d lists 28 Mar '10

28 Mar '10

Hello, After spending the afternoon trying to get this working, I've decided to reach out for some help (tried google - no luck!). I have what I believe is a very simple syslog-ng configuration where I am trying to capture syslog events from Cisco routers via TCP. I first tested via UDP, and all works good. The router establishes a TCP connection with my syslog-ng (viewable via netstat -nap, and in the /var/log/messages): Mar 25 17:07:53 logger syslog-ng[8943]: Syslog connection accepted; fd='11', client='AF_INET(X.X.X.254:56868)', local='AF_INET(X.X.X.5:2002)' On the router I do a "config term" & "exit" to generate a log. tcpdump on the syslog-ng machine shows traffic as expected: 17:18:44.891080 IP X.X.X.254.61241 > 10.240.0.5.2002: . 3679441367:3679441450(83) ack 1811354960 win 4128 17:18:44.891134 IP X.X.X.5.2002> 10.240.0.254.61241: . ack 83 win 5840 And a strace: poll([{fd=6, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=8, events=POLLIN}], 4, 456650) = 1 ([{fd=8, revents=POLLIN}]) gettimeofday({1269559238, 85688}, NULL) = 0 read(8, "<189>145: *Mar 25 23:41:37.033: "..., 7739) = 83 read(8, 0x9f94618, 7656) = -1 EAGAIN (Resource temporarily unavailable) gettimeofday({1269559238, 86032}, NULL) = 0 poll([{fd=6, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=8, events=POLLIN}], 4, 449113 But nothing ever shows up in the log files! Here's my syslog-ng.conf. Any pointers or ideas? @version: 3.0 options { use_dns(no); }; source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); }; source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; destination d_messages { file("/var/log/messages"); }; destination d_cisco { file("/var/log/cisco.log"); }; log { source(t_net); destination(d_messages); }; log { source(s_local); destination(d_messages); }; And for what it is worth the relevant lines from my test cisco: logging host X.X.X.5 transport tcp port 2002 And syslog-ng version: syslog-ng 3.0.5 Installer-Version: 3.0.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#f2fd7457627a88eb9602a73a5c16cd8b2434b667 Compile-Date: Dec 7 2009 14:52:42 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: off Thanks in advance!

4 7

processing stops after config change
by Curt Shaffer 26 Mar '10

26 Mar '10

I have a pretty simple configuration file on my syslog-ng-2.1.3.1 running on CentOS 5.4 64bit and 8GB of RAM. I'm sending logs to 3 destinations. One is a mysql database, one is a flat file, and one is to a log correlation engine. All was running fine and out of nowhere one day, I noticed that syslog-ng kept hanging. I would restart the process and it would be fine for a few seconds then crash. I noticed that when I would start it, it would jump to 100% CPU and stay there until crash. I did some log viewing and straced the process and I noticed that logs were getting to the process, but they never got to their destinations. I started disabling one destination at a time until I found the culprit. The problem was with the destination going to the log correlation via udp 514. If I comment out that source and destination, the system runs like a champ to the other destinations. I'm confused as to what happened, because this was working for at least 3 months solid. I am posting the config here, if anyone can see something or has some suggestions, I would appreciate it. #begin syslog-ng.conf source s_all{ unix-stream("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); }; destination d_lce { udp("xxx.xxx.xxx.xxx"); } log { source(s_all); destination(d_lce); }; Just to reiterate, I do have two other destinations setup and they run just fine if I comment out this destination and log statement. Every time I uncomment this destination and log statement and restart syslog-ng, I reproduce the problem. Any suggestions would be great, and if you need more info, just ask and I can present it. The first thing I wanted to check is that the config looks sane, though again it worked fine for months. Thanks Curt

2 2

forwarding host only gives name, not IP
by Jan Schaumann 24 Mar '10

24 Mar '10

Hello, I have the following setup: +---+ hostA ---+ V | +---> syslog1 ----> syslog2 hostB ---+ That is, hostA, hostB and syslog1 are all logging to syslog1, syslog1 then takes all the messages and forwards them to syslog2. On syslog2, I'd like to get the IP addresses from each of the three hosts it's receiving messages from. syslog2 has "use_dns(no)" as a global option. When I have "keep_hostname(no)" or "chain_hostnames(no)" on syslog2, I only get syslog1's IP address in my logs, even for messages from hostA or hostB. If I put "keep_hostname(yes)" or "chain_hostnames(yes)" on syslog2, I get the IP addresses from hostA or hostB, but from syslog1 I get the short hostname (ie "syslog1"). If I add "use_fqdn(yes)" to syslog1's config, I get "syslog1.full.domain" on syslog2. However, I'd like for messages from syslog1 to arrive with the IP address, not the hostname on syslog2. I'm sure I'm missing something that'll be obvious in hindsight... just what? Thanks! -Jan

1 0

Question RE: Timezone in filename
by Brian Donaldson 23 Mar '10

23 Mar '10

I've configured syslog-ng to include the timezone in the log filename -- but the value used is "07:00". Any way to get "PDT" instead?

2 1

Spoofing SRC of forwarded syslog
by AM M84 23 Mar '10

23 Mar '10

Hi, I want to know if it is possible to spoof(UDP) the original source IP of a message that has already been forwarded. Server1 (Original Source)--> Server2 (Forwarding Syslog Server )--> Server3 (Server Source Spoofing Server1) --> Final Dest. Thanks, AMM

2 2

facility 10 plus?
by fedora fedora 23 Mar '10

23 Mar '10

Hi, I have some odd devices using syslog facility 24, and there is no way I can change it! In this case, is it possible letting syslog-ng use facility 24? right now it gives me, 24 is not supported error and quits. Thanks

3 3

syslog-ng client messages delayed
by mark peters 22 Mar '10

22 Mar '10

Hi, We are running into a issue where syslog-ng clients who are configured to forward to a central syslog-ng destination server that are reasonably chatty are getting there messages delayed at the source (ie. 'tcpdump'ing the client shows it is sending messages from N minutes ago, where N is anything from 5 to 60 minutes on average sometimes more). We have also seen a small percentage of loss at the destination. On these 'delayed' clients we also see the resident memory footprint on the growing. Clients are running 1.6.12 (have tried 2.0.8 also - same symptons). The central syslog-ng server is running 2.0.8. relevant config entries are: syslog-ng client: options { sync (1); time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; destination d_central { tcp("log2.xyz.com" port(5140)); }; filter f_local5 { facility(local5); }; log { source(s_sys); filter(f_local5); destination(d_central); flags(final); }; syslog-ng central server: options { time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_expire (28800); use_fqdn (no); create_dirs (yes); dir_perm (0755); perm (0644); keep_hostname (no); use_time_recvd (no); stats_freq(600); flush_timeout(16000); flush_lines(256); time_sleep(80); }; destination d_server { file("/logs/server.$LEVEL.log"); }; filter f_server { facility(local5) and program("server"); }; log { source(s_sys); filter(f_server); destination(d_server); flags(final); }; to give a rough idea of message volume the server is receiving approximately 50 million log lines per day on average any help/information would be appreciated thanks -------------------------------------- Easy + Joy + Powerful = Yahoo! Bookmarks x Toolbar http://pr.mail.yahoo.co.jp/toolbar/

3 4

log failback groups
by Patrick H. 22 Mar '10

22 Mar '10

How do you do groups of failbacks? For example log { filter(f_filter1); destination(d_file1); }; log { filter(f_filter1); destination(d_file2); }; <-- log here only if the above fails log { filter(f_filter2); destination(d_sql1); }; <-- may include messages from the above 2 lines log { filter(f_filter2); destination(d_sql2); }; <-- log here only if d_sql1 fails So, say lines 1 and 3 both fail, lines 2 and 3 should both start working. If I put a fallback flag on d_file2 and d_sql2, and d_sql1 fails, d_sql2 wont kick in on matches that d_file1 is still taking. It seems like there should be a way to do "log { destination(...) or destination(...); };", or "log {...} or log {...};".

7 12

syslog-ng 3.1.0 has been released
by devel＠balabit.hu 22 Mar '10

22 Mar '10

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.1.0 SUMMARY : new feature release DATE : Mar 22, 2010 ------------------------------------------------------------------------------ DESCRIPTION: This is the first stable release of syslog-ng Open Source Edition 3.1. syslog-ng 3.1 is the first so called "feature release" of syslog-ng, as such its support period ends when either the next feature (named 3.2) or the next stable (named 4.0) version is published. Please note that this announcement contains features that were already published in one of the earlier BETA releases. If you are upgrading from 3.1beta2, please see the list of changes relative to that version at the end of the "CHANGES" block below. CHANGES SINCE v3.0: New features: * Support for patterndb v2 and v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT. * Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message. * Support for message tags: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering. * Added support for rewriting structured data. * Macros and name-value pairs got a little tighter integration, in filters where syslog-ng 3.0 was limited to only use name-value pairs, with 3.1 you can also use macros. * Enhanced dynamic name-value performance by a factor of three. * Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for multiple characters as the end of the string. * Added non-standard and non-portable facility codes (range 10-15), decouple syslog-ng facility name information from the system used to compile syslog-ng on. * Added pcre support in the binary packages of syslog-ng. An updated administrator's guide is available on the BalaBit documentation page at: http://www.balabit.com/support/documentation/ CHANGES SINCE v3.1beta2: Bugfixes: * Fixed Solaris 10 SMF script to properly handle svcadm refresh requests. * Fixed a possible segmentation fault for unix-dgram/unix-stream destinations on some (non-Linux) platforms. * Fixed processing empty log entries when using the syslog() protocol. * Fixed processing partially received syslog messages. Other changes: * It is now possible to add comments before the @version header in the configuration file. * Decrease the frequency of gettimeofday() calls during syslog-ng operation, as gettimeofday() is expensive to call on some platforms. * The SQL destination will only attempt to INSERT a given log message 3 times, and after that it gives up, instead of trying indefinitely. * Tru64 portability fixes. * The functional test program was improved to check the new RFC5424 syslog protocol. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… The documentation of the syslog-ng Open Source Edition is available in The syslog-ng Open Source Edition Administrator's Guide at http://www.balabit.com/support/documentation/

1 0

Server conf for syslog-ng.conf
by Lozada, Agustin T 19 Mar '10

19 Mar '10

Hi, Can somebody show to me a sample to set-up for server of syslog-ng.conf? I'm trying to use this config but its complaining on s_network: @version:3.0 options { time_reap(30); mark_freq(10); keep_hostname(yes); }; source s_local { unix-stream("/dev/log"); internal();}; source s_network { syslog(transport(tcp))}; destination d_logs { file( "/var/log/syslog-ng/logs.txt" owner("root") group("root") perm(0777) ); }; log { source(s_local); source(s_network); destination(d_logs); }; What Im missing here? Thanks and regards, Agustin Lozada UNIX System Admin 713-207-2474

3 4

Problem with sql destination
by yavuz 19 Mar '10

19 Mar '10

Hi all, I want to insert logs directly to mysql database, I am using two destinations one is a file(for testing purposes) and the second is sql. Logs are written to file but not sql destination. I couldn't find the problem because don't know where to look. The configuration file is attached. thanks in advance. // ps.

3 3

Specific log messages have wrong hostname
by Christopher Jon Caldwell 18 Mar '10

18 Mar '10

All of the syslog messages sent by our Solaris servers that contain output from the service processor are getting the wrong hostname assigned to them - the log messages get filed under the hostname of the receiving syslog-ng server. They all share the same process name "SC Alert". The packets look correctly formed so I am assuming it is the space in the process name. Any way to fix this without dropping the messages completely using something like bad_hostname? We are running 2.1.11a Enterprise Edition. Here is the relevant part of my syslog-ng.conf: options { sync (0); create_dirs (yes); keep_hostname (yes); check_hostname (yes); chain_hostnames (no); bad_hostname ("\t"); normalize_hostnames (yes); mark_freq (10); }; source src_net { udp(port(514)); }; destination d_hosts { file("/var/log/systems/$HOST_FROM/$FACILITY/$YEAR/$MONTH/$FACILITY\-$YEAR$MONTH$DAY" perm(0644) dir_perm(0755) create_dirs(yes)); ... log { source(src_net); destination(d_hosts); destination(d_splunk); flags(fallback); }; And here is an example packet from snoop. ETHER: ----- Ether Header ----- ETHER: ETHER: Packet 36 arrived at 12:09:18.28026 ETHER: Packet size = 169 bytes ETHER: Destination = 0:3:ba:71:22:65, ETHER: Source = 0:21:28:4:ec:b7, ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----- IP: IP: Version = 4 IP: Header length = 20 bytes IP: Type of service = 0x00 IP: xxx. .... = 0 (precedence) IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: .... ..0. = not ECN capable transport IP: .... ...0 = no ECN congestion experienced IP: Total length = 155 bytes IP: Identification = 25576 IP: Flags = 0x4 IP: .1.. .... = do not fragment IP: ..0. .... = last fragment IP: Fragment offset = 0 bytes IP: Time to live = 255 seconds/hops IP: Protocol = 17 (UDP) IP: Header checksum = f267 IP: Source address = 10.244.236.183, vienna IP: Destination address = 10.241.34.101, auctor.backup.es.gwu.edu IP: No options IP: UDP: ----- UDP Header ----- UDP: UDP: Source port = 32832 UDP: Destination port = 514 (SYSLOG) UDP: Length = 135 UDP: Checksum = F5D8 UDP: SYSLOG: ----- SYSLOG: ----- SYSLOG: SYSLOG: Priority: <29> (daemon.notice) SYSLOG: "<29>Mar 12 12:09:18 SC Alert: [ID 362536 daemon.notice] Audi" SYSLOG: -- Christopher Caldwell Senior Engineer, Technology Operations and Engineering The George Washington University caldwell @ gwu . edu | +1 202.994.4674 (w) | +1 202.409.0878 (c) PGP key ID: 0x0A0EC46C "Quis custodiet ipsos custodes?" - Juvenal

4 3

subst() pcre escape sequences
by Patrick H. 18 Mar '10

18 Mar '10

Ok, I'm trying to lowercase a certain macro with a pcre rewrite rule. Unfortunately it doesnt seem to be observing pcre escape sequences. subst('^(.+)$','\L$1\E',value('FROM') flags('store-matches') type('pcre')); (the FROM macro is created from the named match of an earlier pcre filter() It ends up just putting a literal '\L' and '\E' at the start and end of the string. Is this a bug, or am I doing something wrong, or is there another way to do this?

3 3

syslog-ng to trigger SNMP traps
by Xuan Zhong 18 Mar '10

18 Mar '10

Hi, I wrote a script to filter out the matching syslog messages and send as SNMP traps. In syslog-ng.conf, I use: filter f_snmp { match("XXXX" value("MESSAGE")) destination d_snmp(program("send_trap.sh")) If I use logger to manually generate the messages, the traps can be sent successfully. But if the exact same syslog message that was generated by a program, the traps were not triggered. I use syslog-ng version 3.0.5-1 x86_64. Can anyone provide any insight on this issue? Thanks Xuan _________________________________________________________________ Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:…

2 1

possible memleak or bad configuration?
by Andreas Sartori 15 Mar '10

15 Mar '10

hello, we have setup a central logging server. currently we are logging firewalls and some webserver / mailserver for testing purpose. the memory usage on the logging server is badly increasing. after 2 days of operation we are at 6.8 gb ram usage. can someone help out, what information do you need to help? thanks in advance. -andy ------------ @version:3.0 # # configuration file for syslog-ng, customized for remote logging # options { owner("root"); group("root"); perm(0600); dir_perm(0750); create_dirs(yes); log_fifo_size(10000); }; ################################################################################################ ######################### SOURCES ############################## ################################################################################################ # Syslog internal logging source s_internal { internal(); }; destination d_syslognglog { file("/var/log/syslog-ng.log"); }; log { source(s_internal); destination(d_syslognglog); }; # Remote logging source s_remote { tcp(ip(0.0.0.0) max-connections(20) port(514) keep_hostname(yes)); udp(ip(0.0.0.0) port(514) use_dns(no) log_fetch_limit(500) log_iw_size(1000)); }; ################################################################################################ ######################### FILTER ############################## ################################################################################################ filter http-official { netmask(xxx.xxx.xxx.47/255.255.255.255) or netmask(xxx.xxx.xxx.48/255.255.255.255) or netmask(xxx.xxx.xxx.167/255.255.255.255) or netmask(xxx.xxx.xxx.46/255.255.255.255) or netmask(xxx.xxx.xxx.52/255.255.255.255) or netmask(xxx.xxx.xxx.25/255.255.255.255) or netmask(xxx.xxx.xxx.26/255.255.255.255); }; filter mail-proxy-internal { netmask(10.10.9.20/255.255.255.255) and not program("perdition"); }; filter mail-relay-internal { netmask(10.10.9.30/255.255.255.255); }; filter mail-relay-alpha-external-out { netmask(xxx.xxx.xxx.59/255.255.255.255) and facility(local1); }; filter mail-relay-beta-external-out { netmask(xxx.xxx.xxx.60/255.255.255.255) and facility(local1); }; filter mail-relay-alpha-external-in { netmask(xxx.xxx.xxx.59/255.255.255.255) and facility(mail); }; filter mail-relay-beta-external-in { netmask(xxx.xxx.xxx.60/255.255.255.255) and facility(mail); }; filter mail-proxy-node1-external { netmask(xxx.xxx.xxx.18/255.255.255.255) and not program("perdition"); }; filter mail-proxy-node2-external { netmask(xxx.xxx.xxx.22/255.255.255.255) and not program("perdition"); }; filter vpn { netmask(10.20.40.0/255.255.255.0); }; filter fw-intern-all { netmask(10.10.20.1/255.255.255.255); }; filter fw-intern-security { netmask(10.10.20.1/255.255.255.255) and match("security" value(".classifier.class") type("string")); }; filter fw-intern-info { netmask(10.10.20.1/255.255.255.255) and match("informational" value(".classifier.class") type("string")); }; filter fw-intern-rest { netmask(10.10.20.1/255.255.255.255) and not match("security" value(".classifier.class") type("string")) and not match("informational" value(".classifier.class") type("string")); }; filter fw-extern-all { netmask(10.80.11.20/255.255.255.255); }; filter fw-extern-security { netmask(10.80.11.20/255.255.255.255) and match("security" value(".classifier.class") type("string")); }; filter fw-extern-info { netmask(10.80.11.20/255.255.255.255) and match("informational" value(".classifier.class") type("string")); }; filter fw-extern-rest { netmask(10.80.11.20/255.255.255.255) and not match("security" value(".classifier.class") type("string")) and not match("informational" value(".classifier.class") type("string")); }; filter fw-extern-new { netmask(10.80.11.30/255.255.255.255); }; ################################################################################################ ######################### PARSER ############################## ################################################################################################ parser pattern_db_fwint { db_parser( file("/etc/syslog-ng/fw-int_patterndb.xml") ); }; parser pattern_db_fwext { db_parser( file("/etc/syslog-ng/fw-ext_patterndb.xml") ); }; ################################################################################################ ######################### DESTINATIONS ############################## ################################################################################################ destination http-log { file("/logging/server/web/$HOST" template("$MSGONLY\n") template-escape(no) owner("root") group("root") perm(0644)); }; destination mail-out { file("/logging/server/mail/mail-out_$MONTH.log"); }; destination mail-in { file("/logging/server/mail/mail-in_$MONTH.log"); }; destination vpn { file("/logging/network/vpn_$MONTH.log" flush_lines(10)); }; destination fw-intern-all { file("/logging/network/fw-intern_$MONTH.log" flush_lines(10)); }; destination fw-extern-all { file("/logging/network/fw-extern_$MONTH.log" flush_lines(10)); }; destination fw-extern-new { file("/logging/network/fw-new_$MONTH.log" flush_lines(10)); }; destination dump { file("/logging/network/dump.log" template ("$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC, $HOST, $FIREWALL_SEQ, $MSGHDR, 0, $FIREWALL_IO, $FIREWALL_PROTO, $FIREWALL_SCR_LAN, $FIREWALL_SRC_IP, $FIREWALL_SRC_PORT, $FIREWALL_DST_LAN, $FIREWALL_DST_IP, $FIREWALL_DST_PORT, $FIREWALL_NAT_SRC_IP, $FIREWALL_NAT_DST_IP, $FIREWALL_RULE, $FIREWALL_REASON, $FIREWALL_DURATION\n")); # file("/logging/network/dump.log" template ("$MSGHDR\n") flush_lines(5)); }; ################################################################################################ ######################### FINAL-LOGS ############################## ################################################################################################ ##### TO FILE log { source(s_remote); filter(http-official); destination(http-log); }; log { source(s_remote); filter(mail-proxy-internal); destination(mail-out); }; log { source(s_remote); filter(mail-relay-internal); destination(mail-out); }; log { source(s_remote); filter(mail-relay-alpha-external-out); destination(mail-out); }; log { source(s_remote); filter(mail-relay-beta-external-out); destination(mail-out); }; log { source(s_remote); filter(mail-proxy-node1-external); destination(mail-out); }; log { source(s_remote); filter(mail-proxy-node2-external); destination(mail-out); }; log { source(s_remote); filter(mail-relay-alpha-external-in); destination(mail-in); }; log { source(s_remote); filter(mail-relay-beta-external-in); destination(mail-in); }; log { source(s_remote); filter(vpn); destination(vpn); }; log { source(s_remote); filter(fw-intern-all); destination(fw-intern-all); }; log { source(s_remote); filter(fw-extern-new); destination(fw-extern-new); }; log { source(s_remote); filter(fw-extern-all); destination(fw-extern-all); flags(final); };

3 4

Wrong hostname on syslog-messages.
by Christopher Jon Caldwell 12 Mar '10

12 Mar '10

All of the syslog messages sent by our Solaris servers that contain output from the service processor are getting the wrong hostname assigned to them - the log messages get filed under the hostname of the receiving syslog-ng server. Our second syslog-ng server is correctly identifying the hostname with the same configuration. We are running 2.1.11a Enterprise Edition. Here is the relevant part of my syslog-ng.conf: options { sync (0); create_dirs (yes); keep_hostname (no); check_hostname (yes); chain_hostnames (no); bad_hostname ("\t"); normalize_hostnames (yes); mark_freq (10); }; source src_net { udp(port(514)); }; destination d_hosts { file("/var/log/systems/$HOST_FROM/$FACILITY/$YEAR/$MONTH/$FACILITY\-$YEAR$MONTH$DAY" perm(0644) dir_perm(0755) create_dirs(yes)); ... log { source(src_net); destination(d_hosts); destination(d_splunk); flags(fallback); }; And here is an example packet from snoop. ETHER: ----- Ether Header ----- ETHER: ETHER: Packet 36 arrived at 12:09:18.28026 ETHER: Packet size = 169 bytes ETHER: Destination = 0:3:ba:71:22:65, ETHER: Source = 0:21:28:4:ec:b7, ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----- IP: IP: Version = 4 IP: Header length = 20 bytes IP: Type of service = 0x00 IP: xxx. .... = 0 (precedence) IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: .... ..0. = not ECN capable transport IP: .... ...0 = no ECN congestion experienced IP: Total length = 155 bytes IP: Identification = 25576 IP: Flags = 0x4 IP: .1.. .... = do not fragment IP: ..0. .... = last fragment IP: Fragment offset = 0 bytes IP: Time to live = 255 seconds/hops IP: Protocol = 17 (UDP) IP: Header checksum = f267 IP: Source address = 10.244.236.183, vienna IP: Destination address = 10.241.34.101, auctor.backup.es.gwu.edu IP: No options IP: UDP: ----- UDP Header ----- UDP: UDP: Source port = 32832 UDP: Destination port = 514 (SYSLOG) UDP: Length = 135 UDP: Checksum = F5D8 UDP: SYSLOG: ----- SYSLOG: ----- SYSLOG: SYSLOG: Priority: <29> (daemon.notice) SYSLOG: "<29>Mar 12 12:09:18 SC Alert: [ID 362536 daemon.notice] Audi" SYSLOG: -- Christopher Caldwell Senior Engineer, Technology Operations and Engineering The George Washington University caldwell @ gwu . edu | +1 202.994.4674 (w) | +1 202.409.0878 (c) PGP key ID: 0x0A0EC46C "Quis custodiet ipsos custodes?" - Juvenal

1 0

Compiling with SQL Support
by Gene Matusovsky 10 Mar '10

10 Mar '10

How do I compile SYSLOG-NG with SQL support? I run ./configure --enable-sql option, but the configure script wasn't able to find my libdbi and libdbd drivers. It looks like the libdbi and libdbd authors do not supply .pc files. I see that my libdbi libraries were successfully installed in /usr/local/lib and libdbd libraries were successfully installed /usr/local/lib/dbd How do I tell the configure script to use these libraries? Should I create .pc files for libdbi and libdbd and put them in my PKG_CONFIG path? -Thanks for any suggestions, Gene Matusovsky The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain Cablevision proprietary information, which is privileged, confidential, or subject to copyright belonging to Cablevision. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof.

2 2

Re: [syslog-ng] queue management
by Fekete Róbert 08 Mar '10

08 Mar '10

Hi, AFAIK, unfortunately there is currently no way to manipulate the queue. Regards, Robert On Monday, March 08, 2010 18:12 CET, "Eric Snow" <esnow(a)verio.net> wrote: > If disk based buffering allows me to yank the message at the front of the queue, so that it doesn't hold up the ones behind it, then yes that is what I want. The statistics and ability to look at the queue are good, but I need to be able to manipulate the queue. > > OR > > I need to be able to have some form of exception handling in my syslog-ng configuration. Something that will catch a failure in a log and allow me to handle that failure intelligently. > > -eric > > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler > Sent: Saturday, March 06, 2010 2:09 AM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] queue management > > On Fri, 2010-03-05 at 17:49 -0500, Eric Snow wrote: > > Hi, > > > > > > > > I am looking to manage my output buffers. I have one destination that > > sometimes has trouble and I want to be able to manage the messages > > that got buffered while I fix my flaky destination. Is there any tool > > I can use to manage those buffered messages? If not, isn’t there any > > exception handling that I can apply to a log such that errors or other > > failures for that log can be redirected to another destination? I > > want to avoid clogging up my output queue and especially to avoid > > losing any messages. Thanks! > > you mean you have disk based buffering? in the upcoming 3.1 version (due > to be released within 2 weeks), there's a tool called dqtool to dump the > contents of the queue file, although I'm not sure that's what you meant. > there are further disk buffer related improvements in the queue for 3.2, > and also a way to send the messages to a failed destination to an > alternate server. > > also, syslog-ng is counting the number of messages in a given queue, > using its statistics framework. it is the stored counter you can see for > a given destination. this gets reported in the 'log statistics' message, > or via the syslog-ng control socket (/opt/syslog-ng/var/syslog-ng.ctl). > > > -- > Bazsi > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you. > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html >

1 0