March 2010 - syslog-ng - lists.balabit.hu

Server conf for syslog-ng.conf
by Lozada, Agustin T 19 Mar '10

19 Mar '10

Hi, Can somebody show to me a sample to set-up for server of syslog-ng.conf? I'm trying to use this config but its complaining on s_network: @version:3.0 options { time_reap(30); mark_freq(10); keep_hostname(yes); }; source s_local { unix-stream("/dev/log"); internal();}; source s_network { syslog(transport(tcp))}; destination d_logs { file( "/var/log/syslog-ng/logs.txt" owner("root") group("root") perm(0777) ); }; log { source(s_local); source(s_network); destination(d_logs); }; What Im missing here? Thanks and regards, Agustin Lozada UNIX System Admin 713-207-2474

3 4

Problem with sql destination
by yavuz 19 Mar '10

19 Mar '10

Hi all, I want to insert logs directly to mysql database, I am using two destinations one is a file(for testing purposes) and the second is sql. Logs are written to file but not sql destination. I couldn't find the problem because don't know where to look. The configuration file is attached. thanks in advance. // ps.

3 3

Specific log messages have wrong hostname
by Christopher Jon Caldwell 18 Mar '10

18 Mar '10

All of the syslog messages sent by our Solaris servers that contain output from the service processor are getting the wrong hostname assigned to them - the log messages get filed under the hostname of the receiving syslog-ng server. They all share the same process name "SC Alert". The packets look correctly formed so I am assuming it is the space in the process name. Any way to fix this without dropping the messages completely using something like bad_hostname? We are running 2.1.11a Enterprise Edition. Here is the relevant part of my syslog-ng.conf: options { sync (0); create_dirs (yes); keep_hostname (yes); check_hostname (yes); chain_hostnames (no); bad_hostname ("\t"); normalize_hostnames (yes); mark_freq (10); }; source src_net { udp(port(514)); }; destination d_hosts { file("/var/log/systems/$HOST_FROM/$FACILITY/$YEAR/$MONTH/$FACILITY\-$YEAR$MONTH$DAY" perm(0644) dir_perm(0755) create_dirs(yes)); ... log { source(src_net); destination(d_hosts); destination(d_splunk); flags(fallback); }; And here is an example packet from snoop. ETHER: ----- Ether Header ----- ETHER: ETHER: Packet 36 arrived at 12:09:18.28026 ETHER: Packet size = 169 bytes ETHER: Destination = 0:3:ba:71:22:65, ETHER: Source = 0:21:28:4:ec:b7, ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----- IP: IP: Version = 4 IP: Header length = 20 bytes IP: Type of service = 0x00 IP: xxx. .... = 0 (precedence) IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: .... ..0. = not ECN capable transport IP: .... ...0 = no ECN congestion experienced IP: Total length = 155 bytes IP: Identification = 25576 IP: Flags = 0x4 IP: .1.. .... = do not fragment IP: ..0. .... = last fragment IP: Fragment offset = 0 bytes IP: Time to live = 255 seconds/hops IP: Protocol = 17 (UDP) IP: Header checksum = f267 IP: Source address = 10.244.236.183, vienna IP: Destination address = 10.241.34.101, auctor.backup.es.gwu.edu IP: No options IP: UDP: ----- UDP Header ----- UDP: UDP: Source port = 32832 UDP: Destination port = 514 (SYSLOG) UDP: Length = 135 UDP: Checksum = F5D8 UDP: SYSLOG: ----- SYSLOG: ----- SYSLOG: SYSLOG: Priority: <29> (daemon.notice) SYSLOG: "<29>Mar 12 12:09:18 SC Alert: [ID 362536 daemon.notice] Audi" SYSLOG: -- Christopher Caldwell Senior Engineer, Technology Operations and Engineering The George Washington University caldwell @ gwu . edu | +1 202.994.4674 (w) | +1 202.409.0878 (c) PGP key ID: 0x0A0EC46C "Quis custodiet ipsos custodes?" - Juvenal

4 3

subst() pcre escape sequences
by Patrick H. 18 Mar '10

18 Mar '10

Ok, I'm trying to lowercase a certain macro with a pcre rewrite rule. Unfortunately it doesnt seem to be observing pcre escape sequences. subst('^(.+)$','\L$1\E',value('FROM') flags('store-matches') type('pcre')); (the FROM macro is created from the named match of an earlier pcre filter() It ends up just putting a literal '\L' and '\E' at the start and end of the string. Is this a bug, or am I doing something wrong, or is there another way to do this?

3 3

syslog-ng to trigger SNMP traps
by Xuan Zhong 18 Mar '10

18 Mar '10

Hi, I wrote a script to filter out the matching syslog messages and send as SNMP traps. In syslog-ng.conf, I use: filter f_snmp { match("XXXX" value("MESSAGE")) destination d_snmp(program("send_trap.sh")) If I use logger to manually generate the messages, the traps can be sent successfully. But if the exact same syslog message that was generated by a program, the traps were not triggered. I use syslog-ng version 3.0.5-1 x86_64. Can anyone provide any insight on this issue? Thanks Xuan _________________________________________________________________ Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:…

2 1

possible memleak or bad configuration?
by Andreas Sartori 15 Mar '10

15 Mar '10

hello, we have setup a central logging server. currently we are logging firewalls and some webserver / mailserver for testing purpose. the memory usage on the logging server is badly increasing. after 2 days of operation we are at 6.8 gb ram usage. can someone help out, what information do you need to help? thanks in advance. -andy ------------ @version:3.0 # # configuration file for syslog-ng, customized for remote logging # options { owner("root"); group("root"); perm(0600); dir_perm(0750); create_dirs(yes); log_fifo_size(10000); }; ################################################################################################ ######################### SOURCES ############################## ################################################################################################ # Syslog internal logging source s_internal { internal(); }; destination d_syslognglog { file("/var/log/syslog-ng.log"); }; log { source(s_internal); destination(d_syslognglog); }; # Remote logging source s_remote { tcp(ip(0.0.0.0) max-connections(20) port(514) keep_hostname(yes)); udp(ip(0.0.0.0) port(514) use_dns(no) log_fetch_limit(500) log_iw_size(1000)); }; ################################################################################################ ######################### FILTER ############################## ################################################################################################ filter http-official { netmask(xxx.xxx.xxx.47/255.255.255.255) or netmask(xxx.xxx.xxx.48/255.255.255.255) or netmask(xxx.xxx.xxx.167/255.255.255.255) or netmask(xxx.xxx.xxx.46/255.255.255.255) or netmask(xxx.xxx.xxx.52/255.255.255.255) or netmask(xxx.xxx.xxx.25/255.255.255.255) or netmask(xxx.xxx.xxx.26/255.255.255.255); }; filter mail-proxy-internal { netmask(10.10.9.20/255.255.255.255) and not program("perdition"); }; filter mail-relay-internal { netmask(10.10.9.30/255.255.255.255); }; filter mail-relay-alpha-external-out { netmask(xxx.xxx.xxx.59/255.255.255.255) and facility(local1); }; filter mail-relay-beta-external-out { netmask(xxx.xxx.xxx.60/255.255.255.255) and facility(local1); }; filter mail-relay-alpha-external-in { netmask(xxx.xxx.xxx.59/255.255.255.255) and facility(mail); }; filter mail-relay-beta-external-in { netmask(xxx.xxx.xxx.60/255.255.255.255) and facility(mail); }; filter mail-proxy-node1-external { netmask(xxx.xxx.xxx.18/255.255.255.255) and not program("perdition"); }; filter mail-proxy-node2-external { netmask(xxx.xxx.xxx.22/255.255.255.255) and not program("perdition"); }; filter vpn { netmask(10.20.40.0/255.255.255.0); }; filter fw-intern-all { netmask(10.10.20.1/255.255.255.255); }; filter fw-intern-security { netmask(10.10.20.1/255.255.255.255) and match("security" value(".classifier.class") type("string")); }; filter fw-intern-info { netmask(10.10.20.1/255.255.255.255) and match("informational" value(".classifier.class") type("string")); }; filter fw-intern-rest { netmask(10.10.20.1/255.255.255.255) and not match("security" value(".classifier.class") type("string")) and not match("informational" value(".classifier.class") type("string")); }; filter fw-extern-all { netmask(10.80.11.20/255.255.255.255); }; filter fw-extern-security { netmask(10.80.11.20/255.255.255.255) and match("security" value(".classifier.class") type("string")); }; filter fw-extern-info { netmask(10.80.11.20/255.255.255.255) and match("informational" value(".classifier.class") type("string")); }; filter fw-extern-rest { netmask(10.80.11.20/255.255.255.255) and not match("security" value(".classifier.class") type("string")) and not match("informational" value(".classifier.class") type("string")); }; filter fw-extern-new { netmask(10.80.11.30/255.255.255.255); }; ################################################################################################ ######################### PARSER ############################## ################################################################################################ parser pattern_db_fwint { db_parser( file("/etc/syslog-ng/fw-int_patterndb.xml") ); }; parser pattern_db_fwext { db_parser( file("/etc/syslog-ng/fw-ext_patterndb.xml") ); }; ################################################################################################ ######################### DESTINATIONS ############################## ################################################################################################ destination http-log { file("/logging/server/web/$HOST" template("$MSGONLY\n") template-escape(no) owner("root") group("root") perm(0644)); }; destination mail-out { file("/logging/server/mail/mail-out_$MONTH.log"); }; destination mail-in { file("/logging/server/mail/mail-in_$MONTH.log"); }; destination vpn { file("/logging/network/vpn_$MONTH.log" flush_lines(10)); }; destination fw-intern-all { file("/logging/network/fw-intern_$MONTH.log" flush_lines(10)); }; destination fw-extern-all { file("/logging/network/fw-extern_$MONTH.log" flush_lines(10)); }; destination fw-extern-new { file("/logging/network/fw-new_$MONTH.log" flush_lines(10)); }; destination dump { file("/logging/network/dump.log" template ("$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC, $HOST, $FIREWALL_SEQ, $MSGHDR, 0, $FIREWALL_IO, $FIREWALL_PROTO, $FIREWALL_SCR_LAN, $FIREWALL_SRC_IP, $FIREWALL_SRC_PORT, $FIREWALL_DST_LAN, $FIREWALL_DST_IP, $FIREWALL_DST_PORT, $FIREWALL_NAT_SRC_IP, $FIREWALL_NAT_DST_IP, $FIREWALL_RULE, $FIREWALL_REASON, $FIREWALL_DURATION\n")); # file("/logging/network/dump.log" template ("$MSGHDR\n") flush_lines(5)); }; ################################################################################################ ######################### FINAL-LOGS ############################## ################################################################################################ ##### TO FILE log { source(s_remote); filter(http-official); destination(http-log); }; log { source(s_remote); filter(mail-proxy-internal); destination(mail-out); }; log { source(s_remote); filter(mail-relay-internal); destination(mail-out); }; log { source(s_remote); filter(mail-relay-alpha-external-out); destination(mail-out); }; log { source(s_remote); filter(mail-relay-beta-external-out); destination(mail-out); }; log { source(s_remote); filter(mail-proxy-node1-external); destination(mail-out); }; log { source(s_remote); filter(mail-proxy-node2-external); destination(mail-out); }; log { source(s_remote); filter(mail-relay-alpha-external-in); destination(mail-in); }; log { source(s_remote); filter(mail-relay-beta-external-in); destination(mail-in); }; log { source(s_remote); filter(vpn); destination(vpn); }; log { source(s_remote); filter(fw-intern-all); destination(fw-intern-all); }; log { source(s_remote); filter(fw-extern-new); destination(fw-extern-new); }; log { source(s_remote); filter(fw-extern-all); destination(fw-extern-all); flags(final); };

3 4

Wrong hostname on syslog-messages.
by Christopher Jon Caldwell 12 Mar '10

12 Mar '10

All of the syslog messages sent by our Solaris servers that contain output from the service processor are getting the wrong hostname assigned to them - the log messages get filed under the hostname of the receiving syslog-ng server. Our second syslog-ng server is correctly identifying the hostname with the same configuration. We are running 2.1.11a Enterprise Edition. Here is the relevant part of my syslog-ng.conf: options { sync (0); create_dirs (yes); keep_hostname (no); check_hostname (yes); chain_hostnames (no); bad_hostname ("\t"); normalize_hostnames (yes); mark_freq (10); }; source src_net { udp(port(514)); }; destination d_hosts { file("/var/log/systems/$HOST_FROM/$FACILITY/$YEAR/$MONTH/$FACILITY\-$YEAR$MONTH$DAY" perm(0644) dir_perm(0755) create_dirs(yes)); ... log { source(src_net); destination(d_hosts); destination(d_splunk); flags(fallback); }; And here is an example packet from snoop. ETHER: ----- Ether Header ----- ETHER: ETHER: Packet 36 arrived at 12:09:18.28026 ETHER: Packet size = 169 bytes ETHER: Destination = 0:3:ba:71:22:65, ETHER: Source = 0:21:28:4:ec:b7, ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----- IP: IP: Version = 4 IP: Header length = 20 bytes IP: Type of service = 0x00 IP: xxx. .... = 0 (precedence) IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: .... ..0. = not ECN capable transport IP: .... ...0 = no ECN congestion experienced IP: Total length = 155 bytes IP: Identification = 25576 IP: Flags = 0x4 IP: .1.. .... = do not fragment IP: ..0. .... = last fragment IP: Fragment offset = 0 bytes IP: Time to live = 255 seconds/hops IP: Protocol = 17 (UDP) IP: Header checksum = f267 IP: Source address = 10.244.236.183, vienna IP: Destination address = 10.241.34.101, auctor.backup.es.gwu.edu IP: No options IP: UDP: ----- UDP Header ----- UDP: UDP: Source port = 32832 UDP: Destination port = 514 (SYSLOG) UDP: Length = 135 UDP: Checksum = F5D8 UDP: SYSLOG: ----- SYSLOG: ----- SYSLOG: SYSLOG: Priority: <29> (daemon.notice) SYSLOG: "<29>Mar 12 12:09:18 SC Alert: [ID 362536 daemon.notice] Audi" SYSLOG: -- Christopher Caldwell Senior Engineer, Technology Operations and Engineering The George Washington University caldwell @ gwu . edu | +1 202.994.4674 (w) | +1 202.409.0878 (c) PGP key ID: 0x0A0EC46C "Quis custodiet ipsos custodes?" - Juvenal

1 0

Compiling with SQL Support
by Gene Matusovsky 10 Mar '10

10 Mar '10

How do I compile SYSLOG-NG with SQL support? I run ./configure --enable-sql option, but the configure script wasn't able to find my libdbi and libdbd drivers. It looks like the libdbi and libdbd authors do not supply .pc files. I see that my libdbi libraries were successfully installed in /usr/local/lib and libdbd libraries were successfully installed /usr/local/lib/dbd How do I tell the configure script to use these libraries? Should I create .pc files for libdbi and libdbd and put them in my PKG_CONFIG path? -Thanks for any suggestions, Gene Matusovsky The information transmitted in this email and any of its attachments is intended only for the person or entity to which it is addressed and may contain Cablevision proprietary information, which is privileged, confidential, or subject to copyright belonging to Cablevision. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this in error, please contact the sender immediately and delete and destroy the communication and all of the attachments you have received and all copies thereof.

2 2

Re: [syslog-ng] queue management
by Fekete Róbert 08 Mar '10

08 Mar '10

Hi, AFAIK, unfortunately there is currently no way to manipulate the queue. Regards, Robert On Monday, March 08, 2010 18:12 CET, "Eric Snow" <esnow(a)verio.net> wrote: > If disk based buffering allows me to yank the message at the front of the queue, so that it doesn't hold up the ones behind it, then yes that is what I want. The statistics and ability to look at the queue are good, but I need to be able to manipulate the queue. > > OR > > I need to be able to have some form of exception handling in my syslog-ng configuration. Something that will catch a failure in a log and allow me to handle that failure intelligently. > > -eric > > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler > Sent: Saturday, March 06, 2010 2:09 AM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] queue management > > On Fri, 2010-03-05 at 17:49 -0500, Eric Snow wrote: > > Hi, > > > > > > > > I am looking to manage my output buffers. I have one destination that > > sometimes has trouble and I want to be able to manage the messages > > that got buffered while I fix my flaky destination. Is there any tool > > I can use to manage those buffered messages? If not, isn’t there any > > exception handling that I can apply to a log such that errors or other > > failures for that log can be redirected to another destination? I > > want to avoid clogging up my output queue and especially to avoid > > losing any messages. Thanks! > > you mean you have disk based buffering? in the upcoming 3.1 version (due > to be released within 2 weeks), there's a tool called dqtool to dump the > contents of the queue file, although I'm not sure that's what you meant. > there are further disk buffer related improvements in the queue for 3.2, > and also a way to send the messages to a failed destination to an > alternate server. > > also, syslog-ng is counting the number of messages in a given queue, > using its statistics framework. it is the stored counter you can see for > a given destination. this gets reported in the 'log statistics' message, > or via the syslog-ng control socket (/opt/syslog-ng/var/syslog-ng.ctl). > > > -- > Bazsi > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you. > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html >

1 0

syslog-ng plugins
by Balazs Scheidler 06 Mar '10

06 Mar '10

Hi, This is a repost of my blog article which I've found important enough to post here as well. The original article is at: http://bazsi.blogs.balabit.com/2010/03/plugins-branch-updated.html It is about a git branch in the syslog-ng OSE 3.1 git repository, as such it is available at: git://git.balabit.hu/bazsi/syslog-ng-3.1.git Plugins branch updated ---------------------- Since the last post, I could hack a couple of hours on the plugins branch, which now compiles. The plugin framework is capable for supporting a quite important core functionality: all socket like sources/destinations are now found in an external plugin called "afsocket". The reason I've started with afsocket is to make syslog-ng a bit less dependant on OpenSSL. A couple of distributions didn't include syslog-ng 3.0 in their current releases, because it uses OpenSSL from /usr, while syslog-ng should remain in the root directory. By separating afsocket from the syslog-ng core, I can compile afsocket with and without TLS support, which can be put into separate packages. Thus syslog-ng can operate without OpenSSL. And the same plugin framework will enable me to create a wide variety of plugins. My ideas: * Plugins for all syslog-ng components (source, destination, filter, rewrite, parser) * Python scriptability (a simple correllation engine in Python?) * macro transformation functions, for example: $(stripslashes $macro), usable anywhere in templates and stripslashes a plugin that is invoked whenever such an expansion occurs * Hooks for transforming the log message as it enters syslog-ng (to fix parsing errors for example), Do you have other ideas? Please post them as comments or as emails to the mailing list. Again, this functionality is experimental, and I'm still going to rebase the current code and will probably be integrated to syslog-ng 3.2. I got to release 3.1 final first though. :) -- Bazsi

1 0