August 2009 - syslog-ng - lists.balabit.hu

PostgreSQL Invalid Encoding Errors
by David Blewett 20 Aug '09

20 Aug '09

Hi All: I'm continually getting into a loop of receiving messages like this in my PostgreSQL log files: 2009-06-19 09:17:36.465 EDT 5846 syslog@syslog: ERROR: invalid byte sequence for encoding "UTF8": 0xa8 2009-06-19 09:17:36.465 EDT 5846 syslog@syslog: HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". 2009-08-01 00:00:05.216 EDT 3986 syslog@syslog: ERROR: invalid byte sequence for encoding "UTF8": 0xc446 2009-08-01 00:00:05.216 EDT 3986 syslog@syslog: HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". I'm using a centralized logging system, and just turned on logging to syslog for our hardware firewall. I'm positive it was this change that is resulting in this behavior, but I really do want to continue pulling this data from the firewall. It appears that random messages from the firewall will come in as invalid UTF8, and somehow get stuck in a loop with syslog-ng continually trying to insert the offending line and PostgreSQL continually refusing. I have several log files that are > 500MB now because of this. Is there a way to force syslog-ng to drop these messages, and perhaps log the error? I am using PostgreSQL 8.3.6. syslog-ng --version syslog-ng 3.0.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#08c9bf9d98e4e021756adc12c42605840140ba8b Compile-Date: Jul 8 2009 12:16:03 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: on syslog-ng.conf: options { chain_hostnames(off); flush_lines(0); stats_level(2); stats_freq(43200); frac_digits(5); ts_format(iso); }; source src { unix-stream( "/dev/log" max-connections(40) ); internal(); udp(port(514)); tcp(port(5140) keep-alive(yes)); }; destination d_sql { sql( type(pgsql) host("10.233.93.18") username("syslog") password("*****") database("syslog") table("facility_$FACILITY") columns("host", "sourceip", "priority", "lvl", "tag", "rcvd", "sent", "program", "msg") values("$HOST", "$SOURCEIP", "$PRIORITY", "$LEVEL", "$TAG", "$R_ISODATE", "$S_ISODATE", "$PROGRAM", "$MSGONLY") indexes("host", "rcvd", "sent", "program", "msg") ); }; destination console_all { file("/dev/tty12"); }; destination messages { file("/var/log/messages"); }; log { source(src); #destination(messages); destination(console_all); destination(d_sql); };

2 3

db-parser: tag and value support
by ILLES, Marton 18 Aug '09

18 Aug '09

Hi, I had some time recently and did some db-parser improvements in my syslog-ng 3.1 branch. Some of the features are already integrated to 3.1 mainline by Bazsi, but I though it is worthwhile to write about them. I have posted it to my blog (http://marci.blogs.balabit.com/) but here it goes as well. In one of my earlier post I wrote about adding tag support to syslog-ng. It has been already merged to the 3.1 mainline by Bazsi. To make the feature even more useful the version 2 patterndb had support for specifying tags in each rule to tag the matching messages. So far this functionality was missing, but now it is possible to tag messages directly from the patterndb configuration so you can latter filter based on these tags. It is more versatile than using classification as you can specify multiple tags and tag matching is way more faster than matching the class of the message. (Though it is not possible to use the tags in macros as you can do with classes. eg.: you can use the ${.classifier.class} macro while you can not include the tags) Besides tags patterndb gained support for using static value assignment to messages. With patterndb version 3 (which is backward compatible with version 2) it is possible to specify named values in each rule. When a message matches a given rule all the specified values are assigned to the message (just like names parsers like @STRING:myvalue@) and these values can be latter used in filters or in macros. (See my previous post on parsers.) Also it is possible to use macros in values, so you can refer back to any parsed part of the message. It is probably most useful if you want to specially store the messages in some structured form where you need to fill all required values. With these addition you can use rules like this: <rule id="1" provider="balabit" class="system"> <patterns> <pattern>my pattern @STRING:my_parsed_value@</pattern> </patterns> <tags> <tag>tag1</tag> <tag>tag2</tag> </tags> <values> <value name="myvalue1">foo</value> <value name="myvalue2">bar ${myvalue1}</value> </values> </rule> Version 3 of the patterndb has been also extended so it can hold example messages for a given pattern. It is really handy if you need to test your pattern automatically or simply you want to document the original message that you created the pattern for. With the examples you can also specify expected values that are parsed or set from the message. The previous example can be extended this way: <rule id="1" provider="balabit" class="system"> <patterns> <pattern>my pattern @STRING:my_parsed_value@</pattern> </patterns> <tags> <tag>tag1</tag> <tag>tag2</tag> </tags> <values> <value name="myvalue1">foo</value> <value name="myvalue2">bar ${myvalue1}</value> </values> <examples> <test_message program="programname">my pattern test</test_message> <test_values> <test_value name="myvalue1">foo</test_value> <test_value name="myvalue2">bar foo</test_value> <test_value name="my_parser_value">test</test_value> </test_values> <test_tags> <test_tag>tag1</test_tag> <test_tag>tag2</test_tag> </test_tags> <example> </examples> </rule> You can find the source code in my 3.1 git tree at http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=summary Happy tagging! M -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

1 0

Unable to run syslog-ng 3.0.4 as non-root on solaris 10
by Jeffrey Psolla 18 Aug '09

18 Aug '09

Hi, Yesterday I upgraded syslog-ng on our central log server from 2.0.5 to 3.0.4 . The OS is solaris 10. Prior to the upgrade I was able to run syslog-ng as a non-root user with the following command: /usr/local/sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -u syslogng -g syslogng -p /etc/syslog-ng/syslog-ng.pid After the upgrade I get the following error using the same command: Error binding socket; addr='AF_INET(0.0.0.0:514)', error='Permission denied (13)' Error initializing source driver; source='gateway', id='gateway#0' Error initializing message pipeline; If I remove the -u and -g arguments the daemon runs without issues. Same physical server, OS, and user account prior to the upgrade. I tried the verbose and debug arguments, when starting syslog-ng, but only get the above error. The syslog.conf file is posted below. I compiled from source using the following options: ./configure --enable-debug --disable-ipv6 --enable-dynamic-linking --enable-pcre --enable-ssl --disable-glibtest --disable-tcp-wrapper Thanks, Jeff ******************************* /usr/local/sbin/syslog-ng --version ******************************* Compile-Date: Aug 13 2009 16:25:43 Enable-Threads: off Enable-Debug: on Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on ***************************** syslog-ng.conf ***************************** @version:3.0 # # options { use_dns(persist_only); dns_cache_hosts(/etc/hosts); keep_hostname(yes); check_hostname(yes); #normalize_hostnames(yes); stats_freq(3600); log_fifo_size(2000); flush_lines(5); bad_hostname(Corrupt); bad_hostname(disk); bad_hostname(drive); }; # local for messages coming in locally source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # Gateway for UDP Syslog messages. This is the default source gateway { udp(ip("0.0.0.0") port(514)); }; template default_template { template("$DATE $HOST $PROGRAM[$PID]: $MESSAGE\n"); template_escape(no); }; # Where to write the incoming log files to. destination hosts { file("/opt/log/clients/new_clients/$HOST/$YEAR-$MONTH-$DAY.$FACILITY" owner(syslogng) group(syslogng) perm(0644) dir_perm(0755) dir_group(syslogng) dir_owner(syslogng) create_dirs(yes) template (default_template)); }; destination backuphost1 { tcp("xxx.xxx.xxx.xxx" port(1468)); }; destination backuphost2 { udp("xxx.xxx.xxx.xxx " port(514)); }; filter Windows_filter { program(MSWinEventLog) and match("Name: Installer" value("$MESSAGE") flags("ignore-case")) or match("Name: MGLdap" value("$MESSAGE") flags("ignore-case")) or match("Special privileges assigned to new logon" value("$MESSAGE")) or match("Authentication Ticket Request" value("$MESSAGE")) or match("Process: Kerberos" value("$MESSAGE")); }; filter RSS_filter { match("Bad protocol version" value("$MESSAGE")) or match("Did not receive identification string" value("$MESSAGE")) or facility(user); }; log { source(gateway); filter(Windows_filter); flags(final); }; log { source(gateway); filter(RSS_filter); flags(final); }; log { source(gateway); destination(hosts); destination(backuphost1); }; log { source(local); destination(hosts); destination(backuphost1); destination(backuphost2); }; ________________________________ ----------------------------------------------------------------------- This e-mail is intended for the addressee shown. It contains information that is confidential and protected from disclosure. Any review, dissemination or use of this transmission or its contents by persons or unauthorized employees of the intended organizations is strictly prohibited. The contents of this email do not necessarily represent the views or policies of PSCU Financial Services.

4 3

program destination problem (again...)
by Alberto Sierra 18 Aug '09

18 Aug '09

hi there, i know this is been discussed like a million times already but i'm stuck and can't get around this. i'm using a program destination in my syslog-ng config, like this: destination test_log { file("/var/log/testlog"); }; destination sshd_alerts { program("/usr/local/bin/ssh_alert_by_email.sh" template("$DATE $HOST $PROGRAM $MSGONLY")); }; filter sshd { program("sshd"); }; filter login_accepted { match("Accepted password|Accepted publickey"); }; log { source(s_all); filter(sshd); filter(login_accepted); destination(sshd_alerts); destination(test_log); }; and the script as follows: #!/bin/bash while read line ; do echo $line >> /tmp/testlog done that's it, it logs to the destination(test_log) but the script does nothing. i followed a similar thread: https://lists.balabit.hu/pipermail/syslog-ng/2008-March/011512.html and the script works well interactively in the shell. I think i hit a dead end here... btw version 2.0.9 -- Alberto Sierra

5 6

Can't make Makefike of syslog-ng-3.0.4. OSE
by seki-norihiko＠mdis.co.jp 18 Aug '09

18 Aug '09

Hello everyone. I am user of syslog-ng. I wanna use syslog-ng instead of syslog on Red Hat Linux 5.2. CPU is Intel Xeon 64-bit architecture. I downloaded "eventlog_0.2.10.tar.gz" and "syslog-ng-3.0.4.tar.gz" Open Source Edition, from www.balabit.com. Thank you. I can make install eventlog_0.2.10. But I can't make Makefike of syslog-ng-3.0.4. I show error at below [root@hostname syslog-ng-3.0.4]# make make all-recursive ・・・ logmatcher.c: In function ‘log_matcher_pcre_re_compile’: logmatcher.c:534: error: ‘PCRE_NEWLINE_ANYCRLF’ undeclared (first use in this function) logmatcher.c:534: error: (Each undeclared identifier is reported only once logmatcher.c:534: error: for each function it appears in.) Would you please teach me what I should do to solve this error? I can't info about this error on the web. I need help. Thank you. Nori.

3 3

port 514 not used
by stephen.greenfield＠wachovia.com 15 Aug '09

15 Aug '09

I configure syslog-ng to use udp port 514, on the syslog server receiving log messages. When I issue a `netstat -lp | grep syslog` it shows different ports but never 514. Any ideas why? /steve

3 4

SOLVED: program destination problem (again...)
by Alberto Sierra 13 Aug '09

13 Aug '09

Ok, i finally found the problem, i don't know if it is only related to the "template" option within the program directive, but it must end with a newline character "\n" for the program to process every line. like this: program("/usr/local/bin/ssh_alert_by_email.sh" template("$DATE $HOST $PROGRAM $MSGONLY\n")); }; *note the \n after $MSGONLY for the curious, this is the script that i implemented to get a notification for every ssh successful login: #!/bin/bash recipient="alerts(a)example.com" processLog() { subject=$(echo $1 |awk '{print "Successful ssh login by " $9, "on " $4}') body=$(echo $1 |awk '{printf ("%s %s %s", "Date: " $1, $2, $3 "\\n" "Hostname: " $4 "\\n" "Username: " $9 "\\n" "Source host: " $11); }') /bin/echo -e $body | mail -e -s "$subject" $recipient } while read msg ; do processLog "$msg" done On Thu, Aug 13, 2009 at 9:44 AM, Alberto Sierra<albertosierra(a)aesetres.com> wrote: > 0 S root 20465 1 0 80 0 - 443 - 11:26 pts/0 > 00:00:00 /bin/sh -c /usr/local/bin/ssh_alert_by_email.sh > 0 S root 20466 20465 0 80 0 - 443 - 11:26 pts/0 > 00:00:00 /bin/bash /usr/local/bin/ssh_alert_by_email.sh > 5 S root 20468 1 0 80 0 - 572 - 11:26 ? > 00:00:00 /sbin/syslog-ng -p /var/run/syslog-ng.pid > > the PID is not changing, (unless the syslog-ng is restarted of > course), the debug.log shows the program runs until syslog-ng is > restarted as well. but it still sends nothing to the /tmp/testlog > file. > > > > On Thu, Aug 13, 2009 at 3:47 AM, Fegan, Joe<Joe.Fegan(a)hp.com> wrote: >> In "ps -elf" do you see your script? Does the pid stay the same as time advances, or does it change (which would mean it's exiting and being replaced with a new instance by syslog-ng automatically). You could add a start and end marker to see if it's starting at all and if/when it's exiting. Like: >> >> #!/bin/bash >> echo "$0 started `date`" >> /tmp/debug.log >> while read line ; do >> echo $line >> /tmp/testlog >> done >> echo "$0 exited `date`" >> /tmp/debug.log >> >> >> >> -----Original Message----- >> From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alberto Sierra >> Sent: 13 August 2009 07:26 >> To: syslog-ng(a)lists.balabit.hu >> Subject: [syslog-ng] program destination problem (again...) >> >> hi there, >> >> i know this is been discussed like a million times already but i'm >> stuck and can't get around this. >> >> i'm using a program destination in my syslog-ng config, like this: >> destination test_log { file("/var/log/testlog"); }; >> destination sshd_alerts { >> program("/usr/local/bin/ssh_alert_by_email.sh" template("$DATE $HOST >> $PROGRAM $MSGONLY")); }; >> >> filter sshd { program("sshd"); }; >> filter login_accepted { match("Accepted password|Accepted publickey"); }; >> >> >> log { >> source(s_all); >> filter(sshd); >> filter(login_accepted); >> destination(sshd_alerts); >> destination(test_log); >> }; >> >> and the script as follows: >> >> #!/bin/bash >> while read line ; do >> echo $line >> /tmp/testlog >> done >> >> that's it, it logs to the destination(test_log) but the script does nothing. >> >> i followed a similar thread: >> https://lists.balabit.hu/pipermail/syslog-ng/2008-March/011512.html >> >> and the script works well interactively in the shell. I think i hit a >> dead end here... btw version 2.0.9 >> >> -- >> Alberto Sierra >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.campin.net/syslog-ng/faq.html >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.campin.net/syslog-ng/faq.html >> >> > > > > -- > Alberto Sierra Reales [aesetres] > IT Consultant > Cel. 8319-1805 > -- Alberto Sierra Reales [aesetres] IT Consultant Cel. 8319-1805

1 0

syslog-ng and oracle
by Andreas Sartori 12 Aug '09

12 Aug '09

hello, we are currently setting up a new syslog server and switching from a sun fire v880 with syslog-ng and plain files to a new linux based (centos 5.3) syslog server with oracle as logging backend. we do not want to use pipes to write to the database, so we compiled libdbi-dbd-oracle with the oracle instantclient 10.2.0.4-1. syslog-ng is starting up fine (not complaining about the oracle backend), but we do have some problem with the inserts to the database. our trial syslog-ng configuration: destination d_sql_fwint { sql(type(oracle) username("xxx") password("xxx") database("dashb") table("SYSLOG_FWINT") columns("MSG_RCV_TIME varchar2(16)", "HOSTNAME varchar2(256)", "MESSAGE varchar2(2048)") values("$R_DATE", "$HOST", "$MSGONLY") }; when running in debug mode we get the following output: Running SQL query; query='INSERT INTO SYSLOG_FWINT (MSG_RCV_TIME, HOSTNAME, MESSAGE) VALUES (\'Aug 11 21:24:14\', \'gw20\', \'Built outbound TCP connection 144547326429398234 for LAN230:10.20.30.17/43825 (10.20.30.17/43825) to outside:10.10.10.102/80 (10.10.10.102/80)\')' this cant work sind oracle is not accepting \ as escape char, and in this case we do not need an escape at all. the sql syntax is right without the \. since templates are not working when using the sql driver, does anyone has an idea how to fix that? are we missing something? oracle setup is right, tnsnames or works and we can connect to the database with sqlplus. the syslog-ng init script as the environment variables set (ORACLE_HOME, LD_LIBRARY_PATH) thanks for any help -andy

2 5

syslog-ng exits when system time set backwards
by Kirill Stepanchuk 12 Aug '09

12 Aug '09

Hello, I've googled the mailing list, but could find any related threads. On FreeBSD, syslog-ng 1.6.8 exits when the system time is set to a value earlier than the previous value. Has anyone else encountered this before? Is there a fix or a configuration issue? $ ps auxwww | grep syslog-ng root 1040 0.0 0.1 1516 1208 ?? Ss 10:15AM 0:00.08 /usr/local/sbin/syslog-ng kstepanchuk 6529 0.0 0.0 1568 980 p0 R+ 10:38AM 0:00.00 grep syslog-ng $ sudo date 200501010000 Sat Jan 1 00:00:00 EST 2005 $ ps auxwww | grep syslog-ng mfeeney 6533 0.0 0.0 1612 996 p0 S+ 12:00AM 0:00.00 grep syslog-ng Thanks,

2 1

3.0.3: Accessing unix-dgram source appears to break all logging
by Markus Stalder 06 Aug '09

06 Aug '09

Hello list, my first post to the list, so hello everybody! :-) I'm on Ubuntu 8.04.2 amd64 and trying to upgrade from syslog-ng 2.0.9-1ubuntu1 to syslog-ng 3.0.3. I've both tried the on balabit.com provided amd64 .deb package and also created an own amd64 .deb with the default settings in /debian using dpkg buildpackage. I have the following two sources in my config: source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; source src { internal(); unix-dgram("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; Logging works just fine with the s_all source. However, as soon as I access the src source, by, for example destination temp_logfile { fifo("/tmp/.tivoli/.logfile."); }; log { source(src); destination(temp_logfile); }; all logging will cease. Meaning, other than Aug 5 15:04:23 server syslog-ng[17505]: syslog-ng starting up; version='3.0.3' I'm not getting *anything* in my logs after restarting syslog-ng. If I then comment destination temp_logfile ... again and restart syslog-ng, logging works again. The exact same config has worked just fine with 2.0.9-1ubuntu1. Any idea what could be wrong here? Thanks a lot! Best regards Markus

4 4