- syslog-ng - lists.balabit.hu

facility 'security'
by Jan Schaumann 04 Sep '09

04 Sep '09

Hello, It appears that syslog-ng does not correctly identify the 'security' facility: $ logger -p security.info oink yields: Aug 25 10:46:43 <d.info> syslog1 oink Note the false facility "d". In src/syslog-names.c, the mapping for 'security' is done thusly: {"security", LOG_AUTH}, /* DEPRECATED */ FreeBSD, however, appears to still use LOG_SECURITY, which leads to syslog-ng falsely categorizing the incoming messages. I'd be able to deal with this if it actually did fall back to LOG_AUTH, but for some reason it shows up as facility "d" (which seems like a string comparison gone awry). -Jan

3 4

multiline messages via unix socket
by Max Arnold 04 Sep '09

04 Sep '09

Hello all! Can someone look at this bugreport: http://bugs.python.org/issue6444 In short, when multiline log messages sent via /dev/log socket they are splitted into multiple log entries. Only first entry will have proper timestamp and facility. This behavior differs from sysklogd and metalog, where multiple lines are concatenated. Even syslog-ng itself concatenates them when used via udp socket. Tested on syslog-ng-2.1.3. Thanks.

2 3

Milliseconds in timestamp and black magic
by Sergei Zhirikov 02 Sep '09

02 Sep '09

Hi, I'm observing something mysterious with milliseconds in timestamps. I set frac_digits() in the global options and then I have a rewriter and a template. Something like this: options { frac_digits(3); }; source s { internal(); } rewrite r { set("$ISODATE", value("MYDATE")); }; template t { template("$ISODATE ($MYDATE) $HOST $MSGHDR$MESSAGE\n"); }; destination d { file("/var/log/test.log" template(t)); }; log { source(s); rewrite(r); destination(d); }; To my total surprise the expanded value of $ISODATE in the log file contains fraction part, but the value of $MYDATE does not! Aren't they supposed to be the same because of the rewrite rule? Or am I doing something wrong? (I'm using syslog-ng 3.0.4 on Linux). -- Thanks, Sergei.

4 5

Using remote file as source
by G. Rayot 27 Aug '09

27 Aug '09

Hi. Using syslog-ng OSE, is it possible to use a remote file (ie: samba share) as source file? Doing it with a local file works like a charm, but using a mounted remote file does not work (with follow_freq set to 1 or not set). The file will be read by syslog-ng only if I make a ls on the folder where the file is mounted, or cat the file. Thnaks. G.

1 0

[Fwd: Re: [Fwd: Re: iptables logging in a separate logfile works not properly]]
by Ralf Heidenreich 27 Aug '09

27 Aug '09

Hello, can anybody help me? regards Ralf Ralf Heidenreich schrieb: > Hello, > > if I use the final flag, postfix loggs to /var/log/messages, not to > /var/log/maillog. > What I am doing wrong? > > regards ralf > Here my config: > options { > sync (0); > time_reopen (10); > log_fifo_size (1000); > long_hostnames (off); > use_dns (no); > use_fqdn (no); > create_dirs (no); > keep_hostname (yes); > }; > > source s_sys { > file ("/proc/kmsg" log_prefix("kernel: ")); > unix-stream ("/dev/log"); > internal(); > # udp(ip(0.0.0.0) port(514)); > }; > destination iptables { file("/var/log/firewall.log"); }; > filter iptables { match("INPUT:"); }; > log { source(s_sys); filter(iptables); > destination(iptables);flags(final); }; > #log { source(s_sys); filter(iptables); destination(iptables); }; > filter noiptables { not match("INPUT:"); }; > log { source(s_sys); filter(noiptables); destination(d_kern); }; > > destination d_cons { file("/dev/console"); }; > destination d_mesg { file("/var/log/messages"); }; > destination d_auth { file("/var/log/secure"); }; > destination d_mail { file("/var/log/maillog" sync(10)); }; > destination d_spol { file("/var/log/spooler"); }; > destination d_boot { file("/var/log/boot.log"); }; > destination d_cron { file("/var/log/cron"); }; > destination d_kern { file("/var/log/messages"); }; > destination d_mlal { usertty("*"); }; > # direct iptables logs into /var/log/firewall.log > > > > filter f_kernel { facility(kern); }; > filter f_default { level(info..emerg) and > not (facility(mail) > or facility(authpriv) > or facility(cron)); }; > filter f_auth { facility(authpriv); }; > filter f_mail { facility(mail); }; > filter f_emergency { level(emerg); }; > filter f_news { facility(uucp) or > (facility(news) > and level(crit..emerg)); }; > filter f_boot { facility(local7); }; > filter f_cron { facility(cron); }; > > #log { source(s_sys); filter(f_kernel); destination(d_cons); }; > log { source(s_sys); filter(f_kernel); destination(d_kern); }; > log { source(s_sys); filter(f_default); destination(d_mesg); }; > log { source(s_sys); filter(f_auth); destination(d_auth); }; > log { source(s_sys); filter(f_mail); destination(d_mail); }; > log { source(s_sys); filter(f_emergency); destination(d_mlal); }; > log { source(s_sys); filter(f_news); destination(d_spol); }; > log { source(s_sys); filter(f_boot); destination(d_boot); }; > log { source(s_sys); filter(f_cron); destination(d_cron); }; > > Ralf Heidenreich schrieb: >> Hello, >> >> can you give me an example, how to use the final flag? >> thanks in advance. >> >> regards >> Ralf >> >> Szalay Attila schrieb: >>> Hi, >>> >>> On Fri, 2009-05-29 at 14:15 +0200, Ralf Heidenreich wrote: >>>> Hello, >>>> >>>> I have the following line in iptables: >>>> iptables -A INPUT -j LOG --log-prefix "INPUT: " >>>> In syslog-ng (/etc/syslog-ng/syslog-ng.conf) I have the following config: >>> [...] >>>> filter iptables { match("INPUT:"); }; >>>> log { source(s_sys); filter(iptables); destination(iptables); }; >>> [...] >>>> filter f_default { level(info..emerg) and >>>> not (facility(mail) >>>> or facility(authpriv) >>>> or facility(cron)); }; >>> [...] >>>> log { source(s_sys); filter(f_default); destination(d_mesg); }; >>> syslog-ng do not stop the evaluation when a destination is found. And >>> because of this and because the second log statement is matching to the >>> line the syslog-ng store it to the message too. >>> >>> If you want to order the syslog-ng to stop you can use the final >>> directive. >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> ______________________________________________________________________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >>> FAQ: http://www.campin.net/syslog-ng/faq.html >>> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.campin.net/syslog-ng/faq.html >> > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

1 0

Re: [syslog-ng] syslog-ng 3.0 seem to chop some of the long lines off
by Joe Hansen 26 Aug '09

26 Aug '09

The application is custom in-house. I'm really not sure what RFC they've followed. However your suggestion about using 'no-multi-line' seems to have take care of the split lines. Thanks, Jo --- On Fri, 6/19/09, Balazs Scheidler <bazsi(a)balabit.hu> wrote: From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] syslog-ng 3.0 seem to chop some of the long lines off To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Received: Friday, June 19, 2009, 4:38 AM On Wed, 2009-06-17 at 12:10 -0700, Joe Hansen wrote: > We're seeing lines in /var/log/messages that are chopped off. They > should look like this: > > Jun 17 00:34:36 bxe1001 WORKER[18808]: DEBUG,email header > info,3893336992,1941292826,CPSendWork::ProcessConverterSuccessMessage,"man_id=551797512,srp_id=GBISXAPC00S07,from=From: ""info(a)callcentres.net"" <info(a)callcentres.net> ,to=To: ""Graeme Ross"" <gross(a)mobileemail.vodafone.com.au>,subject=Turn-key call centre for lease. Launceston, Tasmania","worker/CPSendWork.cpp:858" > > But they look like this: > > Jun 17 00:34:36 bxe1001 WORKER[18808]: DEBUG,email header > info,3893336992,1941292826,CPSendWork::ProcessConverterSuccessMessage,"man_id=551797512,srp_id=GBISXAPC00S07,from=From: ""info(a)callcentres.net"" <info(a)callcentres.net> > > > Anyone seen this? > > Thanks > > > Well, can you tell us a little bit more on your configuration? What protocol do you use on the source side? Do you happen to use the new RFC5424 style protocol, or the legacy one? What is generating those messages, can you show us a tcpdump/strace snippet that shows how that frame is travelling the network? Also, syslog-ng 3.0 does not remove embedded NL characters by default, maybe the rest of the message continues on the next line? You can reenable the previous behaviour by using the 'no-multi-line' flag for your source or destination (e.g. you can change the multi-line handling not just for a given source, but also handle the same message differently in different destinations). -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html __________________________________________________________________ Get a sneak peak at messages with a handy reading pane with All new Yahoo! Mail: http://ca.promos.yahoo.com/newmail/overview2/

2 3

new db-parser() utility: pdbtool
by ILLES, Marton 23 Aug '09

23 Aug '09

Hi, As I wrote in my previous post I kept myself busy with patterndb/db-parser enhancements. Recently I added a small utility called pdbtool to help working with pattern databases. Creating, maintaining and especially troubleshooting patterns are probably not the most easy task if you try it with the first time. pdbtool is a collection of small utilities bounded into one binary similar to samba's net utility. You can find pdbtool in my 3.1 syslog-ng git repo (http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=summary) hopefully Bazsi is gonna merge it soon mainline. MATCH The first command included in pdbtool is called "match" which can be used to test match rules in a pattern database. Debuging or troubleshooting patterns was somewhat complicated as you had to feed the messages into syslog-ng to see if a given pattern matches a message. The match command does the same. It tries to match a given message against patterndb rules and also evaluates parser while extracting parsed part of the message. To test rules simple rune pdbtool with the message and optionally with the program name to test: $ pdbtool match -p patterndb.xml -P sshd -M "Accepted publickey for marci from 127.0.0.1 port 59357 ssh2" match returns with 0 or 1 return code and prints the assigned values as well: .classifier.rule_id=fc44a9fe-75fd-11dd-9bba-001e6806451b .classifier.class=system I also plan to add some more verbose debug output if you specify debug option, so you can see how the given message is matched against the tree. Till than you can still test and debug your ruleset. DUMP The dump command can be used to dump the radix tree built from the patterns. This way you can see how your patterns are represented in syslog-ng and it might also help you tracking down problems with patterns. The dump utility can dump the tree used for matching the PROGRAM or the MSG parts. To see the patterns for the 'sshd' PROGRAM one should run the following command: $ pdbtool dump -p patterndb.xml -P 'sshd' The output looks like this (just part of the whole output): 'p' 'assword for' @QSTRING:@ 'from' @QSTRING:@ 'port ' @NUMBER:@ rule_id='fc49054e-75fd-11dd-9bba-001e6806451b' ' ssh' rule_id='fc55cf86-75fd-11dd-9bba-001e6806451b' '2' rule_id='fc4b7982-75fd-11dd-9bba-001e6806451b' 'ublickey for' @QSTRING:@ 'from' @QSTRING:@ 'port ' @NUMBER:@ rule_id='fc4d377c-75fd-11dd-9bba-001e6806451b' ' ssh' rule_id='fc5441ac-75fd-11dd-9bba-001e6806451b' '2' rule_id='fc44a9fe-75fd-11dd-9bba-001e6806451b' MERGE The third command "merge" can be used to convert and merge pattern databases into one big file which can be loaded into syslog-ng. It makes sense to store the patterns in separate files per applications or on other grouping bases, however they need to be merged as syslog-ng can utilize only one file per db-parser() instance. My idea is to store patterns in different files along with the syslog-ng configuration and merge them automatically (or manually) into one big file which is referenced from syslog-ng configuration. By default syslog-ng reads patterndb from the /var/lib/syslog-ng/patterndb.xml (depending on the compile time parameters) file which can be the output of a merge of the separate patterndb files. Having separate files help maintaining and troubleshooting patterns and also helps distributing or moving patterns from one node to an other. Besides merging the patterndb files the "merge" command takes care of converting/upgrading of older patterndb to the latest version (as of this writing version 3). This way you can easily use your old patterns with newer syslog-ng versions without any problem. Of course it might make sense to convert them with the tool as some features (like values, tags etc.) are only provided by newer versions. (The tool can be used to merge only 1 file, which case it upgrades it only if necessary.) I still need to create a man page for the utility as probably that is the first source of information most people is looking at. I hope you find the tool useful and as always any feedback, comment is very welcomed. Marton -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

1 0

PostgreSQL Invalid Encoding Errors
by David Blewett 20 Aug '09

20 Aug '09

Hi All: I'm continually getting into a loop of receiving messages like this in my PostgreSQL log files: 2009-06-19 09:17:36.465 EDT 5846 syslog@syslog: ERROR: invalid byte sequence for encoding "UTF8": 0xa8 2009-06-19 09:17:36.465 EDT 5846 syslog@syslog: HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". 2009-08-01 00:00:05.216 EDT 3986 syslog@syslog: ERROR: invalid byte sequence for encoding "UTF8": 0xc446 2009-08-01 00:00:05.216 EDT 3986 syslog@syslog: HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". I'm using a centralized logging system, and just turned on logging to syslog for our hardware firewall. I'm positive it was this change that is resulting in this behavior, but I really do want to continue pulling this data from the firewall. It appears that random messages from the firewall will come in as invalid UTF8, and somehow get stuck in a loop with syslog-ng continually trying to insert the offending line and PostgreSQL continually refusing. I have several log files that are > 500MB now because of this. Is there a way to force syslog-ng to drop these messages, and perhaps log the error? I am using PostgreSQL 8.3.6. syslog-ng --version syslog-ng 3.0.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#08c9bf9d98e4e021756adc12c42605840140ba8b Compile-Date: Jul 8 2009 12:16:03 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: on syslog-ng.conf: options { chain_hostnames(off); flush_lines(0); stats_level(2); stats_freq(43200); frac_digits(5); ts_format(iso); }; source src { unix-stream( "/dev/log" max-connections(40) ); internal(); udp(port(514)); tcp(port(5140) keep-alive(yes)); }; destination d_sql { sql( type(pgsql) host("10.233.93.18") username("syslog") password("*****") database("syslog") table("facility_$FACILITY") columns("host", "sourceip", "priority", "lvl", "tag", "rcvd", "sent", "program", "msg") values("$HOST", "$SOURCEIP", "$PRIORITY", "$LEVEL", "$TAG", "$R_ISODATE", "$S_ISODATE", "$PROGRAM", "$MSGONLY") indexes("host", "rcvd", "sent", "program", "msg") ); }; destination console_all { file("/dev/tty12"); }; destination messages { file("/var/log/messages"); }; log { source(src); #destination(messages); destination(console_all); destination(d_sql); };

2 3

db-parser: tag and value support
by ILLES, Marton 18 Aug '09

18 Aug '09

Hi, I had some time recently and did some db-parser improvements in my syslog-ng 3.1 branch. Some of the features are already integrated to 3.1 mainline by Bazsi, but I though it is worthwhile to write about them. I have posted it to my blog (http://marci.blogs.balabit.com/) but here it goes as well. In one of my earlier post I wrote about adding tag support to syslog-ng. It has been already merged to the 3.1 mainline by Bazsi. To make the feature even more useful the version 2 patterndb had support for specifying tags in each rule to tag the matching messages. So far this functionality was missing, but now it is possible to tag messages directly from the patterndb configuration so you can latter filter based on these tags. It is more versatile than using classification as you can specify multiple tags and tag matching is way more faster than matching the class of the message. (Though it is not possible to use the tags in macros as you can do with classes. eg.: you can use the ${.classifier.class} macro while you can not include the tags) Besides tags patterndb gained support for using static value assignment to messages. With patterndb version 3 (which is backward compatible with version 2) it is possible to specify named values in each rule. When a message matches a given rule all the specified values are assigned to the message (just like names parsers like @STRING:myvalue@) and these values can be latter used in filters or in macros. (See my previous post on parsers.) Also it is possible to use macros in values, so you can refer back to any parsed part of the message. It is probably most useful if you want to specially store the messages in some structured form where you need to fill all required values. With these addition you can use rules like this: <rule id="1" provider="balabit" class="system"> <patterns> <pattern>my pattern @STRING:my_parsed_value@</pattern> </patterns> <tags> <tag>tag1</tag> <tag>tag2</tag> </tags> <values> <value name="myvalue1">foo</value> <value name="myvalue2">bar ${myvalue1}</value> </values> </rule> Version 3 of the patterndb has been also extended so it can hold example messages for a given pattern. It is really handy if you need to test your pattern automatically or simply you want to document the original message that you created the pattern for. With the examples you can also specify expected values that are parsed or set from the message. The previous example can be extended this way: <rule id="1" provider="balabit" class="system"> <patterns> <pattern>my pattern @STRING:my_parsed_value@</pattern> </patterns> <tags> <tag>tag1</tag> <tag>tag2</tag> </tags> <values> <value name="myvalue1">foo</value> <value name="myvalue2">bar ${myvalue1}</value> </values> <examples> <test_message program="programname">my pattern test</test_message> <test_values> <test_value name="myvalue1">foo</test_value> <test_value name="myvalue2">bar foo</test_value> <test_value name="my_parser_value">test</test_value> </test_values> <test_tags> <test_tag>tag1</test_tag> <test_tag>tag2</test_tag> </test_tags> <example> </examples> </rule> You can find the source code in my 3.1 git tree at http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=summary Happy tagging! M -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

1 0

Unable to run syslog-ng 3.0.4 as non-root on solaris 10
by Jeffrey Psolla 18 Aug '09

18 Aug '09

Hi, Yesterday I upgraded syslog-ng on our central log server from 2.0.5 to 3.0.4 . The OS is solaris 10. Prior to the upgrade I was able to run syslog-ng as a non-root user with the following command: /usr/local/sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -u syslogng -g syslogng -p /etc/syslog-ng/syslog-ng.pid After the upgrade I get the following error using the same command: Error binding socket; addr='AF_INET(0.0.0.0:514)', error='Permission denied (13)' Error initializing source driver; source='gateway', id='gateway#0' Error initializing message pipeline; If I remove the -u and -g arguments the daemon runs without issues. Same physical server, OS, and user account prior to the upgrade. I tried the verbose and debug arguments, when starting syslog-ng, but only get the above error. The syslog.conf file is posted below. I compiled from source using the following options: ./configure --enable-debug --disable-ipv6 --enable-dynamic-linking --enable-pcre --enable-ssl --disable-glibtest --disable-tcp-wrapper Thanks, Jeff ******************************* /usr/local/sbin/syslog-ng --version ******************************* Compile-Date: Aug 13 2009 16:25:43 Enable-Threads: off Enable-Debug: on Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on ***************************** syslog-ng.conf ***************************** @version:3.0 # # options { use_dns(persist_only); dns_cache_hosts(/etc/hosts); keep_hostname(yes); check_hostname(yes); #normalize_hostnames(yes); stats_freq(3600); log_fifo_size(2000); flush_lines(5); bad_hostname(Corrupt); bad_hostname(disk); bad_hostname(drive); }; # local for messages coming in locally source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # Gateway for UDP Syslog messages. This is the default source gateway { udp(ip("0.0.0.0") port(514)); }; template default_template { template("$DATE $HOST $PROGRAM[$PID]: $MESSAGE\n"); template_escape(no); }; # Where to write the incoming log files to. destination hosts { file("/opt/log/clients/new_clients/$HOST/$YEAR-$MONTH-$DAY.$FACILITY" owner(syslogng) group(syslogng) perm(0644) dir_perm(0755) dir_group(syslogng) dir_owner(syslogng) create_dirs(yes) template (default_template)); }; destination backuphost1 { tcp("xxx.xxx.xxx.xxx" port(1468)); }; destination backuphost2 { udp("xxx.xxx.xxx.xxx " port(514)); }; filter Windows_filter { program(MSWinEventLog) and match("Name: Installer" value("$MESSAGE") flags("ignore-case")) or match("Name: MGLdap" value("$MESSAGE") flags("ignore-case")) or match("Special privileges assigned to new logon" value("$MESSAGE")) or match("Authentication Ticket Request" value("$MESSAGE")) or match("Process: Kerberos" value("$MESSAGE")); }; filter RSS_filter { match("Bad protocol version" value("$MESSAGE")) or match("Did not receive identification string" value("$MESSAGE")) or facility(user); }; log { source(gateway); filter(Windows_filter); flags(final); }; log { source(gateway); filter(RSS_filter); flags(final); }; log { source(gateway); destination(hosts); destination(backuphost1); }; log { source(local); destination(hosts); destination(backuphost1); destination(backuphost2); }; ________________________________ ----------------------------------------------------------------------- This e-mail is intended for the addressee shown. It contains information that is confidential and protected from disclosure. Any review, dissemination or use of this transmission or its contents by persons or unauthorized employees of the intended organizations is strictly prohibited. The contents of this email do not necessarily represent the views or policies of PSCU Financial Services.

4 3