- syslog-ng - lists.balabit.hu

Re: [syslog-ng] syslog-ng 3.0 seem to chop some of the long lines off
by Joe Hansen 26 Aug '09

26 Aug '09

The application is custom in-house. I'm really not sure what RFC they've followed. However your suggestion about using 'no-multi-line' seems to have take care of the split lines. Thanks, Jo --- On Fri, 6/19/09, Balazs Scheidler <bazsi(a)balabit.hu> wrote: From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] syslog-ng 3.0 seem to chop some of the long lines off To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Received: Friday, June 19, 2009, 4:38 AM On Wed, 2009-06-17 at 12:10 -0700, Joe Hansen wrote: > We're seeing lines in /var/log/messages that are chopped off. They > should look like this: > > Jun 17 00:34:36 bxe1001 WORKER[18808]: DEBUG,email header > info,3893336992,1941292826,CPSendWork::ProcessConverterSuccessMessage,"man_id=551797512,srp_id=GBISXAPC00S07,from=From: ""info(a)callcentres.net"" <info(a)callcentres.net> ,to=To: ""Graeme Ross"" <gross(a)mobileemail.vodafone.com.au>,subject=Turn-key call centre for lease. Launceston, Tasmania","worker/CPSendWork.cpp:858" > > But they look like this: > > Jun 17 00:34:36 bxe1001 WORKER[18808]: DEBUG,email header > info,3893336992,1941292826,CPSendWork::ProcessConverterSuccessMessage,"man_id=551797512,srp_id=GBISXAPC00S07,from=From: ""info(a)callcentres.net"" <info(a)callcentres.net> > > > Anyone seen this? > > Thanks > > > Well, can you tell us a little bit more on your configuration? What protocol do you use on the source side? Do you happen to use the new RFC5424 style protocol, or the legacy one? What is generating those messages, can you show us a tcpdump/strace snippet that shows how that frame is travelling the network? Also, syslog-ng 3.0 does not remove embedded NL characters by default, maybe the rest of the message continues on the next line? You can reenable the previous behaviour by using the 'no-multi-line' flag for your source or destination (e.g. you can change the multi-line handling not just for a given source, but also handle the same message differently in different destinations). -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html __________________________________________________________________ Get a sneak peak at messages with a handy reading pane with All new Yahoo! Mail: http://ca.promos.yahoo.com/newmail/overview2/

2 3

new db-parser() utility: pdbtool
by ILLES, Marton 23 Aug '09

23 Aug '09

Hi, As I wrote in my previous post I kept myself busy with patterndb/db-parser enhancements. Recently I added a small utility called pdbtool to help working with pattern databases. Creating, maintaining and especially troubleshooting patterns are probably not the most easy task if you try it with the first time. pdbtool is a collection of small utilities bounded into one binary similar to samba's net utility. You can find pdbtool in my 3.1 syslog-ng git repo (http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=summary) hopefully Bazsi is gonna merge it soon mainline. MATCH The first command included in pdbtool is called "match" which can be used to test match rules in a pattern database. Debuging or troubleshooting patterns was somewhat complicated as you had to feed the messages into syslog-ng to see if a given pattern matches a message. The match command does the same. It tries to match a given message against patterndb rules and also evaluates parser while extracting parsed part of the message. To test rules simple rune pdbtool with the message and optionally with the program name to test: $ pdbtool match -p patterndb.xml -P sshd -M "Accepted publickey for marci from 127.0.0.1 port 59357 ssh2" match returns with 0 or 1 return code and prints the assigned values as well: .classifier.rule_id=fc44a9fe-75fd-11dd-9bba-001e6806451b .classifier.class=system I also plan to add some more verbose debug output if you specify debug option, so you can see how the given message is matched against the tree. Till than you can still test and debug your ruleset. DUMP The dump command can be used to dump the radix tree built from the patterns. This way you can see how your patterns are represented in syslog-ng and it might also help you tracking down problems with patterns. The dump utility can dump the tree used for matching the PROGRAM or the MSG parts. To see the patterns for the 'sshd' PROGRAM one should run the following command: $ pdbtool dump -p patterndb.xml -P 'sshd' The output looks like this (just part of the whole output): 'p' 'assword for' @QSTRING:@ 'from' @QSTRING:@ 'port ' @NUMBER:@ rule_id='fc49054e-75fd-11dd-9bba-001e6806451b' ' ssh' rule_id='fc55cf86-75fd-11dd-9bba-001e6806451b' '2' rule_id='fc4b7982-75fd-11dd-9bba-001e6806451b' 'ublickey for' @QSTRING:@ 'from' @QSTRING:@ 'port ' @NUMBER:@ rule_id='fc4d377c-75fd-11dd-9bba-001e6806451b' ' ssh' rule_id='fc5441ac-75fd-11dd-9bba-001e6806451b' '2' rule_id='fc44a9fe-75fd-11dd-9bba-001e6806451b' MERGE The third command "merge" can be used to convert and merge pattern databases into one big file which can be loaded into syslog-ng. It makes sense to store the patterns in separate files per applications or on other grouping bases, however they need to be merged as syslog-ng can utilize only one file per db-parser() instance. My idea is to store patterns in different files along with the syslog-ng configuration and merge them automatically (or manually) into one big file which is referenced from syslog-ng configuration. By default syslog-ng reads patterndb from the /var/lib/syslog-ng/patterndb.xml (depending on the compile time parameters) file which can be the output of a merge of the separate patterndb files. Having separate files help maintaining and troubleshooting patterns and also helps distributing or moving patterns from one node to an other. Besides merging the patterndb files the "merge" command takes care of converting/upgrading of older patterndb to the latest version (as of this writing version 3). This way you can easily use your old patterns with newer syslog-ng versions without any problem. Of course it might make sense to convert them with the tool as some features (like values, tags etc.) are only provided by newer versions. (The tool can be used to merge only 1 file, which case it upgrades it only if necessary.) I still need to create a man page for the utility as probably that is the first source of information most people is looking at. I hope you find the tool useful and as always any feedback, comment is very welcomed. Marton -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

1 0

PostgreSQL Invalid Encoding Errors
by David Blewett 20 Aug '09

20 Aug '09

Hi All: I'm continually getting into a loop of receiving messages like this in my PostgreSQL log files: 2009-06-19 09:17:36.465 EDT 5846 syslog@syslog: ERROR: invalid byte sequence for encoding "UTF8": 0xa8 2009-06-19 09:17:36.465 EDT 5846 syslog@syslog: HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". 2009-08-01 00:00:05.216 EDT 3986 syslog@syslog: ERROR: invalid byte sequence for encoding "UTF8": 0xc446 2009-08-01 00:00:05.216 EDT 3986 syslog@syslog: HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". I'm using a centralized logging system, and just turned on logging to syslog for our hardware firewall. I'm positive it was this change that is resulting in this behavior, but I really do want to continue pulling this data from the firewall. It appears that random messages from the firewall will come in as invalid UTF8, and somehow get stuck in a loop with syslog-ng continually trying to insert the offending line and PostgreSQL continually refusing. I have several log files that are > 500MB now because of this. Is there a way to force syslog-ng to drop these messages, and perhaps log the error? I am using PostgreSQL 8.3.6. syslog-ng --version syslog-ng 3.0.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#08c9bf9d98e4e021756adc12c42605840140ba8b Compile-Date: Jul 8 2009 12:16:03 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: on syslog-ng.conf: options { chain_hostnames(off); flush_lines(0); stats_level(2); stats_freq(43200); frac_digits(5); ts_format(iso); }; source src { unix-stream( "/dev/log" max-connections(40) ); internal(); udp(port(514)); tcp(port(5140) keep-alive(yes)); }; destination d_sql { sql( type(pgsql) host("10.233.93.18") username("syslog") password("*****") database("syslog") table("facility_$FACILITY") columns("host", "sourceip", "priority", "lvl", "tag", "rcvd", "sent", "program", "msg") values("$HOST", "$SOURCEIP", "$PRIORITY", "$LEVEL", "$TAG", "$R_ISODATE", "$S_ISODATE", "$PROGRAM", "$MSGONLY") indexes("host", "rcvd", "sent", "program", "msg") ); }; destination console_all { file("/dev/tty12"); }; destination messages { file("/var/log/messages"); }; log { source(src); #destination(messages); destination(console_all); destination(d_sql); };

2 3

db-parser: tag and value support
by ILLES, Marton 18 Aug '09

18 Aug '09

Hi, I had some time recently and did some db-parser improvements in my syslog-ng 3.1 branch. Some of the features are already integrated to 3.1 mainline by Bazsi, but I though it is worthwhile to write about them. I have posted it to my blog (http://marci.blogs.balabit.com/) but here it goes as well. In one of my earlier post I wrote about adding tag support to syslog-ng. It has been already merged to the 3.1 mainline by Bazsi. To make the feature even more useful the version 2 patterndb had support for specifying tags in each rule to tag the matching messages. So far this functionality was missing, but now it is possible to tag messages directly from the patterndb configuration so you can latter filter based on these tags. It is more versatile than using classification as you can specify multiple tags and tag matching is way more faster than matching the class of the message. (Though it is not possible to use the tags in macros as you can do with classes. eg.: you can use the ${.classifier.class} macro while you can not include the tags) Besides tags patterndb gained support for using static value assignment to messages. With patterndb version 3 (which is backward compatible with version 2) it is possible to specify named values in each rule. When a message matches a given rule all the specified values are assigned to the message (just like names parsers like @STRING:myvalue@) and these values can be latter used in filters or in macros. (See my previous post on parsers.) Also it is possible to use macros in values, so you can refer back to any parsed part of the message. It is probably most useful if you want to specially store the messages in some structured form where you need to fill all required values. With these addition you can use rules like this: <rule id="1" provider="balabit" class="system"> <patterns> <pattern>my pattern @STRING:my_parsed_value@</pattern> </patterns> <tags> <tag>tag1</tag> <tag>tag2</tag> </tags> <values> <value name="myvalue1">foo</value> <value name="myvalue2">bar ${myvalue1}</value> </values> </rule> Version 3 of the patterndb has been also extended so it can hold example messages for a given pattern. It is really handy if you need to test your pattern automatically or simply you want to document the original message that you created the pattern for. With the examples you can also specify expected values that are parsed or set from the message. The previous example can be extended this way: <rule id="1" provider="balabit" class="system"> <patterns> <pattern>my pattern @STRING:my_parsed_value@</pattern> </patterns> <tags> <tag>tag1</tag> <tag>tag2</tag> </tags> <values> <value name="myvalue1">foo</value> <value name="myvalue2">bar ${myvalue1}</value> </values> <examples> <test_message program="programname">my pattern test</test_message> <test_values> <test_value name="myvalue1">foo</test_value> <test_value name="myvalue2">bar foo</test_value> <test_value name="my_parser_value">test</test_value> </test_values> <test_tags> <test_tag>tag1</test_tag> <test_tag>tag2</test_tag> </test_tags> <example> </examples> </rule> You can find the source code in my 3.1 git tree at http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=summary Happy tagging! M -- Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D

1 0

Unable to run syslog-ng 3.0.4 as non-root on solaris 10
by Jeffrey Psolla 18 Aug '09

18 Aug '09

Hi, Yesterday I upgraded syslog-ng on our central log server from 2.0.5 to 3.0.4 . The OS is solaris 10. Prior to the upgrade I was able to run syslog-ng as a non-root user with the following command: /usr/local/sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -u syslogng -g syslogng -p /etc/syslog-ng/syslog-ng.pid After the upgrade I get the following error using the same command: Error binding socket; addr='AF_INET(0.0.0.0:514)', error='Permission denied (13)' Error initializing source driver; source='gateway', id='gateway#0' Error initializing message pipeline; If I remove the -u and -g arguments the daemon runs without issues. Same physical server, OS, and user account prior to the upgrade. I tried the verbose and debug arguments, when starting syslog-ng, but only get the above error. The syslog.conf file is posted below. I compiled from source using the following options: ./configure --enable-debug --disable-ipv6 --enable-dynamic-linking --enable-pcre --enable-ssl --disable-glibtest --disable-tcp-wrapper Thanks, Jeff ******************************* /usr/local/sbin/syslog-ng --version ******************************* Compile-Date: Aug 13 2009 16:25:43 Enable-Threads: off Enable-Debug: on Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: on Enable-Sun-Door: on Enable-IPv6: off Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on ***************************** syslog-ng.conf ***************************** @version:3.0 # # options { use_dns(persist_only); dns_cache_hosts(/etc/hosts); keep_hostname(yes); check_hostname(yes); #normalize_hostnames(yes); stats_freq(3600); log_fifo_size(2000); flush_lines(5); bad_hostname(Corrupt); bad_hostname(disk); bad_hostname(drive); }; # local for messages coming in locally source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; # Gateway for UDP Syslog messages. This is the default source gateway { udp(ip("0.0.0.0") port(514)); }; template default_template { template("$DATE $HOST $PROGRAM[$PID]: $MESSAGE\n"); template_escape(no); }; # Where to write the incoming log files to. destination hosts { file("/opt/log/clients/new_clients/$HOST/$YEAR-$MONTH-$DAY.$FACILITY" owner(syslogng) group(syslogng) perm(0644) dir_perm(0755) dir_group(syslogng) dir_owner(syslogng) create_dirs(yes) template (default_template)); }; destination backuphost1 { tcp("xxx.xxx.xxx.xxx" port(1468)); }; destination backuphost2 { udp("xxx.xxx.xxx.xxx " port(514)); }; filter Windows_filter { program(MSWinEventLog) and match("Name: Installer" value("$MESSAGE") flags("ignore-case")) or match("Name: MGLdap" value("$MESSAGE") flags("ignore-case")) or match("Special privileges assigned to new logon" value("$MESSAGE")) or match("Authentication Ticket Request" value("$MESSAGE")) or match("Process: Kerberos" value("$MESSAGE")); }; filter RSS_filter { match("Bad protocol version" value("$MESSAGE")) or match("Did not receive identification string" value("$MESSAGE")) or facility(user); }; log { source(gateway); filter(Windows_filter); flags(final); }; log { source(gateway); filter(RSS_filter); flags(final); }; log { source(gateway); destination(hosts); destination(backuphost1); }; log { source(local); destination(hosts); destination(backuphost1); destination(backuphost2); }; ________________________________ ----------------------------------------------------------------------- This e-mail is intended for the addressee shown. It contains information that is confidential and protected from disclosure. Any review, dissemination or use of this transmission or its contents by persons or unauthorized employees of the intended organizations is strictly prohibited. The contents of this email do not necessarily represent the views or policies of PSCU Financial Services.

4 3

program destination problem (again...)
by Alberto Sierra 18 Aug '09

18 Aug '09

hi there, i know this is been discussed like a million times already but i'm stuck and can't get around this. i'm using a program destination in my syslog-ng config, like this: destination test_log { file("/var/log/testlog"); }; destination sshd_alerts { program("/usr/local/bin/ssh_alert_by_email.sh" template("$DATE $HOST $PROGRAM $MSGONLY")); }; filter sshd { program("sshd"); }; filter login_accepted { match("Accepted password|Accepted publickey"); }; log { source(s_all); filter(sshd); filter(login_accepted); destination(sshd_alerts); destination(test_log); }; and the script as follows: #!/bin/bash while read line ; do echo $line >> /tmp/testlog done that's it, it logs to the destination(test_log) but the script does nothing. i followed a similar thread: https://lists.balabit.hu/pipermail/syslog-ng/2008-March/011512.html and the script works well interactively in the shell. I think i hit a dead end here... btw version 2.0.9 -- Alberto Sierra

5 6

Can't make Makefike of syslog-ng-3.0.4. OSE
by seki-norihiko＠mdis.co.jp 18 Aug '09

18 Aug '09

Hello everyone. I am user of syslog-ng. I wanna use syslog-ng instead of syslog on Red Hat Linux 5.2. CPU is Intel Xeon 64-bit architecture. I downloaded "eventlog_0.2.10.tar.gz" and "syslog-ng-3.0.4.tar.gz" Open Source Edition, from www.balabit.com. Thank you. I can make install eventlog_0.2.10. But I can't make Makefike of syslog-ng-3.0.4. I show error at below [root@hostname syslog-ng-3.0.4]# make make all-recursive ・・・ logmatcher.c: In function ‘log_matcher_pcre_re_compile’: logmatcher.c:534: error: ‘PCRE_NEWLINE_ANYCRLF’ undeclared (first use in this function) logmatcher.c:534: error: (Each undeclared identifier is reported only once logmatcher.c:534: error: for each function it appears in.) Would you please teach me what I should do to solve this error? I can't info about this error on the web. I need help. Thank you. Nori.

3 3

port 514 not used
by stephen.greenfield＠wachovia.com 15 Aug '09

15 Aug '09

I configure syslog-ng to use udp port 514, on the syslog server receiving log messages. When I issue a `netstat -lp | grep syslog` it shows different ports but never 514. Any ideas why? /steve

3 4

SOLVED: program destination problem (again...)
by Alberto Sierra 13 Aug '09

13 Aug '09

Ok, i finally found the problem, i don't know if it is only related to the "template" option within the program directive, but it must end with a newline character "\n" for the program to process every line. like this: program("/usr/local/bin/ssh_alert_by_email.sh" template("$DATE $HOST $PROGRAM $MSGONLY\n")); }; *note the \n after $MSGONLY for the curious, this is the script that i implemented to get a notification for every ssh successful login: #!/bin/bash recipient="alerts(a)example.com" processLog() { subject=$(echo $1 |awk '{print "Successful ssh login by " $9, "on " $4}') body=$(echo $1 |awk '{printf ("%s %s %s", "Date: " $1, $2, $3 "\\n" "Hostname: " $4 "\\n" "Username: " $9 "\\n" "Source host: " $11); }') /bin/echo -e $body | mail -e -s "$subject" $recipient } while read msg ; do processLog "$msg" done On Thu, Aug 13, 2009 at 9:44 AM, Alberto Sierra<albertosierra(a)aesetres.com> wrote: > 0 S root 20465 1 0 80 0 - 443 - 11:26 pts/0 > 00:00:00 /bin/sh -c /usr/local/bin/ssh_alert_by_email.sh > 0 S root 20466 20465 0 80 0 - 443 - 11:26 pts/0 > 00:00:00 /bin/bash /usr/local/bin/ssh_alert_by_email.sh > 5 S root 20468 1 0 80 0 - 572 - 11:26 ? > 00:00:00 /sbin/syslog-ng -p /var/run/syslog-ng.pid > > the PID is not changing, (unless the syslog-ng is restarted of > course), the debug.log shows the program runs until syslog-ng is > restarted as well. but it still sends nothing to the /tmp/testlog > file. > > > > On Thu, Aug 13, 2009 at 3:47 AM, Fegan, Joe<Joe.Fegan(a)hp.com> wrote: >> In "ps -elf" do you see your script? Does the pid stay the same as time advances, or does it change (which would mean it's exiting and being replaced with a new instance by syslog-ng automatically). You could add a start and end marker to see if it's starting at all and if/when it's exiting. Like: >> >> #!/bin/bash >> echo "$0 started `date`" >> /tmp/debug.log >> while read line ; do >> echo $line >> /tmp/testlog >> done >> echo "$0 exited `date`" >> /tmp/debug.log >> >> >> >> -----Original Message----- >> From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alberto Sierra >> Sent: 13 August 2009 07:26 >> To: syslog-ng(a)lists.balabit.hu >> Subject: [syslog-ng] program destination problem (again...) >> >> hi there, >> >> i know this is been discussed like a million times already but i'm >> stuck and can't get around this. >> >> i'm using a program destination in my syslog-ng config, like this: >> destination test_log { file("/var/log/testlog"); }; >> destination sshd_alerts { >> program("/usr/local/bin/ssh_alert_by_email.sh" template("$DATE $HOST >> $PROGRAM $MSGONLY")); }; >> >> filter sshd { program("sshd"); }; >> filter login_accepted { match("Accepted password|Accepted publickey"); }; >> >> >> log { >> source(s_all); >> filter(sshd); >> filter(login_accepted); >> destination(sshd_alerts); >> destination(test_log); >> }; >> >> and the script as follows: >> >> #!/bin/bash >> while read line ; do >> echo $line >> /tmp/testlog >> done >> >> that's it, it logs to the destination(test_log) but the script does nothing. >> >> i followed a similar thread: >> https://lists.balabit.hu/pipermail/syslog-ng/2008-March/011512.html >> >> and the script works well interactively in the shell. I think i hit a >> dead end here... btw version 2.0.9 >> >> -- >> Alberto Sierra >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.campin.net/syslog-ng/faq.html >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.campin.net/syslog-ng/faq.html >> >> > > > > -- > Alberto Sierra Reales [aesetres] > IT Consultant > Cel. 8319-1805 > -- Alberto Sierra Reales [aesetres] IT Consultant Cel. 8319-1805

1 0

syslog-ng and oracle
by Andreas Sartori 12 Aug '09

12 Aug '09

hello, we are currently setting up a new syslog server and switching from a sun fire v880 with syslog-ng and plain files to a new linux based (centos 5.3) syslog server with oracle as logging backend. we do not want to use pipes to write to the database, so we compiled libdbi-dbd-oracle with the oracle instantclient 10.2.0.4-1. syslog-ng is starting up fine (not complaining about the oracle backend), but we do have some problem with the inserts to the database. our trial syslog-ng configuration: destination d_sql_fwint { sql(type(oracle) username("xxx") password("xxx") database("dashb") table("SYSLOG_FWINT") columns("MSG_RCV_TIME varchar2(16)", "HOSTNAME varchar2(256)", "MESSAGE varchar2(2048)") values("$R_DATE", "$HOST", "$MSGONLY") }; when running in debug mode we get the following output: Running SQL query; query='INSERT INTO SYSLOG_FWINT (MSG_RCV_TIME, HOSTNAME, MESSAGE) VALUES (\'Aug 11 21:24:14\', \'gw20\', \'Built outbound TCP connection 144547326429398234 for LAN230:10.20.30.17/43825 (10.20.30.17/43825) to outside:10.10.10.102/80 (10.10.10.102/80)\')' this cant work sind oracle is not accepting \ as escape char, and in this case we do not need an escape at all. the sql syntax is right without the \. since templates are not working when using the sql driver, does anyone has an idea how to fix that? are we missing something? oracle setup is right, tnsnames or works and we can connect to the database with sqlplus. the syslog-ng init script as the environment variables set (ORACLE_HOME, LD_LIBRARY_PATH) thanks for any help -andy

2 5

syslog-ng exits when system time set backwards
by Kirill Stepanchuk 12 Aug '09

12 Aug '09

Hello, I've googled the mailing list, but could find any related threads. On FreeBSD, syslog-ng 1.6.8 exits when the system time is set to a value earlier than the previous value. Has anyone else encountered this before? Is there a fix or a configuration issue? $ ps auxwww | grep syslog-ng root 1040 0.0 0.1 1516 1208 ?? Ss 10:15AM 0:00.08 /usr/local/sbin/syslog-ng kstepanchuk 6529 0.0 0.0 1568 980 p0 R+ 10:38AM 0:00.00 grep syslog-ng $ sudo date 200501010000 Sat Jan 1 00:00:00 EST 2005 $ ps auxwww | grep syslog-ng mfeeney 6533 0.0 0.0 1612 996 p0 S+ 12:00AM 0:00.00 grep syslog-ng Thanks,

2 1

3.0.3: Accessing unix-dgram source appears to break all logging
by Markus Stalder 06 Aug '09

06 Aug '09

Hello list, my first post to the list, so hello everybody! :-) I'm on Ubuntu 8.04.2 amd64 and trying to upgrade from syslog-ng 2.0.9-1ubuntu1 to syslog-ng 3.0.3. I've both tried the on balabit.com provided amd64 .deb package and also created an own amd64 .deb with the default settings in /debian using dpkg buildpackage. I have the following two sources in my config: source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; source src { internal(); unix-dgram("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; Logging works just fine with the s_all source. However, as soon as I access the src source, by, for example destination temp_logfile { fifo("/tmp/.tivoli/.logfile."); }; log { source(src); destination(temp_logfile); }; all logging will cease. Meaning, other than Aug 5 15:04:23 server syslog-ng[17505]: syslog-ng starting up; version='3.0.3' I'm not getting *anything* in my logs after restarting syslog-ng. If I then comment destination temp_logfile ... again and restart syslog-ng, logging works again. The exact same config has worked just fine with 2.0.9-1ubuntu1. Any idea what could be wrong here? Thanks a lot! Best regards Markus

4 4

syslog-ng 3.0.4 has been released
by devel＠balabit.hu 05 Aug '09

05 Aug '09

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 3.0.4 SUMMARY : new stable release DATE : Aug 5, 2009 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (3.0.4) has been released. For latest fixes in the 3.0.x branch you are recommended to upgrade to this version. CHANGES: 3.0.4 Wed, 05 Aug 2009 12:51:31 +0200 Bugfixes: * The use of the tcp6 destination caused aborts during startup, which has now been fixed. * The window size used for flow control could be decremented to a negative value in case multiple incoming connections overflowed the queue. This could cause an abort if a SIGHUP was received at this time. * Fixed a possible segmentation fault under Solaris if the syslog() driver was used without explicitly specifying a transport() option. * Fixed parse error handling when loading the patterndb. * Fixed the handling of door initialization failures on Solaris. Other changes: * Fixed performance degradation when using the host_override/program_override options. DOWNLOAD: You can download the source or binary packages from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…

1 0

IPv6 destination host + port causes assertion failure in gsockaddr.h
by Alexei Aliaev 05 Aug '09

05 Aug '09

Hi, I am trying to solve an issue with syslog-ng 3.0.3 in IPv6 environment... I have a destination specified in my config file as such: destination collector { tcp6("fdab:228b:77af:6e7e:206:5bff:fef6:e2af" port(514)); }; When I run syslog-ng, I get this: file gsockaddr.h: line 86 (g_sockaddr_inet_get_sa): assertion failed: (g_sockaddr_inet_check(s)) Aborted I tracked down that this is caused by the destination definition above. The destination address is of the same box and port 514 is open and listening. If I remove the port, everything seems to start-up fine but it tries to connect to the specified IP to port 0 and I keep getting connection refused messages. Are there any known issues that could explain this? -- Alexey

2 7

Re: [syslog-ng] 3.0.3: Accessing unix-dgram source appears to break all logging
by Markus Stalder 05 Aug '09

05 Aug '09

> But who and what is reading the fifo /tmp/.tivoli/.logfile. in your > configuration? Opening a fifo for write will hang until someone else opens > it for read, that's how named pipes work... Thanks for your reply. As said before, the config worked just fine with 2.0.9. Also, the following config snippet will produce the exact same effect, ALL logging will stop, and there's no fifo involved at all: destination dings { udp("172.30.10.204" port(514) ); file("/var/monitoring/chk/logs/chk.log"); }; filter f_local5 { facility(local5); }; filter f_info { level(info); }; log { source(src); filter(f_local5); filter(f_info); destination(dings); }; So, it got to be about the source, since only this snippet and the previous one refer to the unix-dgram source "src". If I comment them, everything works fine, and I got a few dozen other log-entries in the config which refer to the unix-stream source "s_all". Again, *ALL* logging stops! Or, put differently: syslog-ng stops to log anything. It stops working. Thanks, Markus

1 0

log analyzer
by dreamywoman 04 Aug '09

04 Aug '09

hi everybody I would like to know that what web analyzer i can integrate with syslog-ng ? Best Roya

3 2

analyzer
by dreamywoman 02 Aug '09

02 Aug '09

Hi all Tanx for ur reply but I have installed php-syslog-ng but i need something more. I need one analyzer that realize attacks because I have different mailer. Best Roya

1 0

syslog-ng stops accepting new connections every 100-110 minutes
by Matt Pinkham 31 Jul '09

31 Jul '09

For the last 24 hours on versions 2.0.4, 2.1.4, & 3.0.3 syslog-ng will stop taking new connections via a listening port every 100-110 minutes (aka it will hang up immediately). It will never recover on its own and has to be restarted. I haven't figured out the exact interval but hopefully that will be close enough to work with (note that the traffic is fairly low -- 10 mesgs/sec -- 500K-600K data/min). I had a program logging data locally via /dev/log into a named directory and then moved this program to a remote server. That remote server does not seem to be having an issue. I have observed this issue on two separate servers (RHEL4.[image: Cool] that were taking this data feed. I have tried with flush_lines/sync & time_reopen commented out with no difference as well as log_fifo_size, log_mesg_size,so_recvbuf commented out. There are no obvious messages about why syslog-ng stops working (even with debug and verbose enabled). Note that these two servers (that stop working) are behind an Alteon 2424 switch (although I have other feeds to other servers working fine behind this switch). Ideas? Need more data? ==syslog will stop accepting connections== [root@server]# telnet localhost 514 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. Connection closed by foreign host. ==top section of 3.0 syslog-ng.conf== options { flush_lines (0); time_reopen (10); log_fifo_size (10000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); dir_perm (0755); perm (0644); chain_hostnames(no); keep_hostname (yes); stats_freq (3600); log_msg_size(65536); }; source remote { udp(ip(0.0.0.0) port(514) so_rcvbuf(1048576)); tcp(ip(0.0.0.0) port(514) max-connections(50) so_rcvbuf(1048576)); }; ==logging data like this=== filter f_data { match("Data:"); }; destination d_data { file("/var/log/data/data-$R_MONTH$R_DAY$R_HOUR$R_MIN"); }; log { source(remote); filter(f_data); destination(d_data); };

3 6

Problem with static linking glib on Solaris
by kdajka 30 Jul '09

30 Jul '09

Hi, I'm trying to compile latest stable syslog-ng (3.0.3) release on Solaris 10. I have compiled glib (2.10.3) with ./configure --enable-static --with-libiconv --prefix=/usr/local/inp/glib. Eventlog was installed by Sun administrator in my company: $pkginfo |grep -i event application SMCevtlog eventlog Which is installed to /usr/local/lib Everything is fine if I run configure like this: LD_LIBRARY_PATH=/usr/local/inp/glib/lib:$LD_LIBRARY_PATH PKG_CONFIG_PATH=/usr/local/inp/glib/lib/pkgconfig/:/usr/local/lib/pkgconfig ./configure but if I try compile it with static: LD_LIBRARY_PATH=/usr/local/inp/glib/lib:$LD_LIBRARY_PATH PKG_CONFIG_PATH=/usr/local/inp/glib/lib/pkgconfig/:/usr/local/lib/pkgconfig ./configure --enable-static-linking I'm getting this error: checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for a BSD-compatible install... ./install-sh -c checking whether build environment is sane... ^Z ... ... checking for pkg-config... (cached) /usr/bin/pkg-config checking pkg-config is at least version 0.14... yes checking for GLIB - version >= 2.10.1... yes (version 2.10.3) checking for EVTLOG... yes checking for static GLib libraries... no checking sanity checking Glib headers... yes configure: error: static GLib libraries not found (a file named libglib-2.0.a), either link GLib dynamically using the --enable-dynamic-linking or install a static GLib It says that it couldn't find libglib-2.0.a, even though it is located on my system: $ ls -l /usr/local/inp/glib/lib/libglib-2.0.* -rw-r--r-- 1 webadm webadm 1103212 Jul 29 09:58 /usr/local/inp/glib/lib/libglib-2.0.a -rwxr-xr-x 1 webadm webadm 1236 Jul 29 09:58 /usr/local/inp/glib/lib/libglib-2.0.la lrwxrwxrwx 1 webadm webadm 23 Jul 29 09:58 /usr/local/inp/glib/lib/libglib-2.0.so -> libglib-2.0.so.0.1000.3 lrwxrwxrwx 1 webadm webadm 23 Jul 29 09:58 /usr/local/inp/glib/lib/libglib-2.0.so.0 -> libglib-2.0.so.0.1000.3 -rwxr-xr-x 1 webadm webadm 940008 Jul 29 09:58 /usr/local/inp/glib/lib/libglib-2.0.so.0.1000.3 Is there another variable in which I can specify non-standard place of libglib-2.0.a? My config.log http://pastebin.com/d29c03773

2 2

Errors in Installing Syslog-ng
by Ashish Shrimali 30 Jul '09

30 Jul '09

Hi Team, I'm new to Syslog-ng. I was trying to install Syslog-ng on one of our servers and got the errors as attached. Can you please help me out in this ? Thanks & Regards, Ashish Shrimali -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of syslog-ng-request(a)lists.balabit.hu Sent: Wednesday, July 29, 2009 3:34 PM To: Ashish Shrimali Subject: Welcome to the "syslog-ng" mailing list (Digest mode) Welcome to the syslog-ng(a)lists.balabit.hu mailing list! To post to this list, send your email to: syslog-ng(a)lists.balabit.hu General information about the mailing list is at: https://lists.balabit.hu/mailman/listinfo/syslog-ng If you ever want to unsubscribe or change your options (eg, switch to or from digest mode, change your password, etc.), visit your subscription page at: https://lists.balabit.hu/mailman/options/syslog-ng/ashish_shrimali%40infosy… You can also make such adjustments via email by sending a message to: syslog-ng-request(a)lists.balabit.hu with the word `help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You must know your password to change your options (including changing the password, itself) or to unsubscribe. It is: syslog-ng Normally, Mailman will remind you of your lists.balabit.hu mailing list passwords once every month, although you can disable this if you prefer. This reminder will also include instructions on how to unsubscribe or change your account options. There is also a button on your options page that will email your current password to you. **************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS******** End of Disclaimer ********INFOSYS***

2 1

local-remote syslog-ng log collection problem
by linux linux 29 Jul '09

29 Jul '09

hi, I am new in syslog-ng. I want to make my syslog-ng.conf file in such way so it can collect data from remote and local process. After modifying the syslog-ng.conf i am able to collect the log, but problem is am getting the log with localhost insteed of local server ip and remote server ip. But if i restart the syslo-ng and the process then am getting log with my server ip for only that particular process that i have restarted. Could anyone plese let me know why i have to restart the syslog-ng and daemons/processes everytime, though i did not make any change in syslog-ng.conf file. I prepared the bootable iso with my syslog-ng changes so syslog-ng and daemons should pick up my cganges when i install os on the server. Am ii missing something in syslog-ng.conf file or i need to add something in each precess/daemon? Below is code for collect local-remore log. udp(ip(0.0.0.0) port(514)); destination remote {udp("remote server ip" port(514));}; log { source(all); destination(remote); }; Thanks Masami

1 0

Questions about Facility, Priority and Level
by Clayton Dukes 28 Jul '09

28 Jul '09

Hey Guys, According to the documentation: $LEVEL = Importance level of the message represented as a number: 6 - Success, 5 - Informational, 4- Warning, or 3 - Error). $FACILITY = The facility sending the message. $PRI = Priority header of the message, storing the facility and the level of the message. Questions: What is the difference between LEVEL and PRI in syslog-ng? it doesn't seem to be storing anything but the level? When I look in the database, it shows identical information about the two: mysql> select distinct priority from logs; +----------+ | priority | +----------+ | alert | | crit | | debug | | emerg | | err | | info | | notice | | warning | +----------+ mysql> select distinct level from logs; +---------+ | level | +---------+ | err | | info | | notice | | debug | | warning | | crit | | emerg | | alert | +---------+ According to the RFC: The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. For example, a kernel message (Facility=0) with a Severity of Emergency (Severity=0) would have a Priority value of 0. Also, a "local use 4" message (Facility=20) with a Severity of Notice (Severity=5) would have a Priority value of 165. In the PRI part of a syslog message, these values would be placed between the angle brackets as <0> and <165> respectively. The only time a value of "0" will follow the "<" is for the Priority value of "0". Otherwise, leading "0"s MUST NOT be used. So, if that's the case, couldn't I just grab the PRI from the message and store only that in the table and use the code on my end to derive the FAC and SEV (I assume severity = level in syslog-ng)? Also, is there a way to store these as integers from syslog-ng? There's a finite number of all of these, it seems that it would be better/faster to store them as an integer in the table and reference that in my code. Or I could use enum - not sure which is faster. Can I safely drop LEVEL or PRI and only keep one? it seems odd to have two columns for the same thing :-) Thanks! ______________________________________________________________ Clayton Dukes ______________________________________________________________

3 6

Syslog-ng seems to be not parsing correctly if the PROGRAM macro parsing fails.
by Yu Watanabe 27 Jul '09

27 Jul '09

Hello all. I am using syslog-ng v 2.0.5. However, I am gathering logs from the cisco catalyst switches, but when I tried to use the PROGRAM macro it seems not be working properly. In Cisco switches , there seems to be messages that program is not included in the message that is sent from the device. I would like to know how does syslog-ng parses the messages that does not have the PROGRAM name included and what would happen if we use the PROGRAM macro for these message. Following is the proper message: Jul 27 13:17:11 l2swtich 128: %SYS-5-CONFIG_I: Configured from console by console **** There are logs that does not have this part. Regards, Yu Watanabe

3 4

Fastest, most scaleable solution
by Clayton Dukes 25 Jul '09

25 Jul '09

Hi folks, I'm thinking about completely re-writing my app (php-syslog-ng) in order to provide high end scalability and a better (ajax) front end. At the base of it, my biggest concern is scalability. I want to write the backend to be able to handle many thousands of messages per second. In your opinion, what is the best way to accomplish this? Should I: Log everything to MySQL directly? If so, can MySQL handle that insert rate? Log everything to disk and use flat files to do analysis and reporting? Log everything to disk and use MySQL's load_data_infile? Other suggestions? Also, is anyone out there using MySQL for this level of insert rates? Do you have any recommendations on table structures, indexes, etc.? -- ______________________________________________________________ Clayton Dukes ______________________________________________________________

3 6

Syslog-ng log viewer for Postgres
by Craig Ayliffe 24 Jul '09

24 Jul '09

Hi All, Does anyone have any suggestions for a web based viewer like php-syslog-ng but for a Postgres backend? Regards, -- Craig Ayliffe

3 2