May 2009 - syslog-ng - lists.balabit.hu

[Bug 42] New: capabilities, chown, chmod
by bugzilla＠bugzilla.balabit.com 20 Dec '12

20 Dec '12

https://bugzilla.balabit.com/show_bug.cgi?id=42 Summary: capabilities, chown, chmod Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zbyniu(a)pld-linux.org Type of the Report: bug Estimated Hours: 0.0 Let's take a look at syslog-ng-3.0.1/src/affile.c lines 60-83 1. CAP_SYS_ADMIN is needed only for /proc/kmsg, it is added w/o check 2. CAP_DAC_READ_SEARCH should be added only if open fail with errno 13 2a. CAP_DAC_OVERRIDE should be added only if open fail with errno 13 and with CAP_DAC_READ_SEARCH set 3. fchown needs CAP_CHOWN unconditionaly 4. fchmod needs CAP_FOWNER if file owner != euid (root here) 5. all caps should be restored summary: - CAP_SYS_ADMIN and CAP_DAC_OVERRIDE are set always even if unnecessary, and permanently - owner, group and perm doesn't work -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 10

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 38] New: Setting values of the Structured Data block.
by bugzilla＠bugzilla.balabit.com 12 Apr '10

12 Apr '10

https://bugzilla.balabit.com/show_bug.cgi?id=38 Summary: Setting values of the Structured Data block. Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: c.haase(a)ifu.com Type of the Report: enhancement Estimated Hours: 0.0 Please add the functionality to set values inside the Structured Data block. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 37] New: Syslog-ng macros don't work in filters
by bugzilla＠bugzilla.balabit.com 12 Apr '10

12 Apr '10

https://bugzilla.balabit.com/show_bug.cgi?id=37 Summary: Syslog-ng macros don't work in filters Product: syslog-ng Version: 3.0.x Platform: Other OS/Version: Other Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vincent.panel(a)telindus.be Type of the Report: --- Estimated Hours: 0.0 Several documented macros don't work when used in filters. It's the case for $SEC, $MSGHDR. Already reported on the syslog-ng mailing list and acknowleged as a syslog-ng bug. I've tried it on Sparc/Solaris (but can't choose it in the platform menu of bugzilla...) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng client messages delayed
by mark peters 22 Mar '10

22 Mar '10

Hi, We are running into a issue where syslog-ng clients who are configured to forward to a central syslog-ng destination server that are reasonably chatty are getting there messages delayed at the source (ie. 'tcpdump'ing the client shows it is sending messages from N minutes ago, where N is anything from 5 to 60 minutes on average sometimes more). We have also seen a small percentage of loss at the destination. On these 'delayed' clients we also see the resident memory footprint on the growing. Clients are running 1.6.12 (have tried 2.0.8 also - same symptons). The central syslog-ng server is running 2.0.8. relevant config entries are: syslog-ng client: options { sync (1); time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; destination d_central { tcp("log2.xyz.com" port(5140)); }; filter f_local5 { facility(local5); }; log { source(s_sys); filter(f_local5); destination(d_central); flags(final); }; syslog-ng central server: options { time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_expire (28800); use_fqdn (no); create_dirs (yes); dir_perm (0755); perm (0644); keep_hostname (no); use_time_recvd (no); stats_freq(600); flush_timeout(16000); flush_lines(256); time_sleep(80); }; destination d_server { file("/logs/server.$LEVEL.log"); }; filter f_server { facility(local5) and program("server"); }; log { source(s_sys); filter(f_server); destination(d_server); flags(final); }; to give a rough idea of message volume the server is receiving approximately 50 million log lines per day on average any help/information would be appreciated thanks -------------------------------------- Easy + Joy + Powerful = Yahoo! Bookmarks x Toolbar http://pr.mail.yahoo.co.jp/toolbar/

3 4

Expected performance
by Christopher Bland 04 Dec '09

04 Dec '09

Hey guys, I know it's hard to compare apples to apples when you start talking about performance and hardware but I would appreciate some feedback. I am currently have a 32Bit HP Proliant DL380 with 2 2.8Ghz cpus and 4G of memory running Fedora 10. At present I have 150 hosts generating between 10-15G worth of logs per day. I dump all of my logs to a Mysql database so that I can use php-syslog-ng. The OS is build on a 1.2T raid 5 disk array. To cut down on I/O I have the database writing to a ext3 filesystem while the rest of the box uses LVM. I have mounted the database partition with noatime and implemented all of the mysqltuner suggestions. My box is performing slow like molasses. My config looks like this: options { sync (0); time_reopen (10); log_fifo_size (1000); chain_hostnames(no); long_hostnames (off); keep_hostname(no); use_dns (yes); dns_cache(yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); }; source s_everything { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); udp(); }; destination d_database { pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" sync(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination d_asa { file("/var/log/syslog-ng/network/asa/$YEAR/$MONTH/$YEAR-$MONTH-$DAY" owner(root) group(staff) perm(0650) dir_perm(0750) create_dirs(yes)); }; #filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; filter f_asa { host("asain-temp1"); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; log { source(s_everything); filter(f_asa); destination(d_asa); }; log { source(s_everything); destination(d_database); }; my.cnf looks like this: [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql old_passwords=1 skip-innodb skip-bdb skip-name-resolve table_cache = 128 tmp_table_size = 256M max_heap_table_size = 256M query_cache_size = 128M query_cache_limit = 4M read_rnd_buffer_size = 1M thread_cache_size = 8 [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [ndbd] connect-string="nodeid=2;host=localhost:1186" [ndb_mgm] connect-string="host=localhost:1186" My load looks like this while inserting data: Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie Cpu(s): 20.4%us, 2.3%sy, 0.1%ni, 75.9%id, 0.7%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3634412k total, 3507692k used, 126720k free, 20224k buffers Swap: 8388600k total, 64k used, 8388536k free, 3246292k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6868 mysql 20 0 165m 20m 5088 S 91.9 0.6 1488:41 mysqld 7679 root 20 0 4140 1932 948 S 15.6 0.1 158:18.24 syslog-ng 7705 root 20 0 8456 1756 1384 S 5.9 0.0 79:18.84 mysql 21296 root 20 0 2556 996 740 R 2.0 0.0 0:00.01 top 1 root 20 0 2012 780 568 S 0.0 0.0 0:01.62 init 2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/0 4 root 15 -5 0 0 0 S 0.0 0.0 0:17.46 ksoftirqd/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 6 root RT -5 0 0 0 S 0.0 0.0 0:00.18 migration/1 7 root 15 -5 0 0 0 S 0.0 0.0 0:26.76 ksoftirqd/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 9 root RT -5 0 0 0 S 0.0 0.0 0:00.13 migration/2 10 root 15 -5 0 0 0 S 0.0 0.0 0:22.02 ksoftirqd/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 12 root RT -5 0 0 0 S 0.0 0.0 0:00.11 migration/3 13 root 15 -5 0 0 0 S 0.0 0.0 0:11.79 ksoftirqd/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 15 root 15 -5 0 0 0 S 0.0 0.0 0:04.69 events/0 16 root 15 -5 0 0 0 S 0.0 0.0 0:02.86 events/1 17 root 15 -5 0 0 0 S 0.0 0.0 0:02.80 events/2 18 root 15 -5 0 0 0 S 0.0 0.0 0:02.91 events/3 19 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 khelper 95 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0 96 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/1 97 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/2 98 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/3 100 root 15 -5 0 0 0 S 0.0 0.0 0:00.20 kblockd/0 101 root 15 -5 0 0 0 S 0.0 0.0 0:00.14 kblockd/1 102 root 15 -5 0 0 0 S 0.0 0.0 0:00.33 kblockd/2 103 root 15 -5 0 0 0 S 0.0 0.0 0:12.47 kblockd/3 105 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid 106 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify The box gets almost unusable when I do a query to retrieve data from the database? Again, I would appreciate any thoughts or suggestions . -Chris Bland

7 8

rewriteable fields
by chris packham 03 Dec '09

03 Dec '09

Hi List, Just looking for a bit of clarification on exactly what fields I can rewrite in a message. Basically I've got an annoying debug message coming out of a program and while I could filter it off to a separate file, what I really want to do is drop its level to debug (from warning) and let my existing filters take care of it. The docs say "Setting a field can operate on any value available via macros, e.g., HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers" so modifying LEVEL is not listed but its not explicitly excluded either. I tried "rewrite r_norm {set("1", value("LEVEL")); };" but that didn't work. However "rewrite r_norm {set("FOO", value("MESSAGE")); };" and "rewrite r_norm {set("BAR", value("PROGRAM")); };" work as expected. Are there limitations as to which parts of a message I can rewrite? Are there some fuller examples hanging around? Thanks, Chris

3 2

syslog-ng OSE 2.1.4 released
by Balazs Scheidler 16 Jun '09

16 Jun '09

Hi, I've released syslog-ng OSE 2.1.4 with the following changes: 2.1.4 Tue, 17 Mar 2009 13:35:00 +0100 Bugfixes: * Fixed a possible DoS condition triggered by a destination port unreachable ICMP packet received from a UDP destination. syslog-ng started eating all available memory and CPU until it crashed if this happened. * Fixed the rate at which files regular were read using the file() source. * Report connection breaks as a write error instead of reporting POLLERR as the write error path reports more sensible information in the logs. The release is available at: https://www.balabit.com/network-security/syslog-ng/opensource-logging-syste… -- Bazsi

4 5

Stripping the original hostname /ip from the syslog message
by Shashank Vinchurkar 12 Jun '09

12 Jun '09

Hi, We have a setup where multiple syslog-ng servers send logs to a central syslog-ng server. Finally this central syslog-ng server sends the consolidated logs to an outside server. The outside server can be any server accepting standard syslog messages. The first group of servers are running in the internal network and don't have any hostname associated with them. Also the ip address is internal and does not make sense to outside world. My requirement is that the outside server should only see the ip address of the syslog-ng server which consolidates the messages from these syslog-ng servers. But I always see the ip address of the syslog-ng server which originated the message. Is there anyway to get rid of this? I tried playing with the keep_hostname, long_hostname, chain_hostname and bad_hostname options but I still see the ip address of the originating server. Thanks in advance for the help. -Shashank

3 5

udp drops
by Jan Schaumann 06 Jun '09

06 Jun '09

Hello, I have a FreeBSD 6.2 (amd64) host where I'd like to replace the stock syslogd with syslog-ng (3.0.2). This host receives a lot of syslog messages per second from a large number of clients via UDP. The stock syslogd configuration is trivial: *.* /var/log/all This host currently drops about 2-4% of all UDP packets, syslog takes about 50-65% of one CPU. A drop-in replacement configuration using syslog-ng: options { create_dirs(yes); use_dns(no); }; template t_default { template("${DATE} <${FACILITY}.${PRIORITY}> ${HOST} ${MSG}\n"); }; source s_standard { file("/dev/klog"); internal(); udp(); unix-dgram("/var/run/log"); }; destination d_all { file("/var/log/all" template(t_default) ); }; # *.* /var/log/all log { source(s_standard); destination(d_all); }; syslog-ng uses about 90% of one CPU and drops between 15% and 20% of UDP packets (and, based on traffic patterns and logfile size, concurrent logfile rotation etc. even as high as 30%). I've tried a number of things to improve this performance, including: log_fetch_limit(100); log_iw_size(10000); flush_lines(100000); flush_timeout(10); in the global options and log_fifo_size(100000) in the destination definition with flags(flow-control) in the log definition. The best I was able to get with these numbers was 5-7% of UDP drops (ie still double of what the stock syslogd drops). I also tried adjusting "so_rcvbuf" for UDP with no noticable difference. Now consider that I did not do any sysctl tuning, as those should equally influence the stock syslog and I'm trying to sort out why one performs so significantly better than the other. Leaving aside any of the things I can do with syslog-ng further down the road (such as filtering and intentionally dropping certain messages), what can I do to get syslog-ng up to the same performance as the stock syslogd? Many thanks in advance for any pointers and help. -Jan

4 8