July 2009 - syslog-ng - lists.balabit.hu

[Bug 42] New: capabilities, chown, chmod
by bugzilla＠bugzilla.balabit.com 20 Dec '12

20 Dec '12

https://bugzilla.balabit.com/show_bug.cgi?id=42 Summary: capabilities, chown, chmod Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: zbyniu(a)pld-linux.org Type of the Report: bug Estimated Hours: 0.0 Let's take a look at syslog-ng-3.0.1/src/affile.c lines 60-83 1. CAP_SYS_ADMIN is needed only for /proc/kmsg, it is added w/o check 2. CAP_DAC_READ_SEARCH should be added only if open fail with errno 13 2a. CAP_DAC_OVERRIDE should be added only if open fail with errno 13 and with CAP_DAC_READ_SEARCH set 3. fchown needs CAP_CHOWN unconditionaly 4. fchmod needs CAP_FOWNER if file owner != euid (root here) 5. all caps should be restored summary: - CAP_SYS_ADMIN and CAP_DAC_OVERRIDE are set always even if unnecessary, and permanently - owner, group and perm doesn't work -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 10

[Bug 14] New: Maintain symlinks to current and previous log output files...
by bugzilla＠bugzilla.balabit.com 28 Sep '11

28 Sep '11

https://bugzilla.balabit.com/show_bug.cgi?id=14 Summary: Maintain symlinks to current and previous log output files... Product: syslog-ng Version: 2.0.x Platform: PC URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218688 OS/Version: Linux Status: NEW Severity: minor Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: sasa(a)debian.org Type of the Report: enhancement Estimated Hours: 0.0 >From bugs.debian.org: I'm hoping that upstream might consider adding functionality to syslog-ng to allow it to track current and previous log files and maintain symlinks to them, as a user option. This replicates one of the options that I (and others) find useful in cronolog. This has been previously discussed on the syslog-ng mailing list, though apparently nothing fully came of it. The archive of that discussion is at: https://lists.balabit.hu/pipermail/syslog-ng/2002-June/003452.html Obviously, it's probably somewhat of a low priority, but it would be nice. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 38] New: Setting values of the Structured Data block.
by bugzilla＠bugzilla.balabit.com 12 Apr '10

12 Apr '10

https://bugzilla.balabit.com/show_bug.cgi?id=38 Summary: Setting values of the Structured Data block. Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: c.haase(a)ifu.com Type of the Report: enhancement Estimated Hours: 0.0 Please add the functionality to set values inside the Structured Data block. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 37] New: Syslog-ng macros don't work in filters
by bugzilla＠bugzilla.balabit.com 12 Apr '10

12 Apr '10

https://bugzilla.balabit.com/show_bug.cgi?id=37 Summary: Syslog-ng macros don't work in filters Product: syslog-ng Version: 3.0.x Platform: Other OS/Version: Other Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vincent.panel(a)telindus.be Type of the Report: --- Estimated Hours: 0.0 Several documented macros don't work when used in filters. It's the case for $SEC, $MSGHDR. Already reported on the syslog-ng mailing list and acknowleged as a syslog-ng bug. I've tried it on Sparc/Solaris (but can't choose it in the platform menu of bugzilla...) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

syslog-ng client messages delayed
by mark peters 22 Mar '10

22 Mar '10

Hi, We are running into a issue where syslog-ng clients who are configured to forward to a central syslog-ng destination server that are reasonably chatty are getting there messages delayed at the source (ie. 'tcpdump'ing the client shows it is sending messages from N minutes ago, where N is anything from 5 to 60 minutes on average sometimes more). We have also seen a small percentage of loss at the destination. On these 'delayed' clients we also see the resident memory footprint on the growing. Clients are running 1.6.12 (have tried 2.0.8 also - same symptons). The central syslog-ng server is running 2.0.8. relevant config entries are: syslog-ng client: options { sync (1); time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; destination d_central { tcp("log2.xyz.com" port(5140)); }; filter f_local5 { facility(local5); }; log { source(s_sys); filter(f_local5); destination(d_central); flags(final); }; syslog-ng central server: options { time_reopen (10); log_fifo_size (2097152); long_hostnames (off); use_dns (yes); dns_cache (yes); dns_cache_expire (28800); use_fqdn (no); create_dirs (yes); dir_perm (0755); perm (0644); keep_hostname (no); use_time_recvd (no); stats_freq(600); flush_timeout(16000); flush_lines(256); time_sleep(80); }; destination d_server { file("/logs/server.$LEVEL.log"); }; filter f_server { facility(local5) and program("server"); }; log { source(s_sys); filter(f_server); destination(d_server); flags(final); }; to give a rough idea of message volume the server is receiving approximately 50 million log lines per day on average any help/information would be appreciated thanks -------------------------------------- Easy + Joy + Powerful = Yahoo! Bookmarks x Toolbar http://pr.mail.yahoo.co.jp/toolbar/

3 4

Expected performance
by Christopher Bland 04 Dec '09

04 Dec '09

Hey guys, I know it's hard to compare apples to apples when you start talking about performance and hardware but I would appreciate some feedback. I am currently have a 32Bit HP Proliant DL380 with 2 2.8Ghz cpus and 4G of memory running Fedora 10. At present I have 150 hosts generating between 10-15G worth of logs per day. I dump all of my logs to a Mysql database so that I can use php-syslog-ng. The OS is build on a 1.2T raid 5 disk array. To cut down on I/O I have the database writing to a ext3 filesystem while the rest of the box uses LVM. I have mounted the database partition with noatime and implemented all of the mysqltuner suggestions. My box is performing slow like molasses. My config looks like this: options { sync (0); time_reopen (10); log_fifo_size (1000); chain_hostnames(no); long_hostnames (off); keep_hostname(no); use_dns (yes); dns_cache(yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); }; source s_everything { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); udp(); }; destination d_database { pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" sync(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination d_asa { file("/var/log/syslog-ng/network/asa/$YEAR/$MONTH/$YEAR-$MONTH-$DAY" owner(root) group(staff) perm(0650) dir_perm(0750) create_dirs(yes)); }; #filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; filter f_asa { host("asain-temp1"); }; log { source(s_sys); filter(f_filter2); destination(d_mesg); }; log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; log { source(s_everything); filter(f_asa); destination(d_asa); }; log { source(s_everything); destination(d_database); }; my.cnf looks like this: [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql old_passwords=1 skip-innodb skip-bdb skip-name-resolve table_cache = 128 tmp_table_size = 256M max_heap_table_size = 256M query_cache_size = 128M query_cache_limit = 4M read_rnd_buffer_size = 1M thread_cache_size = 8 [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [ndbd] connect-string="nodeid=2;host=localhost:1186" [ndb_mgm] connect-string="host=localhost:1186" My load looks like this while inserting data: Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie Cpu(s): 20.4%us, 2.3%sy, 0.1%ni, 75.9%id, 0.7%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 3634412k total, 3507692k used, 126720k free, 20224k buffers Swap: 8388600k total, 64k used, 8388536k free, 3246292k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6868 mysql 20 0 165m 20m 5088 S 91.9 0.6 1488:41 mysqld 7679 root 20 0 4140 1932 948 S 15.6 0.1 158:18.24 syslog-ng 7705 root 20 0 8456 1756 1384 S 5.9 0.0 79:18.84 mysql 21296 root 20 0 2556 996 740 R 2.0 0.0 0:00.01 top 1 root 20 0 2012 780 568 S 0.0 0.0 0:01.62 init 2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/0 4 root 15 -5 0 0 0 S 0.0 0.0 0:17.46 ksoftirqd/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 6 root RT -5 0 0 0 S 0.0 0.0 0:00.18 migration/1 7 root 15 -5 0 0 0 S 0.0 0.0 0:26.76 ksoftirqd/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 9 root RT -5 0 0 0 S 0.0 0.0 0:00.13 migration/2 10 root 15 -5 0 0 0 S 0.0 0.0 0:22.02 ksoftirqd/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 12 root RT -5 0 0 0 S 0.0 0.0 0:00.11 migration/3 13 root 15 -5 0 0 0 S 0.0 0.0 0:11.79 ksoftirqd/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 15 root 15 -5 0 0 0 S 0.0 0.0 0:04.69 events/0 16 root 15 -5 0 0 0 S 0.0 0.0 0:02.86 events/1 17 root 15 -5 0 0 0 S 0.0 0.0 0:02.80 events/2 18 root 15 -5 0 0 0 S 0.0 0.0 0:02.91 events/3 19 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 khelper 95 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0 96 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/1 97 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/2 98 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/3 100 root 15 -5 0 0 0 S 0.0 0.0 0:00.20 kblockd/0 101 root 15 -5 0 0 0 S 0.0 0.0 0:00.14 kblockd/1 102 root 15 -5 0 0 0 S 0.0 0.0 0:00.33 kblockd/2 103 root 15 -5 0 0 0 S 0.0 0.0 0:12.47 kblockd/3 105 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid 106 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify The box gets almost unusable when I do a query to retrieve data from the database? Again, I would appreciate any thoughts or suggestions . -Chris Bland

7 8

rewriteable fields
by chris packham 03 Dec '09

03 Dec '09

Hi List, Just looking for a bit of clarification on exactly what fields I can rewrite in a message. Basically I've got an annoying debug message coming out of a program and while I could filter it off to a separate file, what I really want to do is drop its level to debug (from warning) and let my existing filters take care of it. The docs say "Setting a field can operate on any value available via macros, e.g., HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers" so modifying LEVEL is not listed but its not explicitly excluded either. I tried "rewrite r_norm {set("1", value("LEVEL")); };" but that didn't work. However "rewrite r_norm {set("FOO", value("MESSAGE")); };" and "rewrite r_norm {set("BAR", value("PROGRAM")); };" work as expected. Are there limitations as to which parts of a message I can rewrite? Are there some fuller examples hanging around? Thanks, Chris

3 2

Re: [syslog-ng] syslog-ng 3.0 seem to chop some of the long lines off
by Joe Hansen 26 Aug '09

26 Aug '09

The application is custom in-house. I'm really not sure what RFC they've followed. However your suggestion about using 'no-multi-line' seems to have take care of the split lines. Thanks, Jo --- On Fri, 6/19/09, Balazs Scheidler <bazsi(a)balabit.hu> wrote: From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] syslog-ng 3.0 seem to chop some of the long lines off To: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Received: Friday, June 19, 2009, 4:38 AM On Wed, 2009-06-17 at 12:10 -0700, Joe Hansen wrote: > We're seeing lines in /var/log/messages that are chopped off. They > should look like this: > > Jun 17 00:34:36 bxe1001 WORKER[18808]: DEBUG,email header > info,3893336992,1941292826,CPSendWork::ProcessConverterSuccessMessage,"man_id=551797512,srp_id=GBISXAPC00S07,from=From: ""info(a)callcentres.net"" <info(a)callcentres.net> ,to=To: ""Graeme Ross"" <gross(a)mobileemail.vodafone.com.au>,subject=Turn-key call centre for lease. Launceston, Tasmania","worker/CPSendWork.cpp:858" > > But they look like this: > > Jun 17 00:34:36 bxe1001 WORKER[18808]: DEBUG,email header > info,3893336992,1941292826,CPSendWork::ProcessConverterSuccessMessage,"man_id=551797512,srp_id=GBISXAPC00S07,from=From: ""info(a)callcentres.net"" <info(a)callcentres.net> > > > Anyone seen this? > > Thanks > > > Well, can you tell us a little bit more on your configuration? What protocol do you use on the source side? Do you happen to use the new RFC5424 style protocol, or the legacy one? What is generating those messages, can you show us a tcpdump/strace snippet that shows how that frame is travelling the network? Also, syslog-ng 3.0 does not remove embedded NL characters by default, maybe the rest of the message continues on the next line? You can reenable the previous behaviour by using the 'no-multi-line' flag for your source or destination (e.g. you can change the multi-line handling not just for a given source, but also handle the same message differently in different destinations). -- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html __________________________________________________________________ Get a sneak peak at messages with a handy reading pane with All new Yahoo! Mail: http://ca.promos.yahoo.com/newmail/overview2/

2 3

IPv6 destination host + port causes assertion failure in gsockaddr.h
by Alexei Aliaev 05 Aug '09

05 Aug '09

Hi, I am trying to solve an issue with syslog-ng 3.0.3 in IPv6 environment... I have a destination specified in my config file as such: destination collector { tcp6("fdab:228b:77af:6e7e:206:5bff:fef6:e2af" port(514)); }; When I run syslog-ng, I get this: file gsockaddr.h: line 86 (g_sockaddr_inet_get_sa): assertion failed: (g_sockaddr_inet_check(s)) Aborted I tracked down that this is caused by the destination definition above. The destination address is of the same box and port 514 is open and listening. If I remove the port, everything seems to start-up fine but it tries to connect to the specified IP to port 0 and I keep getting connection refused messages. Are there any known issues that could explain this? -- Alexey

2 7

syslog-ng stops accepting new connections every 100-110 minutes
by Matt Pinkham 31 Jul '09

31 Jul '09

For the last 24 hours on versions 2.0.4, 2.1.4, & 3.0.3 syslog-ng will stop taking new connections via a listening port every 100-110 minutes (aka it will hang up immediately). It will never recover on its own and has to be restarted. I haven't figured out the exact interval but hopefully that will be close enough to work with (note that the traffic is fairly low -- 10 mesgs/sec -- 500K-600K data/min). I had a program logging data locally via /dev/log into a named directory and then moved this program to a remote server. That remote server does not seem to be having an issue. I have observed this issue on two separate servers (RHEL4.[image: Cool] that were taking this data feed. I have tried with flush_lines/sync & time_reopen commented out with no difference as well as log_fifo_size, log_mesg_size,so_recvbuf commented out. There are no obvious messages about why syslog-ng stops working (even with debug and verbose enabled). Note that these two servers (that stop working) are behind an Alteon 2424 switch (although I have other feeds to other servers working fine behind this switch). Ideas? Need more data? ==syslog will stop accepting connections== [root@server]# telnet localhost 514 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. Connection closed by foreign host. ==top section of 3.0 syslog-ng.conf== options { flush_lines (0); time_reopen (10); log_fifo_size (10000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); dir_perm (0755); perm (0644); chain_hostnames(no); keep_hostname (yes); stats_freq (3600); log_msg_size(65536); }; source remote { udp(ip(0.0.0.0) port(514) so_rcvbuf(1048576)); tcp(ip(0.0.0.0) port(514) max-connections(50) so_rcvbuf(1048576)); }; ==logging data like this=== filter f_data { match("Data:"); }; destination d_data { file("/var/log/data/data-$R_MONTH$R_DAY$R_HOUR$R_MIN"); }; log { source(remote); filter(f_data); destination(d_data); };

3 6