- syslog-ng - lists.balabit.hu

Logrotate Questions And Solicitation For Ideas...
by Vadim Pushkin 21 Jul '06

21 Jul '06

Hello All; I've got a syslog-1.6.9 server running quite well with phpsyslog-ng. I am trying to rotate my logs and maintain just 14 days worth of logs within my MySql DB, an issue that I am having and will hopefully get fixed within the phpSyslog-NG group. With regards to flat file logs, the issue that I am writing here for, I wish to have 30 days worth of data within my log files (flat files under /var/log). The problem that I am having is rotating under syslog-ng, *NOT* MySql. Can anyone please provide me with a good method for rotating logs, where my logs are kept under various subdirs, depending on source address? Attached is my syslog-ng.conf, in case anyone cares to read it to determine what my dir-layouts are, also I am looking for help on why some of my source hosts are getting into two locations, I am sure that it is bacause of the "filter" rules in my config. Does the below make sense? # Trying to make all hosts that start with sw or contain sw1, or sw2, # get placed where filter F_switch should go. filter F_switch { host("sw*") or host("sw1") or host("sw2"); }; # Trying to place logs for all hosts that start with fw to go to where filter F_firewall should go. filter F_firewall { host("^fw*"); }; Please, I am sorry for this question if it is in the FAQ, I am having problem finding this info. Thank you all! .vp

7 12

Feature Request: conf.d-like folders or include statement
by JP Senior 20 Jul '06

20 Jul '06

I'm finding that I'm in need of a recursive conf.d-like configuration folder for syslog-ng. The reason I'm asking this is because I have many hosts, each requiring a separate configuration from the other, and my syslog-ng.conf file is getting pretty large, messy, and unmanageable. If it could iterate/enumerate /etc/syslog-ng/conf.d and add each of those to the 'running' config, that would be amazing. I think this could also work, in a less versatile function, by an 'include' statement in syslog-ng.conf. Ie: include { includefile(/etc/syslog-ng/extra/hostname.conf); }; Am I on the right track, or is there a function to do this? :) -JP Senior

2 1

Re: syslog-ng Digest, Vol 15, Issue 10
by dwarnock＠harbrook.net 19 Jul '06

19 Jul '06

I am on holiday from the 17th August returning on Monday the 31st August. If your query is urgent please contact the office on +44 (0)1423 508060 or Neil Harvey on +44 (0)15395 64734 (nharvey(a)harbrook.net) Regards David Warnock

1 0

why this messages from syslog?
by Maurizio B 18 Jul '06

18 Jul '06

why I get this messages from syslog-ng 2.0rc1? Log statistics; processed='source(s_local)=19', processed='source(s_remote)=0', processed='destination(dsl_console)=0', processed='destination(dsl_maillog)=0', processed='destination(ds_emerg)=0', processed='destination(ds_alert)=0', processed='destination(df_uucp)=0', processed='destination(ds_err)=0', processed='destination(dfs_err)=0', processed='destination(dsl_messages)=16', processed='destination(dfs_crit)=0', is it a misconfiguration? thanks in advance

2 1

Re: Logging to program destinations has odd prefix?
by John Caruso 16 Jul '06

16 Jul '06

At Friday 11:14 AM 9/17/2004, Balazs Scheidler wrote: >On Thu, 2004-09-16 at 19:36, Lars Kellogg-Stedman wrote: >> Text logged to a program via the program() destination has a prefix of the >> form <nnn>, which I'm assuming is the encoding of the syslog level and >> facility. Great! But I don't want it :). >> >> Is there any way to prevent this text from being logged? Sure, I can write >> a "wrapper" program to edit these messages before they reach the intended >> destination, but my template doesn't ask for this information, so I'm not >> sure it should be there. > >I think this was fixed in one of the recent versions. I just ran into this bug while using syslog-ng 1.6.0rc1 with a program destination on a Solaris system. After finding the above discussion from 2004 I upgraded to 1.6.11...but the bug is still there. Here are some samples of lines handed off from syslog-ng to the external program: <6>Jul 7 14:53:50 sys1 kernel: SysRq : Emergency Sync <4>Jul 7 14:53:50 sys1 kernel: Emergency Sync complete <6>Jul 7 14:53:51 sys2 kernel: SELinux: initialized (dev 0:15, type nfs), uses genfs_contexts I'm getting around this by editing out the prefix, but it would be good if this were fixed in syslog-ng itself. - John

2 7

syslog-ng 2.0rc1 released
by Balazs Scheidler 15 Jul '06

15 Jul '06

Dear syslog-ng users, I'm proud to announce that syslog-ng 2.0rc1 has been released. The new release is based on a completely rewritten code base, and aims to be a drop-in replacement for syslog-ng 1.6.x. Highlights over 1.6.x: - Support for extended timestamps with year, timezone information and subsecond granularity. - Flow control support: enabling flow control end-to-end, on the whole message path leaves TCP connection breaks the only source of message loss. - Better performance: various data structures and algorithms have been optimized. - IPv6 and multicast support. - and more... This release has already been tested in production environment, with high performance demands. Here is a list of user-visible changes since 1.9.11: 2.0rc1 Sat, 08 Jul 2006 13:52:50 +0200 Some minor new features: * Added LEVEL_NUM and FACILITY_NUM macros. * Added optional() support for destination files to avoid flooding the log if a given destination cannot be opened. Bugfixes: * Fixed priority range filter evaluation. * Portability fixes for various OSs (Solaris, AIX and HP-UX) and architectures (64bit CPUs) * In the case of a broken connection don't reconnect immediately, but wait some time specified by time_reopen() to match 1.6.x behaviour. * Fixed a reproducible segmentation fault in UNIX domain socket destinations. * Fixed dir_group() handling. Other changes: * Documentation updates. -- Bazsi

1 0

Bug in syntax checking in 2.0-rc1
by Paul Krizak 14 Jul '06

14 Jul '06

I'm using 2.0-rc1 and found the following bug in syntax checking: $ syslog-ng -d -F -f /opt/mpd/vtec/etc/syslog-ng.conf Error in configuration, unresolved destination reference; filter='d_syslog_pipe' But running: $ syslog-ng -s -f /opt/mpd/vtec/etc/syslog-ng.conf ; echo $? 0 Returns 0, i.e. "syntax is OK!" This just caused me a great deal of headaches, because I use a script to generate my syslog-ng.conf and I couldn't figure out why syslog-ng refused to startup...the script checked the syntax at the end and it returned fine! I believe that the -s switch should mean "if it returns zero, then when you fire up syslog-ng, it *will* start up and begin functioning." -- Paul Krizak 5900 E. Ben White Blvd. MS 625 Advanced Micro Devices Austin, TX 78741 Linux/Unix Systems Engineering Phone: (512) 602-8775 Microprocessor Solutions Sector

3 3

RE: [syslog-ng] Syslog-ng dropped packets.
by Vaibhav Goel 14 Jul '06

14 Jul '06

The other thing I forgot to mention is that yes it is syslog-ng on the receving end that is dropping packets. We are logging a lot of web data on a loghost. We have no sending syslog-ng process. We have our application that connects directly to the syslog-ng port. It is a java application and uses log4j. We wrote a custom "TCP appender" to connect to port 514 and dump the log messages there. Now it is possible I guess that its dropping messages but unlikely. We can do more investigation in that. -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: July 14, 2006 8:27 AM To: Syslog-ng users' and developers' mailing list Subject: RE: [syslog-ng] Syslog-ng dropped packets. On Fri, 2006-07-14 at 06:21 -0700, Vaibhav Goel wrote: > Hi Bazsi > > We are still having issues with dropped packets under heavy load. We > increased the size of rmem_default and rmem_max drastically but we are > not seeing much of an improvement....Maybe about 50% or so. Currently, > the sizes are > > $ > cat /proc/sys/net/core/rmem_default > 786426 > $ > cat /proc/sys/net/core/rmem_max > 1572852 > > Do you have any other suggestions? We are running syslog-ng 1.6.11. > > All the logging is done over TCP (except 1 UDP source but that doesn't > generate a lot of logging data). It would be important to know whether it is syslog-ng itself that drops messages or something else in the system. Do you have STATS lines in your logs? That shows how many lines were dropped by syslog-ng itself. It is also possible that it is the sending syslog-ng process that drops messages (in the case of TCP), you can verify this by checking the STATS line of the sending syslog-ng process. You can improve this situation somewhat by increasing log_fifo_size on the sender. 1.6.x sends messages on its TCP destination without flow control, which means that if the bandwidth of a TCP channel is less than the messages received, it could intentionally drop messages, that's what is shown in the DROPPED line. 2.0 supports flow control, which means that syslog-ng is trying to slow its inputs down to the rate of its output pipes. This could slow down applications at the end, but this is the only way to avoid dropping messages. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

RE: [syslog-ng] Syslog-ng dropped packets.
by Vaibhav Goel 14 Jul '06

14 Jul '06

Yup... Below are some of the stats. As you can see, in peak hours, the rate of the dropped messages is huge. I have increased the log_fifo_size in the global options to 4096 (from 2000) options { sync (0); time_reopen (10); log_fifo_size (4096); long_hostnames (on); use_dns (yes); } Jul 13 15:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 9445 Jul 13 15:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 18944 Jul 13 15:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 10517 Jul 13 15:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 12071 Jul 13 16:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 17505 Jul 13 16:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 12226 Jul 13 16:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 8276 Jul 13 16:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 13717 Jul 13 16:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 14688 Jul 13 16:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 4136 Jul 13 17:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 5506 Jul 13 17:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 2328 Jul 13 17:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1754 Jul 13 17:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 3253 Jul 13 17:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 2449 Jul 13 17:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1023 Jul 13 18:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1868 Jul 13 18:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1541 Jul 13 18:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1863 Jul 13 18:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 380 Jul 13 18:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1340 Jul 13 18:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1241 Jul 13 19:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1693 Jul 13 19:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1858 Jul 13 19:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 19:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 651 Jul 13 19:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 739 Jul 13 19:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1894 Jul 13 20:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 294 Jul 13 20:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 20:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 586 Jul 13 20:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 996 Jul 13 20:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 203 Jul 13 20:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 594 Jul 13 21:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 2365 Jul 13 21:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1158 Jul 13 21:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 1065 Jul 13 21:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 949 Jul 13 21:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 2341 Jul 13 21:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 98 Jul 13 22:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 382 Jul 13 22:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 50 Jul 13 22:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 84 Jul 13 22:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 293 Jul 13 22:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 54 Jul 13 22:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 351 Jul 13 23:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 6 Jul 13 23:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 493 Jul 13 23:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 23:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 23:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 13 23:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 00:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 01:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 02:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 03:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 04:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 05:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 06:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 623 Jul 14 07:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 07:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 0 Jul 14 07:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 10755 Jul 14 07:39:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 6718 Jul 14 07:49:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 17484 Jul 14 07:59:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 30382 Jul 14 08:09:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 32052 Jul 14 08:19:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 47914 Jul 14 08:29:55 s_syslogng@clxp01ml syslog-ng[29324]: STATS: dropped 44719 Jul 14 08:41:31 s_syslogng@clxp01ml syslog-ng[8094]: STATS: dropped 37327 Jul 14 08:51:31 s_syslogng@clxp01ml syslog-ng[8094]: STATS: dropped 49299 -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: July 14, 2006 8:27 AM To: Syslog-ng users' and developers' mailing list Subject: RE: [syslog-ng] Syslog-ng dropped packets. On Fri, 2006-07-14 at 06:21 -0700, Vaibhav Goel wrote: > Hi Bazsi > > We are still having issues with dropped packets under heavy load. We > increased the size of rmem_default and rmem_max drastically but we are > not seeing much of an improvement....Maybe about 50% or so. Currently, > the sizes are > > $ > cat /proc/sys/net/core/rmem_default > 786426 > $ > cat /proc/sys/net/core/rmem_max > 1572852 > > Do you have any other suggestions? We are running syslog-ng 1.6.11. > > All the logging is done over TCP (except 1 UDP source but that doesn't > generate a lot of logging data). It would be important to know whether it is syslog-ng itself that drops messages or something else in the system. Do you have STATS lines in your logs? That shows how many lines were dropped by syslog-ng itself. It is also possible that it is the sending syslog-ng process that drops messages (in the case of TCP), you can verify this by checking the STATS line of the sending syslog-ng process. You can improve this situation somewhat by increasing log_fifo_size on the sender. 1.6.x sends messages on its TCP destination without flow control, which means that if the bandwidth of a TCP channel is less than the messages received, it could intentionally drop messages, that's what is shown in the DROPPED line. 2.0 supports flow control, which means that syslog-ng is trying to slow its inputs down to the rate of its output pipes. This could slow down applications at the end, but this is the only way to avoid dropping messages. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

log statistics messages
by Maurizio B 14 Jul '06

14 Jul '06

Hello, I've successfully replaced syslog on a Fedora5 box with syslog-ng 2.0rc1. My "newbie" concern is about the file /var/log/messages filling up with log statistics messages (every ten minutes) like these: Log statistics; processed='source(s_local)=19', processed='source(s_remote)=0', processed='destination(dsl_console)=0', processed='destination(dsl_maillog)=0', processed='destination(ds_emerg)=0', processed='destination(ds_alert)=0', processed='destination(df_uucp)=0', processed='destination(ds_err)=0', processed='destination(dfs_err)=0', processed='destination(dsl_messages)=16', processed='destination(dfs_crit)=0', How can I avoid such records to be recorded? Thanks Maurizio

1 0