- syslog-ng - lists.balabit.hu

fix for 3.0 compile error
by Joe Shaw 18 Nov '08

18 Nov '08

Hi, I was getting an error compiling 3.0 because there was a reference to self->fd->fd which doesn't exist anymore. Attached is a patch which fixes that. Thanks, Joe

2 1

fix build without openssl
by Joe Shaw 18 Nov '08

18 Nov '08

Hi, Currently syslog-ng 3.0 from git fails to build if you don't have the right version of OpenSSL installed, even if you pass in --disable-ssl. Attached is a patch which fixes it. Thanks, Joe

2 1

Response to CVE-2008-5110
by Douglas E. Warner 18 Nov '08

18 Nov '08

Just wanted to make sure others were aware of this CVE [1] and get a response on the patch put up by Debian and if a release is planned to address this. -Doug [1] http://nvd.nist.gov/nvd.cfm?cvename=CVE-2008-5110

2 1

Re: [syslog-ng] I/O error occurred while reading; fd='4',error='Operation not permitted (1)'
by Florian Grandel 17 Nov '08

17 Nov '08

Hi Matt, Bazsi and all, I saw that the thread was not correctly attached. So here is the original post I am refering to. https://lists.balabit.hu/pipermail/syslog-ng/2007-January/009686.html Florian

1 0

keep_hostname as source option
by joël Winteregg 16 Nov '08

16 Nov '08

I All, I saw this post related to keep_hostname option: https://lists.balabit.hu/pipermail/syslog-ng/2002-November/004075.html And I'm wondering if there is now a possible keep_hostname usage as a non-global option (which would override the given global option) ? A bit as given here: source syslog { udp( port(514); keep_hostname(no); ); }; Thanks in advance, Regards, Joël

2 1

syslog-ng 2.1.2 has been released
by devel＠balabit.hu 16 Nov '08

16 Nov '08

------------------------------------------------------------------------------ PACKAGE : syslog-ng VERSION : 2.1.2 SUMMARY : new stable release DATE : Nov 16, 2008 ------------------------------------------------------------------------------ DESCRIPTION: A new stable version of syslog-ng Open Source Edition (2.1.2) has been released. For latest fixes in the 2.1.x branch you are recommended to upgrade to this version. CHANGES: 2.1.2 Wed, 12 Nov 2008 12:25:37 +0100 Bugfixes: * Fixed handling windows sizes above 65535 by changing the internal reference counter type to a 32 bit integer. * Fixed the reported port number in log statistics messages. * Fixed the internal representation of "facility" values, earlier an 8 bit value was used for both the priority and facility values, and although systems do not define facilities in this range, the LOG_FACMASK macro defines two additional bits. The type of this field was changed to 16 bits. * Fixed filter evaluation, the evaluation in AND and OR operators had a problem, it evaluated the right-hand-size of expressions first, and then the left-hand-side. This is wrong, and it easily ruins the concept of extracting information from messages via regexps. * Fixed a segmentation fault in the configuration parser on OpenBSD/amd64. All configuration options that expected a number in the parser may cause a crash, as the strdup() function went undeclared on OpenBSD, causing it to be auto-declared as a function returning "int". As a 32 bit int cannot hold a 64 bit pointer, this may cause a crash if the high-order bits of the pointer are lost. * Fixed a possible "stall" of the syslog-ng main loop, when SQL destinations are used and flow control is enabled on the log path feeding the SQL destination. * Fixed a possible 100% CPU usage case on HP-UX. HP-UX may return POLLERR on its own without the other flags set, this was not properly handled by syslog-ng, causing it to spin on the CPU. Other changes: * Instead of relying on BalaBit's version of libdbi, adapt the configure script to libdbi upstream. This currently means that instead of dbi-0.8.pc, syslog-ng is looking for dbi.pc, just as the upstream installs the pkg-config file. * Added --enable-mixed-linking option to the configure script. This functionally equivalent of the default linking option in syslog-ng 2.0. This change only makes this possibility more explicit. * Fixed some compilation warnings. * Removed the build dependency on flex 2.5.34. BINARY DOWNLOAD: If you have a binary subscription, you can download the latest binaries from: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… OR, if you have a platform that is supported by apt-get, use the following apt sources to fetch the latest releases: Debian GNU/Linux ---------------- sarge: deb https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ debian-sarge/syslog-ng-2.1 syslog-ng etch: deb https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ debian-etch/syslog-ng-2.1 syslog-ng RedHat Enterprise Linux ----------------------- RHEL-4 rpm https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ rhel-4/syslog-ng-2.1 syslog-ng SUSE 10 ------- SUSE 10.0 rpm https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ suse-10.0/syslog-ng-2.1 syslog-ng SUSE 10.1 rpm https://USERNAME:PASSWORD@apt.balabit.com/syslog-ng/open-source/ suse-10.1/syslog-ng-2.1 syslog-ng HTTP can also be used in the place of HTTPS If your version of apt-get does not support the HTTPS protocol. When using plain HTTP, the username and password will not be encrypted. SOURCE DOWNLOAD: The latest versions of syslog-ng in source format can be found at: http://www.balabit.com/downloads/files/syslog-ng/sources/2.1/src/

1 0

Send auditd info to syslog-ng in RHEL
by Phusion 13 Nov '08

13 Nov '08

Does anyone know a way to send auditd information to syslog-ng? I have a server running RHEL5 and by default it stores the info to a log file defined in the config file. By default it points to /var/log/audit/audit.log. Let me know if you have experience with this or ideas of things to try. Phusion

2 1

snmptrapd and HOST macro mapping
by joël Winteregg 13 Nov '08

13 Nov '08

Dear Syslog-ng users, I'm trying to understand (using syslog-ng community source code and some testing) if it is possible to map a given hostname located into MSGONLY macro to the FULLHOST or HOST macro. I'm trying to achieve this in order to fetch the hostname (SNMP trap source) provided by my snmptrapd collector (which is located on the same server as syslog-ng). The snmptrapd collector is logging its trap using syslog API as follow (snmptrapd source code): int log_handler_syslog( netsnmp_log_handler* logh, int pri, const char *str){ /* * XXX * We've got three items of information to work with: * Is the syslog currently open? * What ident string to use? * What facility to log to? * * We've got two "magic" locations (imagic & magic) plus the token */ if (!(logh->imagic)) { const char *ident = logh->token; int facility = (int)(intptr_t)logh->magic; if (!ident) ident = netsnmp_ds_get_string(NETSNMP_DS_LIBRARY_ID, NETSNMP_DS_LIB_APPTYPE); openlog(ident, LOG_CONS | LOG_PID, facility); logh->imagic = 1; } syslog( pri, "%s", str ); return 1; } As provided by syslog.h, there is no way to set the hostname used into syslog message (it is automatically set by the API). So my logs, provided by snmptrapd, look like the following (where HOST macro is related to the server hosting syslog-ng and snmptrapd): 12/11/2008 16:57:14 SYSLOG-COLLECTOR LEVEL=info snmptrapd[29592]: wlc02.mydomain.com Cold Start In order to be able to get wlc02.mydomain.com as $HOST or $FULLHOST macro, I set: chain_hostnames(yes); keep_hostname(yes); and I formated my snmptrapd logs MSGONLY content (using snmptrad config statements) as RFC 3164 need it: 12/11/2008 16:58:00 SYSLOG-COLLECTOR LEVEL=info snmptrapd[29592]: Nov 12 16:57:59 wlc02.mydomain.com Cold Start I thought that the above log would be seen as a relayed message by syslog-ng allowing me to fetch wlc02.mydomain.com into $HOST or $FULLHOST. Unfortunately this config provide me the following string into $FULLHOST macro: s_source@SYSLOG-COLLECTOR Does syslog-ng only use relay config statements (keep_hostname, etc.) when the log source is defined as udp() or tcp() ? I saw another hint about this here: http://osdir.com/ml/syslog-ng/2004-10/msg00062.html It says that: "If you still have your snmptrapd hostname, then you have to modify snmptrapd to use the original hostname when it sends the message about the trap" But as given above (syslog API usage into snmptrapd) how can I set the hostname of the syslog message ? Any hint is welcome ! Thanks in advance, Joël

2 6

Does syslog-ng have the ability of controlling repeated lines?
by Mao Rui 13 Nov '08

13 Nov '08

I always got multiple lines of repeated messages. Syslog can merge them into a line like 'last message repeated n times'. But syslog-ng did not. Is there any way to make syslog-ng working in the same way? Best regards, Mao Rui

3 2

fails if dns error resolve hostname destination forwarding
by JV 12 Nov '08

12 Nov '08

Tried searching in google and the mailing archives... I'm on syslog-ng-2.0.9 and it refuses to start if i have a filter like filter f_11 { level(info..emerg); }; destination d_8 { udp("sysremote" port(514)); }; log { source(s_sys); filter(f_11); destination(d_8); }; and sysremote is not resolvable via my DNS or /etc/hosts You'd think syslog-ng-2.0.9 would continue to run and do the other filters which is to log locally instead of refusing to run at all. Thats how the builtin syslog works i believe. This limits our ability to swap out replace syslog with syslog-ng since local logging should not fail just because dns is unreachable ? Am I missing something. Thanks in advance.

4 8

syslog-ng refuses to open/write destination file
by Peck, Jonathan 12 Nov '08

12 Nov '08

Hello... Any help would be greatly appreciated. I have configured syslog-ng to add a custom application filter and log destination. Let me preface all works when I configure the destination to be in /var/log. However, due to operational needs the file needs to be written off to a directory in /usr/local/ssl/<application location>. Here's the tricky part. The /usr/local/ssl/<application> is actually being exported by a NFS server as a share. I thought perhaps that was the issue as I do see a "cannot open file for writing, permission denied" vomited by syslog-ng and stamped when I kick start the application. I checked the permissions on directory and the log file. They are all good. I am able to touch, create, destroy, move, copy etc data on that directory. It only appears syslog-ng is having the trouble. What makes it more goofy is when I manipulate how I kick start the app (via shell or daemon) sometimes I see the permission error sometimes I do not. Ultimately syslog-ng still just stamps everything to 'messages'. Question, does syslog-ng have a configuration for writing off your destination file to an NFS share? I've never had this problem appear before. Perhaps someone has some experience to lend a hand? Appreciate it in advance! Cheers - Jonathan ----------------------------------------- The information in this message may be proprietary and/or confidential, and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify First Data immediately by replying to this message and deleting it from your computer.

3 3

Re: [syslog-ng] syslog-ng takes 100% CPU when network fails
by D S, Manu (STSD) 06 Nov '08

06 Nov '08

This patch fixes the issue. Thanks! - Manu > > Hi, > > > > We are running syslog-ng 2.0.9 on a HP-UX 11.31 server. We > have configured this system as a client to forward logs to a remote > server. When there is a network failure ( simulated by ifconfig down ) > syslog-ng starts to consume CPU and even after the network comes back, > it does not forward any log messages and continues to hog CPU. > > > > We did system call tracing using tusc and found that "poll()" > gets "POLLERR" event from TCP socket descriptor, but syslog-ng does not > call any socket calls for the TCP, only calls "gettimeofday()". > > > > In the logs given, TCP connection to server is disconnected > at 13:10:30. From that time, poll() receives POLLERR on the TCP socket > (fd=6) and starts loop on gettimeofday(). Attached are the sar, > netstat and tusc logs. > > First of all, Thanks for the detailed error report. > > As I see the problem seems to be caused by the fact that HP-UX returns > POLLERR only without the other bits (e.g. POLLHUP) syslog-ng would > handle this gracefully if either the other bits would be set, or > there'd > be some pending messages to send, in which case a normal write() error > would occur. > > This patch should fix the problem, although I only compile-tested it. > I'd appreciate if you could test this patch in your environment. > > diff --git a/src/logwriter.c b/src/logwriter.c > index bb82b43..7a5fcf7 100644 > --- a/src/logwriter.c > +++ b/src/logwriter.c > @@ -139,6 +139,13 @@ log_writer_fd_dispatch(GSource *source, > log_writer_broken(self->writer, NC_CLOSE); > return FALSE; > } > + else if (self->pollfd.revents & (G_IO_ERR)) > + { > + msg_error("POLLERR occurred while idle", > + evt_tag_int("fd", self->fd->fd), > + NULL); > + log_writer_broken(self->writer, NC_WRITE_ERROR); > + } > else if (self->writer->queue->length || self->writer->partial) > { > if (!log_writer_flush_log(self->writer, self->fd)) > > > > > > -- > Bazsi >

2 1

Problems compiling on RHEL 5 (flex error)
by Douglas E. Warner 06 Nov '08

06 Nov '08

I'm getting the following error trying to build packages for RHEL5: flex `test -f cfg-lex.l || echo './'`cfg-lex.l "cfg-lex.l", line 1: unrecognized '%' directive "cfg-lex.l", line 3: bad character: # "cfg-lex.l", line 3: unknown error processing section 1 "cfg-lex.l", line 3: bad character: " "cfg-lex.l", line 3: bad character: . "cfg-lex.l", line 3: bad character: " "cfg-lex.l", line 5: bad character: } make[2]: *** [cfg-lex.c] Error 1 make[2]: Any ideas on what I can do here? -Doug

2 4

[PATCH] anonymizing filter
by Micah Anderson 06 Nov '08

06 Nov '08

Hello, A couple years ago this patch was submitted to the list for consideration for inclusion into syslog-ng. I am writing this email again to request that it be considered again. The patch provides a simple replace which enables you to strip out IP addresses from your logs before they are written to disk. The patch has been included in the Debian stable distribution, and currently is included in both Debian Sid and Lenny (unstable and testing). It has had a very wide testing base and is non-intrusive, it has existed since 2004 and has been adapted to work with the newer syslog-ng. The goal of this patch is to give an organization the means to implement site logging policies, by allowing for easy control over exactly what data is retained in the logfiles. When I first requested consideration for inclusion the reactions were some suggestions for improvement (which were done), some side discussions about the various states of data retention laws, and a general agreement that this patch is non-intrusive and had a valid use case (at least in the U.S., but also likely in other countries as well[0]). The side-discussions about data-retention laws were mostly around specific geographic localities that were considering laws that would make stripping of addresses illegal, or had already mandated such things. Although these were interesting discussions, as EU data retention laws would prohibit many people from making such configuration changes to their syslog-ng.conf, they were tangential to the point because this patch does not cause those to break such laws. On the other side of the pond, in the U.S., the EFF[1] has made it very clear that this mechanism of anonymizing logs is perfectly (a) legal in the U.S., and (b) advisable. There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain. The current climate in the U.S. makes this problem so much more important now than it was many years ago. Having the ability to implement a site-policy that enables an organization to decide if the trade-off between privacy and analysis is worthwhile. This patch allows organizations to have that choice if they feel that it is more important to avoid retaining sensitive data rather than having a full history of everything logged. Please accept this patch[2], Micah [0] EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html [1] The EFF is the major civil liberties internet watchdog in the US, their "Best Practices for Online Service Providers" can be found here: http://www.eff.org/osp, they explicitly link to our patch as a recommendation [2] The latest patch can be found at https://code.autistici.org/trac/privacy/browser/trunk/syslog-ng

6 11

Re: [syslog-ng] Encoding provlem when writing to postgres
by ml＠bortal.de 05 Nov '08

05 Nov '08

Hi, where can i get syslog-ng 3.0? I cant find it on http://www.balabit.com/downloads/files/syslog-ng/sources/ Thanks, Mario Am 15.10.2008 um Uhr haben Sie geschrieben: > On Wed, 2008-10-15 at 09:53 +0200, ml(a)bortal.de wrote: > > Hello List, > > > > How can i fix this? I am running postgresql-8.1 with Debian 4.0. > > Probably the syslog client sends non-utf8 characters in its message. > syslog-ng 2.0 and 2.1 do not really care about the message contents. > > 3.0 has support for various encodings and utf8 validation like this: > > This assumes that the input is latin1: > > source s_net { udp(encoding("iso-8859-1")); }; > > And this one enforces valid utf8 messages > > source s_net { udp(flags(validate-utf8)); }; > > I was planning to add another flag, but this is not yet implemented: > > source s_net { udp(flags(force-utf8)); }; > > Which would enforce valid utf8 sequences by changing the input. > > As an alternative don't use utf8 as the character encoding of your > database, use latin1, that'll permit any kind of data in the database. >

3 7

syslog-ng .2.1.1 AIX module libglib-2.0.a(libglib-2.0.so.0) could not be loaded
by CentralRegion IPAdmin 05 Nov '08

05 Nov '08

Hi I have configured and compiled syslog-ng 2.1.1 on AIX 5.2 using the DEFAULT configure options. I can start the binary on the device where I have compiled it, but when I transfer it to another AIX device I'm getting an error saying "Could not load program ./syslog-ng.2.1.1: Dependent module libglib-2.0.a(libglib-2.0.so.0) could not be loaded. Could not load module libglib-2.0.a(libglib-2.0.so.0). System error: No such file or directory" I have tried the same with version 2.0.9 - This works fine. Do you have an idea what I have to change for 2.1.1? Thank You! Werner Keuler

4 6

No such DBI driver; type='pgsql'
by ml＠bortal.de 05 Nov '08

05 Nov '08

Hello List, i successfully compiled version 2.9.0+20081022. I used: LDFLAGS=-L/usr/local/lib LIBS=-ldbi ./configure --enable-sql But when i start syslog-ng i get this error: ------------------------------------------------------------------------- syslog-db:~# syslog-ng -F -v -f /etc/syslog-ng.conf -e -v Configuration file has no version number, assuming syslog-ng 2.1 format. Please add @version: maj.min to the beginning of the file; WARNING: global: the default value of chain_hostnames is changing to 'no' in version 3.0, please update your configuration accordingly; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='stats', change='stats_freq' WARNING: input: sources do not remove new-line characters from messages by default in version 3.0, please add 'no-multi-line' flag to your configuration if you want to retain this functionality; WARNING: file source: default value of follow_freq in file sources is changing in 3.0 to '1' for all files except /proc/kmsg; WARNING: template: the default value for template-escape is changing to 'no' in version 3.0, please update your configuration file accordingly; syslog-ng starting up; version='2.9.0+20081022' Database thread started; No such DBI driver; type='pgsql' My Syslog Config looks like this: ========================== options { chain_hostnames(off); sync(0); stats(43200); }; source src { unix-stream("/dev/log"); internal(); pipe("/proc/kmsg"); }; source src_remote { tcp(ip("foo") port(514) max-connections(20)); }; destination d_sql { # source s_net { udp(encoding("iso-8859-1")); }; # source s_net { udp(flags(validate-utf8)); }; sql(type(pgsql) host("212.87.146.204") username("foo") password("bar") database("syslogng") table("msg_table") columns("msg_rcv_time", "msg_sent_time", "hostname", "msg_facility", "msg_priority", "msg_text") values($R_ISODATE, $S_ISODATE, $HOST, $FACILITY, $PRIORITY, $MSG) ); }; destination messages { file("/var/log/messages"); }; log { source(src_remote); source(src); destination(messages); destination(d_sql); }; Any Idea how to fix this? Cheers, Mario

2 1

Real Time Alerting message from Syslog-ng!
by Wilson Lai 05 Nov '08

05 Nov '08

Hi, I am implementing the real time alert messages in Syslog-ng by using the following script : # TO=wilsonlai(a)abc.com # while read line # do # hostname=`echo $line | awk '{print $4}'`; # echo $line | /bin/mail -s "Syslog Alert Messages from HOST : $hostname" $TO # done My question is while using readline, sometimes the hostname could not be displayed correctly and give the hostname “Report” as the following event message collected : Nov 5 15:57:19 report server (mssqlserver)[error] 106 Scheduled operations cannot be created. SQL Server Agent is not running. How could it be fixed? Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email ( : wilsonlai(a)macausjm.com

1 0

Re: [syslog-ng] filter on src ip address.
by Esben Laursen 29 Oct '08

29 Oct '08

> -----Oprindelig meddelelse----- > Fra: Leonard Mills [mailto:lenm@ops.corpnet.sel.sony.com] > Sendt: 28. oktober 2008 22:24 > Til: Esben Laursen > Emne: Re: [syslog-ng] filter on src ip address. > > Esben wrote: > >Now the next question is, can I have the syslog-ng accept everyting, but > >"auto" generate a new file for each different source addresses? > > Sure, you can use $HOST as a part of the path. Something like: > > destination cacheflow { file("/c1/logs/$HOST/cacheflow.log" > template("$DATE $HOST $PROGRAM: [$FACILITY.$PRIORITY] $MSG\n")); }; > > (all one line) works just fine. Hmm, looks pretty easy.. So the $HOST is the "remote" source host and not "localhost" right? Where did you find the variable list? It was not in the man 5 syslog-ng.conf file.. Cheers Esben

2 1

Re: [syslog-ng] filter on src ip address.
by Esben Laursen 28 Oct '08

28 Oct '08

Hi Leo, Thanks for your quick reply.. Im looking at it, but it does not seem to make much sense to me.. Am I correct to assume that destination is a sort of container for what goes where? I extracted this from my Debian syslog-ng install source s_all { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" log_prefix("kernel: ")); # use the following line if you want to receive remote UDP logging messages # (this is equivalent to the "-r" syslogd flag) # udp(); }; ###### # destinations # some standard log files destination df_auth { file("/var/log/auth.log"); }; # auth,authpriv.* /var/log/auth.log log { source(s_all); filter(f_auth); destination(df_auth); }; So, I could do something like: Source MyAppliance { udp(<src IP>) }; Destination MyAppliance_log { file("/var/log/MyAppliance"); }; Log { Source(MyAppliance); Destination(MyAppliance_log); }; Does that make sense? Now the next question is, can I have the syslog-ng accept everyting, but "auto" generate a new file for each different source addresses? Cheers Esben > -----Oprindelig meddelelse----- > Fra: Leonard Mills [mailto:lenm@ops.corpnet.sel.sony.com] > Sendt: 28. oktober 2008 21:53 > Til: Esben Laursen > Emne: Re: [syslog-ng] filter on src ip address. > > You should probably start with: > > man syslog-ng.conf > > Pay particular attention to the section "DESTINATIONS" > What you want to do is _very_ easy with NG. > > Len > > On Tue, 28 Oct 2008, Esben Laursen wrote: > > > Hi Guys, > > > > I've been looking at syslog-ng's website, but cannot seem to find the > correct info. > > > > I need to have a syslog server that gather logs from different > appliances devices. Currently I just a regular syslogd, the problem is > that I need to have every appliance in a different file, and that does > syslogd not support. > > > > I've looked at the webpage for it appear that syslog-ng support this, I > look at the feature list, it says something like this: > > > > Filtering for sending IP address, check for syslog-ng. > > > > However I have no idea how to configure it or if I misunderstood the > meaning of the feature list. > > > > Could someone please point me in the correct direction? > > > > Cheers > > > > Esben > > > > > __________________________________________________________________________ > ____ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > > >

1 0

Kevin Gorski/IS/NEU is out of the office.
by K.Gorski＠neu.edu 28 Oct '08

28 Oct '08

I will be out of the office starting 10/28/2008 and will not return until 10/30/2008. I will respond to your message when I return Thursday morning.

1 0

filter on src ip address.
by Esben Laursen 28 Oct '08

28 Oct '08

Hi Guys, I've been looking at syslog-ng's website, but cannot seem to find the correct info. I need to have a syslog server that gather logs from different appliances devices. Currently I just a regular syslogd, the problem is that I need to have every appliance in a different file, and that does syslogd not support. I've looked at the webpage for it appear that syslog-ng support this, I look at the feature list, it says something like this: Filtering for sending IP address, check for syslog-ng. However I have no idea how to configure it or if I misunderstood the meaning of the feature list. Could someone please point me in the correct direction? Cheers Esben

1 0

Re: [syslog-ng] Reload does not free up RAM
by Evan Rempel 27 Oct '08

27 Oct '08

Evan Rempel wrote: > Balazs Scheidler wrote: >> On Mon, 2008-10-06 at 11:38 -0700, Evan Rempel wrote: >>> Balazs Scheidler wrote: >>>> On Sun, 2008-10-05 at 13:49 -0700, Evan Rempel wrote: >>>>> Balazs Scheidler wrote: >>>>>>>>>>> We have cases where a syslog-ng buffers quite a bit of data (2GB or so) before >>>>>>>>>>> it is able to flush the data to the destination. Currently, the only way to "free up" that RAM >>>>>>>>>>> is to restart syslog-ng, which means that some log messages are lost during the restart >>>>>>>>>>> window. >>>>>> Both PE and OSE frees up disk/memory when possible. If they don't, then >>>>>> that's probably a leak. >>> No, it isn't a memory leak as the process can be months old and only grows when >>> it buffers messages. >>> >>>>>> So if your syslog-ng process is 2GB in size, and you didn't specify an >>>>>> overly large log_fifo_size(), it's probably because of a leak. >>> >> >>>>> I have a large log_fifo_size() but the process starts small, grows when it needs >>>>> to buffer, and does NOT get small again when the destination is available. >>>> Hmm... that might be related to the malloc implementation. Which >>>> platform are you using? >>>> >>>>>>> 2. The buffer space should be thrown away only after the message >>>>>>> has been delivered to the destination. >>>>>> I see, this is what should happen now. >>> >> >>>>> This does not appear to be happening. >>>>> I am using OSE 2.0.8 Have there been memory fixes/changes since then? >>>> Not that I know of. >>>> >>>>> Is there a way to get buffer count statistics out? How many messages are >>>>> in each fifo buffer? >>>> In the experimental 3.0 branch, yes there is. In current 2.0, I'm afraid >>>> there's none. >>>> >>>> Is there a limit on the growth of syslog-ng? I mean if it gets large >>>> once and then empties its buffers, it stays large, but does it start >>>> growing again, or it stays steadily at the size it was before? >>> I have not specifically tested this scenario, but we do track processes >>> and memory usage, and it appears that once the space is allocated, it does >>> not grow unless a larger set of messages gets buffered. >>> >>> >>>> Again, the platform matters a lot, how the local malloc implementation >>>> takes care about fragmentation, whether it shrinks the process once >>>> memory is freed. >>> This is running on Redhat AS 3 (2.4 kernel) and Redhat AS 4 (2.6 kernel) >>> >>> On the Redhat 4 system the glibc packages installed are >>> >>> glibc-common-2.3.4-2.36 >>> glibc-devel-2.3.4-2.36 >>> glibc-headers-2.3.4-2.36 >>> glib2-2.4.7-1 >>> glibc-kernheaders-2.4-9.1.100.EL >>> >> hmm.. this version should be all right, at least in my experience glibc >> tries to release memory back to the system once the application freed it >> using free(). >> >> Maybe it is some kind of memory fragmentation issue. How easy is this to >> reproduce? Does it require months of running, or you have a simple >> recipe to create it? > > We are encountering this fairly easily in production, so I should be able to > verify a recipe. It will take me a couple of days to "get to it", but will report back. I thought that this deserved some attention... make a destination destination erempel { pipe("/var/log/syslog.pipes/erempel" log_fifo_size(1000000) perm(0666) ); }; and set up some logging to it. I directed only *.debug stuff to mine. filter f_debug { priority(debug); }; log { source(local); filter(f_debug); destination(erempel); flags(final); }; Start a "tail -f /var/log/syslog.pipes/erempel" and then generate some logs ls -1 /usr/bin | logger -p user.debug and I see the stuff pop out from the "tail -f" Cancel the tail process and product a whole bunch more logs. ls -1 /usr/bin | logger -p user.debug repeatedly while watching the size of the syslog process. Tue Oct 7 08:56:09 PDT 2008 root 23265 1.7 0.1 4004 1396 ? Ss 08:53 0:02 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:14 PDT 2008 root 23265 1.7 0.1 4004 1412 ? Ss 08:53 0:02 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:19 PDT 2008 root 23265 1.7 0.1 4004 1412 ? Ss 08:53 0:02 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:24 PDT 2008 root 23265 2.5 1.2 15524 12840 ? Ss 08:53 0:04 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:29 PDT 2008 root 23265 3.6 3.1 35384 32668 ? Ss 08:53 0:06 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:34 PDT 2008 root 23265 4.7 5.1 55904 53264 ? Ss 08:53 0:08 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:40 PDT 2008 root 23265 5.7 7.0 75724 73100 ? Ss 08:53 0:10 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:45 PDT 2008 root 23265 5.6 7.1 76504 73864 ? Ss 08:53 0:10 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:50 PDT 2008 root 23265 6.8 7.1 76504 73864 ? Ss 08:53 0:13 /usr/local/sbin/syslog-ng Tue Oct 7 08:56:55 PDT 2008 root 23265 6.6 7.1 76504 73864 ? Ss 08:53 0:13 /usr/local/sbin/syslog-ng The intent is to start the "rail -f" again to read all of the records and see if the memory gets smaller, but when I start the tail -f again, I get NOTHING. Syslog-ng seems to have closed the pipe destination and is NOT reopening it. Fortunately, the author produced a HUP signal that will cause all outputs to be closed and opened again. So, I do a HUP, and only the LAST 10 records are displayed. Where did all of the other records go? If I do ls -1 /usr/bin | logger -p user.debug I see the full list from the "tail -f" Back to memory. The memory never gets smaller. Now I am beginning to thing that the records got orphaned inside of syslog-ng such that they never made it to the destination, and as such are never candidates for memory reclaiming. I think I have stumbled on a bug for record loss, and I would like to state again that ALL destinations should follow the time_reopen() setting. -- Evan Rempel

3 2

syslog-ng takes 100% CPU when network fails
by D S, Manu (STSD) 26 Oct '08

26 Oct '08

Hi, We are running syslog-ng 2.0.9 on a HP-UX 11.31 server. We have configured this system as a client to forward logs to a remote server. When there is a network failure ( simulated by ifconfig down ) syslog-ng starts to consume CPU and even after the network comes back, it does not forward any log messages and continues to hog CPU. We did system call tracing using tusc and found that "poll()" gets "POLLERR" event from TCP socket descriptor, but syslog-ng does not call any socket calls for the TCP, only calls "gettimeofday()". In the logs given, TCP connection to server is disconnected at 13:10:30. From that time, poll() receives POLLERR on the TCP socket (fd=6) and starts loop on gettimeofday(). Attached are the sar, netstat and tusc logs. Thanks, Manu

2 1

syslog-ng and multiple interfaces
by brandon askme 24 Oct '08

24 Oct '08

Hello- I have successfully moved all of our servers over to syslog-ng. All but a few are reporting to the remote "logging" server I have setup. What I have found is that the clients that are not reporting have multiple interfaces, IE eth0, eth0:1, eth1 etc.. Thus the current firewall rules are not setup to allow all the ips. So my question is, can I configure syslog-ng to only use a certain ip/interface when sending the logs to the remote logging server? The clients and the remote logging server are running Linux. Thanks -Brandon

3 2