- syslog-ng - lists.balabit.hu

Noob Question
by Chuck 15 Dec '07

15 Dec '07

I used this product a long time ago and it looks like its come a long way. I had one question - I want to deploy syslog-ng on my syslog box and then have all my other servers send standarad syslog message to it remotely (my enviornment in a mix of rhel 4, rhel 5, solaris 10, and sles 9) Can syslog-ng parse the standard syslog messages and route them to individual files per server w/o adding syslog-ng itself to the client systems? (otherwise leavfe syslogd intact on all my servers except the syslog box itself) Thanks, Chuck

3 2

Loss of messages: what is the limits ?
by FILEUX arnaud 13 Dec '07

13 Dec '07

Dear all, I realized some bench tests in order to determine the limits supported in tcp connection with a syslog-ng (client/server mode) and what is the optimal configuration (buffer size,etc..). My setup is as follows: 2 syslog-ng (client A - server B) on dedicated servers with 8Go RAM and quadripro CPU. Client A is configured as follow: destination to_serverB { tcp("x.x.x.x" port(1999)); }; destination to_clientA { file("/var/log/syslog-ng.log"); }; Server B: destination all { file("/var/log/syslog-ng.log"); }; On the client A i have a a C program (named stressyslog) who send send 10.000 to 500.000 messages on 10 to 100 threads and the delay between sending is 1ms to 10 ms to Client A with : syslog(LOG_NOTICE, "go %d pThread for %d messages and %d ms", iThreads, iMessages, iDelay); I make a diff between clientA and serverB and i lose some lines on ServerB (delai less than 5ms and thread greater than 50) example: delay: 2ms ; messages: 50000; thread 100; Server B has 4947888 lines in syslog-ng.log Client A has 5000087 lines in syslog-ng.log Another problem is syslog-ng never alert if some line are lost beteween Client A and Server B. I have tried to increase the buffer size (10.000 and 100.000) but the same problem. If someone can help me i would be graceful. arnaud _____________________________________________________________________________ Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail http://mail.yahoo.fr

3 2

Eric Savidan/France/Equant is out of the office. : Eric Savidan/France/Equant is out of the office. : EricSavidan/France/Equant is out of the office. : Lossof messages: what is thelimits ?
by eric.savidan＠orange-ftgroup.com 12 Dec '07

12 Dec '07

I will be out of the office starting 12/12/2007 and will not return until 13/12/2007.

1 0

Eric Savidan/France/Equant is out of the office. : Eric Savidan/France/Equant is out of the office. : Lossof messages: what is the limits ?
by eric.savidan＠orange-ftgroup.com 12 Dec '07

12 Dec '07

I will be out of the office starting 12/12/2007 and will not return until 13/12/2007.

1 0

Eric Savidan/France/Equant is out of the office. : Loss of messages: what is the limits ?
by eric.savidan＠orange-ftgroup.com 12 Dec '07

12 Dec '07

I will be out of the office starting 12/12/2007 and will not return until 13/12/2007.

1 0

Re: [syslog-ng] December 79% OFF
by VIAGRA � Official Site 10 Dec '07

10 Dec '07

1 0

Patch for following logfiles
by Charles G Waldman 10 Dec '07

10 Dec '07

Sirs: Several months ago I submitted the following patch to syslog-ng, which makes the "file" source work even if the file is not present when syslog-ng starts up. (Consider the scenario where syslog-ng starts up before some other service which logs to a file, and that file has not yet been created yet.) This patch causes syslog-ng to treat the transition from "nonexistent" to "existent" the same as if the file were moved/renamed. We are using the patched version of syslog-ng in our project and find this modification helpful. It is a very non-intrusive change, and I cannot see any negative side-effects. I would like to know if there is any chance for this patch to be incorporated into the official distribution. Thanks for your consideration. Charles G Waldman High-Energy Physics University of Chicago cgw(a)hep.uchicago.edu Patch: (also available as http://repo.mwt2.org/viewvc/patches/syslog-ng-file-source.patch ) --- syslog-ng-2.0.4/src/affile.c 2007-04-19 14:37:16.000000000 -0500 +++ syslog-ng-2.0.4-cgw2/src/affile.c 2007-06-07 11:41:00.000000000 -0500 @@ -134,7 +134,7 @@ gint fd; msg_verbose("Follow-mode file source moved, tracking of the new file is started", - evt_tag_str("file", self->filename->str), + evt_tag_str("filename", self->filename->str), NULL); log_pipe_deinit(self->reader, NULL, NULL); @@ -170,14 +170,26 @@ { AFFileSourceDriver *self = (AFFileSourceDriver *) s; gint fd; + gboolean file_opened, open_deferred=FALSE; log_reader_options_init(&self->reader_options, cfg); - if (affile_sd_open_file(self, &fd)) + file_opened = affile_sd_open_file(self, &fd); + + if (!file_opened && self->reader_options.follow_freq > 0) + { + msg_info("Follow-mode file source not found, deferring open", + evt_tag_str("filename", self->filename->str), + NULL); + open_deferred = TRUE; + fd = -1; + } + + if (file_opened || open_deferred) { self->reader = log_reader_new(fd_read_new(fd, 0), LR_LOCAL | LR_NOMREAD, s, &self->reader_options); - if (persist) + if (persist && file_opened) { gchar *str; off_t cur_pos; --- syslog-ng-2.0.4/src/logreader.c 2007-04-20 15:11:07.000000000 -0500 +++ syslog-ng-2.0.4-cgw2/src/logreader.c 2007-06-07 11:26:12.000000000 -0500 @@ -100,29 +100,32 @@ struct stat st, followed_st; off_t pos; - pos = lseek(self->fd->fd, 0, SEEK_CUR); - if (pos == (off_t) -1) - { - msg_error("Error invoking seek on followed file", - evt_tag_errno("error", errno), - NULL); - return FALSE; - } - - if (fstat(self->fd->fd, &st) < 0) - { - msg_error("Error invoking fstat() on followed file", - evt_tag_errno("error", errno), - NULL); - return FALSE; - } + if (self->fd->fd >= 0) + { + pos = lseek(self->fd->fd, 0, SEEK_CUR); + if (pos == (off_t) -1) + { + msg_error("Error invoking seek on followed file", + evt_tag_errno("error", errno), + NULL); + return FALSE; + } + + if (fstat(self->fd->fd, &st) < 0) + { + msg_error("Error invoking fstat() on followed file", + evt_tag_errno("error", errno), + NULL); + return FALSE; + } - if (pos < st.st_size) - return TRUE; + if (pos < st.st_size) + return TRUE; + } if (self->reader->options->follow_filename && stat(self->reader->options->follow_filename, &followed_st) != -1) { - if (st.st_ino != followed_st.st_ino) + if (self->fd->fd < 0 || st.st_ino != followed_st.st_ino) { /* file was moved and we are at EOF, follow the new file */ log_pipe_notify(self->reader->control, &self->reader->super.super, NC_FILE_MOVED, self);

5 8

Please help.........Its very URGENT-----Output logs contains double line for each process
by Rohit Saxena 10 Dec '07

10 Dec '07

Hi, I m getting the logs according the Hostname.....but there is a problem...I m getting two lines for every process in the logs... Following is the example for the same...... Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: Shutting down NFS services: succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: Shutting down NFS services: succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: Starting NFS services: succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: Starting NFS services: succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: rpc.rquotad startup succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: rpc.rquotad startup succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfsd[8005]: nfssvc_versbits: +2 +3 +4 Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfsd[8005]: nfssvc_versbits: +2 +3 +4 Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: rpc.nfsd startup succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: rpc.nfsd startup succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: rpc.mountd startup succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c nfs: rpc.mountd startup succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c rpcidmapd: rpc.idmapd -SIGHUP succeeded Dec 8 19:09:10 nic-761c892613c/nic-761c892613c rpcidmapd: rpc.idmapd -SIGHUP succeeded Please check the above output and suggest me the changes to be made in the syslog-ng.conf file so as to get the proper output of the log....... _________________________________________________________________ Post free property ads on Yello Classifieds now! www.yello.in http://ss1.richmedia.in/recurl.asp?pid=221

3 2

missing logs when SIGHUP
by Martin Ervius 05 Dec '07

05 Dec '07

Hi, It seems that sometimes when syslog-ng receives a SIGHUP (like for example after logrotating or config change) it loses track of the logs that has been received, but are not yet written to disk. Except for this problem, syslog-ng can receive ~35000 logs/sec on the same machine without any loss at all. I have been able to reproduce the situation in a test environment on both FreeBSD 6.1 (Opteron) with syslog-ng 2.0.3 and on Ubuntu Linux 7.10 (i386) with syslog-ng 2.0.5. On linux, it appears that the number of logs lost equals the size of the log_fetch_limit parameter, because the number of lost logs are in multiples of this value. To see what happens I used strace to follow syslog-ng and dump read and write calls while sending numbered logs at a rate of 1000/sec. Only one machine was sending logs and they were transferred from a different machine over tcp. During this time I sent a SIGHUP to the receiving syslog with two seconds apart (since it doesn't occur with every SIGHUP). log_fetch_limit is set to 100, log_fifo_size to 51200 and log_msg_size is 2048. Flow-control is not activated since it didn't appear to make any difference in this particular case. This is the output of 'strace -s 65536 -e write,read' with the irrelevant parts omitted (I captured a similar output on FreeBSD with ktrace, but is not included in this mail) ## read from tcp socket (only beginning and end to save space) read(5, "<135>Nov 27 18:36:34 tiger logger: hejsan 9388\n<135>Nov 27 18:36:34 tiger logger: hejsan 9389\n<135>Nov 27 18:36:34 tiger logger: hejsan 9390\n<135>Nov 27 18:36:34 tiger logger: hejsan 9391\n<135>Nov 27 18:36:34 tiger logger: hejsan 9392\n<135>Nov 27 18:36:34 tiger logger: hejsan 9393\n<135>Nov 27 18:36:34 tiger logger: hejsan 9394\n<135>Nov 27 18:36:34 tiger logger: hejsan 9395\n<135>Nov 27 18:36:34 tiger logger: hejsan 9396\n<135>Nov 27 18:36:34 tiger logger: hejsan 9397\n<135>Nov 27 18:36:34 tiger logger: hejsan 9398\n<135>Nov 27 18:36:34 tiger logger: hejsan 9399\n<135>Nov 27 18:36:34 tiger logger: hejsan 9400\n<135>Nov 27 18:36:34 tiger logger: hejsan 9401\n<135>Nov 27 18:36:34 tiger logger: hejsan 9402\n<135>Nov 27 18:36:34 tiger logger: hejsan 9403\n<135>Nov 27 18:36:34 tiger logger: hejsan 9404\n<135>Nov 27 18:36:34 tiger logger: hejsan 9405\n<135>Nov 27 18:36:34 tiger logger: hejsan 9406\n<135>Nov 27 18:36:34 tiger logger: hejsan 9407\n<135>Nov 27 18:36:34 tiger logger: hejsan 9408\n<135>Nov 27 18:36:34 tiger logger: hejsan 9409\n<135>Nov 27 18:36:34 tiger logger: hejsan 9410\n . . . <135>Nov 27 18:36:34 tiger logger: hejsan 9504\n<135>Nov 27 18:36:34 tiger logger: hejsan 9505\n<135>Nov 27 18:36:3 4 tiger logger: hejsan 9506\n<135>Nov 27 18:36:34 tiger logger: hejsan 9507\n<135>Nov 27 18:36:34 tiger logger: hej ", 8192) = 5678 ## then follows all the writes until a SIGHUP is received ... write(6, "Nov 27 18:36:34 tiger logger: hejsan 9401\n", 42) = 42 write(6, "Nov 27 18:36:34 tiger logger: hejsan 9402\n", 42) = 42 write(6, "Nov 27 18:36:34 tiger logger: hejsan 9403\n", 42) = 42 write(6, "Nov 27 18:36:34 tiger logger: hejsan 9404\n", 42) = 42 write(6, "Nov 27 18:36:34 tiger logger: hejsan 9405\n", 42) = 42 write(6, "Nov 27 18:36:34 tiger logger: hejsan 9406\n", 42) = 42 write(6, "Nov 27 18:36:34 tiger logger: hejsan 9407\n", 42) = 42 write(6, "Nov 27 18:36:34 tiger logger: hejsan 9408\n", 42) = 42 --- SIGHUP (Hangup) @ 0 (0) --- ## After the config file is re-read, syslog-ng continues to read new logs from the socket, with unwritten logs from 9409 to 9507 in the buffer (9508 is partly read before the SIGHUP) read(5, "san 9508\n<135>Nov 27 18:36:34 tiger logger: hejsan 9509\n<135>Nov 27 18:36:34 tiger logger: hejsan 9510\n<135>Nov 27 18:36:34 tiger logger: hejsan 9511\n<135>Nov 27 18:36:34 tiger logger: hejsan 9512\n<135>Nov 27 18:36:34 tiger logger: hejsan 9513\n<135>Nov 27 18:36:34 tiger logger: hejsan 9514\n<135>Nov 27 18:36:34 tiger logger: hejsan 9515\n.... ## And finally new writes are done, but starting at 9508. The logs ranging from 9409 to 9507 are now lost and are consequently missing in the destination file. write(4, "Nov 27 18:36:34 tiger logger: hejsan 9508\n", 42) = 42 write(4, "Nov 27 18:36:34 tiger logger: hejsan 9509\n", 42) = 42 write(4, "Nov 27 18:36:34 tiger logger: hejsan 9510\n", 42) = 42 write(4, "Nov 27 18:36:34 tiger logger: hejsan 9511\n", 42) = 42 ----------------------------------------------------------------------------------------- I have experimented with different parameter settings, but am still losing logs at SIGHUP. I hope this information is enough if someone is able to diagnose the issue. If not, I can send my particular config etc on request. Any input on this is appreciated, has anyone else noticed this behaviour or is it maybe a configuration issue with this setup? // Martin

2 7

keep-alive question
by Josh Endries 04 Dec '07

04 Dec '07

I'm having remote connections drop with 1.6. I have my hosts log to localhost stunnel to my server. The problem seems to be when hosts rotate and HUP their daemons, it breaks the stunnel connection. I believe my log server is working after putting keep-alive on it; when I reload it, I don't see messages indicating reconnects. It doesn't on the clients, though, it gave me a syntax error. It seems keep-alive only works in source sections but not in destination sections. Is this correct? The manual online seems to suggest it would work with any tcp stream... (the man page that came with my syslog-ng doesn't mention it at all). Is this different in 2.0? Thanks, J

2 2