February 2007 - syslog-ng - lists.balabit.hu

log_msg_size() not used for syslog-ng internal messages
by Wolfram Schlich 12 Feb '07

12 Feb '07

Hi, it seems that syslog-ng does not use log_msg_size() for its own generated messages, because "Log statistics" messages are cut at exactly 1024 characters, even if 'log_msg_size(32768)' is set in syslog-ng.conf. The used syslog-ng version is 2.0.2. -- Regards, Wolfram Schlich <wschlich(a)gentoo.org> Gentoo Linux * http://dev.gentoo.org/~wschlich/

2 2

websphere
by Brian Loe 12 Feb '07

12 Feb '07

Anyone doing anything with websphere logs - especially websphere on AIX?

3 2

Log daemon for high volume of logs
by Federico Petronio 12 Feb '07

12 Feb '07

Dear list members, I have a syslog (traditional) server that receives logs from a variety of sources. Recently I added a new host to those that sends, but this one generate high volume of logs. After adding this new one I notice that many of the logs (form the new host and from the previous never arrives to the syslog files). I did a little research and came to the idea that syslog is not able to manage the large amount of logs. I would like to know if syslog-ng is better than syslog in that subject. I know syslog-ng is better in other subjects, but currently I am really urge to solve the high volume problem and that's why I interested mainly in knowing about high volume support. Thank you very much. Best regards, -- Federico Petronio petrus(a)activesec.biz

4 4

Re: [syslog-ng] user-bound UDP socket
by bryanh＠giraffe-data.com 11 Feb '07

11 Feb '07

>I am somewhat reluctant to include such a patch, so you need external >tools to actually set the fd up, For me, the ability to use an external tool to create the socket is a benefit, not a burden. I do it even where there is no privilege separation issue, because I don't think individual servers should all duplicate the same common functions; I want to rely on syslog-ng to handle syslog and socketexec to handle setting up sockets. >and it does not handle reloads. Yes, it's broken that way. And thanks for pointing it out, because it would have caused me grief. The fix looks simple enough, though: use dup() to provide syslog-ng something to close while the user's socket remains unmolested. >I would rather use some kind of dynamic capability management. (e.g. a >minimal set of syslog-ng would run as root, while the actual message >processing would happen in a restricted part. That would close the window of vulnerability substantially, but is still in a different league from a program that you don't give privileges to at all. And it adds to syslog-ng the complexity of understanding the privilege system on whatever system it's running on. Not all of them do the classic uid zero/nonzero thing. (I have some that use Linux capabilities, and the program that execs syslog-ng knows what to do with them). -- Bryan Henderson San Jose, California

1 0

I need help to understand a log in mysql db
by Fabian Pucciarelli 11 Feb '07

11 Feb '07

Can someone with a little bit more experience handling syslogs help me understanding this log: lux | syslog | notice | notice | 2d | 2007-02-11 | 13:14:10 | syslog-ng | syslog-ng[28472]: Log statistics; dropped='pipe(/tmp/mysql.pipe)=0', processed='center(queued)=133', processed='center(received)=33', processed='destination(d_mysql)=133', processed='source(net)=0', processed='source(src)=33' | 3577 | I'm getting many many of them and don't know why. TIA. -- Regards, Fabian Pucciarelli

1 0

RE: [syslog-ng] How to convert syslog-ng logs into xml format
by Padmanabhan, Rajeesh (GE Healthcare) 11 Feb '07

11 Feb '07

Hi Alex, Thanks a lot for your quick response. Infact my role here to act as 'Tech-monkey' :) I have a syslog-ng clients & server. Clients should be able to transfer logs in xml format to server. & server stores in a local location(xml file). Idea is to upload this to web & backup, which can read data in xml fomat. Could you please explain me to create a template that generate logs in xml format. Cheers! Rajeesh -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alexander Clouter Sent: Sunday, February 11, 2007 4:21 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] How to convert syslog-ng logs into xml format Hi, Padmanabhan, Rajeesh (GE Healthcare) <Rajeesh.Padmanabhan(a)ge.com> [20070211 15:40:27 +0530]: > > Hi All, > > I need to convert syslog-ng logs into xml format. Could you please > help me to do the same. If someone can send me a sample format, would > be great. > Oh dear...sounds like someone caught 'xml fever', it happens to all of us at some stage and we usually recover from the illness when its too late and the project is already using it... XML is something thats abused more often than its used properly. Its useful if you have two rather different systems that have no common data exchange mechanism between them and so XML is used as a 'common' format. Bear in mind that this means is at both ends you have to get programming tech-monkeys to write scripts to convert the data to XML and then back from XML into another format. In situations like this you do not care that it could take two weeks of solid CPU crunching to import the data, just as long as it gets there... Now as you have not given much detail about why you need the files in an XML format or what you plan on doing with the files the follow format is probably 'suitable': <?xml version="1.0" encoding="UTF-8"?> <syslog> <message>first syslog message</message> <message>second syslog message</message> ... </syslog> I'm finding it hard to think of a situation where XML and syslog data would be a good idea given that its rather easy to parse and store syslog data already. People want to archive and easily search (usually with nothing more than grep) their syslog logs and in an XML format searching your logs *will* slow down by at least a factor of 1000. A far more suitable format, if flat text files is not good enough, is a SQL database; probably the only real useful application of an SQL database[1]. I'm unsure how to help you recover from your XML fever, it was difficult for me to grow out of. I learn from my mistakes and unfortunately for me when you base a project around XML the mistakes (for me at least) do not have an effect till much later on. There is nothing on the Internet as suggested reading but I am happy to discuss this off list if you want to. I did some digging around to try to see if there was anything commercial doing this and found[2] something and was shocked at how they were doing it...its a bad schema and seems to be a solution to something that is not actually a problem. *Everything* that deals with logging data should support syslog built logfiles (you might need to use syslog-ng with its template() command to structure the logfile to be in a format that the import engine expects though), if it does not you should look for some other software. My gut feelings tell me this is an internal homebrew project though.. I hope the 'fever' breaks soon :) Cheers Alex [1] thats another rant of mine though :) [2] http://www.kiwisyslog.com/kb/idx/4/125/article/ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

3 2

How to convert syslog-ng logs into xml format
by Padmanabhan, Rajeesh (GE Healthcare) 11 Feb '07

11 Feb '07

Hi All, I need to convert syslog-ng logs into xml format. Could you please help me to do the same. If someone can send me a sample format, would be great. Thanks in advance. -Rajeesh

2 1

Analysis of logs and alerting
by Muath Al Khalaf 11 Feb '07

11 Feb '07

Hi, I am implementing a syslog-ng centralized correlation server. The server receives the log entries from some servers, apps and devices. Is there any good product (commercial or open-source) with GUI or web interface to analyze these logs and give real time alerts with strong data mining tools? I need a tool which could be used by both help desk non-technical staff and technical engineers. It should give basic alerting on applications' or devices' failures first for help desk, and then engineers could dig deeper into these alerts to understand the problem. Kind regards

4 3

Problems using match() filter
by PATEL, Leena, GBM 09 Feb '07

09 Feb '07

Hi I'm having problem using the match() filter in my syslog-ng.conf file. I have a log file below which i'm trying to seperate the events to different log files based on the filed variables "[commonservices]" and "[fx]". eb 9 13:05:28 lonrs00341/lonrs00341 [commonservices] 2007-02-09 13:05:28,131 [Thread-19] DEBUG - StringCache Party Preference Total Cache refreshed in 250ms. Feb 9 13:05:28 lonrs00341/lonrs00341 [commonservices] 2007-02-09 13:05:28,771 [Thread-27] DEBUG - Sent heartbeat Feb 9 13:05:47 lonrs00341/lonrs00341 [fx] 2007-02-09 13:05:47,242 [QuoteStore cleanup] DEBUG - [2007-02-09 13:05:47.242] [lonrs00341] [DEBUG] [QuoteStore cleanup] [unknown] [] [] [] QuoteStore: Running cleanup thread Feb 9 13:05:47 lonrs00341/lonrs00341 [fx] 2007-02-09 13:05:47,812 [Thread-7] DEBUG - [2007-02-09 13:05:47.812] [lonrs00341] [DEBUG] [Thread-7] [unknown] [] [] [] QuoteStoreNotification from [lonrs00320] to add trade 376771,Source=RBSTrade,state=Start Feb 9 13:05:47 lonrs00341/lonrs00341 [fx] 2007-02-09 13:05:47,812 [Thread-7] DEBUG - [2007-02-09 13:05:47.812] [lonrs00341] [DEBUG] [Thread-7] [unknown] [] [] [] QuoteStore: Added Quote '10065.EUR.GBP.2/13/07 12:00 AM', tradeId=376771 my syslog-ng.conf looks like this filter f_common { match("[commonservices]"); }; filter f_fx { match("[fx]"); }; filter f_core { match("[core]"); }; log { source(remote); filter(f_common); destination(common); }; log { source(remote); filter(f_fx); destination(fx); }; log { source(remote); filter(f_core); destination(core); }; however the 3 seperate log files are created but events are still all mixed up in each log file instead of being filtered accordingly. Some help would be appreciated. Thanks Leena Regards Leena Patel | Royal Bank of Scotland Global Banking & Markets | E-Commerce | Desk: +44 (0) 207 085 0692 *********************************************************************************** The Royal Bank of Scotland plc. Registered in Scotland No 90312. Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB. Authorised and regulated by the Financial Services Authority This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer. Internet e-mails are not necessarily secure. The Royal Bank of Scotland plc does not accept responsibility for changes made to this message after it was sent. Whilst all reasonable care has been taken to avoid the transmission of viruses, it is the responsibility of the recipient to ensure that the onward transmission, opening or use of this message and any attachments will not adversely affect its systems or data. No responsibility is accepted by The Royal Bank of Scotland plc in this regard and the recipient should carry out such virus and other checks as it considers appropriate. Visit our websites at: www.rbs.com www.rbsgc.com www.rbsmarkets.com ***********************************************************************************

2 1

user-bound UDP socket
by bryanh＠giraffe-data.com 09 Feb '07

09 Feb '07

Here's a patch that adds an option to the UDP source to specify the file descriptor of an already bound UDP socket, in lieu of having syslog-ng create and bind the socket. This makes it possible to run syslog-ng without the privilege required to bind to reserved UDP port numbers (such as the standard syslog port number). I exec syslog-ng with no privileges at all, except write permission to the log files. Aside from the config file processing, all the patch does is skip the call to afsocket_open_socket() if the user supplied a file descriptor, and move the settings of the nonblock and close-on-exec options from afsocket_open_socket() to afsocket_setup_socket(). -- Bryan Henderson San Jose, California diff --rec --unif syslog-ng-2.0.2+20070202/src/afinet.c syslog-ng/src/afinet.c --- syslog-ng-2.0.2+20070202/src/afinet.c 2007-02-01 23:11:48.000000000 +0000 +++ syslog-ng/src/afinet.c 2007-02-02 17:26:26.000000000 +0000 @@ -233,6 +233,15 @@ afinet_resolve_name(self->bind_addr, ip); } +void +afinet_sd_set_fd(LogDriver *s, int fd) +{ + AFSocketSourceDriver *self = (AFSocketSourceDriver *) s; + + self->user_fd = fd; + self->user_bound_socket = TRUE; +} + static gboolean afinet_sd_setup_socket(AFSocketSourceDriver *s, gint fd) { diff --rec --unif syslog-ng-2.0.2+20070202/src/afinet.h syslog-ng/src/afinet.h --- syslog-ng-2.0.2+20070202/src/afinet.h 2007-02-01 23:11:40.000000000 +0000 +++ syslog-ng/src/afinet.h 2007-02-02 17:26:26.000000000 +0000 @@ -42,6 +42,7 @@ LogDriver *afinet_sd_new(gint af, gchar *host, gint port, guint flags); void afinet_sd_set_localport(LogDriver *self, gint port, gchar *service, gchar *proto); void afinet_sd_set_localip(LogDriver *self, gchar *ip); +void afinet_sd_set_fd(LogDriver *s, int fd); #define afinet_sd_set_auth(a,b) #define afinet_sd_set_mac(a,b) --- syslog-ng-2.0.2+20070202/src/afsocket.c 2007-02-01 23:11:47.000000000 +0000 +++ syslog-ng/src/afsocket.c 2007-02-09 03:00:27.000000000 +0000 @@ -157,6 +157,9 @@ gboolean afsocket_setup_socket(gint fd, SocketOptions *sock_options, AFSocketDirection dir) { + g_fd_set_nonblock(fd, TRUE); + g_fd_set_cloexec(sock, TRUE); + if (dir & AFSOCKET_DIR_RECV) { if (sock_options->rcvbuf) @@ -183,8 +186,6 @@ else sock = socket(bind_addr->sa.sa_family, SOCK_DGRAM, 0); - g_fd_set_nonblock(sock, TRUE); - g_fd_set_cloexec(sock, TRUE); if (sock != -1) { if (g_bind(sock, bind_addr) != G_IO_STATUS_NORMAL) @@ -390,9 +391,6 @@ return TRUE; } - g_fd_set_nonblock(new_fd, TRUE); - g_fd_set_cloexec(new_fd, TRUE); - msg_verbose("Syslog connection accepted", evt_tag_str("from", g_sockaddr_format(peer_addr, buf1, sizeof(buf1))), evt_tag_str("to", g_sockaddr_format(self->bind_addr, buf2, sizeof(buf2))), @@ -492,8 +490,13 @@ { if (!self->connections) { - if (!afsocket_open_socket(self->bind_addr, !!(self->flags & AFSOCKET_STREAM), &sock)) - return self->super.optional; + if (self->user_bound_socket) + sock = self->user_fd; + else { + if (!afsocket_open_socket(self->bind_addr, !!(self->flags & AFSOCKET_STREAM), &sock)) + return self->super.optional; + } + } self->fd = -1; @@ -633,6 +636,7 @@ self->max_connections = 10; self->listen_backlog = 255; self->flags = flags; + self->user_bound_socket = FALSE; log_reader_options_defaults(&self->reader_options); } diff --rec --unif syslog-ng-2.0.2+20070202/src/afsocket.h syslog-ng/src/afsocket.h --- syslog-ng-2.0.2+20070202/src/afsocket.h 2007-02-01 23:11:47.000000000 +0000 +++ syslog-ng/src/afsocket.h 2007-02-02 17:26:26.000000000 +0000 @@ -61,6 +61,8 @@ guint source_id; LogReaderOptions reader_options; + gboolean user_bound_socket; + int user_fd; GSockAddr *bind_addr; gint max_connections; gint num_connections; diff --rec --unif syslog-ng-2.0.2+20070202/src/cfg-grammar.y syslog-ng/src/cfg-grammar.y --- syslog-ng-2.0.2+20070202/src/cfg-grammar.y 2007-02-01 23:11:47.000000000 +0000 +++ syslog-ng/src/cfg-grammar.y 2007-02-02 17:26:26.000000000 +0000 @@ -94,7 +94,7 @@ /* socket related options */ %token KW_KEEP_ALIVE KW_MAX_CONNECTIONS -%token KW_LOCALIP KW_IP KW_LOCALPORT KW_PORT KW_DESTPORT +%token KW_LOCALIP KW_IP KW_LOCALPORT KW_PORT KW_DESTPORT KW_FD %token KW_IP_TTL KW_SO_BROADCAST KW_IP_TOS KW_SO_SNDBUF KW_SO_RCVBUF KW_SO_KEEPALIVE /* misc options */ @@ -371,6 +371,7 @@ : source_afinet_option | KW_LOCALPORT '(' string ')' { afinet_sd_set_localport(last_driver, 0, $3, "udp"); free($3); } | KW_PORT '(' string ')' { afinet_sd_set_localport(last_driver, 0, $3, "udp"); free($3); } + | KW_FD '(' NUMBER ')' { afinet_sd_set_fd(last_driver, $3); } ; source_afinet_option @@ -904,4 +905,4 @@ last_reader_options = NULL; last_writer_options = NULL; last_template = NULL; -} \ No newline at end of file +} diff --rec --unif syslog-ng-2.0.2+20070202/src/cfg-lex.l syslog-ng/src/cfg-lex.l --- syslog-ng-2.0.2+20070202/src/cfg-lex.l 2007-02-01 23:11:47.000000000 +0000 +++ syslog-ng/src/cfg-lex.l 2007-02-02 17:26:26.000000000 +0000 @@ -119,6 +119,7 @@ { "localport", KW_LOCALPORT }, { "port", KW_PORT }, { "destport", KW_DESTPORT }, + { "fd", KW_FD }, { "ip_ttl", KW_IP_TTL }, { "ip_tos", KW_IP_TOS }, { "so_broadcast", KW_SO_BROADCAST },

2 1