January 2007 - syslog-ng - lists.balabit.hu

AIX startup
by Brian Loe 27 Mar '08

27 Mar '08

Okay, my syslog-ng instance died at some point last night and because I don't have it setup properly for respawning... Can anyone running syslog-ng on AIX 5.* give me some tips for this? I'm feeding pipes so i have to start instances of SEC as soon as I start syslog-ng - but the SEC scripts appear to "close" so if they're configured to respawn I'll have 2000 instances of them in an hour.

7 21

incorrect firewall logs with syslogng-2.0.0 [security error?]
by Jacek Kalinski 12 Mar '08

12 Mar '08

Hello, After upgradeing from syslogng-1.6.11 to 2.0.0 I've got a strange messages in /var/log/messages file: Nov 29 08:14:05 denise WINDOW=16985 RES=0x00 ACK URGP=0 Nov 29 11:10:19 denise PT=34536 WINDOW=16985 RES=0x00 ACK URGP=0 Nov 29 16:42:16 denise W=0 RES=0x00 RST URGP=0 Nov 29 21:37:15 denise 116 ID=34901 DF PROTO=TCP SPT=3584 DPT=80 WINDOW=65535 RES=0x00 ACK FIN URGP=0 These are truncated iptables logs. Because in firewall.log file are simmilar entries to truncated one, I think full file should be: Nov 29 11:10:19 denise firewallp=INVALID:1 a=DROP IN=eth1 OUT= MAC=00:30:4f:36:2b:dc:00:04:9a:2c:7f:20:08:00 SRC=193.41.230.81 DST=xxx.xxx.xxx.x LEN=576 TOS=0x00 PREC=0x00 TTL=117 ID=3495 PROTO=TCP SPT=443 DPT=34536 WINDOW=16985 RES=0x00 ACK URGP=0 (xxx.xxx.xxx.x is a real server IP address - one 'x' is one digit) Config file with interesting lines: options { dir_perm(0700); perm(0600); chain_hostnames(no); stats(1200); keep_hostname(yes); dir_owner(root); dir_group(root); owner(root); group(root); }; source local { unix-stream("/dev/log" max_connections(300)); file("/proc/kmsg"); # udp(ip(0.0.0.0) port(514)); internal(); }; # ..... # authpriv.none;cron.none;mail.none;news.none -/var/log/messages filter f_2 { not facility(news); }; filter f_3 { not facility(local5); }; filter f_4 { not facility(local6); }; filter f_5 { not facility(authpriv); }; filter f_6 { not facility(cron); }; filter f_7 { not facility(mail); }; filter f_notlocal7 { not facility(local7); }; filter f_notfirewall { not match("firewallp="); }; destination d_2 { file("/var/log/messages" create_dirs(yes)); }; log { source(local); filter(f_2); filter(f_4); filter(f_5); filter(f_6); filter(f_7); filter(f_notimapd); filter(f_notpopa3d); filter(f_notnamed); filter(f_notfirewall); destination(d_2); }; #..... # firewallp.* -/var/log/firewall.log filter f_16 { level(warning) and match("firewallp="); }; destination d_12 { file("/var/log/firewall.log" create_dirs(yes)); }; log { source(local); filter(f_16); destination(d_12); }; #..... Do you know where could be an error? It is a production server so I can't debug syslog-ng on this machine (sorry). NOTE: the only truncated lines I can find are iptables (AFAIR they are logged by /proc/kmesg) entries. All other lines are complete. And it is probably a critical error in security. PS: simple statistics line from syslog-ng: Nov 30 00:00:06 denise syslog-ng[32682]: Log statistics; dropped='pipe(/dev/tty12)=0', processed='center(queued)=758289', processed='center(received)=369020', processed='destination(d_maillogimapd)=4245', processed='destination(d_5)=1', processed='destination(d_4)=0', processed='destination(d_3)=14259', processed='destination(d_2)=14259', processed='destination(d_13)=0', processed='destination(d_1)=6000', processed='destination(d_12)=6000', processed='destination(d_11)=369020', processed='destination(d_10)=36', processed='destination(d_8)=0', processed='destination(d_maillogpopa3d)=103224', processed='destination(d_7)=241245', processed='destination(d_6)=0', processed='source(local)=369020' I think you see that it could be a little hard to debug (with so many messages parsed by syslog-ng) ;) Jacek

3 6

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

afinet.c:: Compilation help needed on HP-UX 11i
by Shamim 02 Mar '07

02 Mar '07

afunix.c: In function 'afunix_sd_init': afunix.c:73: warning: comparison is always true due to limited range of data type if gcc -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -I/usr/local/include/eventlog -D_GNU_SOURCE -g -O2 -Wall -g -MT afinet.o -MD -MP -MF ".deps/afinet.Tpo" \ -c -o afinet.o `test -f 'afinet.c' || echo './'`afinet.c; \ then mv -f ".deps/afinet.Tpo" ".deps/afinet.Po"; \ else rm -f ".deps/afinet.Tpo"; exit 1; \ fi afinet.c: In function 'afinet_setup_socket': afinet.c:156: error: storage size of 'mreq' isn't known afinet.c:156: warning: unused variable 'mreq' *** Error exit code 1 Stop. *** Error exit code 1 Stop. *** Error exit code 1 Stop. # uname -a HP-UX wlggh010 B.11.23 U 9000/785 2006586620 unlimited-user license # Packages installed on my Machine are # swlist |grep gcc gcc 4.1.1 gcc libgcc 4.1.1 libgcc # swlist |grep glib glib 1.2.10 glib glib2 2.12.8 glib2 # --------------------------------- New Yahoo! Mail is the ultimate force in competitive emailing. Find out more at the Yahoo! Mail Championships. Plus: play games and win prizes.

2 9

Log daemon for high volume of logs
by Federico Petronio 12 Feb '07

12 Feb '07

Dear list members, I have a syslog (traditional) server that receives logs from a variety of sources. Recently I added a new host to those that sends, but this one generate high volume of logs. After adding this new one I notice that many of the logs (form the new host and from the previous never arrives to the syslog files). I did a little research and came to the idea that syslog is not able to manage the large amount of logs. I would like to know if syslog-ng is better than syslog in that subject. I know syslog-ng is better in other subjects, but currently I am really urge to solve the high volume problem and that's why I interested mainly in knowing about high volume support. Thank you very much. Best regards, -- Federico Petronio petrus(a)activesec.biz

4 4

Analysis of logs and alerting
by Muath Al Khalaf 11 Feb '07

11 Feb '07

Hi, I am implementing a syslog-ng centralized correlation server. The server receives the log entries from some servers, apps and devices. Is there any good product (commercial or open-source) with GUI or web interface to analyze these logs and give real time alerts with strong data mining tools? I need a tool which could be used by both help desk non-technical staff and technical engineers. It should give basic alerting on applications' or devices' failures first for help desk, and then engineers could dig deeper into these alerts to understand the problem. Kind regards

4 3

concatenated sendmail messages
by Fran Loehmann 02 Feb '07

02 Feb '07

Hi, I am new to syslog-ng and have set up a system using eventlog-0.2.5 and syslog-ng-2.0.1 Local sendmail messages seem to have 2 entries together. I am not sure if something is awry with the config included below, but it seems to only happen with the sendmail entries from sendmail running on the log host. I am trying to write messages to both /var/log/maillog and /var/log/archive/2007-01-25. Messages logged from the sending server seem ok but sendmail running on they log server appear to be on the same line seperated by <22>. Messages in maillog and 2007-01-25 look the same. Thanks for your help, Fran Example: --------- Jan 25 13:20:12 secmgmt-cs01 sendmail[3050]: l0PIKCSY003050: from=<root(a)secmgmt-cs02.secmgmt.pvt>, size=619, class=0, nrcpts=1, msgi d=<200701251820.l0PIKCEu004718(a)secmgmt-cs02.secmgmt.pvt>, proto=ESMTP, daemon=MTA, relay=secmgmt-cs02.secmgmt.pvt [10.200.1.5]<22>Ja n 25 13:20:12 sendmail[3051]: l0PIKCSY003050: to=<lhmn(a)secmgmt-cs01.secmgmt.pvt>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30856, dsn=2.0.0, stat=Sent Config: ------- #********** # Options #********** options { use_dns(persist-only); dns_cache_hosts(/etc/hosts); chain_hostnames(no); keep_hostname(yes); log_fifo_size(4096); log_msg_size(8192); stats(3600); sync(0); time_reopen(10); bad_hostname("sccp|tmm"); }; #********* # Sources #********* ## local source s_local { unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); internal(); }; ## remote source s_remote { udp(); tcp(ip(0.0.0.0) port(514) max-connections(300)); }; #********* # Filters #********* ## Local filters filter f_messages { level(info..emerg); }; filter f_secure { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_cron { facility(cron); }; filter f_emerg { level(emerg); }; filter f_spooler { level(crit..emerg) and facility(uucp, news); }; filter f_local7 { facility(local7); }; #filter f_debug { level(debug); }; ## SEC destination d_sec { program("/usr/local/bin/sec.pl -input=\"-\" -conf=/usr/local/etc/sec.conf"); }; ## Archive - timestamped archives destination d_archive { file("/var/log/archive/$R_YEAR-$R_MONTH-$R_DAY" owner("root") group("logs") dir_group("logs") perm(0640) dir_perm(0750) create_dirs(yes)); }; #***************** # Log statements #***************** # Send all logs to sec.pl log { source(s_local); source(s_remote); destination(d_sec); }; # Send all logs to archive log { source(s_local); source(s_remote); destination(d_archive); }; # Local logs - order DOES matter ! log { source(s_local); filter(f_emerg); destination(d_console); }; log { source(s_local); filter(f_secure); destination(d_secure); flags(final); }; log { source(s_local); filter(f_mail); destination(d_maillog); flags(final); }; log { source(s_local); filter(f_cron); destination(d_cron); flags(final); }; log { source(s_local); filter(f_spooler); destination(d_spooler); }; log { source(s_local); filter(f_local7); destination(d_bootlog); }; log { source(s_local); filter(f_messages); destination(d_messages); }; #log { source(s_local); filter(f_debug); destination(d_debug); };

3 10

Re: [syslog-ng] HAVE_GETUTENT not defined
by bryanh＠giraffe-data.com 01 Feb '07

01 Feb '07

>>> >>> BTW, I recommend not using a standard function name for a local function. >>> >>> E.g. in this case, the local version of getutent is called getutent. That >>> >>> misled me for a while, because I assumed syslog-ng was actually calling >>> >>> standard getutent. ... >>> > >>> >I don't want to add #ifdefs to the calling side of things. I know I >>> >could use macros, but then the calling side would not indicate that >>> >there's a private function implemented. >>> >>> I'm with you on that, but I was suggesting something different. There >>> is today an #ifdef in utils.c like this: >>> ... > >But how would the calling site look like? Today, it's: while ((ut = getutent())) In my scheme, it's: while ((ut = my_getutent())) (on a system that has standard getutent(), my_getutent() is implemented as a call to the system's getutent()). -- Bryan Henderson San Jose, California

2 1

Restarting destination program is dangerous
by bryanh＠giraffe-data.com 30 Jan '07

30 Jan '07

The new feature that restarts a 'program' destination when it exits is pretty dangerous. Whatever caused the program to exit might be persistent, so that every time you start it you exit, causing an infinite loop of forking and execing. I discovered this the first time I tried to use current Syslog-ng, when the specified destination program didn't exist. I think this should be optional behavior. Another idea that would help with this is not to do the restart until you have a message to deliver. That way, you'd have at most one fork per message. In lieu of an option, I am presently running modified Syslog-ng in which I simply remove the log_pipe_init() from afprogram_dd_exit(). Is that sufficient? I also think that, philosophically, Syslog-ng shouldn't be involved in restarting failed programs, because it isn't a problem special to syslog. I think it's better done by a generic monitor/restarter program. For example, I have a program called 'respawn' that spawns a child and restarts it when it dies. It has various parameters to limit how often it's willing to fork. I tell Syslog-ng destination program(respawn logstuff ...). -- Bryan Henderson San Jose, California

1 0

Re: [syslog-ng] HAVE_GETUTENT not defined
by bryanh＠giraffe-data.com 30 Jan '07

30 Jan '07

>>> BTW, I recommend not using a standard function name for a local function. >>> E.g. in this case, the local version of getutent is called getutent. That >>> misled me for a while, because I assumed syslog-ng was actually calling >>> standard getutent. It would be better for it to call syslogng_getutent() >>> and have that call the real getutent() if it's available. > >I don't want to add #ifdefs to the calling side of things. I know I >could use macros, but then the calling side would not indicate that >there's a private function implemented. I'm with you on that, but I was suggesting something different. There is today an #ifdef in utils.c like this: #ifndef HAVE_GETUTENT struct utmp *getutent(void) { ... } #endif What I think works better is: #ifndef HAVE_GETUTENT struct utmp *my_getutent(void) { ... } #else struct utmp *my_getutent(void) { return getutent(); } #endif Or put the #ifdef inside my_getutent() or make my_getutent a macro. In any case, it eliminates more confusion and with less clutter than a comment saying "this getutent might not be the one you're thinking of". -- Bryan Henderson San Jose, California

2 1