- syslog-ng - lists.balabit.hu

Question regarding filters
by Ian Meyer 03 Oct '06

03 Oct '06

I don't want to assume anything about how the program deals with messages, so my question is how do messages that match or do not match a filter cascade down? For instance if I have a filter: filter f_test { program('test'); }; and a destination that specific filter, under what conditions would it NOT make it to said destination? What's going on is I log output from a program, but it's making it's way into /var/log/messages, when it should be going to it's own definition /var/log/services/app/app.log. Does the order of log { } make a difference? filters? destinations? I looked around the faq and the reference manual and didn't find anything explaining the process flow. Thanks in advance! Ian

2 1

syslog drops stats
by Niki Hammler 02 Oct '06

02 Oct '06

Hi syslog-ng pro's, I always get messages like these: Oct 2 09:28:05 nobaq syslog-ng[2207]: STATS: dropped 548 I searched google but I could not find a solution or a good explanation why this happens. If this is true and syslog dropped 548 (!) log messages this would be bad. I think a syslog daemon is an important thing that should NOT just drop messages. I can't imagine that my system is not strong enough. 512MB RAM, 1.7Ghz. I've already set the FIFO size to a high value: $ grep size /etc/syslog-ng/syslog-ng.conf # log_fifo_size(2048); log_fifo_size(32768); log_msg_size(2048); But it does not help. Then I grepped the source to see where the problem comes from. There is a variable pktbuf_dropped_pkts and in the whole source I can not find any reference that increments this var. Is this warning uncritical? Is there any solution on this? Best regards, Niki PS: Debian sarge, v. 1.6.5

2 2

EventViewer to SysLog - looking for opinions
by Tom Valdes 01 Oct '06

01 Oct '06

I recently started evaluating tools to convert Windows Event Viewer messages to SysLog and I'm looking for opinions on the different ones and what to look for. I first started with EvtSys and it worked pretty well but it left out the hostname so it was hard to make server specific swatch statements. I then tried ntsyslog which shared the same problem. I'm now trying Snare (thanks Kevin for the tip) and it looks a lot more flexible as to what type of events get sent and it sends the hostname as well. When I first installed it, it worked fine and send messages.. After fiddling with it, it stopped working correctly. I'm going to start working with it again, but I'd like to here what else people are using. The 2 things the tool should have are: be free and transmit the Hostname. thanks, tom

3 3

RE: [syslog-ng] EventViewer to SysLog - looking for opinions
by Justin Randall 29 Sep '06

29 Sep '06

* I first started with EvtSys and it worked pretty well but it left out the hostname * so it was hard to make server specific swatch statements. I then tried ntsyslog * which shared the same problem. This shouldn't be a problem for Syslog-NG because you can have it do a DNS lookup on the source IP address of a Syslog message to get the hostname, and then use the $FULLHOST_FROM macro in your 'destination' directive to log it to a file including the hostname. Here's a link that explains more regarding the macros you can use to refine log handling: http://www.balabit.com/products/syslog_ng/reference-2.0/syslog-ng.html/i ndex.html#macros If you don't have DNS in your internal network, you can simply make the IP to hostname correlation in your /etc/hosts file. I hope this helps, Justin. _____ From: Tom Valdes [mailto:tom.valdes@gmail.com] Sent: Friday, September 29, 2006 1:06 PM To: Syslog-ng users' and developers' mailing list Subject: [syslog-ng] EventViewer to SysLog - looking for opinions I recently started evaluating tools to convert Windows Event Viewer messages to SysLog and I'm looking for opinions on the different ones and what to look for. I first started with EvtSys and it worked pretty well but it left out the hostname so it was hard to make server specific swatch statements. I then tried ntsyslog which shared the same problem. I'm now trying Snare (thanks Kevin for the tip) and it looks a lot more flexible as to what type of events get sent and it sends the hostname as well. When I first installed it, it worked fine and send messages.. After fiddling with it, it stopped working correctly. I'm going to start working with it again, but I'd like to here what else people are using. The 2 things the tool should have are: be free and transmit the Hostname. thanks, tom

1 0

create_dirs(yes) doesn't actually create directories
by Ian Meyer 29 Sep '06

29 Sep '06

[root@xxxxx ~]# syslog-ng --version syslog-ng 1.6.9 global options from syslog-ng.conf: options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); use_fqdn (no); create_dirs (yes); keep_hostname (yes); }; Here's three lines from the config, the /var/log/services directory exists, but the network dir doesn't get created. From what I understand of the reference manual.. create_dirs(yes) *should* create them, yes? I have about 30 directories, and I'd rather not create them by hand. :) Permissions on all directories above /var/log/services are correct. (rwxr-x-r-x) Any insight would be appreciated! Thanks in advance, Ian

2 1

log reporting
by Brian Loe 28 Sep '06

28 Sep '06

Anyone using scripts (like cislog) for reporting on your logs? Anyone using cislog 1.2? If so, can you help me with getting my log format to match up with what is expected? I have SEC running against pipes but wouldn't mind something like cislog to run against yesterday's logs counting and reporting on what is found. Possibly catching things my SEC rules aren't accounting for...

1 0

Syslog IP information is incorrect
by Tom Valdes 28 Sep '06

28 Sep '06

I have some machines behind a firewall VLAN of 10.0.240.0 sending logs to a Linux Syslog server on the 10.0.230.0 network. The 2 machines are 10.0.240.71 and 10.0.240.72 and the Syslog server is 10.0.230.222. They are Windows and I am using the Eventlog to Syslog utility from Purdue University ( https://engineering.purdue.edu/ECN/Resources/Documents/UNIX/evtsys) to convert the Windows event logs to Syslog. Syslog is getting the information, however, any information from the 2 machines are coming in as 10.0.230.1. ------- Sep 28 11:37:54 10.0.230.1 Service Control ....... <---- This machine is actually 10.0.240.71 ------- Is there a way to get Syslog to read the correct IP information? or does Syslog simply not pass correct host information through a router? thanks, tom

3 3

RE: [syslog-ng] Syslog-ng load testing
by Vaibhav Goel 28 Sep '06

28 Sep '06

Thanks Nate, I am on a similar path. Already hacked out a perl script and I will share it with the world when its more meaningful. So far its just generically opening a socket to syslog-ng and sending messages to it like crazy. Already seeing some interesting results.... -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Nate Campi Sent: September 28, 2006 10:12 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Syslog-ng load testing On Thu, Sep 28, 2006 at 09:36:10AM -0700, Vaibhav Goel wrote: > > I am planning to do some capacity and load testing our syslog-ng servers > over TCP. Towards that end, I will be writing a script to send packets > to a syslog-ng host. Has anyone done anything similar? I was just > thinking of writing a simple script that will open multiple sockets to > the server and send a message (based upon our current calculated average > message size). Any other suggestions are appreciated. The graphs here: http://www.campin.net/syslog-ng/faq.html#perf Are from a perl script that simply looks in syslog-ng logs for a particular "starting test" string that includes how many test messages are going to be sent, then looks for that many messages all with a sequence number appended to real-world syslog messages (a collection of about 50 messages I grabbed), then reports any numbers missing from the sequence when it gets near the total number of messages (doesn't depend on the actual last log making it in case it's lost), and reports on the time period between when the first and last message were logged to the file. Pretty simple. I did the log scraper in perl to generate those graphs via gd::graph or something like that, but you could just spit out CSV and graph it with something like (*gasp*) Excel. I can't share those scripts since they were written while working for a large, evil corporation, but the logic is quite simple. I had it measure CPU and also network stats (network stats were a separate graph not pictured on the site) but that's certainly not mandatory. Just make sure that if you measure CPU that you also see how much CPU the testing/analysis script uses - might be more than you'd think. -- Nate There are two ways to write error-free programs. Only the third one works. -Anon. _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

DNS and hosts
by Hari Sekhon 28 Sep '06

28 Sep '06

I have a logserver with a mysql back-end and php-syslog-ng front-end. I was doing some work on DNS (migration to another machine, another version) yesterday and it was down for a little while. I noticed today that in the logserver interface I have loads of ip addresses instead of hostnames. So I had to go through and change the fields for all those hosts manually, which was quite annoying. In order to stop this from happening again I was wondering what steps I could take. My logserver conf options section is as follows: options { chain_hostnames(off); sync(10); stats(43200); keep_hostname(no); use_dns(yes); dns_cache(yes); create_dirs(yes); log_fifo_size(5000); }; If I change keep_hostname() to yes will I still get validation via dns or not? I think not judging from the docs. I was wondering if perhaps it would use dns and if unavailable it would use the name from the logs. Wishful thinking? On a DNS front, I was wondering if I could just copy a hosts file with all the dns names in it to the /etc/hosts of the linux system running the logserver. Would this work? Would syslog-ng obey the nsswitch of the linux system and use the hosts file first? Or does it have to do a dns request when use_dns(yes) is the options{}; ? This way, I'll never have this problem again if I need to fiddle the DNS server. I already have a hosts file generated when I update my dns server records so this is ready to go if it will work... All feedback welcome. -h -- Hari Sekhon

3 4

Forwarding to a Loglogic device
by Dukes Clayton 28 Sep '06

28 Sep '06

Argh... Can someone tell me why a sniffer shows no traffic passing to the destination when using this config? I've tried everything I can think of...what am I doing wrong? ############################## # Loglogic dump ############################## options { sync(0); log_fifo_size(1024); chain_hostnames(no); use_fqdn(yes); use_dns(yes); dns_cache(yes); dns_cache_expire(300); dns_cache_expire_failed(10); dns_cache_size(1024); keep_hostname(yes); }; source snet { tcp(); udp(); }; filter f_loglogic { level(debug..emerg) and facility(local0, local1, local2, local3, local4, local6, local7); }; destination LogLogic { tcp("10.24.2.39" port(514) log_fifo_size(60000) template("<$PRI> $R_DATE $SOURCEIP $MSG\n") ); }; log { source(snet); filter(f_loglogic); destination(LogLogic); }; ############################## # END Loglogic dump ############################## When I telnet to my router and generate a SYS-5-CONFIG, I see the packet come into the server: #----Start capture tcpdump dst port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 10:17:33.211813 IP 10.26.31.2.52935 > syslog.net.syslog: SYSLOG local7.notice, length: 105 148 packets captured 297 packets received by filter 0 packets dropped by kernel #----End capture But when I try to sniff packets going to the destination server, I see nothing: /etc/init.d/syslog-ng restart * Stopping syslog-ng ... [ ok ] * Starting syslog-ng ... #----Start capture - note that the entries below are from syslog-ng restarting, so I know it establishes a connection... tcpdump dst host 10.24.2.39 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 10:24:40.401037 IP syslog.net.56631 > .shell: F 1487912722:1487912722(0) ack 91904650 win 1460 <nop,nop,timestamp 866281169 157842259> 10:24:40.407670 IP syslog.net.56631 > .shell: . ack 2 win 1460 <nop,nop,timestamp 866281177 158151244> 10:24:43.014989 IP syslog.net.42848 > .shell: S 1805017512:1805017512(0) win 5840 <mss 1460,sackOK,timestamp 866283785 0,nop,wscale 2> 10:24:43.015885 IP syslog.net.42848 > .shell: . ack 425051406 win 1460 <nop,nop,timestamp 866283786 158153852> #----End capture What am I missing here? Thanks! Clayton

1 1