October 2007 - syslog-ng - lists.balabit.hu

Microsoft SQL
by Mike Lott 16 Oct '07

16 Oct '07

Hi all I was just wondering whether it was possible to get Syslog-NG to write to a MS SQL database cluster. I understand that the Premium Edition supports more DB's, but did not see any mention of MS SQL. Any ideas, or has anyone achieved this? Thanks, Mike

3 2

Wrong timestamp for incoming messages!
by Wilson Lai 16 Oct '07

16 Oct '07

Dear ALL, After I chrooted for syslog-ng, the timestamp displayed in the tables are wrong. How could I fix the localtime()? Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com

3 2

Ronald Jones is out of the office.
by Ronald Jones 15 Oct '07

15 Oct '07

I will be out of the office starting 10/15/2007 and will not return until 10/18/2007. I will respond to your message when I return. ----------------------------------------- ________________________________________________________ DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

1 0

Run syslog-NG as a non root user!
by Wilson Lai 15 Oct '07

15 Oct '07

Dear ALL, How could I start the Sylog-NG service as a non-root user? Is there any guideline? Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com

3 2

Mysql.pipe connection check by script!
by Wilson Lai 15 Oct '07

15 Oct '07

Dear ALL, I am writing a script to check the existence of the “MySQL –u root –password=password syslog < /tmp/mysql.pipe &”. I have tried to grep the string as the following command, #> ps aux | grep “mysql –u root” | grep “syslog” When I try to run this command, the PS cannot not show the command but “-bash”. How could I use “ps” command to show the full detail of this “-bash”. Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com

2 1

facilities problem
by Mathieu Cron 12 Oct '07

12 Oct '07

Hi, I have a problem with syslog-ng facilities. I use a program called SEC to customize logs. so my logs come to my syslog-ng server, which send them to SEC by running it, and then, SEC gives them back to syslog-ng throught a pipe, and then, syslog-ng send them into a mysql database. in the config file, I have : source src_reseau { udp(ip(0.0.0.0) port(513)); }; source src_solaris { sun-streams("/dev/log" door("/etc/.syslog_door")); }; destination d_sec { program("/var/hote/sec-2.4.1/sec.pl -input=- -conf=/etc/syslog-ng/sec.conf -log=/var/adm/sec.log"); }; log { source(src_reseau); source(src_solaris); destination(d_sec); }; and after SEC, it returns to syslog-ng source src_sec { pipe("/tmp/src_sec.pipe"); }; destination d_mysql { program("mysql -h xxx -u xxx --password=xxx syslog >/dev/null" template ("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ('$HOST' , 'auth' , '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; log { source(src_sec); destination(da_mysql); }; The problem is : Before comming back throught the pipe, facilities and levels are corrects, but after a pipe (or a file, because I experienced the same problem with a file input), facilities and levels are user.notice in all cases... Does syslog-ng incapable of reading facilities throught files and pipes, or did I missed something? Thanks in advance -- Mathieu Cron Mathieu.Cron(a)univ-rennes1.fr Centre de Ressources Informatiques http://www.univ-rennes1.fr/ Universite de Rennes 1 Tel : (33) 02.23.23.68.15 263 Av du General Leclerc CS 74205 Fax : (33) 02.23.23.71.11 35042 RENNES Cedex

2 1

Re: [syslog-ng] Logs out of control
by Brian Long 11 Oct '07

11 Oct '07

You are trying to do this the hard way. Just install splunk and let it read all your logs from a fifo. ------------------------------ Message: 2 Date: Wed, 10 Oct 2007 13:22:04 -0400 From: John Hala <john.hala(a)villanova.edu> Subject: Re: [syslog-ng] Logs out of control To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <294949F5411DFC418D1DEC84BE825AE11DB300AD6C(a)VUEX2.vuad.villanova.edu> Content-Type: text/plain; charset="us-ascii" I have 170 hosts right now and plan on going up to 250. Currently getting 1.5 GB of logs per day. I think I should try to partition by day. Using php-syslog-ng, I believe most searches will either be by date or by host, or a combination of both. What type of partitioning do you recommend? I was thinking HASH partitioning. With HASH, I won't have to specify the range or exact days to partition by, whereas with HASH, I just specify the column date. In the table for the date column, I went with 'datetime'. I don't quite understand how to partition that out. Here's my newbie idea of how it should be done, however, I don't quite understand how to implement... Meaning, is it possible to partition a table that already has data in it, and if so, how to specify that particular table to partition because in the examples it just shows creating a table and following it by the partition code... ie | host | varchar(128) | YES | MUL | NULL | | | facility | varchar(10) | YES | MUL | NULL | | | priority | varchar(10) | YES | MUL | NULL | | | level | varchar(10) | YES | | NULL | | | tag | varchar(10) | YES | | NULL | | | datetime | datetime | YES | MUL | NULL | | | program | varchar(15) | YES | MUL | NULL | | | msg | text | YES | MUL | NULL | | | seq | bigint(20) unsigned | NO | PRI | NULL | auto_increment | logs | CREATE TABLE `logs` ( `host` varchar(128) DEFAULT NULL, `facility` varchar(10) DEFAULT NULL, `priority` varchar(10) DEFAULT NULL, `level` varchar(10) DEFAULT NULL, `tag` varchar(10) DEFAULT NULL, `datetime` datetime DEFAULT NULL, `program` varchar(15) DEFAULT NULL, `msg` text, `seq` bigint(20) unsigned NOT NULL AUTO_INCREMENT, PRIMARY KEY (`seq`), KEY `host` (`host`), KEY `program` (`program`), KEY `datetime` (`datetime`), KEY `priority` (`priority`), KEY `facility` (`facility`), FULLTEXT KEY `msg` (`msg`) ) ENGINE=MyISAM AUTO_INCREMENT=44728025 DEFAULT CHARSET=latin1 MAX_ROWS=4294967295 AVG_ROW_LENGTH=50 so the partition code would go like... PARTITION BY HASH(DAY(timedate)) PARTITIONS 365; PARTITION BY HASH(HOST(host)) PARTITIONS 500; --> (I am just giving a large number that I don't think the number of hosts will reach) I'm pretty certain, I'm way off, lol. Any help would be greatly appreciated! Thanks, John -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Valdis.Kletnieks(a)vt.edu Sent: Saturday, October 06, 2007 12:27 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Logs out of control On Fri, 05 Oct 2007 15:33:30 EDT, John Hala said: > how would you cut up the partition... by day? This would of course depend on your local requirements - but units such as "by day", "by week", "by month" certainly come to mind. You will also want to consider how easy it is to implement local policy/legal requirements such as "logs of XYZ events *must* be kept 180 days", "logs of ABC *must* be discarded after 90 days", and so on. Also add in things like "how many records per day", and "how many machines", and so on. You may want to consider using slightly larger time units, such as "month" or "last 30 days", to simplify your life when you're asked to produce log entries for "ABC for the last week", or consider some other partition such as "per machine per month" if that makes sense in your network. If it helps any, I also manage a server that tracks IDS incidents, and the useful quantities there are "last 2 hours", "last 24 hours", "this month", and "forever". ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 30, Issue 8 ****************************************

2 1

Syslog-ng acting in relay mode
by James B Horwath 10 Oct '07

10 Oct '07

I was testing syslog-ng compiled with spoof_source compiled in. This was the premium version used in evaluation mode. When I use the spoof_source option to relay log data, syslog-ng will not allow me to write data locally. Is this the case whenever syslog-ng is using spoof_source acting a relay. Is it possible to write the data locally on the relay host? Please advise, Jim ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

6 7

parser
by Yüce Sungur 10 Oct '07

10 Oct '07

I am developing a reporting application based on syslog-ng log files, this is a fw log file, there is a filter in syslog-ng definition matching the "teardown" keyword, then it writes to a pipe and there is my perl code running in background and parsing the logs and inserts to db, when I check for the log file with grep -c -i teardown and my db entries there is a difference of % 0.3 you think using program() in syslog-ng is more efficient than reading pipe,or instead of reading a pipe and filtering with syslog-ng, I can tail with perl the log file, and filter by perl? any ideas?? and maximum log entries per second is around 500 and filetred results must be the half,, Yüce SUNGUR Türkiye İş Bankası A.Ş. Genel Müdürlük Bilgi İşlem Müdürlüğü İŞ Kuleleri Kule-1 Kat 3 34330 Levent/İstanbul Tel: +90 (212) 316 88 49 Fax:+90 (212) 316 0938 Cep:+90 532 748 5466 Yuce.sungur(a)isbank.com.tr

2 1

Re: [syslog-ng] E-mail alert from Syslog-NG! (Valdis.Kletnieks@vt.edu
by Wilson Lai 10 Oct '07

10 Oct '07

Dear Valdis, Could OSSEC do the same thing as I want? Thanks. Regards, Wilson Lai System Engineer IT Dept., SJM Office ( : (853)2978585 Mobile ( : (853)66506709 Email +: : wilsonlai(a)macausjm.com -----Original Message----- From: syslog-ng-request(a)lists.balabit.hu [mailto:syslog-ng-request@lists.balabit.hu] Sent: Tuesday, October 09, 2007 6:00 PM To: syslog-ng(a)lists.balabit.hu Subject: syslog-ng Digest, Vol 30, Issue 6 Send syslog-ng mailing list submissions to syslog-ng(a)lists.balabit.hu To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request(a)lists.balabit.hu You can reach the person managing the list at syslog-ng-owner(a)lists.balabit.hu When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Source-file not working (fabian marcos) 2. Re: E-mail alert from Syslog-NG! (Valdis.Kletnieks(a)vt.edu) 3. Re: Trying to build RPM (Balazs Scheidler) 4. Re: Source-file not working (Balazs Scheidler) ---------------------------------------------------------------------- Message: 1 Date: Mon, 8 Oct 2007 03:09:50 -0700 (PDT) From: fabian marcos <ositoll(a)yahoo.com> Subject: [syslog-ng] Source-file not working To: syslog-ng(a)lists.balabit.hu Message-ID: <255430.34734.qm(a)web50902.mail.re2.yahoo.com> Content-Type: text/plain; charset="iso-8859-1" Hi everyone, I have a problems with a ?source-file?. Syslog-ng can?t read my ?source-file?. I don?t know why, please help me. This is my simple syslog-ng.conf file (Vers- 1.6.11) on my Solaris 8 (Sparc.117350-16); options { mark(600); sync(0); use_dns(yes); create_dirs(yes); }; source src_tail { file("/var/log/syslog-ng/mar" ); internal(); }; source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); }; destination d_loghost_localhost { udp("10.10.10.48" port(514)); file ("/var/log/syslog-ng/$YEAR.$MONTH.$DAY/localhost.log"); }; log { source(src_tail); source(s_local); destination(d_loghost_localhost) ; }; I can see on the remote central server log "10.10.10.48" that it is working with the internal messages; 15:11:50.193397 10.10.1.36.33055 > 10.10.10.48.syslog: udp 92 (DF) 0x0000 4500 0078 df47 4000 fd11 7ec5 0a0a 0124 E..x.G@...~....$ 0x0010 0a0a 0a30 811f 0202 0064 9f5e 3c34 353e ...0.....d.^<45> 0x0020 4f63 7420 2034 2031 353a 3131 3a35 3020 Oct..4.15:11:50. 0x0030 7372 635f 7461 696c 4061 7070 7331 6d6e src_tail@testhos 0x0040 3120 7379 736c 6f67 2d6e 675b 3139 3933 t.syslog-ng[1993 0x0050 305d 0] I make a test in the local server #logger -p local3.info test1 and I can see the message on tcpdump in the remote server; 15:22:58.946246 10.10.1.36.33318 > 10.10.10.48.syslog: udp 78 (DF) 0x0000 4500 006a 014e 4000 fd11 5ccd 0a0a 0124 E..j.N@...\....$ 0x0010 0a0a 0a30 8226 0202 0056 852d 3c31 3538 ...0.&...V.-<158 0x0020 3e4f 6374 2020 3420 3135 3a32 323a 3538 >Oct..4.15:22:58 0x0030 2073 5f6c 6f63 616c 4061 7070 7331 6d6e .s_local@testhos 0x0040 3120 6d61 7266 6162 6961 3a20 5b49 4420 t.marcos:.[ID. 0x0050 3730 70 The file destination local is writing only the internal() but nothing about my file ?/var/log/syslog-ng/mar? ; #tail /var/log/syslog-ng/$YEAR.$MONTH.$DAY/localhost.log Oct 4 15:22:54 src_tail@testhost syslog-ng[23738]: syslog-ng version 1.6.11 starting Oct 4 15:32:54 src_tail@testhost syslog-ng[23738]: STATS: dropped 0 This test script is running ?while true; do date >>/var/log/syslog-ng/mar; sleep 5; done &? and it is writing every 5 seconds on my ?source file? but I can see nothing on the remote host and nothing in local host (root@testhost# snoop -d hme0 10.10.10.48) or local file. root@testhost # ps -ef|grep syslog root 28281 1 0 Sep 19 ? 0:00 /usr/sbin/syslogd root 28310 1 1 16:09:21 ? 0:00 /usr/local/sbin/syslog-ng -f /etc/syslog-ng.conf root@testhost # ls -la /var/log/syslog-ng/mar -rwxrwxrwx 1 root other 64042 Oct 4 16:09 /var/log/syslog-ng/mar Can you help me? Thanks in advance, Marcos Fabian. PS- Also when I include the option ?follow_freq(1)? on the syslog-ng.conf ; source s_tail { file("/var/log/apache/access.log" follow_freq(1) flags(no-parse)); }; I have the next error; # /usr/local/sbin/syslog-ng -d -v /etc/syslog-ng.conf syntax error at 10 Parse error reading configuration file, exiting. (line 10) --------------------------------- Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.

1 0