I was testing syslog-ng compiled with spoof_source compiled in. This was the premium version used in evaluation mode. When I use the spoof_source option to relay log data, syslog-ng will not allow me to write data locally. Is this the case whenever syslog-ng is using spoof_source acting a relay. Is it possible to write the data locally on the relay host? Please advise, Jim ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.
Hello,
I was testing syslog-ng compiled with spoof_source compiled in. This was the premium version used in evaluation mode.
When I use the spoof_source option to relay log data, syslog-ng will not allow me to write data locally. Is this the case whenever syslog-ng is using spoof_source acting a relay. Is it possible to write the data locally on the relay host?
spoof_source is usable for UDP destinations. If all you want is to omit the host information, you can use a custom template for the local file destinations. regards, Sandor -------------------------------------------------------- NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.
On Fri, 2007-09-28 at 08:53 -0400, James B Horwath wrote:
I was testing syslog-ng compiled with spoof_source compiled in. This was the premium version used in evaluation mode.
When I use the spoof_source option to relay log data, syslog-ng will not allow me to write data locally. Is this the case whenever syslog-ng is using spoof_source acting a relay. Is it possible to write the data locally on the relay host?
The premium edition starts in relay/client mode without a license. In this mode messages received from the network cannot be written to disk locally, of course disk buffer is not affected by this restriction, e.g. you can do things like: network messages -> relay (disk buffer) -> network Spoof source is completely independent of the above case, it is only used when sending out messages. For more information on the way licensing works in syslog-ng PE, read this description: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s03.html -- Bazsi
Hello, I wanted to get some guidelines on how other people are controlling their syslogs with regards to I/O performance on the database side. I am running php-syslog-ng on a separate webserver and it takes a very long time to do a search and return the results. Long time meaning 30 to 60 minutes or even longer as the logs grow. I have indexed the database by message, but it did not help. The I/O on the database server is just unmanageable. Possibilities that I'm looking into are getting 15K hard drives, change the RAID from 5 to 3, add more memory the controller's cache. Thanks, John
Might I suggest table partitioning by date? - billn On Fri, 5 Oct 2007, John Hala wrote:
Hello,
I wanted to get some guidelines on how other people are controlling their syslogs with regards to I/O performance on the database side.
I am running php-syslog-ng on a separate webserver and it takes a very long time to do a search and return the results. Long time meaning 30 to 60 minutes or even longer as the logs grow. I have indexed the database by message, but it did not help. The I/O on the database server is just unmanageable.
Possibilities that I'm looking into are getting 15K hard drives, change the RAID from 5 to 3, add more memory the controller's cache.
Thanks, John _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Thanks for the suggestion. Would you use the HASH partitioning mode, and how would you cut up the partition... by day? -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Bill Nash Sent: Friday, October 05, 2007 11:25 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Logs out of control Might I suggest table partitioning by date? - billn On Fri, 5 Oct 2007, John Hala wrote:
Hello,
I wanted to get some guidelines on how other people are controlling their syslogs with regards to I/O performance on the database side.
I am running php-syslog-ng on a separate webserver and it takes a very long time to do a search and return the results. Long time meaning 30 to 60 minutes or even longer as the logs grow. I have indexed the database by message, but it did not help. The I/O on the database server is just unmanageable.
Possibilities that I'm looking into are getting 15K hard drives, change the RAID from 5 to 3, add more memory the controller's cache.
Thanks, John _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Fri, 05 Oct 2007 15:33:30 EDT, John Hala said:
how would you cut up the partition... by day?
This would of course depend on your local requirements - but units such as "by day", "by week", "by month" certainly come to mind. You will also want to consider how easy it is to implement local policy/legal requirements such as "logs of XYZ events *must* be kept 180 days", "logs of ABC *must* be discarded after 90 days", and so on. Also add in things like "how many records per day", and "how many machines", and so on. You may want to consider using slightly larger time units, such as "month" or "last 30 days", to simplify your life when you're asked to produce log entries for "ABC for the last week", or consider some other partition such as "per machine per month" if that makes sense in your network. If it helps any, I also manage a server that tracks IDS incidents, and the useful quantities there are "last 2 hours", "last 24 hours", "this month", and "forever".
I have 170 hosts right now and plan on going up to 250. Currently getting 1.5 GB of logs per day. I think I should try to partition by day. Using php-syslog-ng, I believe most searches will either be by date or by host, or a combination of both. What type of partitioning do you recommend? I was thinking HASH partitioning. With HASH, I won't have to specify the range or exact days to partition by, whereas with HASH, I just specify the column date. In the table for the date column, I went with 'datetime'. I don't quite understand how to partition that out. Here's my newbie idea of how it should be done, however, I don't quite understand how to implement... Meaning, is it possible to partition a table that already has data in it, and if so, how to specify that particular table to partition because in the examples it just shows creating a table and following it by the partition code... ie | host | varchar(128) | YES | MUL | NULL | | | facility | varchar(10) | YES | MUL | NULL | | | priority | varchar(10) | YES | MUL | NULL | | | level | varchar(10) | YES | | NULL | | | tag | varchar(10) | YES | | NULL | | | datetime | datetime | YES | MUL | NULL | | | program | varchar(15) | YES | MUL | NULL | | | msg | text | YES | MUL | NULL | | | seq | bigint(20) unsigned | NO | PRI | NULL | auto_increment | logs | CREATE TABLE `logs` ( `host` varchar(128) DEFAULT NULL, `facility` varchar(10) DEFAULT NULL, `priority` varchar(10) DEFAULT NULL, `level` varchar(10) DEFAULT NULL, `tag` varchar(10) DEFAULT NULL, `datetime` datetime DEFAULT NULL, `program` varchar(15) DEFAULT NULL, `msg` text, `seq` bigint(20) unsigned NOT NULL AUTO_INCREMENT, PRIMARY KEY (`seq`), KEY `host` (`host`), KEY `program` (`program`), KEY `datetime` (`datetime`), KEY `priority` (`priority`), KEY `facility` (`facility`), FULLTEXT KEY `msg` (`msg`) ) ENGINE=MyISAM AUTO_INCREMENT=44728025 DEFAULT CHARSET=latin1 MAX_ROWS=4294967295 AVG_ROW_LENGTH=50 so the partition code would go like... PARTITION BY HASH(DAY(timedate)) PARTITIONS 365; PARTITION BY HASH(HOST(host)) PARTITIONS 500; --> (I am just giving a large number that I don't think the number of hosts will reach) I'm pretty certain, I'm way off, lol. Any help would be greatly appreciated! Thanks, John -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Valdis.Kletnieks@vt.edu Sent: Saturday, October 06, 2007 12:27 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Logs out of control On Fri, 05 Oct 2007 15:33:30 EDT, John Hala said:
how would you cut up the partition... by day?
This would of course depend on your local requirements - but units such as "by day", "by week", "by month" certainly come to mind. You will also want to consider how easy it is to implement local policy/legal requirements such as "logs of XYZ events *must* be kept 180 days", "logs of ABC *must* be discarded after 90 days", and so on. Also add in things like "how many records per day", and "how many machines", and so on. You may want to consider using slightly larger time units, such as "month" or "last 30 days", to simplify your life when you're asked to produce log entries for "ABC for the last week", or consider some other partition such as "per machine per month" if that makes sense in your network. If it helps any, I also manage a server that tracks IDS incidents, and the useful quantities there are "last 2 hours", "last 24 hours", "this month", and "forever".
participants (6)
-
Balazs Scheidler
-
Bill Nash
-
Geller, Sandor (IT)
-
James B Horwath
-
John Hala
-
Valdis.Kletnieks＠vt.edu