Wow, it sounds like logging to files and then bulk loading into SQL is going to be the best route to go. Is LOAD DATA a syslog-ng command?<br><br><div class="gmail_quote">On Thu, Mar 10, 2011 at 3:49 PM, Martin Holste <span dir="ltr"><<a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="im">> Perhaps it is adding necessary complexity that can be done in a<br>
> separate script, but perhaps after writing X lines to a log file or<br>
> simply if it can be detected that a new file is created, as suggested by<br>
> Martin's 1-minute log file example, a command could be spawned that<br>
> will do the bulk load of the old one? Might need to be careful to<br>
> limit the number of spawned bulk loads is all.<br>
><br>
</div>That could be useful and it would be fairly simple if it were<br>
implemented as a command to run on file close, like on_file_close() or<br>
something. That would be a lot fewer moving parts than running a<br>
script as a daemon or from cron. I think the tough part for syslog-ng<br>
is that it would have to know whether it's going to reopen the file or<br>
not, which wouldn't necessarily be obvious unless you're using<br>
time-based filenames.<br>
<br>
BTW, if you use /dev/shm on Linux as your directory for temp files for<br>
the bulk loading, the logs never hit local disk and everything stays<br>
in RAM. There are of course some dangers in doing that, but<br>
file(/dev/shm/$MINUTE) + bulk load really can't be beat for DB speed.<br>
I get around 100k logs/sec inserted into MySQL using LOAD DATA. I<br>
would expect similar performance from MS-SQL, depending on the number<br>
of indexes on the table you're loading into, which is actually where<br>
your bottleneck is.<br>
<div><div></div><div class="h5">______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</div></div></blockquote></div><br>