[syslog-ng] AMQP destination driver

[Martin Holste]

My recommendation would be to write a very simple program that does it
and use the program() driver to send your data and the JSON template
to pre-format it as JSON.  You could probably do exactly what you're
describing in 10 lines or less of Perl/Python/Ruby/etc.  Here's one in
Perl:

syslog-ng json template: $(format_json --select .classifier.* --select
usracct.* --exclude *.*id)
destination d_rabbitmq { program("/usr/local/bin/to_rabbitmq.pl"); };

#!/usr/bin/perl
use strict;
use Net::RabbitMQ;
use JSON;
my $mq = Net::RabbitMQ->new();
$mq->connect("localhost", { user => "guest", password => "guest" });
$mq->channel_open(1);
while (<>){
  $mq->publish(1, "queuename", $_ ); # message is already JSON from Syslog-NG
}
$mq->disconnect();

<shameless project promotion>
Logstash is nice and well polished.  If you find that you are getting
good results from parsing and classifying your logs with pattern-db
but ElasticSearch is too slow when you start getting higher volumes or
you want additional features like per-log permissions, transforms, and
other plugins, please check out ELSA
(enterprise-log-search-and-archive.googlecode.com).
</shameless project promotion>

On Tue, Feb 7, 2012 at 2:29 PM, Thomas Wollner <tw at wollner-net.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi List,
>
> some days ago I found logstash and I`m very impressed about the
> capabilities. It took me half an hour to have a running system and
> with patternized messages it is real fun to search and find logs.
> My plans are to classify messages with patterndb and deliver
> json-format to logstash via amqp for further indexing and searching.
> I think the options to the driver should be the same like monogdb
> destination driver, only the delivery is different. do not insert into
> database, but write it to an amqp enabled message queue like rabbitmq.
>
> are there any plans to create an AMQP destination driver?
> any thoughts on that?
>
>
> best regards,
>
> Tom
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iD8DBQFPMYmRTCCRT+dccOYRAjKmAJ9nfAtCo4aIumsK97ebmXaRM9TO/gCeO+D7
> JiHk0SOdi/E3AkzyBo462/0=
> =Xt93
> -----END PGP SIGNATURE-----
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>