Re: [syslog-ng] syslog-ng as "shipper" into ELK stack
Thanks!I am on vacation this week but I look forward to trying out these suggestions. Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana@gmail.com> Date:10/06/2014 11:42 AM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Mon, Oct 6, 2014 at 9:36 AM, Radu Gheorghe <radu.gheorghe@sematext.com> wrote: Hi Jim, With rabbitmq you have the advantage that you can install the RabbitMQ river and have Elasticsearch pull logs from Rabbit instead of having another [moving] piece pull logs from Rabbit and push them to ES. So you'd have a simpler setup that also makes sure ES isn't overwhelmed (because ES is pulling). Another point that can be an advantage is that with Redis you are limited to the RAM memory available on the machine (all your data need to fit in memory) , with RabbitMQ you have disk persistence that can help in cases where you need to stop ES consumption by any reason. There are some problems with this approach: - the river only runs on one node at a time, which may become a bottleneck - rivers are deprecated (or will be) so the ES side isn't actively maintained. I've seen failover issues (node running the river goes down, another node should start the river but doesn't) which needed river delete + recreate to kick the process in again I didn't know that, I will check it out. Logstash started by recommending RabbitMQ as the queue between two Logstash instances, but now moved to Redis. Apparently the reason is that Redis plays nicely with Logstash, and Rabbit didn't, here's a quote from the guide: "Previous versions of this guide used AMQP via RabbitMQ. Due to the complexity of AMQP as well as performance issues related to the Bunny driver we use, we're now recommending Redis instead." Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Oct 4, 2014 at 5:09 AM, Jim Hendrick <jrhendri@roadrunner.com> wrote: Thanks. Why rabbitmq instead of redis? Is it faster, or does it offer some additional functions? Jim Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Alexandre Biancalana <biancalana@gmail.com> Date:10/03/2014 7:01 PM (GMT-05:00) To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng as "shipper" into ELK stack On Thu, Oct 2, 2014 at 9:33 PM, Jim Hendrick <jrhendri@roadrunner.com> wrote: Hi, I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to test it as a backend search tool for large volumes of logs. I decided to put Redis in front of Logstash as a "broker" for the incoming logs, and syslog-ng as the "shipper" so it looks like this: syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana I've been using the following: syslog-ng => rabbitmq => elasticsearch syslog-ng + patterndb to parse logs and write then in json format on rabbitmq, after that is just use elasticsearch amqp river to consume the queue. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Technically speaking, any non-alphanumeric character will terminate the TAG field at the beginning of the message. This is usually one of : [ or space as in the examples program: this is the message program[123]: this is the message program this is the message In practice though, syslog daemons will send TAGs that contain any character and the syslog-ng Agent for Windows will forward the application name as it shows in the Windows Event Log. In some cases, this TAG will contain a $ character. The patterndb-4.xsd definition disallows the $ character in the program pattern in pattern database files. Can this restriction be removed to allow for the $ or is this a larger issue that I see? Thanks again for all of the support. Evan.
I think the restriction should be removed. Viktor, do you agree? On Oct 20, 2014 12:47 AM, "Evan Rempel" <erempel@uvic.ca> wrote:
Technically speaking, any non-alphanumeric character will terminate the TAG field at the beginning of the message. This is usually one of : [ or space as in the examples
program: this is the message program[123]: this is the message program this is the message
In practice though, syslog daemons will send TAGs that contain any character and the syslog-ng Agent for Windows will forward the application name as it shows in the Windows Event Log. In some cases, this TAG will contain a $ character.
The patterndb-4.xsd definition disallows the $ character in the program pattern in pattern database files.
Can this restriction be removed to allow for the $ or is this a larger issue that I see?
Thanks again for all of the support.
Evan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi! I think we can. But I can't see in the xsd file where we diallow the $ character. patternType is an xs:string which allows every legal character, and I even managed to validate a patterndb file with $ in it's program name with xmllint. Can you please send me an example of non-validating xml file? Best Regards, Viktor On Sat, Oct 25, 2014 at 10:37 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
I think the restriction should be removed. Viktor, do you agree? On Oct 20, 2014 12:47 AM, "Evan Rempel" <erempel@uvic.ca> wrote:
Technically speaking, any non-alphanumeric character will terminate the TAG field at the beginning of the message. This is usually one of : [ or space as in the examples
program: this is the message program[123]: this is the message program this is the message
In practice though, syslog daemons will send TAGs that contain any character and the syslog-ng Agent for Windows will forward the application name as it shows in the Windows Event Log. In some cases, this TAG will contain a $ character.
The patterndb-4.xsd definition disallows the $ character in the program pattern in pattern database files.
Can this restriction be removed to allow for the $ or is this a larger issue that I see?
Thanks again for all of the support.
Evan.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
My mistake. We have a database that is used to produce the actual patterndb.xml file. Since the "class" isn't of functional importance, the code that produced the patterndb.xml file just used the "program pattern" so that all matches for a given program would use the same class. This code filtered out some of the invalid characters, but not the $. The end result is that the "class" was the component that was failing the xml validation because it contained the $. Since there is no real functional difference for what the class is set to, I am happy to ignore this limitation, but if you want to discuss it internally, I wouldn't complain if you removed this limitation on the "class". thanks for looking at this. Evan. On 10/28/2014 02:51 AM, Tusa Viktor wrote:
Hi!
I think we can. But I can't see in the xsd file where we diallow the $ character. patternType is an xs:string which allows every legal character, and I even managed to validate a patterndb file with $ in it's program name with xmllint. Can you please send me an example of non-validating xml file?
Best Regards, Viktor
On Sat, Oct 25, 2014 at 10:37 PM, Balazs Scheidler <bazsi77@gmail.com <mailto:bazsi77@gmail.com>> wrote:
I think the restriction should be removed. Viktor, do you agree?
On Oct 20, 2014 12:47 AM, "Evan Rempel" <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
Technically speaking, any non-alphanumeric character will terminate the TAG field at the beginning of the message. This is usually one of : [ or space as in the examples
program: this is the message program[123]: this is the message program this is the message
In practice though, syslog daemons will send TAGs that contain any character and the syslog-ng Agent for Windows will forward the application name as it shows in the Windows Event Log. In some cases, this TAG will contain a $ character.
The patterndb-4.xsd definition disallows the $ character in the program pattern in pattern database files.
Can this restriction be removed to allow for the $ or is this a larger issue that I see?
Thanks again for all of the support.
Evan.
participants (4)
-
Balazs Scheidler
-
Evan Rempel
-
Jim Hendrick
-
Tusa Viktor