syslog-ng to elasticsearch
Hi, I’m having some issues trying to setup (syslog-ng v3.7.2) an elastic search destination. ES 2.2.0. In my syslog-ng.conf file, I have the destination defined as: destination d_elasticsearch { elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog-ng") class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/lib/syslog-ng-java-module-dependency-jars/jars/*.jar:/usr/share/elasticsearch/lib/*.jar:/usr/share/elasticsearch/modules/*.jar") client_mode("transport") server("172.16.100.137") port("9300") cluster("dev-elasticsearch") template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MSG)") ); }; However, in the elastic search logs, I just see an exception through on each connection attempt: [2016-03-04 06:33:00,737][WARN ][transport.netty ] [node-1] exception caught on transport layer [[id: 0xe12086b7, /172.16.100.137:52583 => /172.16.100.137:9300]], closing connection java.lang.IllegalStateException: Message not fully read (request) for requestId [0], action [cluster/state], readerIndex [34] vs expected [49]; resetting at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:120) Has anyone come across this issues previously? Regards, Mike Lewis -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. -------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors UK (“Advisors UK”). Advisors UK provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors UK and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------
Hi Mike, On Fri, Mar 04, 2016 at 11:38:27AM +0000, Mike Lewis wrote:
I’m having some issues trying to setup (syslog-ng v3.7.2) an elastic search destination. ES 2.2.0.
As stated in the last night's thread, syslog-ng 3.7.2 is not compatible with ES2.x. You'll either have to wait for syslog-ng-3.8 or build from source
Ok, thanks. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Fabien Wernli Sent: 04 March 2016 11:43 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng to elasticsearch Hi Mike, On Fri, Mar 04, 2016 at 11:38:27AM +0000, Mike Lewis wrote:
I’m having some issues trying to setup (syslog-ng v3.7.2) an elastic search destination. ES 2.2.0.
As stated in the last night's thread, syslog-ng 3.7.2 is not compatible with ES2.x. You'll either have to wait for syslog-ng-3.8 or build from source ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. -------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors UK (“Advisors UK”). Advisors UK provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors UK and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------
Hi, 3.7.2 does not work with ES 2.2 (due to some internal API change between ES2.1 and 2.2). Maybe a different issue, but you could take a look at: https://github.com/balabit/syslog-ng/issues/967 and another related github issue you can follow: https://github.com/balabit/syslog-ng/issues/970 regards, Laszlo Budai On Fri, Mar 4, 2016 at 12:38 PM, Mike Lewis <MLewis@nephilaadvisors.co.uk> wrote:
Hi,
I’m having some issues trying to setup (syslog-ng v3.7.2) an elastic search destination. ES 2.2.0.
In my syslog-ng.conf file, I have the destination defined as:
destination d_elasticsearch {
elasticsearch(
index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
type("syslog-ng")
class-path("/usr/lib64/syslog-ng/java-modules/*.jar:/usr/lib/syslog-ng-java-module-dependency-jars/jars/*.jar:/usr/share/elasticsearch/lib/*.jar:/usr/share/elasticsearch/modules/*.jar")
client_mode("transport")
server("172.16.100.137")
port("9300")
cluster("dev-elasticsearch")
template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MSG)")
);
};
However, in the elastic search logs, I just see an exception through on each connection attempt:
[2016-03-04 06:33:00,737][WARN ][transport.netty ] [node-1] exception caught on transport layer [[id: 0xe12086b7, / 172.16.100.137:52583 => /172.16.100.137:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (request) for requestId [0], action [cluster/state], readerIndex [34] vs expected [49]; resetting
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:120)
Has anyone come across this issues previously?
Regards,
Mike Lewis
--------------------------------------------------------------------------------------------------------------------------
This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement.
The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------
This email has been sent to you on behalf of Nephila Advisors UK (“Advisors UK”). Advisors UK provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors UK and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement.
The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Budai, László
-
Fabien Wernli
-
Mike Lewis