So this, I assume, is how you get data into elasticsearch (to be viewable by Kibana). This should be where the message gets broken-down into ES fields. I personally have not used this approach and thus cannot offer any further suggestions other than to look at the java code itself... • destination d_elastic { • java( • class_path("/usr/local/lib/syslog-ng/java-modules/elastic.jar:/usr/share/elasticsearch/lib/*.jar:/usr/local/lib/syslog-ng/java-modules/*.jar") • class_name("org.syslog_ng.elasticsearch.ElasticSearchDestination") • • option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}") • option("type", "${SOURCEIP}") • ); • };

On Aug 14, 2015, at 08:00, Jacek Drewniak <jacek.drewniak@oort.in> wrote:

I dont use logstash. Syslog-ng on client and server side. Here is my configs: client: http://pastebin.com/wCVc2hqH server: http://pastebin.com/G6S2YV6S

-- Jacek Drewniak R&D

email: jacek.drewniak@oort.in mobile: +48 696 151 670 website: www.oort.in

AWARDS

Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner

2015-08-14 14:47 GMT+02:00 Robin Blanchard <rblanchard@nephilaadvisors.com>: Since you mention kibana, I assume you are post-processing syslog-ng with logstash? If so, what is your filter sequence/config?

...

On Aug 14, 2015, at 07:40, Jacek Drewniak <jacek.drewniak@oort.in> wrote:

Hello,

I am new in logging world. I am formating my logs according to: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g...

I am using syslog protocol.

For example I am logging this: http://pastebin.com/4UtUYiJJ But it is parsed to fields (I can see this on kibana) : http://pastebin.com/cNX8PZJp

Can You tell me what I am doing wrong? -- Jacek Drewniak R&D

email: jacek.drewniak@oort.in mobile: +48 696 151 670 website: www.oort.in

AWARDS

Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-- Robin P. Blanchard Nephila Advisors Infrastructure Administrator +1 615.823.8516 ext 4516

-------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement.

The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. -------------------------------------------------------------------------------------------------------------------------- ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-- Robin P. Blanchard Nephila Advisors Infrastructure Administrator +1 615.823.8516 ext 4516 -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------

And something else: When I add to client configuration extra _SDATA fields, they appears on kibana parsed correctly. http://pastebin.com/6ZKVTwVS So I assume that syslog-ng passes fields correctly to elastic. -- *Jacek Drewniak* R&D *email*: jacek.drewniak@oort.in *mobile*: *+**48 696 151 670* *website*: www.oort.in AWARDS Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner 2015-08-14 15:44 GMT+02:00 Jacek Drewniak <jacek.drewniak@oort.in>:

Thanks for advises.

Now my configs: http://pastebin.com/G6S2YV6S http://pastebin.com/wCVc2hqH

Sending log: http://pastebin.com/Euhp1Lmz Now its is parsed: http://pastebin.com/x46pk4FF So this didn't help.

Yes, "[TIMER]" part is also part of the message.

@Gyu I don't understand this part about length of message . Do You have link to documentation?

-- *Jacek Drewniak* R&D

*email*: jacek.drewniak@oort.in

*mobile*: *+**48 696 151 670*

*website*: www.oort.in

AWARDS

Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner

2015-08-14 15:10 GMT+02:00 PÁSZTOR György <pasztor@linux.gyakg.u-szeged.hu

...

:

...

Hi,

"Jacek Drewniak" <jacek.drewniak@oort.in> írta 2015-08-14 14:40-kor:

...

I am new in logging world. I am formating my logs according to:

https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g...

...

I am using *syslog* protocol.

For example I am logging this: http://pastebin.com/4UtUYiJJ But it is parsed to fields (I can see this on kibana) : http://pastebin.com/cNX8PZJp

Can You tell me what I am doing wrong?

Your format is not exactly the ietf syslog protocol's format. The beginning is okay, but: <15>1 2015-08-14T12:33:53Z jackahub oortApp - -

Until this point it seems okay. And now the real but: "{_SDATA:{meta:{sequenceId:jackaSEQ,hubId:123456789}}" should be formatted in this way: [meta sequenceId="jackaSEQ" hubId="123456789"]

Assuming that the "[TIMER]" part is also part of the message.

Also, please care about the transport protocol. Eg. if your transfer this over tcp/tls channel, then you have to prefix the whole with the length of this message in bytes eg. print SOCK "".length($message)." ".$message;

Cheers, Gyu

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

It works! Thank You for explaination. -- *Jacek Drewniak* R&D *email*: jacek.drewniak@oort.in *mobile*: *+**48 696 151 670* *website*: www.oort.in AWARDS Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner 2015-08-17 11:10 GMT+02:00 PÁSZTOR György <pasztor@linux.gyakg.u-szeged.hu>:

Hi,

"Jacek Drewniak" <jacek.drewniak@oort.in> írta 2015-08-14 15:44-kor:

...

Thanks for advises.

Now my configs: http://pastebin.com/G6S2YV6S http://pastebin.com/wCVc2hqH

As far as I see, you put the flags(syslog-protocol) to the wrong place. In the "syslog()" source (even if it's tls), you do not have to specify that it is syslog-protocol, since that is the defaults. flags(syslog-protocol) is needed only, when you do not have framing. But since it's tls, and the other side is a syslog-ng too, It is not needed to define flags(syslog-protocol) there too. Where you need the flags(syslog-protocol) is the file() source, since if you do not define otherwise, the default is that every line is a new message. That's why the complete line appears as the "$MESSAGE" part.

So suggested changes: * s_hub1 & s_hub2: add flags(syslog-protocol) * d_tls: remove flags(syslog-protocol) * s_tls: remove flags(syslog-protocol)

...

Sending log: http://pastebin.com/Euhp1Lmz Now its is parsed: http://pastebin.com/x46pk4FF So this didn't help.

See above!

...

Yes, "[TIMER]" part is also part of the message.

@Gyu I don't understand this part about length of message . Do You have link to documentation?

The format/ protocol specification is not part of the syslog-documentation. Since it's a protocol, they are defined in RFCs. The ietf syslog protocol itself is defined in RFC5424, and the transport related things, eg. the framing is defined in RFC5425 and RFC5426. However, the new information, which was not clear for me, for the first sight: You want to receive ietf syslog protocol from files.

One important notice: Since the source are files, and you want to receive syslog-protocol from them, but framing is not possible to define this way, you can not have multi-line messages / values in the sdata fields, since one line is one message. On udp: One packet is one message, and on tcp/tls you can have framing (unless you disable it explicitly by defining flags(syslog-protocol) ), as I already referenced the related RFCs about that.

Cheers, Gyu

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq