Fields don't appear on kibana.
Hello, When I am putting new fields to elasticsearch for example using rewrite, they don't appear on kibana. But when I prefix name this fields by ".SDATA.meta" - they appear. For example: http://pastebin.com/8teCTC0x -- fields dosn't appears - kibana thinks thats are empty: http://pastebin.com/ikXMesWj -- fields appears. This same with geoip. fields. This is my kibana fields: http://i.imgur.com/pWIhsHf.png?1 Can You help me with the problem? Jacek -- *Jacek Drewniak* R&D *email*: jacek.drewniak@oort.in *mobile*: *+**48 696 151 670* *website*: www.oort.in AWARDS Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner
Hi Jacek, On Tue, Sep 01, 2015 at 10:55:13AM +0200, Jacek Drewniak wrote:
When I am putting new fields to elasticsearch for example using rewrite, they don't appear on kibana. But when I prefix name this fields by ".SDATA.meta" - they appear.
Well it depends on where you set these fields. If you do it on the host with the elasticsearch destination instance, they should appear (provided you've got the right `message_template`). However if you set them on the remote host sending the data using RFC5424, then you need to prepend the STATA bit, otherwise syslog-ng won't send them over to the elasticsearch writer.
Unfortunately I am doing this on host with elasticsearch, that is why I don't know what is wrong. Message template? My server config is: http://pastebin.com/FJzD6n77 -- *Jacek Drewniak* R&D *email*: jacek.drewniak@oort.in *mobile*: *+**48 696 151 670* *website*: www.oort.in AWARDS Bluetooth Breakthrough Award Finalist CES 2015 Envisioneering Innovation & Design Award Winner Tech Trailblazers Awards Winner Most exciting company at Bluetooth Media Event in New York 2014 Polish Agency for Enterprise Development Award Winner 2015-09-01 11:10 GMT+02:00 Fabien Wernli <wernli@in2p3.fr>:
Hi Jacek,
On Tue, Sep 01, 2015 at 10:55:13AM +0200, Jacek Drewniak wrote:
When I am putting new fields to elasticsearch for example using rewrite, they don't appear on kibana. But when I prefix name this fields by ".SDATA.meta" - they appear.
Well it depends on where you set these fields. If you do it on the host with the elasticsearch destination instance, they should appear (provided you've got the right `message_template`). However if you set them on the remote host sending the data using RFC5424, then you need to prepend the STATA bit, otherwise syslog-ng won't send them over to the elasticsearch writer.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi again, On Tue, Sep 01, 2015 at 11:28:45AM +0200, Jacek Drewniak wrote:
Unfortunately I am doing this on host with elasticsearch, that is why I don't know what is wrong. Message template?
the `message_template` option of the elastic java module controls the key-values which are sent to ES. For instance: destination d_elastic { java( option("message_template", "$(format-json -s all-nv-pairs -s rfc5424 -x DATE -k ISODATE)") ... } also remove the `option("IP", "${SOURCEIP}"` which has nothing to do in there
The best solution to send dara over the wire between two Syslog-ng instances (e.g. the one getting the logs and the other storing them in elastic) is to use json to encode name-value pairs. E.g. use format-json with some kind of prefix, and parse the json payload using json-parser() on the other side. On Sep 1, 2015 11:10 AM, "Fabien Wernli" <wernli@in2p3.fr> wrote:
Hi Jacek,
On Tue, Sep 01, 2015 at 10:55:13AM +0200, Jacek Drewniak wrote:
When I am putting new fields to elasticsearch for example using rewrite, they don't appear on kibana. But when I prefix name this fields by ".SDATA.meta" - they appear.
Well it depends on where you set these fields. If you do it on the host with the elasticsearch destination instance, they should appear (provided you've got the right `message_template`). However if you set them on the remote host sending the data using RFC5424, then you need to prepend the STATA bit, otherwise syslog-ng won't send them over to the elasticsearch writer.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Balázs, On Wed, Sep 02, 2015 at 07:16:32AM +0200, Scheidler, Balázs wrote:
The best solution to send dara over the wire between two Syslog-ng instances (e.g. the one getting the logs and the other storing them in elastic) is to use json to encode name-value pairs.
That's another way, indeed. What these have in common, though, is that there is no way to transmit TAGS from one syslog-ng instance to another properly (then use tags() filters on the remote end)
This is a huge oversight. I have "complained" about this before. A JSON source (json parser) should append all of the tags from the JSON payload into the current set of TAGS. I'm not sure about syslog protocol (new RFC) if the TAGS is prepended with the .SDATA if the syslog parser will populate the TAGS. I would hope so. Evan. On 09/02/2015 12:00 AM, Fabien Wernli wrote:
Hi Balázs,
On Wed, Sep 02, 2015 at 07:16:32AM +0200, Scheidler, Balázs wrote:
The best solution to send dara over the wire between two Syslog-ng instances (e.g. the one getting the logs and the other storing them in elastic) is to use json to encode name-value pairs. That's another way, indeed. What these have in common, though, is that there is no way to transmit TAGS from one syslog-ng instance to another properly (then use tags() filters on the remote end)
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Evan, On Wed, Sep 02, 2015 at 06:38:53AM -0700, Evan Rempel wrote:
I'm not sure about syslog protocol (new RFC) if the TAGS is prepended with the .SDATA if the syslog parser will populate the TAGS. I would hope so.
I'm afraid it won't
you are right, it is a huge oversight. can you pls suggest an on wire format how this should work? -- Bazsi This is a huge oversight. I have "complained" about this before. A JSON source (json parser) should append all of the tags from the JSON payload into the current set of TAGS. I'm not sure about syslog protocol (new RFC) if the TAGS is prepended with the .SDATA if the syslog parser will populate the TAGS. I would hope so. Evan. On 09/02/2015 12:00 AM, Fabien Wernli wrote:
Hi Balázs,
On Wed, Sep 02, 2015 at 07:16:32AM +0200, Scheidler, Balázs wrote:
The best solution to send dara over the wire between two Syslog-ng instances (e.g. the one getting the logs and the other storing them in elastic) is to use json to encode name-value pairs. That's another way, indeed. What these have in common, though, is that there is no way to transmit TAGS from one syslog-ng instance to another properly (then use tags() filters on the remote end)
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, Sep 04, 2015 at 06:42:06AM +0200, Scheidler, Balázs wrote:
you are right, it is a huge oversight. can you pls suggest an on wire format how this should work?
As far as RFC5424 is concerned, how about using a specific SDATA key with balabit's IANA enterpriseId? For JSON this could be a special prefix like `.syslog_ng.TAGS`
I'm relatively new to RFC5424 and was under the impression that all of the syslog-ng parsed values could/should be sent using the structured data mechanism. This should apply to all macros, not just TAGS. In general, what I am proposing is that any "work" done by syslog-ng on one computer, should be able to be maintained as the event gets passed to another syslog-ng computer. The work should not be lost just because the payload has moved from one system to another. I was hoping to use the syslog-ng pattern database to parse messages at the source, matching source ip addresses, user names etc. and forward that vi RFC5424 to a syslog-ng system that would then feed elastic search, passing all of the parsed values to elastic search for indexing. If that could not be done by RFC5424, I could use a raw JSON payload, and unwrap it at the receiving end, however, the source HOST, TAGS, DATE, TIME etc are all lost in this case as well. For the json parser, I think it would be good to have some kind of option for permitting core macros to be replaced/overwritten. In the case of TAGS, which is a little bit special in the json object because it is converted to a string, rather than a json list, it should be appended to. Evan. On 09/03/2015 09:42 PM, Scheidler, Balázs wrote:
you are right, it is a huge oversight. can you pls suggest an on wire format how this should work?
-- Bazsi
This is a huge oversight. I have "complained" about this before. A JSON source (json parser) should append all of the tags from the JSON payload into the current set of TAGS.
I'm not sure about syslog protocol (new RFC) if the TAGS is prepended with the .SDATA if the syslog parser will populate the TAGS. I would hope so.
Evan.
On 09/02/2015 12:00 AM, Fabien Wernli wrote:
Hi Balázs,
On Wed, Sep 02, 2015 at 07:16:32AM +0200, Scheidler, Balázs wrote:
The best solution to send dara over the wire between two Syslog-ng instances (e.g. the one getting the logs and the other storing them in elastic) is to use json to encode name-value pairs. That's another way, indeed. What these have in common, though, is that there is no way to transmit TAGS from one syslog-ng instance to another properly (then use tags() filters on the remote end)
______________________________________________________________________________
Hi Evan, Currently we are achieving this using rfc5424 by prepending all fields with `.SDATA.` on the sender and stripping it off on the receiver.
thanks for all the feedback. I agree that we should have a transport that transmits all local information to a remote syslog-ng intact. -- Bazsi On Fri, Sep 4, 2015 at 6:18 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Evan,
Currently we are achieving this using rfc5424 by prepending all fields with `.SDATA.` on the sender and stripping it off on the receiver.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Err, sent too soon. I just wanted to add, that I intend to work on this in the future.\ Cheers, Bazsi -- Bazsi On Thu, Sep 10, 2015 at 9:35 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
thanks for all the feedback. I agree that we should have a transport that transmits all local information to a remote syslog-ng intact.
-- Bazsi
On Fri, Sep 4, 2015 at 6:18 PM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Evan,
Currently we are achieving this using rfc5424 by prepending all fields with `.SDATA.` on the sender and stripping it off on the receiver.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, On Fri, Sep 04, 2015 at 08:11:16AM -0700, Evan Rempel wrote:
For the json parser, I think it would be good to have some kind of option for permitting core macros to be replaced/overwritten. In the case of TAGS, which is a little bit special in the json object because it is converted to a string, rather than a json list, it should be appended to.
Just a small addition I though useful in the case of elasticsearch: the fact that TAGS is a coma separated string is in fact Elasticsearch-friendly: if you set up a decent analyzer (e.g. the default), tags *will* get tokenized and split at the coma, so searching for TAGS:foo *will* do what you think. Of course it would be better to have syslog-ng support real arrays, but I'm sure that'll come soonish enough. Cheers
participants (4)
-
Evan Rempel
-
Fabien Wernli
-
Jacek Drewniak
-
Scheidler, Balázs