Thanks to both of you :). Fabien is right I was wondering if there was something I could do on the syslog-ng side to control the index creation.
On 3/09/2019, at 3:06 AM, Attila Szakacs (aszakacs) <Attila.Szakacs@oneidentity.com> wrote:
Thanks Fabien, I think I understand now! 🙂
Answering to Russel:
As far as I know it is not possible to change the mapping type of an already created field in an already created index: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html... When started, syslog-ng does not create the index in ES, it relies on ES to create it itself with the default mapping types. If you want to have an index with custom mappings, you will have to create it yourself, before sending logs to it from syslog-ng.
So if I create an index in ES with the appropriate mapping then it will work. I do this for another thing I use with ES but that does not have daily indexes just a single one. I will have a play and report back with the results — hopefully with some useful code ;). I can live with this… We have some ES experts in house so I will consult.
I can come up with a possible enhancement: We could give the user an option, to set multiple field mapping types when configuring the elasticsearch-http() destination, and if it is set, syslog-ng will try to create the index with the given mapping types before sending the logs. Although, it does not fit really well with the current implementation of elasticsearch-http(), it might be possible, that we can make it work.
What do you think about this idea? Is this what you are looking for? 🙂
this is what I was hoping for ;). Even better if destination code know how the fields were parsed then set them by default. As a software developer for the last 40 odd years I realise that that information probably is not available to the destination interface and that it would be a non trivial to retrofit. Having IP addresses indexed as such is vital for what I am doing as it allow searches by CIDR blocks etc. Same goes for dates and timestamps.
Best regards, Attila From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Fabien Wernli <wernli@in2p3.fr> Sent: Monday, September 2, 2019 10:26 AM To: syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Elasticscearh-http dest wish list
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Hi,
On Mon, Sep 02, 2019 at 08:08:03AM +0000, Attila Szakacs (aszakacs) wrote:
Please correct me, if I misunderstood something.
I think you misunderstood :) Russel was talking about the ES side of things : ES templates. The latter define the data types of fields in Elasticsearch.
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=nhXDR7qKdda4%2Btxq5PG8%2B3TPWnPLxB5z7v4R%2B%2FuJyKc%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=FL8H7deOLn5iDDxURGTz4QYYg2CYcOT5g3DX2NHZftw%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=uFc4qtRKfVG2FLVWWzcz4ndyibDPcfl8lYT3sS8U9zA%3D&reserved=0
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq