[syslog-ng] [FORGED] Elasticscearh-http dest wish list

[Russell Fulton]

Thanks to both of you :). Fabien is right I was wondering if there was something I could do on the syslog-ng side to control the index creation.

> On 3/09/2019, at 3:06 AM, Attila Szakacs (aszakacs) <Attila.Szakacs at oneidentity.com> wrote:
> 
> Thanks Fabien, I think I understand now! 🙂
> 
> Answering to Russel:
> 
> As far as I know it is not possible to change the mapping type of an already created field in an already created index: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#update-mapping
> When started, syslog-ng does not create the index in ES, it relies on ES to create it itself with the default mapping types.
> If you want to have an index with custom mappings, you will have to create it yourself, before sending logs to it from syslog-ng.
> 
 So if I create an index in ES with the appropriate mapping then it will work.  I do this for another thing I use with ES but that does not have daily indexes just a single one.  I will have a play and report back with the results — hopefully with some useful code ;).  I can live with this…

We have some ES experts in house so I will consult.

> I can come up with a possible enhancement:
> We could give the user an option, to set multiple field mapping types when configuring the elasticsearch-http() destination, and if it is set, syslog-ng will try to create the index with the given mapping types before sending the logs. Although, it does not fit really well with the current implementation of elasticsearch-http(), it might be possible, that we can make it work.
> 
> What do you think about this idea? Is this what you are looking for? 🙂

this is what I was hoping for ;).  Even better if destination code know how the fields were parsed then set them by default.   As a software developer for the last 40 odd years I realise that that information probably is not available to the destination interface and that it would be a non trivial to retrofit.

Having IP addresses indexed as such is vital for what I am doing as it allow searches by CIDR blocks etc.   Same goes for dates and timestamps.  


> 
> Best regards,
> Attila
> From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Fabien Wernli <wernli at in2p3.fr>
> Sent: Monday, September 2, 2019 10:26 AM
> To: syslog-ng at lists.balabit.hu <syslog-ng at lists.balabit.hu>
> Subject: Re: [syslog-ng] Elasticscearh-http dest wish list
>  
> CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> Hi,
> 
> On Mon, Sep 02, 2019 at 08:08:03AM +0000, Attila Szakacs (aszakacs) wrote:
> > Please correct me, if I misunderstood something.
> 
> I think you misunderstood :)
> Russel was talking about the ES side of things : ES templates.
> The latter define the data types of fields in Elasticsearch.
> 
> See
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Findices-templates.html&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=wPPnf6uO4gKDYZT1bmZNwAa1dTeBTcOuvg5UxLLUKEE%3D&reserved=0
> 
> ______________________________________________________________________________
> Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=nhXDR7qKdda4%2Btxq5PG8%2B3TPWnPLxB5z7v4R%2B%2FuJyKc%3D&reserved=0
> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=FL8H7deOLn5iDDxURGTz4QYYg2CYcOT5g3DX2NHZftw%3D&reserved=0
> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CAttila.Szakacs%40oneidentity.com%7C8fa318fdd83541e91e2908d72f7f3cb5%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637030095857078847&sdata=uFc4qtRKfVG2FLVWWzcz4ndyibDPcfl8lYT3sS8U9zA%3D&reserved=0
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>