[syslog-ng] [EXTERNAL] Re: Failover/Hi Availability

Mon Jan 18 19:47:57 UTC 2021

Thanks!  The reason we use syslog-ng is so that we can do pre-processing before it gets into Splunk, otherwise we have to send more data to Splunk and that costs more.   Though, if I'm reading these blog posts correctly, with SC4S we can do all of the same filtering that we are doing now.  I will ask around, hopefully there are not additional licensing costs.

-Mark

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Attila Szakacs (aszakacs)
Sent: Monday, January 18, 2021 06:06
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] [EXTERNAL] Re: Failover/Hi Availability

I see, thanks!

You mentioned that you want to send to Splunk. I don't think writing the same file on a network storage is the way to go. Have you read the blogpost about sending to Splunk with syslog-ng through http event collectors (HEC) [1]? I believe, you can send to any number of HEC instances from any number of syslog-ng servers.

Nonetheless, I am not an expert in Splunk, but they have a Slack channel [2], where I have heard there are syslog-ng related discussions. It might be worth a try.

Cheers,
Attila

[1] https://www.splunk.com/en_us/blog/tips-and-tricks/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splunk.html<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.splunk.com%2Fen_us%2Fblog%2Ftips-and-tricks%2Fsyslog-ng-and-hec-scalable-aggregated-data-collection-in-splunk.html&data=04%7C01%7Cmark.faine%40nasa.gov%7Ca6df96e16e9b44c8de4808d8bba99fa3%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637465684496991660%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Bu0Zw3ZaBZCcXqrN4EuxyLvq%2BVuwZAeGZWilsfdJbVs%3D&reserved=0>
[2] https://docs.splunk.com/Documentation/Community/1.0/community/Chat<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.splunk.com%2FDocumentation%2FCommunity%2F1.0%2Fcommunity%2FChat&data=04%7C01%7Cmark.faine%40nasa.gov%7Ca6df96e16e9b44c8de4808d8bba99fa3%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637465684496991660%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bnbS3LjzaktxZkrj%2FOZyuIKzNGw6g%2BGk8CbdYbk9Meg%3D&reserved=0>

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine at nasa.gov<mailto:mark.faine at nasa.gov>>
Sent: Friday, January 15, 2021 4:51 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] [EXTERNAL] Re: Failover/Hi Availability

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

For now it's just an idea for an improved syslog-ng infrastructure to support Splunk.

Not on the same machine but multiple, at least two servers, each with their own syslog-ng instance, receiving data from dozens of hosts (the same hosts) and yes writing to a shared NFS file system.

The only substantial problem I have left in the (as of now) hypothetical scenario is that I need them all to write to the same location so that if one goes down we don't have to do any manual switching of any sort and none of the data fails to be indexed by Splunk.

Thanks,

Mark

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> On Behalf Of Attila Szakacs (aszakacs)
Sent: Friday, January 15, 2021 03:07
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [EXTERNAL] Re: [syslog-ng] Failover/Hi Availability

Hi Mark,

Please elaborate a bit more about your architecture. Relevant config parts are also appreciated.

Are there two syslog-ng servers running on the same machine?

Or is it just one syslog-ng instance with two different log sources (on different ports for example), and there is a host which sends to both log sources?

Maybe there are 2 different machines running their own syslog-ng, but using a network shared file system?

Thanks!

Cheers,

Attila

________________________________

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of Faine, Mark R. (MSFC-IS40)[NICS] <mark.faine at nasa.gov<mailto:mark.faine at nasa.gov>>
Sent: Thursday, January 14, 2021 9:21 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: [syslog-ng] Failover/Hi Availability

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

If I have two Syslog-ng servers receiving logs from the same source and writing to the exact same file system destination,  is there a way to do this without running into issues with the two instances clobbering the files or file locking issues?

Thanks,

-Mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210118/f739edaf/attachment-0001.html>