Hello, I have a syslogNG based siem setup with customized rules like: options { use_dns(no); use_fqdn(no); check_hostname(no); owner(root); group(root); perm(0640); dir_owner(root); dir_group(root); dir_perm(0750); create_dirs(yes); normalize_hostnames(yes); keep_hostname(yes); # disable stats stats_freq(0); }; destination d_net_auth { file("/var/log/corporate/$HOST_FROM/auth.log"); }; ... These settings will not do dns resolution will result that when hosts sending their logs into this SIEM directories will be created by their IP addresses where the logs go. I would like to replicate this server on a second location without using brute methods like rsyncing the whole directory structure daily. I have configured syslogng to keep forwarding the logs to a remote destination which works fine however I can't select the messages based on the same criteria on the new log server because if I use the same config everything will originate from the IP for logserver 1. I need IP based directories on the second loghost as well, everything to be identical. I'm using syslogng 3.12. Is there a workaround for this? Thanks
In path try use like this "/var/log/netlog/app/${HOST}/${PROGRAM}/${YEAR}/${MONTH}/${HOST}-${YEAR}${MONTH}${DAY}.log" On Wed, Nov 13, 2019 at 7:36 PM <freebsd@tango.lu> wrote:
Hello,
I have a syslogNG based siem setup with customized rules like:
options { use_dns(no); use_fqdn(no); check_hostname(no); owner(root); group(root); perm(0640); dir_owner(root); dir_group(root); dir_perm(0750); create_dirs(yes); normalize_hostnames(yes); keep_hostname(yes); # disable stats stats_freq(0); };
destination d_net_auth { file("/var/log/corporate/$HOST_FROM/auth.log"); }; ...
These settings will not do dns resolution will result that when hosts sending their logs into this SIEM directories will be created by their IP addresses where the logs go.
I would like to replicate this server on a second location without using brute methods like rsyncing the whole directory structure daily. I have configured syslogng to keep forwarding the logs to a remote destination which works fine however I can't select the messages based on the same criteria on the new log server because if I use the same config everything will originate from the IP for logserver 1. I need IP based directories on the second loghost as well, everything to be identical.
I'm using syslogng 3.12.
Is there a workaround for this?
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hello, if upgrading syslog-ng is an option for you, then you can use ewmm (introduced in 3.17: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.17.1) to transport your messages between two syslog-ng instances. This way the logs will be identical on the second machine, so every MACRO will produce the same output. if upgrading syslog-ng is not possible in your environment, I would recommend to put the necessary information (The HOST_FROM field in your case.) into a custom SDATA field, - which will be automatically transported by the syslog protocol - and use that on the second server. Br, Laci ________________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Pal, Laszlo <vlad@vlad.hu> Sent: Thursday, November 14, 2019 15:15 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] log server duplication CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. In path try use like this "/var/log/netlog/app/${HOST}/${PROGRAM}/${YEAR}/${MONTH}/${HOST}-${YEAR}${MONTH}${DAY}.log" On Wed, Nov 13, 2019 at 7:36 PM <freebsd@tango.lu<mailto:freebsd@tango.lu>> wrote: Hello, I have a syslogNG based siem setup with customized rules like: options { use_dns(no); use_fqdn(no); check_hostname(no); owner(root); group(root); perm(0640); dir_owner(root); dir_group(root); dir_perm(0750); create_dirs(yes); normalize_hostnames(yes); keep_hostname(yes); # disable stats stats_freq(0); }; destination d_net_auth { file("/var/log/corporate/$HOST_FROM/auth.log"); }; ... These settings will not do dns resolution will result that when hosts sending their logs into this SIEM directories will be created by their IP addresses where the logs go. I would like to replicate this server on a second location without using brute methods like rsyncing the whole directory structure daily. I have configured syslogng to keep forwarding the logs to a remote destination which works fine however I can't select the messages based on the same criteria on the new log server because if I use the same config everything will originate from the IP for logserver 1. I need IP based directories on the second loghost as well, everything to be identical. I'm using syslogng 3.12. Is there a workaround for this? Thanks ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CLaszlo.Szemere%40oneidentity.com%7C22ec230171b043df47c908d7690d29aa%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637093377570931795&sdata=iS%2FUbKcP5u%2FkyBo1pSAtTDtWKttz7%2Bt61UJUf9nsBsU%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CLaszlo.Szemere%40oneidentity.com%7C22ec230171b043df47c908d7690d29aa%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637093377570931795&sdata=PIaX%2BX12PVGNTywCNvQrU2DT8rwqWjvjW%2B9fBchGfdg%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CLaszlo.Szemere%40oneidentity.com%7C22ec230171b043df47c908d7690d29aa%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637093377570931795&sdata=193cWV2J5q375BspsWMFTbcfGqXuBBbchKNCBv54kKo%3D&reserved=0>
Hello, This EWMM sounds more like a well engineered solution, the update might worth it. My sender node: syslog-ng-3.12.1p5 log management solution My receiver node: syslog-ng-3.17.2nb1 Highly portable log management solution I assume for this EWMM both of them has to be 3.17+. Does it also support some sort of SSL transport of the logs over TCP? Thanks. On 2019-11-15 10:17, Laszlo Szemere (lszemere) wrote:
Hello, if upgrading syslog-ng is an option for you, then you can use ewmm (introduced in 3.17: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.17.1) to transport your messages between two syslog-ng instances. This way the logs will be identical on the second machine, so every MACRO will produce the same output.
if upgrading syslog-ng is not possible in your environment, I would recommend to put the necessary information (The HOST_FROM field in your case.) into a custom SDATA field, - which will be automatically transported by the syslog protocol - and use that on the second server.
Br, Laci
________________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Pal, Laszlo <vlad@vlad.hu> Sent: Thursday, November 14, 2019 15:15 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] log server duplication
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
In path try use like this
"/var/log/netlog/app/${HOST}/${PROGRAM}/${YEAR}/${MONTH}/${HOST}-${YEAR}${MONTH}${DAY}.log"
On Wed, Nov 13, 2019 at 7:36 PM <freebsd@tango.lu<mailto:freebsd@tango.lu>> wrote: Hello,
I have a syslogNG based siem setup with customized rules like:
options { use_dns(no); use_fqdn(no); check_hostname(no); owner(root); group(root); perm(0640); dir_owner(root); dir_group(root); dir_perm(0750); create_dirs(yes); normalize_hostnames(yes); keep_hostname(yes); # disable stats stats_freq(0); };
destination d_net_auth { file("/var/log/corporate/$HOST_FROM/auth.log"); }; ...
These settings will not do dns resolution will result that when hosts sending their logs into this SIEM directories will be created by their IP addresses where the logs go.
I would like to replicate this server on a second location without using brute methods like rsyncing the whole directory structure daily. I have configured syslogng to keep forwarding the logs to a remote destination which works fine however I can't select the messages based on the same criteria on the new log server because if I use the same config everything will originate from the IP for logserver 1. I need IP based directories on the second loghost as well, everything to be identical.
I'm using syslogng 3.12.
Is there a workaround for this?
Thanks ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7CLaszlo.Szemere%40oneidentity.com%7C22ec230171b043df47c908d7690d29aa%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637093377570931795&sdata=iS%2FUbKcP5u%2FkyBo1pSAtTDtWKttz7%2Bt61UJUf9nsBsU%3D&reserved=0> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7CLaszlo.Szemere%40oneidentity.com%7C22ec230171b043df47c908d7690d29aa%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637093377570931795&sdata=PIaX%2BX12PVGNTywCNvQrU2DT8rwqWjvjW%2B9fBchGfdg%3D&reserved=0> FAQ: http://www.balabit.com/wiki/syslog-ng-faq<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7CLaszlo.Szemere%40oneidentity.com%7C22ec230171b043df47c908d7690d29aa%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637093377570931795&sdata=193cWV2J5q375BspsWMFTbcfGqXuBBbchKNCBv54kKo%3D&reserved=0>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hello!
I assume for this EWMM both of them has to be 3.17+. Correct. Documentation: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit...
Does it also support some sort of SSL transport of the logs over TCP? Yes. See the "tls()" option in the documentation under the syslog-ng destination/source.
Br, Laci ________________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of freebsd@tango.lu <freebsd@tango.lu> Sent: Friday, November 15, 2019 15:25 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] log server duplication CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Hello, This EWMM sounds more like a well engineered solution, the update might worth it. My sender node: syslog-ng-3.12.1p5 log management solution My receiver node: syslog-ng-3.17.2nb1 Highly portable log management solution I assume for this EWMM both of them has to be 3.17+. Does it also support some sort of SSL transport of the logs over TCP? Thanks. On 2019-11-15 10:17, Laszlo Szemere (lszemere) wrote:
Hello, if upgrading syslog-ng is an option for you, then you can use ewmm (introduced in 3.17: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsyslog-ng%2Fsyslog-ng%2Freleases%2Ftag%2Fsyslog-ng-3.17.1&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=pU2gz79yas7i%2FMTLlqZIPRxHMqC4IwCqBBhHVX8ophM%3D&reserved=0) to transport your messages between two syslog-ng instances. This way the logs will be identical on the second machine, so every MACRO will produce the same output.
if upgrading syslog-ng is not possible in your environment, I would recommend to put the necessary information (The HOST_FROM field in your case.) into a custom SDATA field, - which will be automatically transported by the syslog protocol - and use that on the second server.
Br, Laci
________________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Pal, Laszlo <vlad@vlad.hu> Sent: Thursday, November 14, 2019 15:15 To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] log server duplication
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
In path try use like this
"/var/log/netlog/app/${HOST}/${PROGRAM}/${YEAR}/${MONTH}/${HOST}-${YEAR}${MONTH}${DAY}.log"
On Wed, Nov 13, 2019 at 7:36 PM <freebsd@tango.lu<mailto:freebsd@tango.lu>> wrote: Hello,
I have a syslogNG based siem setup with customized rules like:
options { use_dns(no); use_fqdn(no); check_hostname(no); owner(root); group(root); perm(0640); dir_owner(root); dir_group(root); dir_perm(0750); create_dirs(yes); normalize_hostnames(yes); keep_hostname(yes); # disable stats stats_freq(0); };
destination d_net_auth { file("/var/log/corporate/$HOST_FROM/auth.log"); }; ...
These settings will not do dns resolution will result that when hosts sending their logs into this SIEM directories will be created by their IP addresses where the logs go.
I would like to replicate this server on a second location without using brute methods like rsyncing the whole directory structure daily. I have configured syslogng to keep forwarding the logs to a remote destination which works fine however I can't select the messages based on the same criteria on the new log server because if I use the same config everything will originate from the IP for logserver 1. I need IP based directories on the second loghost as well, everything to be identical.
I'm using syslogng 3.12.
Is there a workaround for this?
Thanks ______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=Gva83uKyWNB5XVL6stmiob4gRgqmNtwH%2BcBrtb7z3PE%3D&reserved=0<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=Gva83uKyWNB5XVL6stmiob4gRgqmNtwH%2BcBrtb7z3PE%3D&reserved=0> Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=CQS3YRGTE7Ws3fWeLzn5A2emkYsloy2OONcfptBJZQQ%3D&reserved=0<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=CQS3YRGTE7Ws3fWeLzn5A2emkYsloy2OONcfptBJZQQ%3D&reserved=0> FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=WvjNhtnYdfC7jELqZtJ2MyFNAZvKE1xqjpw6MJiSMKQ%3D&reserved=0<https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=WvjNhtnYdfC7jELqZtJ2MyFNAZvKE1xqjpw6MJiSMKQ%3D&reserved=0>
______________________________________________________________________________ Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=Gva83uKyWNB5XVL6stmiob4gRgqmNtwH%2BcBrtb7z3PE%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=CQS3YRGTE7Ws3fWeLzn5A2emkYsloy2OONcfptBJZQQ%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=WvjNhtnYdfC7jELqZtJ2MyFNAZvKE1xqjpw6MJiSMKQ%3D&reserved=0
Member info: https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.balabit.hu%2Fmailman%2Flistinfo%2Fsyslog-ng&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=Gva83uKyWNB5XVL6stmiob4gRgqmNtwH%2BcBrtb7z3PE%3D&reserved=0 Documentation: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fsupport%2Fdocumentation%2F%3Fproduct%3Dsyslog-ng&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=CQS3YRGTE7Ws3fWeLzn5A2emkYsloy2OONcfptBJZQQ%3D&reserved=0 FAQ: https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.balabit.com%2Fwiki%2Fsyslog-ng-faq&data=02%7C01%7Claszlo.szemere%40oneidentity.com%7Cd90a3829327d4b2ee78b08d769d7a755%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637094247259923748&sdata=WvjNhtnYdfC7jELqZtJ2MyFNAZvKE1xqjpw6MJiSMKQ%3D&reserved=0
participants (3)
-
freebsd＠tango.lu
-
Laszlo Szemere (lszemere)
-
Pal, Laszlo