Syslog-ng Not Working properly
Hi Team, I am trying to configure syslog-ng in one our linux instance to get NGIPS/FMC data via udp connection on its default port (514). I have configured syslog-ng.conf under /etc/syslog-ng and then we have set SE Linux as Permissive. I am using RHEL 8.7 and syslog version 4.0. Apparently all looked good to me however while checking in the destination path that is mentioned I don't see any directory or logfile from for the said udp connection got created. Below is our observation and steps that we executed, can any of you please help me telling where I went wrong or if I am missing something, there is another testing in pipeline that is stalled for this - 1. Define source, destination and log_file in syslog-ng.conf (file attached). 2. Run the below SELinux command - # ausearch -c 'syslog-ng' --raw | audit2allow -M my-syslogng # semodule -X 300 -i my-syslogng.pp 1. Restart syslog-ng service - # systemctl restart syslog-ng.service (no error message received) 1. Check if the service is running - [cid:image002.png@01D99309.5147A9A0] 1. Check if syslog-ng is listening to udp port 514 - [cid:image003.png@01D99309.5147A9A0] 1. Checked and we have incoming data stream from source using the below command - tcpdump -i any -c10 -nn -A port 514 1. I have went through the syslog-ng troubleshooting steps mentioned in the link - https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... [cid:image004.png@01D9930B.D0B70940] syslog-ng -Fdev command output is also attached. 1. While running the following command got the below output - # watch '/usr/sbin/syslog-ng-ctl stats | grep "^center"' [cid:image005.png@01D99311.15E497D0] 1. # journaltctl command output (first 500 lines) attached 1. Current SE Linux status : [cid:image006.png@01D9931C.B8765370] 1. Our syslog-ng is logging to /var/log/messages and we are getting this message in /var/log/messages - [cid:image007.png@01D9931E.592725A0] Thanks & Regards, Sumanta Banerjee Splunk Admin | CISO | Aviva Group Tel: +91-8420892593 24x7x365: +44 1603 208 582 sumanta.banerjee@aviva.com<mailto:sumanta.banerjee@aviva.com> GlobalCyberSecurityEngineeringTeam@aviva.com<mailto:GlobalCyberSecurityEngineeringTeam@aviva.com> www.aviva.com<http://www.aviva.com> Wipro Technologies - SJP2, Bangalore, India [cid:image001.gif@01D99303.20AF3AC0] Aviva: Internal Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.
Dear All, Can somebody please help me on this - I am trying to configure syslog-ng in one our linux instance to get NGIPS/FMC data via udp connection on its default port (514). I have configured syslog-ng.conf under /etc/syslog-ng and then we have set SE Linux as Permissive. I am using RHEL 8.7 and syslog version 4.0. Apparently all looked good to me however while checking in the destination path that is mentioned in syslog-ng.conf I don't see any directory or logfile from for the said udp connection got created. Below is our observation and steps that we executed, can any of you please help me telling where I went wrong or if I am missing something, there is another testing in pipeline that is stalled for this - 1. Define source, destination and log_file in syslog-ng.conf (file attached). 2. Run the below SELinux command - # ausearch -c 'syslog-ng' --raw | audit2allow -M my-syslogng # semodule -X 300 -i my-syslogng.pp 1. Restart syslog-ng service - # systemctl restart syslog-ng.service (no error message received) 1. Checked if the syslog-ng service is running or not- it is showing as active(running), no error message. 1. Checked if syslog-ng is listening to udp port 514 - it is listening to it. 1. Checked and we have incoming data stream from source using the below command - tcpdump -i any -c10 -nn -A port 514 1. I have went through the syslog-ng troubleshooting steps mentioned in the link (I haven't got any link for 4.0.0 version)- https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... syslog-ng -Fdev command output is also attached. 1. While running the following command got the below output - # watch '/usr/sbin/syslog-ng-ctl stats | grep "^center"' Every 2.0s: /usr/sbin/syslog-ng-ctl stats | grep "^center" np-universal-forwarder-3.splunk: Fri Jun 9 17:05:14 2023 center;;received;a;processed;615 center;;queued;a;processed;615 1. # journaltctl command output (first 500 lines) attached 1. Current SE Linux status is set as Permissive. 2. Our syslog-ng is logging to /var/log/messages and we are getting this message in /var/log/messages - Jun 9 16:57:44 np-universal-forwarder-3.splunk syslog-ng[156121]: Log statistics; processed='global(payload_reallocs)=988', processed='src.journald(s_sys#0,journal)=333', stamp='src.journald(s_sys#0,journal)=1686326226', processed='global(sdata_updates)=0', queued='global(scratch_buffers_bytes)=0', processed='src.internal(s_sys#1)=1', stamp='src.internal(s_sys#1)=1686325664', processed='destination(d_boot)=0', processed='destination(d_kern)=0', processed='source(s_sys)=334', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(internal_queue_length)=0', processed='source(s_network)=0', processed='destination(d_spol)=0', processed='destination(d_mlal)=0', processed='destination(d_splunk)=0', processed='center(received)=334', processed='destination(d_mesg)=34', processed='destination(d_mail)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=300', queued='global(scratch_buffers_count)=0', processed='center(queued)=334', processed='global(msg_clones)=0' Thanks & Regards, Sumanta Banerjee Splunk Admin | CISO | Aviva Group Tel: +91-8420892593 24x7x365: +44 1603 208 582 sumanta.banerjee@aviva.com<mailto:sumanta.banerjee@aviva.com> GlobalCyberSecurityEngineeringTeam@aviva.com<mailto:GlobalCyberSecurityEngineeringTeam@aviva.com> www.aviva.com<http://www.aviva.com> Wipro Technologies - SJP2, Bangalore, India [cid:image001.gif@01D99303.20AF3AC0] Aviva: Internal Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.
Hello Sumanta! Your config looks good. The log about the statistics show that there are no incoming messages on 514 UDP and nothing is written to the files defined in the d_splunk destination. I think you could try to narrow down the scope of the problem with the following ideas. Try to send a message locally to 514 with: echo "foo bar" | nc -u -w0 localhost 514 If it does not work, I would suggest to change the receiving port of the network() source to something larger, like port(12345), and trying again with the following, just to see if the problem only occurs for the 514 port: echo "foo bar" | nc -u -w0 localhost 12345 You should see these kind of logs: [2024-01-15T15:58:46.037255] Incoming log entry; input='foo bar\x0a', msg='0x7f9bb0003020', rcptid='297' ... [2024-01-15T15:58:46.037655] Initializing destination file writer; template='......', filename='......', symlink_as='(null)' ... [2024-01-15T15:58:46.037872] Outgoing message; message='bar' My hunch is that this probably has something to do with SELinux, but unfortunately my knowledge of it is very limited. Regards, Attila
There is a selinux policy setup script in the syslog-ng repository, you might want to look at that to find some ideas. I have seen that some ports are getting enabled by the script with the semanage command, maybe this is what you are missing: https://github.com/syslog-ng/syslog-ng/blob/master/contrib/selinux/syslog_ng... Cheers, Attila On Mon, Jan 15, 2024 at 4:07 PM Attila Szakács <attila.szakacs@axoflow.com> wrote:
Hello Sumanta!
Your config looks good. The log about the statistics show that there are no incoming messages on 514 UDP and nothing is written to the files defined in the d_splunk destination.
I think you could try to narrow down the scope of the problem with the following ideas.
Try to send a message locally to 514 with: echo "foo bar" | nc -u -w0 localhost 514
If it does not work, I would suggest to change the receiving port of the network() source to something larger, like port(12345), and trying again with the following, just to see if the problem only occurs for the 514 port: echo "foo bar" | nc -u -w0 localhost 12345
You should see these kind of logs: [2024-01-15T15:58:46.037255] Incoming log entry; input='foo bar\x0a', msg='0x7f9bb0003020', rcptid='297' ... [2024-01-15T15:58:46.037655] Initializing destination file writer; template='......', filename='......', symlink_as='(null)' ... [2024-01-15T15:58:46.037872] Outgoing message; message='bar'
My hunch is that this probably has something to do with SELinux, but unfortunately my knowledge of it is very limited.
Regards, Attila
Hello Sumanta! Your config looks good. The log about the statistics show that there are no incoming messages on 514 UDP and nothing is written to the files defined in the d_splunk destination. I think you could try to narrow down the scope of the problem with the following ideas. Try to send a message locally to 514 with: echo "foo bar" | nc -u -w0 localhost 514 If it does not work, I would suggest to change the receiving port of the network() source to something larger, like port(12345), and trying again with the following, just to see if the problem only occurs for the 514 port: echo "foo bar" | nc -u -w0 localhost 12345 You should see these kind of logs: [2024-01-15T15:58:46.037255] Incoming log entry; input='foo bar\x0a', msg='0x7f9bb0003020', rcptid='297' ... [2024-01-15T15:58:46.037655] Initializing destination file writer; template='......', filename='......', symlink_as='(null)' ... [2024-01-15T15:58:46.037872] Outgoing message; message='bar' My hunch is that this probably has something to do with SELinux, but unfortunately my knowledge of it is very limited. Regards, Attila On Sun, Jan 14, 2024 at 9:50 PM Sumanta Banerjee <sumanta.banerjee@aviva.com> wrote:
*Hi Team,*
*I am trying to configure syslog-ng in one our linux instance to get NGIPS/FMC data via udp connection on its default port (514). I have configured syslog-ng.conf under /etc/syslog-ng and then we have set SE Linux as Permissive. I am using RHEL 8.7 and syslog version 4.0. Apparently all looked good to me however while checking in the destination path that is mentioned I don’t see any directory or logfile from for the said udp connection got created. *
*Below is our observation and steps that we executed, can any of you please help me telling where I went wrong or if I am missing something, there is another testing in pipeline that is stalled for this – *
1. Define source, destination and log_file in syslog-ng.conf (file attached). 2. Run the below SELinux command –
# ausearch -c 'syslog-ng' --raw | audit2allow -M my-syslogng # semodule -X 300 -i my-syslogng.pp
1. Restart syslog-ng service –
# systemctl restart syslog-ng.service (no error message received)
1. Check if the service is running –
1. Check if syslog-ng is listening to udp port 514 –
1. Checked and we have incoming data stream from source using the below command –
*tcpdump -i any -c10 -nn -A port 514*
1. I have went through the syslog-ng troubleshooting steps mentioned in the link –
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit...
*syslog-ng -Fdev command output is also attached.*
1. While running the following command got the below output - *# watch '/usr/sbin/syslog-ng-ctl stats | grep "^center"'*
1. *# journaltctl command output (first 500 lines) attached*
1. Current SE Linux status :
1. Our syslog-ng is logging to /var/log/messages and we are getting this message in /var/log/messages –
Thanks & Regards,
Sumanta Banerjee
Splunk Admin | CISO | Aviva Group
Tel: +91-8420892593
24x7x365: +44 1603 208 582
sumanta.banerjee@aviva.com
GlobalCyberSecurityEngineeringTeam@aviva.com
www.aviva.com
Wipro Technologies - SJP2, Bangalore, India
Aviva: Internal
Aviva plc, registered Office: St. Helen's, 1 Undershaft, London EC3P 3DQ. Registered in England No. 02468686. www.aviva.com
This message and any attachments may be confidential or legally privileged. If you are not the intended recipient, please telephone or e-mail the sender and delete this message and any attachments from your system. Also, if you are not the intended recipient you must not copy this message or attachments or disclose the contents to any other person. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Aviva.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Attila Szakács
-
Sumanta Banerjee