[syslog-ng] UDP devices stop logging after network services restart.
SZIGETVÁRI János
jszigetvari at gmail.com
Fri Jul 28 13:26:57 UTC 2017
Dear Francesco,
To rule out the possiblity of losing UDP logs, I would ask you to share the
output of the following commands:
# cat /proc/net/udp
# netstat -su
# sysctl net.core.rmem_max
# ps auxfw
# top
# lspci
# /usr/sbin/syslog-ng --preprocess-into ~/syslog-ng.pp.conf
# ip addr show
Furthermore, you may find further useful information about sizing UDP
buffers and other best practice infomation here:
https://www.balabit.com/documents/syslog-ng-pe-6.0-guides/en/syslog-ng-tutorial-udp-source/html-single/index.html
Another useful infomation is that the net.core.rmem_max kernel parameter
has to be at least the double of so-rcvbuf() setting. See man 7 socket for
further information.
In higher traffic UDP logging scenarios 256 MB for the rmem_max, and 128 MB
for so-rcvbuf() might be useful.
Please share the above information with us so that we can get a better
picture of your setup.
Thank you!
Cheers,
János
--
Janos SZIGETVARI
RHCE, License no. 150-053-692
<https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
__ at __˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
2017-07-27 8:50 GMT+02:00 Vincenti Francesco <
Francesco.Vincenti at acciaiterni.it>:
> Hello
>
> I have syslog-ng 3.7 installed on a CentOS7 Server with 4G RAM and 4 CPU.
>
> The server is receiving logs from almost 300 devices, either network
> devices (UDP) and servers (TCP).
>
> The problem I have is the following: to reduce incoming traffic on initial
> two NICs I added two more NICs to the server, where I sent some UDP devices
> and it worked immediately, without any issue.
>
> But when I restarted network services (systemctl restart network) the
> devices stopped to log, they restarted only after I resent them to the
> initial NIC, without any other action on syslog-ng server.
>
> When I started syslog-ng in debug mode to verify this behavior I saw that
> these devices ARE NOT considered by syslog-ng at all, they are not present
> in logs generated by debug, but they REACH the server (tcpdump shows them
> entering).
>
> I'm getting mad about this issue, any suggestion will be welcome.
>
> Following my configuration files:
>
> - /etc/syslog-ng/syslog-ng.conf
>
> # Note: it also sources additional configuration files (*.conf)
>
> # located in /etc/syslog-ng/conf.d/
>
>
>
> options {
>
> flush_lines (0);
>
> time_reopen (10);
>
> log_fifo_size (1000);
>
> chain_hostnames (off);
>
> use_dns (persist_only);
>
> dns-cache-hosts(/etc/hosts);
>
> use_fqdn (no);
>
> create_dirs (yes);
>
> keep_hostname (yes);
>
> owner ("1007");
>
> group ("1007");
>
> dir_owner ("1007");
>
> dir_group ("1007");
>
> dir_perm (0750);
>
> };
>
>
>
> source s_sys {
>
> system();
>
> internal();
>
> # udp(ip(0.0.0.0) port(514));
>
> };
>
>
>
> destination d_cons { file("/dev/console"); };
>
> destination d_mesg { file("/var/log/messages"); };
>
> destination d_auth { file("/var/log/secure"); };
>
> destination d_mail { file("/var/log/maillog" flush_lines(10)); };
>
> destination d_spol { file("/var/log/spooler"); };
>
> destination d_boot { file("/var/log/boot.log"); };
>
> destination d_cron { file("/var/log/cron"); };
>
> destination d_kern { file("/var/log/kern"); };
>
> destination d_mlal { usertty("*"); };
>
> destination d_auth_loc { network ("127.0.0.1" port (601) ); };
>
>
>
> filter f_kernel { facility(kern); };
>
> filter f_default { level(info..emerg) and
>
> not (facility(mail)
>
> or facility(authpriv)
>
> or facility(cron)); };
>
> filter f_auth { facility(authpriv); };
>
> filter f_mail { facility(mail); };
>
> filter f_emergency { level(emerg); };
>
> filter f_news { facility(uucp) or
>
> (facility(news)
>
> and level(crit..emerg)); };
>
> filter f_boot { facility(local7); };
>
> filter f_cron { facility(cron); };
>
>
>
> #log { source(s_sys); filter(f_kernel); destination(d_cons); };
>
> log { source(s_sys); filter(f_kernel); destination(d_kern); };
>
> log { source(s_sys); filter(f_default); destination(d_mesg); };
>
> log { source(s_sys); filter(f_auth); destination(d_auth);
> destination(d_auth_loc); };
>
> log { source(s_sys); filter(f_mail); destination(d_mail); };
>
> log { source(s_sys); filter(f_emergency); destination(d_mlal); };
>
> log { source(s_sys); filter(f_news); destination(d_spol); };
>
> log { source(s_sys); filter(f_boot); destination(d_boot); };
>
> log { source(s_sys); filter(f_cron); destination(d_cron); };
>
>
>
>
>
> # Source additional configuration files (.conf extension only)
>
> @include "/etc/syslog-ng/conf.d/*.conf"
>
>
>
>
>
> # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
>
>
>
> - /etc/syslog-ng/conf.d/network_dev.conf
>
> source s_network_appa
>
> {
>
> network ( ip(0.0.0.0) port (514) transport ("udp") so-rcvbuf (4096000) );
>
> };
>
>
>
> # Destinazioni
>
>
>
> destination d_network_udp { file("/var/log/syslog-ng/
> APPARATI/${YEAR}${MONTH}${DAY}/${HOST}/${YEAR}${MONTH}${DAY}_hh${HOUR}_${HOST}"
> create_dirs(yes) dir_perm(0755) perm(0755) ) ; };
>
>
>
> filter f_appa_rete { not host ("SymantecServer") and not host ("part8")
> and not host ("parti") and not host ("partd") and not host ("part1") and
> not host ("parte") and not host ("part10") and not host ("part11") and not
> host ("part12") and not host ("part3") and not host ("part6") and not host
> ("part8") and not host ("part9") and not host ("parta") and not host
> ("partc") and not host ("partd") and not host ("partf") and not host
> ("partg") and not host ("parth") and not host ("partn") and not host
> ("parto") and not host ("SLITES8736.sercom.aspasiel.it") and not host
> ("172.23.112.47") and not host ("SLITES4557.sercom.aspasiel.it") and not
> host ("172.23.X.Y") and not host ("SLITES4558.sercom.aspasiel.it") and
> not host ("172.23.X.Y") and not host ("SLITES4559.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES4560.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8731.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8732.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8733.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8734.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8735.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("slitdd2727"); };
>
>
>
> log { source(s_network_appa); filter(f_appa_rete);
> destination(d_network_udp); };
>
>
>
> - /etc/syslog-ng/conf.d/server_TCP.conf
>
> source s_server_sop
>
> {
>
> network ( ip(0.0.0.0) port(601) transport("tcp") max-connections(200)
> log_fetch_limit(100) log_iw_size(20000) );
>
> };
>
>
>
> destination d_server_sop
>
> {
>
> file("/var/log/syslog-ng/SERVER/${YEAR}${MONTH}${DAY}/$
> {HOST}/${YEAR}${MONTH}${DAY}_hh${HOUR}_${HOST}" create_dirs(yes)
> dir_perm(0755) perm(0755) flush_lines(100) );
>
> };
>
>
>
> # Indirizzo le destinazioni.
>
> log { source(s_server_sop); destination(d_server_sop);
> flags(flow-control); };
>
>
>
> I changed the following kernel parameters as follow:
>
> net.core.rmem_max = 4096000
>
> net.core.wmem_max = 4096000
>
>
>
>
>
> Thanks in advance
>
>
>
> *Francesco Vincenti*
> *RHCSA Area Data Center Open Source, Quality and Security*
>
> Aspasiel Divisione della Società
> Acciai Speciali Terni S.p.A. con Unico Socio
> Strada di Pentima, 3 – 05100 Terni
>
> francesco.vincenti at acciaiterni.it
> *www.aspasiel.it <http://www.aspasiel.it/>*
>
> Ufficio: +39 0744 203224 <+39%200744%20203224>
> Fax: +39 0744 203444 <+39%200744%20203444>
>
>
>
> This e-mail and any attachments is a confidential correspondence intended
> only for use of the individual or entity named above. If you are not the
> intended recipient or the agent responsible for delivering the message to
> the intended recipient, you are hereby notified that any disclosure,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the error at
> the following email address: helpdesk at aspasiel.it or at Aspasiel Helpdesk
> Team by phone (phone number +390744203555 <+39%200744%20203555>), and
> then delete this message from your system.
>
> P Please consider our environment and think before you print. Thank you! q
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170728/f3e87743/attachment-0001.html>
More information about the syslog-ng
mailing list