[syslog-ng] R: UDP devices stop logging after network services restart.
SZIGETVÁRI János
jszigetvari at gmail.com
Thu Oct 5 02:58:01 UTC 2017
Dear Francesco,
Sorry for the significant delay in answering to the data you sent.
Judging from the data you sent, there are no misconfigurations in regard to
UDP log processing that would explain the behavior you mentioned.
Also your config looks fine.
Tonight tried doing a reproduction but I used 3.5.6, which in turn worked
just fine.
(Although it is true that I did use a different version, and that i used
loggen on the server itself.)
Did you try sending logs to the IPs/ports from the same server? Did it
behave the same way?
Furthermore could you please try the reproduction with a more recent
syslog-ng version? 3.7 is quite old, but 3.5.6 is in deed older.
And in case you already overcame the issue: How did you find out and what
was causing the problem?
Thanks!
Cheers,
János
--
Janos SZIGETVARI
RHCE, License no. 150-053-692
<https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
LinkedIn: linkedin.com/in/janosszigetvari
__ at __˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
2017-07-28 10:04 GMT-04:00 Vincenti Francesco <
Francesco.Vincenti at acciaiterni.it>:
> Dear Jànos
>
> Thank you for you answer.
>
> In this moment syslog-ng is up and running, with the “rebel” network
> devices logging on the original NIC.
>
> Anyway I’m going to send you what you need as attachment.
>
> In ipaddr.txt I changed the IP of the card, for security reason; UDP
> “rebel” devices are logging on ens32 while they should log (and they logged
> before network services restart) on ens35.
>
>
>
> Cheers.
>
>
>
> *Francesco Vincenti*
> *RHCSA Area Data Center Open Source, Quality and Security*
>
> Aspasiel Divisione della Società
> Acciai Speciali Terni S.p.A. con Unico Socio
> Strada di Pentima, 3 – 05100 Terni
>
> francesco.vincenti at acciaiterni.it
> *www.aspasiel.it <http://www.aspasiel.it/>*
>
> Ufficio: +39 0744 203224 <+39%200744%20203224>
> Fax: +39 0744 203444 <+39%200744%20203444>
>
>
>
> *Da:* syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] *Per conto di
> *SZIGETVÁRI János
> *Inviato:* venerdì 28 luglio 2017 15:27
> *A:* Syslog-ng users' and developers' mailing list
> *Oggetto:* Re: [syslog-ng] UDP devices stop logging after network
> services restart.
>
>
>
> Dear Francesco,
>
> To rule out the possiblity of losing UDP logs, I would ask you to share
> the output of the following commands:
>
> # cat /proc/net/udp
>
> # netstat -su
>
> # sysctl net.core.rmem_max
>
> # ps auxfw
>
> # top
>
> # lspci
>
> # /usr/sbin/syslog-ng --preprocess-into ~/syslog-ng.pp.conf
> # ip addr show
>
> Furthermore, you may find further useful information about sizing UDP
> buffers and other best practice infomation here:
> https://www.balabit.com/documents/syslog-ng-pe-6.0-
> guides/en/syslog-ng-tutorial-udp-source/html-single/index.html
>
>
>
> Another useful infomation is that the net.core.rmem_max kernel parameter
> has to be at least the double of so-rcvbuf() setting. See man 7 socket for
> further information.
>
> In higher traffic UDP logging scenarios 256 MB for the rmem_max, and 128
> MB for so-rcvbuf() might be useful.
>
>
>
> Please share the above information with us so that we can get a better
> picture of your setup.
>
> Thank you!
>
> Cheers,
>
> János
>
>
> --
>
> Janos SZIGETVARI
> RHCE, License no. 150-053-692
> <https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
>
> __ at __˚V˚
> Make the switch to open (source) applications, protocols, formats now:
> - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
> - msn -> jabber protocol (Pidgin, Google Talk)
> - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
>
>
>
> 2017-07-27 8:50 GMT+02:00 Vincenti Francesco <Francesco.Vincenti@
> acciaiterni.it>:
>
> Hello
>
> I have syslog-ng 3.7 installed on a CentOS7 Server with 4G RAM and 4 CPU.
>
> The server is receiving logs from almost 300 devices, either network
> devices (UDP) and servers (TCP).
>
> The problem I have is the following: to reduce incoming traffic on initial
> two NICs I added two more NICs to the server, where I sent some UDP devices
> and it worked immediately, without any issue.
>
> But when I restarted network services (systemctl restart network) the
> devices stopped to log, they restarted only after I resent them to the
> initial NIC, without any other action on syslog-ng server.
>
> When I started syslog-ng in debug mode to verify this behavior I saw that
> these devices ARE NOT considered by syslog-ng at all, they are not present
> in logs generated by debug, but they REACH the server (tcpdump shows them
> entering).
>
> I'm getting mad about this issue, any suggestion will be welcome.
>
> Following my configuration files:
>
> - /etc/syslog-ng/syslog-ng.conf
>
> # Note: it also sources additional configuration files (*.conf)
>
> # located in /etc/syslog-ng/conf.d/
>
>
>
> options {
>
> flush_lines (0);
>
> time_reopen (10);
>
> log_fifo_size (1000);
>
> chain_hostnames (off);
>
> use_dns (persist_only);
>
> dns-cache-hosts(/etc/hosts);
>
> use_fqdn (no);
>
> create_dirs (yes);
>
> keep_hostname (yes);
>
> owner ("1007");
>
> group ("1007");
>
> dir_owner ("1007");
>
> dir_group ("1007");
>
> dir_perm (0750);
>
> };
>
>
>
> source s_sys {
>
> system();
>
> internal();
>
> # udp(ip(0.0.0.0) port(514));
>
> };
>
>
>
> destination d_cons { file("/dev/console"); };
>
> destination d_mesg { file("/var/log/messages"); };
>
> destination d_auth { file("/var/log/secure"); };
>
> destination d_mail { file("/var/log/maillog" flush_lines(10)); };
>
> destination d_spol { file("/var/log/spooler"); };
>
> destination d_boot { file("/var/log/boot.log"); };
>
> destination d_cron { file("/var/log/cron"); };
>
> destination d_kern { file("/var/log/kern"); };
>
> destination d_mlal { usertty("*"); };
>
> destination d_auth_loc { network ("127.0.0.1" port (601) ); };
>
>
>
> filter f_kernel { facility(kern); };
>
> filter f_default { level(info..emerg) and
>
> not (facility(mail)
>
> or facility(authpriv)
>
> or facility(cron)); };
>
> filter f_auth { facility(authpriv); };
>
> filter f_mail { facility(mail); };
>
> filter f_emergency { level(emerg); };
>
> filter f_news { facility(uucp) or
>
> (facility(news)
>
> and level(crit..emerg)); };
>
> filter f_boot { facility(local7); };
>
> filter f_cron { facility(cron); };
>
>
>
> #log { source(s_sys); filter(f_kernel); destination(d_cons); };
>
> log { source(s_sys); filter(f_kernel); destination(d_kern); };
>
> log { source(s_sys); filter(f_default); destination(d_mesg); };
>
> log { source(s_sys); filter(f_auth); destination(d_auth);
> destination(d_auth_loc); };
>
> log { source(s_sys); filter(f_mail); destination(d_mail); };
>
> log { source(s_sys); filter(f_emergency); destination(d_mlal); };
>
> log { source(s_sys); filter(f_news); destination(d_spol); };
>
> log { source(s_sys); filter(f_boot); destination(d_boot); };
>
> log { source(s_sys); filter(f_cron); destination(d_cron); };
>
>
>
>
>
> # Source additional configuration files (.conf extension only)
>
> @include "/etc/syslog-ng/conf.d/*.conf"
>
>
>
>
>
> # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
>
>
>
> - /etc/syslog-ng/conf.d/network_dev.conf
>
> source s_network_appa
>
> {
>
> network ( ip(0.0.0.0) port (514) transport ("udp") so-rcvbuf (4096000) );
>
> };
>
>
>
> # Destinazioni
>
>
>
> destination d_network_udp { file("/var/log/syslog-ng/
> APPARATI/${YEAR}${MONTH}${DAY}/${HOST}/${YEAR}${MONTH}${DAY}_hh${HOUR}_${HOST}"
> create_dirs(yes) dir_perm(0755) perm(0755) ) ; };
>
>
>
> filter f_appa_rete { not host ("SymantecServer") and not host ("part8")
> and not host ("parti") and not host ("partd") and not host ("part1") and
> not host ("parte") and not host ("part10") and not host ("part11") and not
> host ("part12") and not host ("part3") and not host ("part6") and not host
> ("part8") and not host ("part9") and not host ("parta") and not host
> ("partc") and not host ("partd") and not host ("partf") and not host
> ("partg") and not host ("parth") and not host ("partn") and not host
> ("parto") and not host ("SLITES8736.sercom.aspasiel.it") and not host
> ("172.23.112.47") and not host ("SLITES4557.sercom.aspasiel.it") and not
> host ("172.23.X.Y") and not host ("SLITES4558.sercom.aspasiel.it") and
> not host ("172.23.X.Y") and not host ("SLITES4559.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES4560.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8731.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8732.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8733.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8734.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("SLITES8735.sercom.aspasiel.it")
> and not host ("172.23.X.Y") and not host ("slitdd2727"); };
>
>
>
> log { source(s_network_appa); filter(f_appa_rete);
> destination(d_network_udp); };
>
>
>
> - /etc/syslog-ng/conf.d/server_TCP.conf
>
> source s_server_sop
>
> {
>
> network ( ip(0.0.0.0) port(601) transport("tcp") max-connections(200)
> log_fetch_limit(100) log_iw_size(20000) );
>
> };
>
>
>
> destination d_server_sop
>
> {
>
> file("/var/log/syslog-ng/SERVER/${YEAR}${MONTH}${DAY}/$
> {HOST}/${YEAR}${MONTH}${DAY}_hh${HOUR}_${HOST}" create_dirs(yes)
> dir_perm(0755) perm(0755) flush_lines(100) );
>
> };
>
>
>
> # Indirizzo le destinazioni.
>
> log { source(s_server_sop); destination(d_server_sop);
> flags(flow-control); };
>
>
>
> I changed the following kernel parameters as follow:
>
> net.core.rmem_max = 4096000
>
> net.core.wmem_max = 4096000
>
>
>
>
>
> Thanks in advance
>
>
>
> *Francesco Vincenti*
> *RHCSA Area Data Center Open Source, Quality and Security*
>
> Aspasiel Divisione della Società
> Acciai Speciali Terni S.p.A. con Unico Socio
> Strada di Pentima, 3 – 05100 Terni
>
> francesco.vincenti at acciaiterni.it
> *www.aspasiel.it <http://www.aspasiel.it/>*
>
> Ufficio: +39 0744 203224 <+39%200744%20203224>
> Fax: +39 0744 203444 <+39%200744%20203444>
>
>
>
> This e-mail and any attachments is a confidential correspondence intended
> only for use of the individual or entity named above. If you are not the
> intended recipient or the agent responsible for delivering the message to
> the intended recipient, you are hereby notified that any disclosure,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the error at
> the following email address: helpdesk at aspasiel.it or at Aspasiel Helpdesk
> Team by phone (phone number +390744203555 <+39%200744%20203555>), and
> then delete this message from your system.
>
> P Please consider our environment and think before you print. Thank you! q
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> This e-mail and any attachments is a confidential correspondence intended
> only for use of the individual or entity named above. If you are not the
> intended recipient or the agent responsible for delivering the message to
> the intended recipient, you are hereby notified that any disclosure,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the error at
> the following email address: helpdesk at aspasiel.it or at Aspasiel Helpdesk
> Team by phone (phone number +390744203555 <+39%200744%20203555>), and
> then delete this message from your system.
>
> P Please consider our environment and think before you print. Thank you! q
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20171004/32e17801/attachment-0001.html>
More information about the syslog-ng
mailing list