August 2017 - syslog-ng - lists.balabit.hu

UDP devices stop logging after network services restart.
by Vincenti Francesco 05 Oct '17

05 Oct '17

Hello I have syslog-ng 3.7 installed on a CentOS7 Server with 4G RAM and 4 CPU. The server is receiving logs from almost 300 devices, either network devices (UDP) and servers (TCP). The problem I have is the following: to reduce incoming traffic on initial two NICs I added two more NICs to the server, where I sent some UDP devices and it worked immediately, without any issue. But when I restarted network services (systemctl restart network) the devices stopped to log, they restarted only after I resent them to the initial NIC, without any other action on syslog-ng server. When I started syslog-ng in debug mode to verify this behavior I saw that these devices ARE NOT considered by syslog-ng at all, they are not present in logs generated by debug, but they REACH the server (tcpdump shows them entering). I'm getting mad about this issue, any suggestion will be welcome. Following my configuration files: - /etc/syslog-ng/syslog-ng.conf # Note: it also sources additional configuration files (*.conf) # located in /etc/syslog-ng/conf.d/ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); chain_hostnames (off); use_dns (persist_only); dns-cache-hosts(/etc/hosts); use_fqdn (no); create_dirs (yes); keep_hostname (yes); owner ("1007"); group ("1007"); dir_owner ("1007"); dir_group ("1007"); dir_perm (0750); }; source s_sys { system(); internal(); # udp(ip(0.0.0.0) port(514)); }; destination d_cons { file("/dev/console"); }; destination d_mesg { file("/var/log/messages"); }; destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" flush_lines(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_kern { file("/var/log/kern"); }; destination d_mlal { usertty("*"); }; destination d_auth_loc { network ("127.0.0.1" port (601) ); }; filter f_kernel { facility(kern); }; filter f_default { level(info..emerg) and not (facility(mail) or facility(authpriv) or facility(cron)); }; filter f_auth { facility(authpriv); }; filter f_mail { facility(mail); }; filter f_emergency { level(emerg); }; filter f_news { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_boot { facility(local7); }; filter f_cron { facility(cron); }; #log { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); filter(f_kernel); destination(d_kern); }; log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_auth); destination(d_auth); destination(d_auth_loc); }; log { source(s_sys); filter(f_mail); destination(d_mail); }; log { source(s_sys); filter(f_emergency); destination(d_mlal); }; log { source(s_sys); filter(f_news); destination(d_spol); }; log { source(s_sys); filter(f_boot); destination(d_boot); }; log { source(s_sys); filter(f_cron); destination(d_cron); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf" # vim:ft=syslog-ng:ai:si:ts=4:sw=4:et: - /etc/syslog-ng/conf.d/network_dev.conf source s_network_appa { network ( ip(0.0.0.0) port (514) transport ("udp") so-rcvbuf (4096000) ); }; # Destinazioni destination d_network_udp { file("/var/log/syslog-ng/APPARATI/${YEAR}${MONTH}${DAY}/${HOST}/${YEAR}${MONTH}${DAY}_hh${HOUR}_${HOST}" create_dirs(yes) dir_perm(0755) perm(0755) ) ; }; filter f_appa_rete { not host ("SymantecServer") and not host ("part8") and not host ("parti") and not host ("partd") and not host ("part1") and not host ("parte") and not host ("part10") and not host ("part11") and not host ("part12") and not host ("part3") and not host ("part6") and not host ("part8") and not host ("part9") and not host ("parta") and not host ("partc") and not host ("partd") and not host ("partf") and not host ("partg") and not host ("parth") and not host ("partn") and not host ("parto") and not host ("SLITES8736.sercom.aspasiel.it<http://SLITES8736.sercom.aspasiel.it>") and not host ("172.23.112.47") and not host ("SLITES4557.sercom.aspasiel.it<http://SLITES4557.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES4558.sercom.aspasiel.it<http://SLITES4558.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES4559.sercom.aspasiel.it<http://SLITES4559.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES4560.sercom.aspasiel.it<http://SLITES4560.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8731.sercom.aspasiel.it<http://SLITES8731.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8732.sercom.aspasiel.it<http://SLITES8732.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8733.sercom.aspasiel.it<http://SLITES8733.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8734.sercom.aspasiel.it<http://SLITES8734.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("SLITES8735.sercom.aspasiel.it<http://SLITES8735.sercom.aspasiel.it>") and not host ("172.23.X.Y") and not host ("slitdd2727"); }; log { source(s_network_appa); filter(f_appa_rete); destination(d_network_udp); }; - /etc/syslog-ng/conf.d/server_TCP.conf source s_server_sop { network ( ip(0.0.0.0) port(601) transport("tcp") max-connections(200) log_fetch_limit(100) log_iw_size(20000) ); }; destination d_server_sop { file("/var/log/syslog-ng/SERVER/${YEAR}${MONTH}${DAY}/${HOST}/${YEAR}${MONTH}${DAY}_hh${HOUR}_${HOST}" create_dirs(yes) dir_perm(0755) perm(0755) flush_lines(100) ); }; # Indirizzo le destinazioni. log { source(s_server_sop); destination(d_server_sop); flags(flow-control); }; I changed the following kernel parameters as follow: net.core.rmem_max = 4096000 net.core.wmem_max = 4096000 Thanks in advance Francesco Vincenti RHCSA Area Data Center Open Source, Quality and Security Aspasiel Divisione della Società Acciai Speciali Terni S.p.A. con Unico Socio Strada di Pentima, 3 - 05100 Terni francesco.vincenti(a)acciaiterni.it<mailto:francesco.vincenti@acciaiterni.it> www.aspasiel.it<http://www.aspasiel.it/> Ufficio: +39 0744 203224 Fax: +39 0744 203444 This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the error at the following email address: helpdesk(a)aspasiel.it<mailto:helpdesk@aspasiel.it> or at Aspasiel Helpdesk Team by phone (phone number +390744203555), and then delete this message from your system. P Please consider our environment and think before you print. Thank you! q

3 6

Bugfix releases?
by Fabien Wernli 29 Sep '17

29 Sep '17

Hi, While I'm very happy with the current rapid development cycle of syslog-ng, I can't stop wondering: what happened to bugfix releases? The quality of the releases is very high indeed, but regressions still appear now and then, and some users (not us) prefer the safe approach of sticking to one major release, and applying only bug fix releases. I think these people may have a hard time nowadays. What's the mailing-list reader's opinion on that? Cheers

2 1

syslog-ng not writing to specified path
by vijay amruth 01 Sep '17

01 Sep '17

Hello all, hope you are all doing great. I have set up syslog-ng on a host and I am able to see packets on tcpdump but syslog-ng is not wriiting to the specified paths. I have checked firewall rules, filters, write permissions at the path. I have another host on the same VLAN with the same config that is able to receive packets and write to the destination. what I am I missing ? *version used:* syslog-ng 3.9.1 Installer-Version: 3.9.1 Revision: Module-Directory: /usr/local/lib/syslog-ng Module-Path: /usr/local/lib/syslog-ng Available-Modules: syslogformat,afsocket,affile,afprog,afuser,afamqp,afmongodb,csvparser,confgen,system-source,linux-kmsg-format,basicfuncs,cryptofuncs,dbparser,json-plugin,afstomp,pseudofile,graphite,sdjournal,kvformat,date,cef,disk-buffer,add-contextual-data Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off syslog-ng -Fvde shows : [2017-08-03T13:57:20.214552] Module loaded and initialized successfully; module='syslogformat' Any help is appreciated. Thanks, Vijay Amrut.

2 1

Creating time lapse videos from log messages
by Czanik, Péter 31 Aug '17

31 Aug '17

Hi, A couple of you asked how I created the time lapse videos visualizing connection attempts to my router. I used the new geoip2 parser of syslog-ng 3.11, Elasticsearch for storage, Kibana for showing the map and OpenShot to convert screenshots into a movie. Read the details from my blog at https://www.balabit.com/blog/creating-time-lapse-videos-log-messages-using-… I'm not really sure if it has any practical use, but definitely a nice eye candy which can raise attention towards security of your network. Do you have any other suggestions for log data to be visualized in a similar way? Bye, Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

host related macros
by Fabien Wernli 28 Aug '17

28 Aug '17

Hi, I have a hard time understanding the behaviour of the multiple host based macros with respect to the value of the use-fqdn(), and keep-hostname() params. First of all, when setting use-fqdn(yes), isn't the network destination supposed to send the fqdn? In our case, tcpdump shows that the short hostname is sent instead. Additionally, what does the following paragraph from the documentation mean? The hostname-related macros (${FULLHOST}, ${FULLHOST_FROM}, ${HOST}, and ${HOST_FROM}) do not have any effect if the keep-hostname() option is disabled Does that mean they are empty if keep-hostname(no) ? Cheers

2 2

laszlo_budai Key expired
by Clayton Dukes 25 Aug '17

25 Aug '17

Are you guys aware of this? W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… ./ Release: The following signatures were invalid: KEYEXPIRED 1503496621 W: Failed to fetch http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUb… The following signatures were invalid: KEYEXPIRED 1503496621 W: Some index files failed to download. They have been ignored, or old ones used instead. [id:image001.png@01D306E3.0FEBC990] Clayton Dukes Founder & CEO LogZilla Corporation 2900 N. Quinlan Park Rd, B240-341 Austin, TX, 78732 Tel: 936-4NetOps (463-8677) Web: www.logzilla.net<http://www.logzilla.net/> [id:image002.png@01D306E3.0FEBC990]<https://twitter.com/logzilla>[id:image003.png@01D306E3.0FEBC990]<https://youtu.be/drg5wv_mgfA>[id:image004.png@01D306E3.0FEBC990]<https://www.linkedin.com/in/lzcdukes/> For NetOps, By NetOps!

2 4

anyone know of a decent but easy to use syslog visualizer
by Smith, Paul (Sr. Admin-InfoSec) 23 Aug '17

23 Aug '17

Does anyone have a simple, easy to use, visualizer for syslogs that allows me to make new graphs quickly? I am in need of getting dashboards from syslog events to our executive teams. They don't know what information they want from these dashboards so I am trying to come up with some pretty graphs they can look at, and create new ones if they don't want that ones I do create. This way I can sketch things up quickly so they can see what can be graphed. We are a small team in a really big shop and I have many other responsibilities. I don't have the time to spend mucking about with this stuff all day. I'm trying to find a simple, easy to use visualizer that does not take me days to set up and configure. I've tried so many articles about setting up things like ELK and others, and have had a problem getting most of them to work. I spend half a day on each one and something is just not quite right because the build instructions are a month or more out of date and there is something that has changed in a dependency. Also, when trying to deal with creating logstash config files, I don't want to have to learn yet another config file syntax that I only need to touch 2-3 times per year. If I touch it at this frequency, I have to spend a day trying to remember what is what in the config file so I can make a new graph. Paul Smith InfoSec Those who don't study history are doomed to repeat it. Those who do study history are doomed to stand by helplessly while everyone else repeats it. Tom Toro

2 1

UDP devices stop logging after network services restart.
by Vincenti Francesco 23 Aug '17

23 Aug '17

Dear Janos I already sent an email with the data you requested that is under analysis from your moderators because it is too big. Anyway, every NIC has a number of dropped RX packages but either their number and their daily growth is not big enough to explain the amount of data lost. Further, netstat -su shows, under UDP, 1 packet receive errors only. I'm going to switch UDP devices again to log on not-working NIC (ens35 according data I've sent to you) to verify how many packages are lost in this case and I'll inform you about the results. Cheers Francesco Vincenti RHCSA Area Data Center Open Source, Quality and Security Aspasiel Divisione della Società Acciai Speciali Terni S.p.A. con Unico Socio Strada di Pentima, 3 - 05100 Terni francesco.vincenti(a)acciaiterni.it<mailto:francesco.vincenti@acciaiterni.it> www.aspasiel.it<http://www.aspasiel.it/> Ufficio: +39 0744 203224 Fax: +39 0744 203444 This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the error at the following email address: helpdesk(a)aspasiel.it<mailto:helpdesk@aspasiel.it> or at Aspasiel Helpdesk Team by phone (phone number +390744203555), and then delete this message from your system. P Please consider our environment and think before you print. Thank you! q

2 2

poll: which operating system do you use?
by Czanik, Péter 22 Aug '17

22 Aug '17

Hi, Here is my next poll: asking for which operating system you use to run syslog-ng. Cast your vote on Twitter at: https://twitter.com/PCzanik/status/899973381549895680 As the number of possible answers is limited by Twitter, if your choice of OS is missing, add it as a comment. The poll is anonymous, but of course comments are not. If you don't want to share your answer with the rest of the world but don't mind sharing with me, answer this e-mail and make sure that the address is set to my address instead of the list... Thanks for your help, Peter Peter Czanik (CzP) <peter.czanik(a)balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik

1 0

Syslog-NG and UTF-8 problem.
by Jason Long 22 Aug '17

22 Aug '17

Hello. I forward my Windows event logs to my Linux Box. I guess the Syslog-NG that I installed it on my Linux box can't support UFT-8 because when the file or folder names are English then I can't see them. In my Windows Box, I'm using "nxlog" that it support UTF-8. How can I add UTF-8 feature to my Syslog-NG on Linux? Thank you.

1 0