- syslog-ng - lists.balabit.hu

Insider 2020-04: syslog-ng RPM; mult-line-timeout; FreeBSD;
by Peter Czanik (pczanik) 09 Apr '20

09 Apr '20

Dear syslog-ng users, This is the 80th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. In these difficult times, when many of us stay at home, we can decide what to do with that bit of extra time that we don’t spend commuting. Some of us turn to entertainment or hobbies, do long-neglected things around the house, or even focus on studying and honing skills. If you ran out of movies to watch and want to spend your time on something more practical, this might be the best time to learn something new about syslog-ng. One starting point for that is the reference documentation, available for each release separately: https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-ed… Another possibility is to check out our ever-growing collection of syslog-ng blogs. These describe a large number of different syslog-ng use cases. Blogs are available at https://www.syslog-ng.com/community/b/blog If you want to deep dive into configuring syslog-ng, here are some of the highlights: * The building blocks of syslog-ng: https://www.syslog-ng.com/community/b/blog/posts/building-blocks-of-syslog-… * Calculate PI with syslog-ng: https://www.syslog-ng.com/community/b/blog/posts/calculate-pi-syslog-ng * Creating heat maps using the new syslog-ng geoip2 parser: https://www.syslog-ng.com/community/b/blog/posts/creating-heat-maps-using-n… * Extend syslog-ng in Python: https://www.syslog-ng.com/community/tags/python Even if it is often not real-time, you can also chat with syslog-ng users and developers. The syslog-ng Gitter channel is a place to ask syslog-ng-related questions, but you are also very welcome to share your experiences about syslog-ng here. A success story not just feels good for developers but can also inspire other users. It is available at https://gitter.im/syslog-ng/syslog-ng NEWS Overview of syslog-ng RPM repositories -------------------------------------- This blog provides you an overview of syslog-ng RPM repositories: why to have unofficial packages at all and what are the advantages and disadvantages of the different unofficial repositories. https://www.syslog-ng.com/community/b/blog/posts/overview-of-syslog-ng-rpm-… Multi-line-timeout: making sure your last multi-line message is not lost -------------------------------- When your application has a problem that it cannot handle, then Java, PHP and other environments often write multi-line error messages. These long messages include information that is useful for developers, but it might be difficult to handle for logging systems. A multi-line log message is usually saved when the beginning of the next message is detected, which can make handling the last message problematic. This is where multi-line-timeout can help. After a pre-configured timeout, the last message is saved by syslog-ng. Losing a message when there are thousands of them is probably not a big deal. But if you are lucky, the error messages are rare. Maybe just a single message in a file. In either case, you can lose most or even all messages. If you do not want to lose these messages, configure multi-line-timeout. https://www.syslog-ng.com/community/b/blog/posts/multi-line-timeout-making-… syslog-ng in FreeBSD ports -------------------------- Recently we explored the different syslog-ng packages available for RPM-based Linux distributions, used by the majority of our Linux users. This time we focus on FreeBSD, the platform of choice for most of our non-Linux users. >From this blog, you can learn about the features that are in the binary syslog-ng packages, how to compile syslog-ng for yourself from ports and the current syslog-ng version policy in ports. NEW WHITE PAPERS * Upgrade to State-of-the-Art Log Management: https://www.syslog-ng.com/whitepaper/log-management-essentials8137754/ WEBINARS * Learn how to easily integrate Azure Sentinel into your log-data strategy: https://www.syslog-ng.com/webcast-ondemand/learn-how-to-easily-integrate-az… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

can not load geoip2 plugin
by Russell Fulton 09 Apr '20

09 Apr '20

HI This is a locally compiled application (version 3.26.1) and I am trying to get the geoip2 stuff working. I had similar problems a week or two back with elasticsearch_http destination plugin. This seems like it might be easier to diagnose since the error says where it is looking for the plugin. rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.26.1/sbin/syslog-ng -s -f /etc/syslog-ng/geoip.conf [2020-04-08T07:30:19.551845] Plugin module not found in 'module-path'; module-path='/usr/local/syslog-ng-3.26.1/lib/syslog-ng', module='geoip2' Error parsing config, syntax error, unexpected LL_ERROR, expecting $end in /etc/syslog-ng/geoip.conf:4:1-5:1: 1 @version: 3.21 2 3 @include "scl.conf" 4-----> @module geoip2 4-----> ^^^^^^^^^^^^^^^ 5-----> # 6 7 options { 8 use_dns (no); 9 use_fqdn (no); rful011@secmgrprd02:~$ ls -l /usr/local/syslog-ng-3.26.1/lib/syslog-ng/libgeoip2-plugin.* -rwxr-xr-x 1 root root 1271 Mar 31 08:57 /usr/local/syslog-ng-3.26.1/lib/syslog-ng/libgeoip2-plugin.la -rwxr-xr-x 1 root root 158912 Mar 31 08:57 /usr/local/syslog-ng-3.26.1/lib/syslog-ng/libgeoip2-plugin.so any ideas why it is not loading the library?

2 2

3.26 RHEL/CentOS 6 packages
by Peter Czanik (pczanik) 02 Apr '20

02 Apr '20

Hi, syslog-ng 3.26 packages are now also available for RHEL/CentOS 6: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng326epel6/ And if you use syslog-ng on RHEL/CentOS I'd appreciate if you could answer a quick poll on Twitter: https://twitter.com/PCzanik/status/1245692669889777667 Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

errors with elasticsearch_http on 2.21 & 3.22
by Russell Fulton 02 Apr '20

02 Apr '20

I have had this config running happily for several months and today I tried to add the geoip2 plugin and managed to break everything. I first realised that I needed to recompile syslog-ng to get the geoip2 plugin but forgot I was running a locally compiled version . I tried apt install syslog-ng-mod-geoip and this broke things badly and I have been trying to recover for the last few hours :( The issue is that any attempt to start syslog-ng with the original config that includes a destination of elasticsearch_http get an error: Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.22/etc/syslog-ng.conf:33:3-33:21 Judging from what I found on th ’Net this is what one would expect if the syslog-ng version was prior to 3.21 and did not support the http dest. I first tried recompiling and installing 3.21 (in a different dir) with geoip2 enabled. That went with out errors but still would not process my config. Aside: At this point I found that in spite of having the geoip2 module included, syslog-ng failed to find it when I copied the setup from the manual. long story short the manual says the module is ‘geoip2’ when in fact it is ‘geoip2-plugin’ and that is the name you have to use in the @module. Weird! I then compiled 3.22 since I had the tar file and this behaved the same. rful011@secmgrprd02:~$ cat short.conf @version: 3.21 @include "scl.conf" @module geoip2-plugin source s_loghost { network( transport("tcp") flags(no-multi-line) port(1514) keep-alive(yes)); }; destination d_elastic { elasticsearch_http( index("auth_${YEAR}.${MONTH}.${DAY}") type("auth") persist-name("auth") template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") url("http://secesprd01.its.auckland.ac.nz:9200/_bulk") ); }; log { source(s_loghost); destination(d_elastic ); flags(flow-control); }; rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng -s -c ~/short.conf Error parsing destination statement, destination plugin elasticsearch_http not found in /usr/local/syslog-ng-3.21/etc/syslog-ng.conf:33:3-33:21: 28 file( "/home/rful011/test.log" template(t_db_parsed_test)); 29 }; 30 31 32 destination d_elastic { 33----> elasticsearch_http( 33----> ^^^^^^^^^^^^^^^^^^ 34 index("auth_${YEAR}.${MONTH}.${DAY}") 35 type("auth") 36 persist-name("auth") 37 template( "$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") 38 url("http://secesprd01.its.auckland.ac.nz:9200/_bulk”) rful011@secmgrprd02:~$ /usr/local/syslog-ng-3.21/sbin/syslog-ng --version syslog-ng 3 (3.21.1) Config version: 3.21 Installer-Version: 3.21.1 Revision: Compile-Date: Mar 29 2020 11:57:39 Module-Directory: /usr/local/syslog-ng-3.21/lib/syslog-ng Module-Path: /usr/local/syslog-ng-3.21/lib/syslog-ng Include-Path: /usr/local/syslog-ng-3.21/share/syslog-ng/include Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,examples,geoip2-plugin,tfgetent,graphite,hook-commands,http,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,mod-python,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,xml Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Systemd: off

3 8

Syslog-NG Mysql Parsing to separate column
by Sync IT 01 Apr '20

01 Apr '20

Hi I am new to syslog-ng. I am receiving my logs perfectly to mysql. now i want to separate the message log into different column in mqsql. Currently i am receiving messages like this R2_DST_RTR forward: in:<pppoe-user1> out:vlan100-Default-Internet, src-mac 11:22:33:44:55:66, proto TCP (ACK,FIN), 172.16.41.159:42670->23.46.16.231:80, NAT (172.16.41.159:42670->9.9.9.9:42670)->23.46.16.231:80, len 52 This is my syslog-ng config source s_net{ udp( ); }; destination d_mtksql { sql(type(mysql) host("localhost") username("dbuser") password("dbpasss") database("syslog") table("${R_YEAR}_${R_MONTH}_${R_DAY}") columns( "id int(11) unsigned not null auto_increment primary key", "host varchar(40) not null", "date datetime", "message text not null") values("0", "$FULLHOST", "$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC", "$MSG") indexes("id")); }; log { source(s_net); destination(d_mtksql); }; i want to separate the message data in different column like- time,username,src mac,src ip,dst ip, nat ip. Any idea how can i achieve that? I am not a pro so if possible kindly write in details how to parse them. Many thanks

2 1

Change Position Information inside Messasge
by William Luiz Ribeiro Vasconcelos Da Silva 27 Mar '20

27 Mar '20

Hello everyone, Is it possible to change a position of an information within the message? From: 2020-03-26; 17: 55: 22; RTCGNMGA0101: RT_SRC_NAT_PBA_ALLOC :; 100.64.1.114:38784-38847; 177.51.112.218; TO: 2020-03-26; 17: 55: 22; RTCGNMGA0101: RT_SRC_NAT_PBA_ALLOC:; 100.64.1.114; 177.51.112.218:38784-38847; Basically, I would like to change the position of the connection port within the message. How to do this? TKS Atenciosamente, WILLIAM LUIZ R V SILVA Mediation Ericsson Mobile +55 11 97979-9886 www.ericsson.com<http://www.ericsson.com/> [Descri??o: Descri??o: Ericsson]<http://www.ericsson.com/> Esta mensagem, incluindo seus anexos, pode conter informa??es privilegiadas e/ou de car?ter confidencial, n?o podendo ser retransmitida sem autoriza??o do remetente. Se voc? n?o ? o destinat?rio ou pessoa autorizada para receb?-la, informamos que o seu uso, divulga??o, c?pia ou arquivamento s?o proibidos. Portanto, se voc? recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conte?do. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 3

variable host filter in events
by Syslogng 25 Mar '20

25 Mar '20

Hi, We receive aggregated syslog from a server (all logs are send from 1 IP). Also all the events are mixed. The name of the host sending the initial traffic is in each event. ex: 2020/03/23 [notice] [user] New original_source=SERV1.example.com Task=0 .... How to recover SER1 which is always preceded by "original_source=" and followed by "example.com" to save it in a file for example /data/serv1.log I don't want a static filter (I know how to do it) but a dynamic one. If a new event arrives at original_source=SERV2.example.com I would like it to automatically create a /data/serv2.log Could you help me please ? thank you in advance Pit

2 3

Problem to Get UDP Packets - Syslog-ng
by William Luiz Ribeiro Vasconcelos Da Silva 25 Mar '20

25 Mar '20

Hello Everyone, I installed syslog-ng on a new machine, however in initial tests, there was no collection of UDP packages by syslog-ng. Here are some points I checked: sudo netstat -plunt | grep -e PID -e syslog Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:601 0.0.0.0:* LISTEN 16169/syslog-ng udp 0 0 10.96.145.42:514 0.0.0.0:* 16169/syslog-ng netstat -anu | grep 514 udp 0 0 10.96.145.42:514 0.0.0.0:* [cgnat@mgalnxa01 etc]$ sudo systemctl status syslog-ng -l ● syslog-ng.service - System Logger Daemon Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-03-25 12:38:08 -03; 5min ago Process: 114207 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS) Main PID: 16169 (syslog-ng) CGroup: /system.slice/syslog-ng.service └─16169 /opt/syslog-ng/libexec/syslog-ng -F --enable-core Mar 25 12:38:08 mgalnxa01 systemd[1]: Starting System Logger Daemon... Here is an example of the package received via tcpdump, but it was not captured by syslog-ng: 10:46:13.529331 IP (tos 0x20, ttl 251, id 33055, offset 0, flags [none], proto UDP (17), length 243) 10.96.145.98.syslog > mgalnxa01.9514: [udp sum ok] SYSLOG, length: 215 Facility user (1), Severity info (6) Msg: Mar 24 13:46:13 2020 RTCGNMGA0103 RT_NAT: RT_SRC_NAT_PBA_ALLOC: Subscriber 100.64.1.6 used/maximum [2/2] blocks, allocates port block [47104-47167] from 177.51.116.146 in source pool PUBLIC-NAT-POOL-1 lsys_id: 0 0x0000: 3c31 343e 4d61 7220 3234 2031 333a 3436 0x0010: 3a31 3320 3230 3230 2052 5443 474e 4d47 0x0020: 4130 3130 3320 5254 5f4e 4154 3a20 5254 0x0030: 5f53 5243 5f4e 4154 5f50 4241 5f41 4c4c 0x0040: 4f43 3a20 5375 6273 6372 6962 6572 2031 0x0050: 3030 2e36 342e 312e 3620 7573 6564 2f6d 0x0060: 6178 696d 756d 205b 322f 325d 2062 6c6f 0x0070: 636b 732c 2061 6c6c 6f63 6174 6573 2070 0x0080: 6f72 7420 626c 6f63 6b20 5b34 3731 3034 0x0090: 2d34 3731 3637 5d20 6672 6f6d 2031 3737 0x00a0: 2e35 312e 3131 362e 3134 3620 696e 2073 0x00b0: 6f75 7263 6520 706f 6f6c 2050 5542 4c49 0x00c0: 432d 4e41 542d 504f 4f4c 2d31 206c 7379 0x00d0: 735f 6964 3a20 30 What I need to analyze / verify, for syslog-ng will capture this type of package and convert it to a file. Atenciosamente, WILLIAM LUIZ R V SILVA Mediation Ericsson Rua Maria Preste Maia, 300 02879-130, Brazil Phone +55 11 2760-3785 Mobile +55 11 97979-9886 wsilva_ericsson(a)timbrasil.com.br<mailto:wsilva_ericsson@timbrasil.com.br> www.ericsson.com<http://www.ericsson.com/> [Descricao: Descricao: Ericsson]<http://www.ericsson.com/> Esta mensagem, incluindo seus anexos, pode conter informacoes privilegiadas e/ou de carater confidencial, nao podendo ser retransmitida sem autorizacao do remetente. Se voce nao e o destinatario ou pessoa autorizada para recebe-la, informamos que o seu uso, divulgacao, copia ou arquivamento sao proibidos. Portanto, se voce recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteudo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

3 8

Remove String from Message + syslog
by William Luiz Ribeiro Vasconcelos Da Silva 25 Mar '20

25 Mar '20

Hello eveyone, We are receiving two strings in the message in UDP packets: "used / maximum [1/2]" or "used / maximum [2/2]". How to configure the subst function, to be able to remove these strings? Atenciosamente, WILLIAM LUIZ R V SILVA Mediation Ericsson Mobile +55 11 97979-9886 wsilva_ericsson(a)timbrasil.com.br<mailto:wsilva_ericsson@timbrasil.com.br> www.ericsson.com<http://www.ericsson.com/> [Descri??o: Descri??o: Ericsson]<http://www.ericsson.com/> Esta mensagem, incluindo seus anexos, pode conter informa??es privilegiadas e/ou de car?ter confidencial, n?o podendo ser retransmitida sem autoriza??o do remetente. Se voc? n?o ? o destinat?rio ou pessoa autorizada para receb?-la, informamos que o seu uso, divulga??o, c?pia ou arquivamento s?o proibidos. Portanto, se voc? recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conte?do. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 3

Disk Buffering
by Ronald Fenner 24 Mar '20

24 Mar '20

I'm going to enable disk buffer but I'd like to place the files on an EBS volume that persists when a server is shutdown. The idea being we'd have a recovery server that would stream out and items left in the buffer. Besides the buffer itself what other files should be on there so that a different instance with the same config file can load the buffer and send the rest of it. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 1