September 2019 - syslog-ng - lists.balabit.hu

Log Rotation based on size
by Alexandre Santos 18 Sep '19

18 Sep '19

Hi All, I noticed that some years ago Elan Ruusamäe made a patch with log rotation by size in https://github.com/pld-linux/syslog-ng/blob/master/log-file-size-limit.patch. This was never included in the master branch. Are there any plans to release a log rotation based on size in syslog-ng? Cheers, Alex

2 1

Re: [syslog-ng] syslog-ng Digest, Vol 173, Issue 18
by Németh Balázs 17 Sep '19

17 Sep '19

Hi Laci, Elaaticsearch is not limited by any means regarding indexable traffic. Should you need any real life example then have a look at my blog: https://balagetech.com/tag/elasticsearch/ This page will show you the feature differences between different licenses: https://www.elastic.co/subscriptions Note that the free version does not have AD integration. (Maybe you could put a WAF before Kibana, or handling the AD integration with Apache + LDAP) ps: I am not paid by anyone to suggest Elasticsearch. It is simply a tool I use. Regards, <syslog-ng-request(a)lists.balabit.hu> ezt írta (időpont: 2019. szept. 17., K, 12:57): > Send syslog-ng mailing list submissions to > syslog-ng(a)lists.balabit.hu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.balabit.hu/mailman/listinfo/syslog-ng > or, via email, send a message with subject or body 'help' to > syslog-ng-request(a)lists.balabit.hu > > You can reach the person managing the list at > syslog-ng-owner(a)lists.balabit.hu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of syslog-ng digest..." > Today's Topics: > > 1. Re: a bit [offtopic] but may not. syslog search solution for > free (Pal, Laszlo) > 2. Re: a bit [offtopic] but may not. syslog search solution for > free (Fabien Wernli) > 3. Re: Enable SNI (Server Name Identification) in TLS > connection (Attila Szakacs (aszakacs)) > > > > ---------- Forwarded message ---------- > From: "Pal, Laszlo" <vlad(a)vlad.hu> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Tue, 17 Sep 2019 10:55:04 +0200 > Subject: Re: [syslog-ng] a bit [offtopic] but may not. syslog search > solution for free > Hi, > > Thanks for the answers. So, I definitely need an onprem solution and yes, > Elastic is a good idea, however based on my previous research even the OS > version is restricted to some amount of daily log. Maybe I'm wrong and in > this case I may give a try again. At this point I don't know how much logs > will I ingest daily but because it is more than 30K devices including core > routers, firewalls and all sort of servers, I don't think any traffic > limited solution will be sufficient. > > Graylog is a very good solution which I already tested but the "free" > version is limited to something like 50 Mbyet per day. > > I'll check this onion thingy, thank you Peter for the heads up > > Vlad > > On Tue, Sep 17, 2019 at 10:01 AM Peter Czanik (pczanik) < > Peter.Czanik(a)oneidentity.com> wrote: > >> Hi, >> >> These are not my personal experiences, but talking to thousands of >> syslog-ng users at different events: >> >> >> - Many are happy with grep and hate anything with a GUI. It works >> relatively well, if you have many small log files (separated by date, host, >> application, etc.), but if you have lots of logs, you can't avoid message >> parsing and indexing for efficient search. >> - Splunk is one of the most popular destinations with syslog-ng. OSE >> users tend to use the free version, which can index up to 500MB of logs a >> day. >> - Elasticsearch is the other most popular destination. With the >> latest version most of the security features are included in the open >> source version for free, so it does not need 3rd party extensions or a >> commercial license any more. >> - Graylog is also getting popular among syslog-ng users. We can feed >> it with structured log messages in multiple ways: either GELF or JSON. >> Free, open source, but commercial extensions and support are available. >> - I did not test recently, but Security Onion is a nice security >> analytic platform which also includes log management with analysis, >> dashboards, etc. It's free, open source, but commercial support is >> available. >> - Logzilla is focused on Cisco network devices and automation but >> also works for generic log management. It is commercial software, but a >> free version is also available. >> >> Of course any time indexing and GUI are involved you need some extra >> hardware compared syslog-ng, so definitely more expensive even if you go >> with the free versions. >> >> Bye, >> >> Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> >> Balabit (a OneIdentity company) / syslog-ng upstream >> https://syslog-ng.com/community/ >> https://twitter.com/PCzanik >> >> ------------------------------ >> *From:* syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of Pal, >> Laszlo <vlad(a)vlad.hu> >> *Sent:* Tuesday, September 17, 2019 9:16 AM >> *To:* Syslog-ng users' and developers' mailing list < >> syslog-ng(a)lists.balabit.hu> >> *Subject:* [syslog-ng] a bit [offtopic] but may not. syslog search >> solution for free >> >> CAUTION: This email originated from outside of the organization. Do not >> follow guidance, click links, or open attachments unless you recognize the >> sender and know the content is safe. >> >> Hi, >> >> Our new central syslog collect&store system are almost complete and >> finally it is based on Syslog-ng OSE. The only issue I want to solve, is >> how to present those logs to the users. Originally I thought it would be >> enough if they will get the directories as NFS exports and they can use >> their favourite grep to search files, but I thought maybe I can implement >> some more user friendly solution. >> >> In the planning phase of the project, I've tested various solutions >> including Graylog, ELK and of course I'm aware of the beauty of SSB :) , >> but all of these solutions are too expensive for this project. >> >> Then I thought maybe if I forward the logs to some database (SQL or >> noSQL) I can try to find some very simple frontend for that DB provides a >> simple search interface for those logs. Nothing fancy is required, but AD >> auth should be an option. >> >> I'm sure I'm not the only one facing this issue, so I hope someone in the >> community can share some previous experience on this >> >> Thanks >> Vlad >> >> >> >> >> ______________________________________________________________________________ >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >> Documentation: >> http://www.balabit.com/support/documentation/?product=syslog-ng >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >> >> > > > ---------- Forwarded message ---------- > From: Fabien Wernli <wernli(a)in2p3.fr> > To: "Syslog-ng users' and developers' mailing list" < > syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Tue, 17 Sep 2019 11:05:18 +0200 > Subject: Re: [syslog-ng] a bit [offtopic] but may not. syslog search > solution for free > On Tue, Sep 17, 2019 at 10:55:04AM +0200, Pal, Laszlo wrote: > > Thanks for the answers. So, I definitely need an onprem solution and yes, > > Elastic is a good idea, however based on my previous research even the OS > > version is restricted to some amount of daily log. Maybe I'm wrong and in > > It's not. By all means grab the -oss packages, they don't contain any > non-free code. > > > > > > ---------- Forwarded message ---------- > From: "Attila Szakacs (aszakacs)" <Attila.Szakacs(a)oneidentity.com> > To: "syslog-ng(a)lists.balabit.hu" <syslog-ng(a)lists.balabit.hu> > Cc: > Bcc: > Date: Tue, 17 Sep 2019 10:57:14 +0000 > Subject: Re: [syslog-ng] Enable SNI (Server Name Identification) in TLS > connection > Hi Raghu, > > Currently we are not sending SNI extension in the Client Hello message. > However, I made a PR to implement this: > https://github.com/balabit/syslog-ng/pull/2930 > > Can you build syslog-ng from source? It would be great, if you tested the > PR. > > Best regards, > Attila > ------------------------------ > *From:* syslog-ng <syslog-ng-bounces(a)lists.balabit.hu> on behalf of > Raghunath Adhyapak <funduraghu(a)gmail.com> > *Sent:* Tuesday, September 17, 2019 9:05 AM > *To:* syslog-ng(a)lists.balabit.hu <syslog-ng(a)lists.balabit.hu> > *Subject:* [syslog-ng] Enable SNI (Server Name Identification) in TLS > connection > > CAUTION: This email originated from outside of the organization. Do not > follow guidance, click links, or open attachments unless you recognize the > sender and know the content is safe. > > Hi, > > I am using TLS over TCP connection to forward my syslog events to a remote > server. > My remote server uses SNI (Server Name Identification) to route > connections/events to one of the available backend servers. > > I observe that syslog-ng doesn't send SNI during TLS handshake. > > How can I enable it? > > My configuration is as follows: > > =================================== > source s_net { syslog(transport(udp) port(1514)); }; > destination d_tcp { > tcp( > "XX.example.net > <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2FXX.example…> > " > port(96) > tls( > peer-verify(required-untrusted) > ca_dir("/etc/syslog-ng/ssl") > > key-file("/etc/syslog-ng/ssl/globaltest/XX.example.net.key.pem") > > cert-file("/etc/syslog-ng/ssl/globaltest/XX.example.net.cert.pem") > ) > ); > }; > log { > source(s_net); > destination(d_tcp); > }; > =================================== > > I want syslog-ng to send XX.example.net > <https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2FXX.example…> > as SNI to my remote server > > Please advise > > Thanks > Raghu > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng >

1 0

a bit [offtopic] but may not. syslog search solution for free
by Pal, Laszlo 17 Sep '19

17 Sep '19

Hi, Our new central syslog collect&store system are almost complete and finally it is based on Syslog-ng OSE. The only issue I want to solve, is how to present those logs to the users. Originally I thought it would be enough if they will get the directories as NFS exports and they can use their favourite grep to search files, but I thought maybe I can implement some more user friendly solution. In the planning phase of the project, I've tested various solutions including Graylog, ELK and of course I'm aware of the beauty of SSB :) , but all of these solutions are too expensive for this project. Then I thought maybe if I forward the logs to some database (SQL or noSQL) I can try to find some very simple frontend for that DB provides a simple search interface for those logs. Nothing fancy is required, but AD auth should be an option. I'm sure I'm not the only one facing this issue, so I hope someone in the community can share some previous experience on this Thanks Vlad

4 4

error while loading shared libraries: libsyslog-ng-3.23.so.0
by Laura Smith 15 Sep '19

15 Sep '19

Hi, I compiled syslog-ng 3.23 on Debian 10 (./configure --enable-systemd --with-systemdsystemunitdir=/etc/systemd/system --enable-mixed-linking) When I try a simple "syslog-ng -F", I get the following error: "syslog-ng: error while loading shared libraries: libsyslog-ng-3.23.so.0: cannot open shared object file: No such file or directory" However.... ldconfig -v | fgrep syslog libsyslog-ng-3.23.so.0 -> libsyslog-ng.so And .... # find / -name "libsyslog-ng-3.23.so.0" -print /usr/local/lib/libsyslog-ng-3.23.so.0 Any ideas ??

3 3

Insider 2019-09: syslog-ng basics; relays; NGINX; Tic-Tac-Toe; sudo; Elastic stack 7; GitHub;
by Peter Czanik (pczanik) 12 Sep '19

12 Sep '19

Dear syslog-ng users, This is the 75th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Building blocks of syslog-ng ---------------------------- Recently I gave a syslog-ng introductory workshop at Pass the SALT conference in Lille, France. I got a lot of positive feedback, so I decided to turn all that content into a blog post. Naturally, I shortened and simplified it, but still managed to get enough material for multiple blog posts. This one gives you an overview of syslog-ng, its major features and an introduction to its configuration. https://www.syslog-ng.com/community/b/blog/posts/building-blocks-of-syslog-… What syslog-ng relays are good for ---------------------------------- While there are some users who run syslog-ng as a stand-alone application, the main strength of syslog-ng is central log collection. In this case the central syslog-ng instance is called the server, while the instances sending log messages to the central server are called the clients. There is a (somewhat lesser known) third type of instance called the relay, too. The relay collects log messages via the network and forwards them to one or more remote destinations after processing (but without writing them onto the disk for storage). A relay can be used for many different use cases. We will discuss a few typical examples below. https://www.syslog-ng.com/community/b/blog/posts/what-syslog-ng-relays-are-… Visualizing NGINX or Apache access logs in Kibana --------------------------------------- This tutorial shows you how to parse NGINX or Apache access logs with syslog-ng and create ECS compatible data in Elasticsearch. https://balagetech.com/visualizing-nginx-access-logs-kibana/ syslog-ng Tic-Tac-Toe --------------------- You can play the game of Tic-Tac-Toe using syslog-ng 3.22.1 or later. Learn how to configure syslog-ng for that: https://kokan.github.io/2019/06/16/syslog-ng-tic-tac-toe.html Alerting on sudo events using syslog-ng --------------------------------------- Why use syslog-ng to alert on sudo events? At the moment, alerting in sudo is limited to E-mail. Using syslog-ng, however, you can send alerts (more precisely, selected logs) to a wide variety of destinations. Logs from sudo are automatically parsed by recent (3.13+) syslog-ng releases, enabling fine-grained alerting. There is a lot of hype around our new Slack destination, so that is what I’ll show here. Naturally, there are many others available as well, including Telegram and, of course, good old E-mail. If something is not yet directly supported by syslog-ng, you can often utilize an HTTP API or write some glue code in Python. >From this blog post you can learn how to build up a syslog-ng configuration step by step and how to use different filters to make sure that you only receive logs (i.e. alerts) that are truly relevant for you. https://www.syslog-ng.com/community/b/blog/posts/alerting-on-sudo-events-us… GitHub and syslog-ng -------------------- As many of you know, the source code of syslog-ng is available on GitHub, just like its issue tracker. We just learned that GitHub itself is running syslog-ng as part of its stack: https://help.github.com/en/enterprise/2.18/admin/installation/log-forwarding syslog-ng with Elastic Stack 7 ------------------------------ For many years, anything I wrote about syslog-ng and Elasticsearch was valid for all available versions. Well, not anymore. With version 7 of Elasticsearch, there are some breaking changes. These changes are mostly related to the fact that Elastic is phasing out type support. This effects mapping (as the _default_ keyword is not used any more) and the syslog-ng configuration as well (as even though type() is a mandatory parameter, you should leave it empty). This blog post is a rewrite of one of my earlier blog posts (about creating a heat map using syslog-ng + Elasticsearch + Kibana), focusing on the changes and the new elasticsearch-http() destination: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-with-elastic-sta… UPCOMING CONFERENCES * One Identity UNITE (https://www.oneidentity.com/unite-conference/unite-phoenix.aspx) syslog-ng booth * All Things Open (https://allthingsopen.org/) syslog-ng talk * Bsides Luxembourg (https://bsideslux.lu/2019/) syslog-ng workshop WEBINARS * How to Leverage NetFlow and vFlow to Better Monitor Network Traffic https://www.syslog-ng.com/event/how-to-leverage-netflow-and-vflow-to-monito… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

cleaning up the unofficial rpm repositories
by Peter Czanik (pczanik) 12 Sep '19

12 Sep '19

Hi, Ever since I started my unofficial rpm packages for (open)SUSE and Fedora/RHEL/CentOS I created a new repository for each new release and never deleted them. Of course OBS and Copr guys did sometimes clean ups, so Fedora and openSUSE packages for long end of life distributions got deleted. But for the rest there are still syslog-ng 3.5 packages on-line. Not everyone likes to update syslog-ng every second month, so I keep the current workflow and create a new repo for each new release. This way your syslog-ng.conf or package management is not accidentally broken at a random time with a new update, you have a chance to test a new release thoroughly before upgrading. On the other hand I plan to keep only a year worth of releases instead of everything. Question: * Is there anyone, who needs packages older than one year? With the one year limit I'd delete anything older than 3.17 and delete 3.17 as well, once 3.24 is out, and so on. Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

5 7

How to enable log file generation by row size or Megabytes?
by William Luiz Ribeiro Vasconcelos Da Silva 09 Sep '19

09 Sep '19

Good afternoon everyone! Which parameters should be enabled in the "syslog-ng.conf" file, because I want the captured log files to be a maximum of 5Mb in size or 1500 lines. Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1

sub-second time digits all 0
by John Chang 09 Sep '19

09 Sep '19

Hello, I am not getting non-zero sub-second timestamp digits. My /etc/syslog-ng/syslog-ng.conf file includes this global configuration: # First, set some global options. options { frac-digits(3); chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); }; My syslog-ng.conf also includes a sub-config file for sending the logs to a remote host, with this configuration: destination d_net { udp("loggerhost" port(30515) frac-digits(3) ); }; log { source(s_src); destination(d_net); }; But all sub-second timestamp digits wind up being only zeroes on the remote "loggerhost", like this: 2019-09-03T21:57:23.000+00:00 10.73.254.255 [info] [sshd] 3284 Accepted password for root from 10.72.0.186 port 50720 ssh2 The sending host is running syslog-ng 3.5.3. The receiving "loggerhost" is running 3.5.6 Thanks in advance for any help you can afford.

2 3

How to parser the message obtained from syslog-ng?
by William Luiz Ribeiro Vasconcelos Da Silva 06 Sep '19

06 Sep '19

Hello every one! The Syslog-ng is writing the log with the layout below: Jun 7 11:54:23 2019 vXXXXX01-node1 RT_XXT: RT_SRC_XXT_PBA_ALLOC: Subscriber 100.64.0.2 But the system that will do the treatment expects to receive the log in the following layout: Jun 7;11:54:23;2019;vxxxxx01-node1;RT_xxT: RT_SRC_NAT_PBA_ALLOC:;Subscriber 100.64.0.2; What setting can I do to be able to do the requested? The expected result is as follows: Current: Jun 7 11:54:23 2019 vXXXXX01-node1 RT_XXT: RT_SRC_XXT_PBA_ALLOC: Subscriber 100.64.0.2 Changed: Jun 7;11:54:23;2019;vxxxxx01-node1;RT_xxT: RT_SRC_NAT_PBA_ALLOC:;Subscriber 100.64.0.2; Tks for the help! Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1

Regular expressions in rewrite
by Faine, Mark R. (MSFC-IS40)[NICS] 06 Sep '19

06 Sep '19

Is there a way to use regular expressions when setting a value in a rewrite rule? Something like: rewrite r_rewrite_set_host{ subst("^[a-z]+\-([a-z]+).*$", "$1", value("HOST")); set("HOST" value("location") condition( filter(f_filter_hostnames) ) ); }; But without changing the actual log message. The goal is to create a variable from a regular expression of the host field in the message that I can then use in a destination path. Thanks, -Mark Mark Faine System Administrator SAIC/NICS 215 Wynn Dr. 5065 Huntsville, AL 35805 256-961-1295 (Desk) 256-617-4861 (Work Cell)

4 4