- syslog-ng - lists.balabit.hu

Rewrite set var value with tag?
by Faine, Mark R. (MSFC-IS40)[NICS] 20 Jan '20

20 Jan '20

Is it possible to do something like: rewrite { set-tag("bar"); }; And then use a rewrite to set the value of a variable to the value of the tag rewrite { set("${TAGS}", value("foo")); }; I need to set a value that I've determined in a log path at the top of the file in several log paths that occur after. It looks the scope for variables is at the log path level so that doesn't help me. I'm also trying to avoid nesting all of the other log paths inside of the top log path where I determine the value. I'm trying to avoid: log{ ... filter{ ... tags("bar"); rewrite{ set("bar", "foo"); }; }; }; log{... log{... Thanks, -Mark

3 3

Go away
by johns spam 16 Jan '20

16 Jan '20

Hello, I am wondering if there is any facility to tell a client to go away? i.e. https://http2.github.io/http2-spec/#GOAWAY I know syslog-ng is not http2 but is there any side-channel mechanism a central server can use to make a client go away? The use case I am thinking is doing a rolling restart of a pool of syslog-ng central server. I think this could also be useful to cycle connections periodically, i.e. to rebalance connections to a central server. Thanks

2 1

librdkafka and messages dropped
by John Skopis 15 Jan '20

15 Jan '20

Hello, We have a syslog-ng aggregator that accepts messages in a few different formats and logs the message format to kafka. The mapping looks something like: format1 -> topic1 format2 -> topic2 ... Each agent will emit messages in ~4 different formats and send to the aggregator. The aggregator is configured with a kafka_c destination with 8 threads/destination. We were previously using the java kafka client without issue. We recently upgraded to 3.24.1 + librdkafka 1.22 After upgrading we are seeing an issue where (seemingly randomly) syslog-ng stops producing messages into kafka. Checking syslog-ng-ctl stats I can see d_kafka_format1 has dropped message counter increasing d_kafka_format1 queued messages counter increases to 80000 (10000*8 threads iw-size) d_kafka_format1 processed messages does not increase There are messages printed to the log that the destination queue is full for all of the rdkafka threads, There are spurious failures to a kafka broker but it seems rdkafka reconnects. Seems like once the destination queue becomes full syslog-ng never recovers and the instance must be restarted. It also seems like rdkafka stops trying to reconnect to kafka (possibly?). Is this expected behavior or is there some bug around reconnecting to kafka after a spurious timeout? Just speculating here but does rdkafka stop trying to reconnect after N times? Are there any examples of destroy/create destination client after failure thresholds? Thanks

2 5

3.25 and my RPM stable repos
by Peter Czanik (pczanik) 13 Jan '20

13 Jan '20

Hi, Version 3.25 of syslog-ng was released a week ago, and it seems to be rock solid. Still, as today is my last working day this year, I feel more comfortable to push it into my stable repos only on the first week of January, when I am back. If you still want to experiment with 3.25 on openSUSE/SLES or fedora/RHEL/CentOS you can do that in the versioned repositories. Note that RHEL 8 / CentOS 8 support is now merged with the rest, it's not in a separate repository any more. suse: https://build.opensuse.org/project/show/home:czanik:syslog-ng325 red hat: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng325/ Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Lines per file + Syslog-ng
by William Luiz Ribeiro Vasconcelos Da Silva 13 Jan '20

13 Jan '20

Hello everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1

Load Balancing question
by Janczuk, George 13 Jan '20

13 Jan '20

I'm new to syslog-ng, so please excuse my ignorance if this a newbie or dumb question (I did do some googling first though - regrettably without success). Where I work we have deployed a syslog-ng OSE cluster (five members) fronted by an F5 LTM. One of our up-stream syslog event sources is a service provider's syslog-ng server acting as a relay and passing on all ingested syslog events to our cluster (via the F5 LTM VIP). The up-steam syslog-ng server aggregates events from hundreds of devices, and therefore the stream is very busy (i.e. never idle/quiet). Therefore, it turns out that the connection from the up-stream relay is NEVER dropped and remains in place almost permanently (certainly days at a time). Consequently, without any cycling of the connection the connection is bound to a single cluster member, and ALL events from the up-stream relay are delivered to that single cluster member and are NOT actually load balanced (with the remaining four members essentially being idle as the volume from the relay greatly eclipses the event volume from other even sources). I suppose the question is whether: a) Can we in some way configure the up-steam syslog-ng relay to cycle the outbound connection after some sort of threshold (be that time based - say every 15 seconds; or volume based - say every 100 events; or even both... - whichever occurs first)?, or b) Can the up-stream syslog-ng keep multiple connections open concurrently and can it then use round-robin or stochastic logic to determine which of the connections to send each relayed event to? I would very much like to hear from people about how to actually load balance an event stream from a syslog-ng relay. NOTE: People might ask the question: "If the up-stream relay is a single node, then why do even need a down-stream cluster, if it's not required upstream"? Two reasons really: 1. The up-steam service provider has deployed dedicated chunky tin (i.e. they've used a vertical scale model), whilst for our syslog-ng cluster we're using a private-cloud with much smaller VMs combined with a horizontal scale model, and 2. The relay is doing a fairly minimal collect and forward, whilst our syslog-ng cluster will be doing more event transformation and enrichment, which is ultimately more CPU intensive. ************** IMPORTANT MESSAGE ***************************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Unsubscribe in the subject line. **************************************************************

3 2

Disk-buffer in kafka-destinations
by Ronald Fenner 10 Jan '20

10 Jan '20

I can't see to me get syslog-ng to start if I add the disk-buffer to the destination. From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block. Is this tied to a specific destination type? We are trying to make out messages going to kafka more reliable when the cluster is unavailable. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 4

Increase number of records per file
by William Luiz Ribeiro Vasconcelos Da Silva 10 Jan '20

10 Jan '20

Good morning everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

1 0

customized rewrite to mysql in syslog-ng
by JAHANZAIB SYED 09 Jan '20

09 Jan '20

I have syslog-ng version 3.25.1 on Ubuntu 18.04 server. All logs from remote mikrotik router device is stored in local file and in mysql DB as well. I want to know if there is any method in which we can customize the incoming message and write it to mysql table in different columns. Incoming message example: Jan 8 13:50:24 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49466->172.217.19.10:443, NAT (172.16.0.2:49466->101.11.11.252:2223)->172.217.19.10:443, len 1023 Jan 8 13:53:09 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49482->58.27.130.12:443, NAT (172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443, len 1152 The relevant part that I want to store is HOSTNAME, DATETIME, AND message part 172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443 * into following tables hostname datatime username userip userport wanip wanport dst-ip dst-port 101.11.11.252 xxxx pppoe-zaib 172.16.0.2 49466 101.11.11.252 2223 172.217.19.10 443 [cid:48adc02d-e37b-47af-aa2d-67e59b25f8d6] Regards, SYED JAHANZAIB <http:///>

2 6

Syslog-NG mysql log with auto increment not working
by JAHANZAIB SYED 08 Jan '20

08 Jan '20

I have Syslog-ng (ver 3.25.1) on Ubuntu 18.4 server. I am using following code to dynamically create date wise table and insert records accordingly. ************************ destination d_mysql { sql(type(mysql) host("localhost") username("root") password("XXXXXXXX") database("syslog") table("${R_YEAR}_${R_MONTH}_${R_DAY}") columns( "id int(20) unsigned not null auto_increment primary key", "host varchar(40) not null", "facility varchar(20)", "priority varchar(10)", "level varchar(10)", "program text", "date date not null", "time time not null", "message text not null") values("", "$FULLHOST", "$FACILITY", "$PRIORITY", "$LEVEL", "$PROGRAM", "$R_YEAR-$R_MONTH-$R_DAY", "$R_HOUR:$R_MIN:$R_SEC", "$MSG",) indexes("host","priority")); }; ************************ Above code auto creates table like this ************************ | Field | Type | Null | Key | Default | Extra | +----------+------------------+------+-----+---------+----------------+ | id | int(11) unsigned | NO | PRI | NULL | auto_increment | | host | varchar(40) | NO | MUL | NULL | | | facility | varchar(20) | YES | | NULL | | | priority | varchar(10) | YES | MUL | NULL | | | level | varchar(10) | YES | | NULL | | | program | text | YES | | NULL | | | date | date | NO | | NULL | | | time | time | NO | | NULL | | | message | text | NO | | NULL | | +----------+------------------+------+-----+---------+----------------+ 9 rows in set (0.01 sec) ************************ Problem is that its not inserting record in the table. I enabled mysql log and I can see only one entry ************************ 2020-01-08T05:26:29.016553Z 167 Query INSERT INTO 2020_01_08 (id, host, facility, priority, level, program, date, time, message) VALUES ('NULL', '101.11.11.252', 'user', 'notice', 'notice', 'script,warning', '2020-01-08', '10:23:30', '10:23:30 jan/08/2020') ************************ but this entry is not recorded in the mysql table, When I remove the ID columns/values from the syslog code, DROP the table, & restart syslog-ng service, then table again recates, and all records inserts fine without problem. When I insert record using phpmyadmin, the record enter fine with this code INSERT INTO `2020_01_08` (`id`, `host`, `facility`, `priority`, `level`, `program`, `date`, `time`, `message`) VALUES (NULL, '2', '2', '2', '2', '2', '2020-01-01', '42:00:00', '2'); I tried to add NULL in the values for ID in the syslog-ng code , but no use, as soon as i remove ID DROP the table, & restart syslog-ng service, all goes fine. How can I configure the auto increment column ? Regards, SYED JAHANZAIB Web : http://aacable.wordpress.com LinkedIn: http://pk.linkedin.com/pub/syed-jahanzaib/24/3b/407 <http:///>

2 2

State of syslog-ng on Mac
by Peter Czanik (pczanik) 20 Dec '19

20 Dec '19

Hi, I just published a blog about the state of syslog-ng on Mac: https://www.syslog-ng.com/community/b/blog/posts/state-of-syslog-ng-on-mac As I write about future plans at the end of the blog, you can see that I'm an almost complete newbie on the Mac (the last time I used it actively was on the Motorola 68040) and it's mostly a free time project. Still, any questions, suggestion or help are very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 3

Add prefix Or suffix inside of FileName
by William Luiz Ribeiro Vasconcelos Da Silva 17 Dec '19

17 Dec '19

Hello everyone, It is possible during syslog-ng processing to insert a suffix, as we have a system that will collect these files. And if we keep the same suffix, we have a big problem inside of prodction enviroment. Here's an example of how the file generation part: destination d_model { file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") owner(syslog) group(syslog) perm (0755) ); }; Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 2

json destination config help
by Zhang, Husen 12 Dec '19

12 Dec '19

Hi community, I'm trying to have syslog-ng to write logs to json. My d_json_syslog-ng.conf: [cid:image001.png@01D5B023.F072D8A0] ├── conf.d │ ├── d_json_syslog-ng.conf │ └── es.conf.bak ├── patterndb.d ├── scl.conf └── syslog-ng.conf

3 5

syslog-ng 3.25 packages for Debian/Ubuntu
by Laszlo Budai 12 Dec '19

12 Dec '19

Hi, syslog-ng 3.25.1[1] packages are available in OBS repositories[2, 3]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 Note that Debian 10 and Ubuntu 19.04 are available from a separate repository[3] and there is no java support on these distributions. The reason is that these OBS builders don't have the required openjdk packages. Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.25.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.25.1
by Laszlo Budai 12 Dec '19

12 Dec '19

3.25.1 Highlights * http-destination: Users now can specify the action for any HTTP result code. Use with response-action(response_code => action) in your http block. Available actions are: success, retry, drop and disconnect. (#3007<https://github.com/syslog-ng/syslog-ng/pull/3007>) * syslog-ng-cfg-db: Added a new script, which can provide the options of sources and destinations queried by the user. This tool can make the configuration of syslog-ng a lot easier. Use with ./syslog-ng-cfg-db.py from the contrib/config_database dir.(#2997<https://github.com/syslog-ng/syslog-ng/pull/2997>) * redis-destination: Improved the performance by 2 orders of magnitude. In our labor environment, now it operates at 25k EPS. (#2972<https://github.com/syslog-ng/syslog-ng/pull/2972>) Features * create-dirs(): Added to pipe() source/destination, and standardize the behavior. (#3018<https://github.com/syslog-ng/syslog-ng/pull/3018>, #2635<https://github.com/syslog-ng/syslog-ng/issues/2635>) * default-network-drivers: Added max-connections() option, to change the limit from 10. (#2961<https://github.com/syslog-ng/syslog-ng/pull/2961>) * checkpoint: Added support for timezone value at the end of timestamps. (#3033<https://github.com/syslog-ng/syslog-ng/pull/3033>) * filter/rewrite: Added disable-jit flag to disable JIT PCRE compilation. (#2992<https://github.com/syslog-ng/syslog-ng/pull/2992>, #2986<https://github.com/syslog-ng/syslog-ng/issues/2986>) * syslog-ng-ctl: Added export-config-graph option to visualize config graph. (#2990<https://github.com/syslog-ng/syslog-ng/pull/2990>) * build/travis: Added ARM64 arch support. (#2967<https://github.com/syslog-ng/syslog-ng/pull/2967>) * build/dbld: Readded CentOS 6 support. (#2860<https://github.com/syslog-ng/syslog-ng/pull/2860>, #2971<https://github.com/syslog-ng/syslog-ng/pull/2971>, #3028<https://github.com/syslog-ng/syslog-ng/pull/3028>) * python: Added Python 3.8 support. (#3017<https://github.com/syslog-ng/syslog-ng/pull/3017>) Bugfixes * tls: Fixed an infinite loop which occured, when a TLS connection broke. (#3026<https://github.com/syslog-ng/syslog-ng/pull/3026>, #3009<https://github.com/syslog-ng/syslog-ng/issues/3009>) * log-block: Fixed an issue, where inline network destinations disjointed the rest of the config. (#2989<https://github.com/syslog-ng/syslog-ng/pull/2989>, #2820<https://github.com/syslog-ng/syslog-ng/issues/2820>) * kafka/network-load-balancer: Fixed a crash when an argument was set to empty. (#3002<https://github.com/syslog-ng/syslog-ng/pull/3002>) * python-source: Fixed a memory corruption during reload. (#3014<https://github.com/syslog-ng/syslog-ng/pull/3014>) * python-destination: Actually use return value of open method. (#2998<https://github.com/syslog-ng/syslog-ng/pull/2998>, #2513<https://github.com/syslog-ng/syslog-ng/issues/2513>) * python-fetcher: Fixed FETCH_NO_DATA and FETCH_TRY_AGAIN constants. (#3012<https://github.com/syslog-ng/syslog-ng/pull/3012>) * python: Fixed python Exception reporting when no Exception happened. (#2995<https://github.com/syslog-ng/syslog-ng/pull/2995>) * telegram: Fixed the syntax error of the use-system-cert-store() option. (#2977<https://github.com/syslog-ng/syslog-ng/pull/2977>) * config: Throw error to single dots, which were ignored before. (#3000<https://github.com/syslog-ng/syslog-ng/pull/3000>) * file-destination: Delay ACKs until messages are written to disk. This fixes message drop on I/O error and message lost in the LogProtoFileWriter in case of a crash, by retrying to send the message. (#2985<https://github.com/syslog-ng/syslog-ng/pull/2985>) * http-destination: Handle global template options values. (#3020<https://github.com/syslog-ng/syslog-ng/pull/3020>) * timeutils: Fixed month and day name parsing, when only the first 2 characters matched. (#3035<https://github.com/syslog-ng/syslog-ng/pull/3035>) * logmsg: Added default PRI value (LOG_USER | LOG_NOTICE) to log messages created without initial parsing. (#2974<https://github.com/syslog-ng/syslog-ng/pull/2974>) * packaging: Added ordering dependencies network.target and network-online.target to the service files. (#2994<https://github.com/syslog-ng/syslog-ng/pull/2994>, #2667<https://github.com/syslog-ng/syslog-ng/issues/2667>) * amqp: Support older (0.7.1) version (#2999<https://github.com/syslog-ng/syslog-ng/pull/2999>) * loggen: Set plugin path in installation time. (#3019<https://github.com/syslog-ng/syslog-ng/pull/3019>) * timeutils/patterndb: Fixed some undefined behaviours. (#2969<https://github.com/syslog-ng/syslog-ng/pull/2969>) * stomp: Fixed a buffer over-read on connection. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) * pseudofile: Fixed a crash, when template() option is not set. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) * wildcard-source: Fixed a crash, when max-files() was set to 0. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) Other changes * syslog-ng-debun: Various maintenance updates and small fixes. (#2993<https://github.com/syslog-ng/syslog-ng/pull/2993>) * scl: Avoid @requires loading the plugins themselves. (#2887<https://github.com/syslog-ng/syslog-ng/pull/2887>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Clément Besnier, Gabor Nagy, jadhavsumit98, Janos Szigetvari, Laszlo Budai, Laszlo Szemere, László Várady, MikeLim, Nikita Uvarov, Norbert Takacs, pabloli, Péter Kókai, Zoltan Pallagi.

1 0

Insider 2019-12: Kibana 7 & GeoIP; PE 6 to 7 upgrade; RHEL 8; Elastic stack;
by Peter Czanik (pczanik) 12 Dec '19

12 Dec '19

Dear syslog-ng users, This is the 77th issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news. NEWS Set up Kibana 7 for syslog-ng & GeoIP ------------------------------------- Version 7 of the Elastic stack was released a few months ago, and brought several breaking changes that affect syslog-ng. In my previous blog post, I gave details about how it affects sending GeoIP information to Elasticsearch. From this blog post you can learn about the Kibana side, which has also changed considerably compared to previous releases. Configuration files for syslog-ng are included, but not explained in depth, as that was already done in previous posts. https://www.syslog-ng.com/community/b/blog/posts/set-up-kibana-7-for-syslog… Upgrading a syslog-ng PE 6 configuration to 7 --------------------------------------------- Upgrading syslog-ng PE 6 to 7 is no more difficult than any major version upgrade. The key here is to use the latest version of PE 7 (at least version 7.0.17), as it has the necessary compatibility fixes to make the upgrade experience smoother. From this blog, you can learn how to upgrade a simple configuration containing a wildcard file source and an RLTP source to work correctly with syslog-ng PE 7. https://www.syslog-ng.com/community/b/blog/posts/upgrading-a-syslog-ng-pe-6… State of syslog-ng on RHEL 8 / CentOS 8 --------------------------------------- Version 8, a new major version of Red Hat Enterprise Linux was released this spring. Now that CentOS 8 is also available, there is a rapidly growing interest in syslog-ng running on these platforms. From this blog, you can learn about the availability of the current syslog-ng package and future plans. https://www.syslog-ng.com/community/b/blog/posts/state-of-syslog-ng-on-rhel… Using syslog-ng with the Elastic stack -------------------------------------- One of the most popular destinations of syslog-ng is Elasticsearch. Any time a new language binding was introduced to syslog-ng, someone implemented an Elasticsearch destination for it. For many years, the official Elasticsearch destination for syslog-ng was implemented in Java. With the recent enhancements to the http() destination of syslog-ng, a new, native C-based implementation called the elasticsearch-http() destination is available. https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-with-the-e… WEBINARS * You can achieve 1m+ UDP messages per second ingestion challenge: https://www.syslog-ng.com/webcast-ondemand/you-can-achieve-1m-udp-messages-… * You can browse recordings of past webinars at https://www.syslog-ng.com/events/ Your feedback and news, or tips about the next issue are welcome. To read this newsletter online, visit: https://syslog-ng.com/blog/ Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

How to syslog-ng execute shell script
by William Luiz Ribeiro Vasconcelos Da Silva 10 Dec '19

10 Dec '19

Hello People, How to configure the syslog-ng.conf file, to call a shell script, for example to do zip file. It is possible to do this? Tks, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 5

Support for netflow logs
by Raghunath Adhyapak 04 Dec '19

04 Dec '19

Hi, I was trying to receive Netflow logs from firewall devices in syslog-ng and then forward to a central server. Does syslog-ng support netflow such that I can validate and filter out all non-netflow log lines? I also dumped some netflow logs to a file and found it to be binary. Therefore I haven't been able to ascertain the format and filtering mechanism. Any pointers on this topic would be helpful. Thanks Raghu

4 7

Its possible compress file using syslog-ng
by William Luiz Ribeiro Vasconcelos Da Silva 03 Dec '19

03 Dec '19

How to configure the syslog-ng.conf to do compress file? Its possible to use this feature inside of file structure: file("/<dir>/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}${SEC}_GMT${TZ}_${UNIQID}.log" template("${PROGRAM};${MESSAGE};\n") owner(user) group(user) perm (0755) Tks, Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

1 0

Structure data set to "-"
by Debjyoti Mukherjee 03 Dec '19

03 Dec '19

Hello Trying to send logs to remote syslog server in RFC 5424 format. The STRUCTURE_DATA should be set to "-". What is the way to the this value to "-" Currently it is coming as [meta sequenceId="21"]. I am using Openwrt and the syslog version is 3.24 Thank you

2 5

Uniqid
by William Luiz Ribeiro Vasconcelos Da Silva 03 Dec '19

03 Dec '19

Hello People, I'm using the uniqid in my syslog-ng.conf , but when using it in the file name, the output was presented in hexadecimal, example: "8e99d7b2(a)000000000000040b.log" How to convert the output to decimal because we have a balancer layer to collect files ending in even numbers and another layer for odd numbers. This is part of syslog-ng.conf: "${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}${SEC}_GMT${TZ}_${UNIQID}.log" Tks Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1

SYSLOGHOST is being replaced with IP
by Raghunath Adhyapak 29 Nov '19

29 Nov '19

Hi, I'm observing that syslog-ng is modifying the SYSLOGHOST in the incoming log line and outputting an IP instead. I would like to retain the incoming hostname in incoming syslog and forward the same information. Here is my incoming log line: <13>Nov 29 04:07:40 BVRM-DC04 AgentDevice=WindowsLog\tAgentLogFile=Security\tPluginVersion=7.2.8.91\tSource=Microsoft-Windows-Security-Auditing\tComputer= BVRM-DC04.xxxxxxxx.com\tOriginatingComputer=172.26.1.60\tUser=\tDomain=\tEventID=4634\tEventIDCode=4634\tEventType=8\tEventCategory=12545\tRecordNumber=166757582\tTimeGenerated=1575029259\tTimeWritten=1575029259\tLevel=Log Always\tKeywords=Audit Success\tTask=SE_ADT_LOGON_LOGOFF\tOpcode=Info\tMessage=An account was logged off. Outgoing log line: <13>Nov 29 04:07:40 172.22.2.55 AgentDevice=WindowsLog\tAgentLogFile=Security\tPluginVersion=7.2.8.91\tSource=Microsoft-Windows-Security-Auditing\tComputer= BVRM-DC04.xxxxxxxx.com\tOriginatingComputer=172.26.1.60\tUser=\tDomain=\tEventID=4634\tEventIDCode=4634\tEventType=8\tEventCategory=12545\tRecordNumber=166757582\tTimeGenerated=1575029259\tTimeWritten=1575029259\tLevel=Log Always\tKeywords=Audit Success\tTask=SE_ADT_LOGON_LOGOFF\tOpcode=Info\tMessage=An account was logged off. FYI, this is log from Windows, but same is happening for syslog from other firewalls as well. My syslog-ng.conf: @version: 3.24 @include "scl.conf" ######################## # Sources ######################## source s_test_net { syslog(transport(udp) port(2514) ); }; ######################## # Destinations ######################## destination d_test { file("/tmp/test.log"); }; ######################## # Log paths ######################## log { source(s_test_net); destination(d_test); }; Thanks Raghu

2 2

syslog-ng packages for RHEL 8 / CentOS 8
by Peter Czanik (pczanik) 27 Nov '19

27 Nov '19

Hi, Today I reached a new milestone with syslog-ng packaging for RHEL 8 / CentOS 8, so here is a quick update: * syslog-ng 3.23 is available in EPEL 8 ( https://fedoraproject.org/wiki/EPEL ). It is missing the C-based Kafka driver (librdkafka in RHEL is too old) and the Java-based destination drivers (Kafka, Elastic, HDFS) due to missing dependencies in RHEL/EPEL. * highly experimental syslog-ng package missing most of the features needing extra dependencies. Core features, Elasticsearch (HTTP) destination are available in the package. You need to first enable the EPEL 8 repository and then add: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng324el8/ * Today I successfully added RHEL 8 support to my git snapshot repository: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ Some dependencies, like recent version of Kafka are available from this repository, the rest from EPEL. As far as I am aware, all OSE features are now supported, except for tcp-wrapper, which was removed from Fedora & RHEL 8. This repository is regularly updated from git. * Once syslog-ng 3.25 is available, the upcoming syslog-ng325 repo will already support RHEL 8 / CentOS 8 and soon after my syslog-ng-stable repo will also receive RHEL 8 support. As usual, any feedback is very welcome and will be handled with best effort. I'd especially welcome feedback on my syslog-ng-githead repo -- even if that's not intended for production environments -- as that's where I prepare for the next syslog-ng release. Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

syntax error when using syslog-ng.conf from debian packaging
by Alexandre Santos 21 Nov '19

21 Nov '19

Hi, I am running into the following error when running the syslog-ng with the syslog-ng.conf from debian packaging: *Error parsing affile, syntax error, unexpected ')', expecting LL_IDENTIFIER or LL_STRING in /usr/local/etc/syslog-ng.conf:68:34-68:35:63 #64 destination d_console { usertty("root"); };6566 # Virtual console.67 #68----> destination d_console_all { file(`tty10`); };68----> ^* Do you have any idea about this? Thanks in advance, Alex

3 4

Re: [syslog-ng] Abort message processing - multiple includes
by Matthias Gruber 19 Nov '19

19 Nov '19

Hi! Good Lord, I think I just found the answer to my question in the Documentation :-/ :-) "This default behavior can be changed using the flags() parameter. Flags apply to individual log paths, they are not global options. " Therefore I think the final breaks out of the log-processing of the actual log-directive, but others "matching" will processed Argh... think I have to cat some of my config-files together... Sorry for "spamming" :-) cheers Matthias P.S. If I am wrong, please correct me (hope dies last ;->) ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Kommanditgesellschaft auf Aktien Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Von: Matthias Gruber/METZLER An: "Syslog-ng users' and developers' mailing list" <syslog-ng(a)lists.balabit.hu> Datum: 19.11.2019 08:54 Betreff: Abort message processing - multiple includes Hi! Perhaps I misunderstood it, or got a bit crossed up, because I dig into syslog-ng-Configs since weeks, but I am unsure about the "flags(final)" statement I have several includes with several log-statements and several sources e.g. (no runnable syntax, only for demonstration-purpose) 0001-A.conf log { source(s_tcp_A); filter(f_filter_A); destination(d_somewhere_A); flags(final); }; 0002-B.conf log { source(s_tcp_A); destination(d_somewhere_BBB); flags(final); }; As I understand it, first of all, its included in "alphabetical order", so in my case first 0001-A.conf, then 0002-B.conf and so on.... But now my "Problem" and question (Yes I would normally do a "if"-Construct to solve that, problem above, but I used that example to understand the flags(final),) If I have a message received by s_tcp_A, filter f_filter_A hits, the rule sends it to d_somewhere_A and do a "Escape" cause of flags(final), does the message still got processes by the include of 0002-B.conf?? Does the flags(final) is only relevant for one "log-rule", or does it ends the whole processing? To be honest, I habe several includes, and a "last" one, which is imported last, with a "hit"-all, in my childish thought, if the flags(final) stopps processing of the message compleetly this hit-all only gets "the junk" where nothing else hists. Why do I do that, and not write it in one file with a large if-elif-else-construct, I do that for the readability since this file would be extremly large. I am thankfull for any help cheers Matthias ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Kommanditgesellschaft auf Aktien Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Persönlich haftende Gesellschafter: Harald Illy, Michael Klaus, Friedrich von Metzler, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 27 515 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

1 0