- syslog-ng - lists.balabit.hu

Load Balancing question
by Janczuk, George 13 Jan '20

13 Jan '20

I'm new to syslog-ng, so please excuse my ignorance if this a newbie or dumb question (I did do some googling first though - regrettably without success). Where I work we have deployed a syslog-ng OSE cluster (five members) fronted by an F5 LTM. One of our up-stream syslog event sources is a service provider's syslog-ng server acting as a relay and passing on all ingested syslog events to our cluster (via the F5 LTM VIP). The up-steam syslog-ng server aggregates events from hundreds of devices, and therefore the stream is very busy (i.e. never idle/quiet). Therefore, it turns out that the connection from the up-stream relay is NEVER dropped and remains in place almost permanently (certainly days at a time). Consequently, without any cycling of the connection the connection is bound to a single cluster member, and ALL events from the up-stream relay are delivered to that single cluster member and are NOT actually load balanced (with the remaining four members essentially being idle as the volume from the relay greatly eclipses the event volume from other even sources). I suppose the question is whether: a) Can we in some way configure the up-steam syslog-ng relay to cycle the outbound connection after some sort of threshold (be that time based - say every 15 seconds; or volume based - say every 100 events; or even both... - whichever occurs first)?, or b) Can the up-stream syslog-ng keep multiple connections open concurrently and can it then use round-robin or stochastic logic to determine which of the connections to send each relayed event to? I would very much like to hear from people about how to actually load balance an event stream from a syslog-ng relay. NOTE: People might ask the question: "If the up-stream relay is a single node, then why do even need a down-stream cluster, if it's not required upstream"? Two reasons really: 1. The up-steam service provider has deployed dedicated chunky tin (i.e. they've used a vertical scale model), whilst for our syslog-ng cluster we're using a private-cloud with much smaller VMs combined with a horizontal scale model, and 2. The relay is doing a fairly minimal collect and forward, whilst our syslog-ng cluster will be doing more event transformation and enrichment, which is ultimately more CPU intensive. ************** IMPORTANT MESSAGE ***************************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Unsubscribe in the subject line. **************************************************************

3 2

Disk-buffer in kafka-destinations
by Ronald Fenner 10 Jan '20

10 Jan '20

I can't see to me get syslog-ng to start if I add the disk-buffer to the destination. From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block. Is this tied to a specific destination type? We are trying to make out messages going to kafka more reliable when the cluster is unavailable. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 4

Increase number of records per file
by William Luiz Ribeiro Vasconcelos Da Silva 10 Jan '20

10 Jan '20

Good morning everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

1 0

customized rewrite to mysql in syslog-ng
by JAHANZAIB SYED 09 Jan '20

09 Jan '20

I have syslog-ng version 3.25.1 on Ubuntu 18.04 server. All logs from remote mikrotik router device is stored in local file and in mysql DB as well. I want to know if there is any method in which we can customize the incoming message and write it to mysql table in different columns. Incoming message example: Jan 8 13:50:24 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49466->172.217.19.10:443, NAT (172.16.0.2:49466->101.11.11.252:2223)->172.217.19.10:443, len 1023 Jan 8 13:53:09 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49482->58.27.130.12:443, NAT (172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443, len 1152 The relevant part that I want to store is HOSTNAME, DATETIME, AND message part 172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443 * into following tables hostname datatime username userip userport wanip wanport dst-ip dst-port 101.11.11.252 xxxx pppoe-zaib 172.16.0.2 49466 101.11.11.252 2223 172.217.19.10 443 [cid:48adc02d-e37b-47af-aa2d-67e59b25f8d6] Regards, SYED JAHANZAIB <http:///>

2 6

Syslog-NG mysql log with auto increment not working
by JAHANZAIB SYED 08 Jan '20

08 Jan '20

I have Syslog-ng (ver 3.25.1) on Ubuntu 18.4 server. I am using following code to dynamically create date wise table and insert records accordingly. ************************ destination d_mysql { sql(type(mysql) host("localhost") username("root") password("XXXXXXXX") database("syslog") table("${R_YEAR}_${R_MONTH}_${R_DAY}") columns( "id int(20) unsigned not null auto_increment primary key", "host varchar(40) not null", "facility varchar(20)", "priority varchar(10)", "level varchar(10)", "program text", "date date not null", "time time not null", "message text not null") values("", "$FULLHOST", "$FACILITY", "$PRIORITY", "$LEVEL", "$PROGRAM", "$R_YEAR-$R_MONTH-$R_DAY", "$R_HOUR:$R_MIN:$R_SEC", "$MSG",) indexes("host","priority")); }; ************************ Above code auto creates table like this ************************ | Field | Type | Null | Key | Default | Extra | +----------+------------------+------+-----+---------+----------------+ | id | int(11) unsigned | NO | PRI | NULL | auto_increment | | host | varchar(40) | NO | MUL | NULL | | | facility | varchar(20) | YES | | NULL | | | priority | varchar(10) | YES | MUL | NULL | | | level | varchar(10) | YES | | NULL | | | program | text | YES | | NULL | | | date | date | NO | | NULL | | | time | time | NO | | NULL | | | message | text | NO | | NULL | | +----------+------------------+------+-----+---------+----------------+ 9 rows in set (0.01 sec) ************************ Problem is that its not inserting record in the table. I enabled mysql log and I can see only one entry ************************ 2020-01-08T05:26:29.016553Z 167 Query INSERT INTO 2020_01_08 (id, host, facility, priority, level, program, date, time, message) VALUES ('NULL', '101.11.11.252', 'user', 'notice', 'notice', 'script,warning', '2020-01-08', '10:23:30', '10:23:30 jan/08/2020') ************************ but this entry is not recorded in the mysql table, When I remove the ID columns/values from the syslog code, DROP the table, & restart syslog-ng service, then table again recates, and all records inserts fine without problem. When I insert record using phpmyadmin, the record enter fine with this code INSERT INTO `2020_01_08` (`id`, `host`, `facility`, `priority`, `level`, `program`, `date`, `time`, `message`) VALUES (NULL, '2', '2', '2', '2', '2', '2020-01-01', '42:00:00', '2'); I tried to add NULL in the values for ID in the syslog-ng code , but no use, as soon as i remove ID DROP the table, & restart syslog-ng service, all goes fine. How can I configure the auto increment column ? Regards, SYED JAHANZAIB Web : http://aacable.wordpress.com LinkedIn: http://pk.linkedin.com/pub/syed-jahanzaib/24/3b/407 <http:///>

2 2

State of syslog-ng on Mac
by Peter Czanik (pczanik) 20 Dec '19

20 Dec '19

Hi, I just published a blog about the state of syslog-ng on Mac: https://www.syslog-ng.com/community/b/blog/posts/state-of-syslog-ng-on-mac As I write about future plans at the end of the blog, you can see that I'm an almost complete newbie on the Mac (the last time I used it actively was on the Motorola 68040) and it's mostly a free time project. Still, any questions, suggestion or help are very welcome! Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

3 3

Add prefix Or suffix inside of FileName
by William Luiz Ribeiro Vasconcelos Da Silva 17 Dec '19

17 Dec '19

Hello everyone, It is possible during syslog-ng processing to insert a suffix, as we have a system that will collect these files. And if we keep the same suffix, we have a big problem inside of prodction enviroment. Here's an example of how the file generation part: destination d_model { file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") owner(syslog) group(syslog) perm (0755) ); }; Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 2

json destination config help
by Zhang, Husen 12 Dec '19

12 Dec '19

Hi community, I'm trying to have syslog-ng to write logs to json. My d_json_syslog-ng.conf: [cid:image001.png@01D5B023.F072D8A0] ├── conf.d │ ├── d_json_syslog-ng.conf │ └── es.conf.bak ├── patterndb.d ├── scl.conf └── syslog-ng.conf

3 5

syslog-ng 3.25 packages for Debian/Ubuntu
by Laszlo Budai 12 Dec '19

12 Dec '19

Hi, syslog-ng 3.25.1[1] packages are available in OBS repositories[2, 3]. List of supported OSs: * Debian 8.0 * Debian 9.0 * Debian 10.0 * Raspbian 9.0 * Raspbian 10.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 * Ubuntu 19.04 * Ubuntu 19.10 Note that Debian 10 and Ubuntu 19.04 are available from a separate repository[3] and there is no java support on these distributions. The reason is that these OBS builders don't have the required openjdk packages. Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.25.1 [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.25.1
by Laszlo Budai 12 Dec '19

12 Dec '19

3.25.1 Highlights * http-destination: Users now can specify the action for any HTTP result code. Use with response-action(response_code => action) in your http block. Available actions are: success, retry, drop and disconnect. (#3007<https://github.com/syslog-ng/syslog-ng/pull/3007>) * syslog-ng-cfg-db: Added a new script, which can provide the options of sources and destinations queried by the user. This tool can make the configuration of syslog-ng a lot easier. Use with ./syslog-ng-cfg-db.py from the contrib/config_database dir.(#2997<https://github.com/syslog-ng/syslog-ng/pull/2997>) * redis-destination: Improved the performance by 2 orders of magnitude. In our labor environment, now it operates at 25k EPS. (#2972<https://github.com/syslog-ng/syslog-ng/pull/2972>) Features * create-dirs(): Added to pipe() source/destination, and standardize the behavior. (#3018<https://github.com/syslog-ng/syslog-ng/pull/3018>, #2635<https://github.com/syslog-ng/syslog-ng/issues/2635>) * default-network-drivers: Added max-connections() option, to change the limit from 10. (#2961<https://github.com/syslog-ng/syslog-ng/pull/2961>) * checkpoint: Added support for timezone value at the end of timestamps. (#3033<https://github.com/syslog-ng/syslog-ng/pull/3033>) * filter/rewrite: Added disable-jit flag to disable JIT PCRE compilation. (#2992<https://github.com/syslog-ng/syslog-ng/pull/2992>, #2986<https://github.com/syslog-ng/syslog-ng/issues/2986>) * syslog-ng-ctl: Added export-config-graph option to visualize config graph. (#2990<https://github.com/syslog-ng/syslog-ng/pull/2990>) * build/travis: Added ARM64 arch support. (#2967<https://github.com/syslog-ng/syslog-ng/pull/2967>) * build/dbld: Readded CentOS 6 support. (#2860<https://github.com/syslog-ng/syslog-ng/pull/2860>, #2971<https://github.com/syslog-ng/syslog-ng/pull/2971>, #3028<https://github.com/syslog-ng/syslog-ng/pull/3028>) * python: Added Python 3.8 support. (#3017<https://github.com/syslog-ng/syslog-ng/pull/3017>) Bugfixes * tls: Fixed an infinite loop which occured, when a TLS connection broke. (#3026<https://github.com/syslog-ng/syslog-ng/pull/3026>, #3009<https://github.com/syslog-ng/syslog-ng/issues/3009>) * log-block: Fixed an issue, where inline network destinations disjointed the rest of the config. (#2989<https://github.com/syslog-ng/syslog-ng/pull/2989>, #2820<https://github.com/syslog-ng/syslog-ng/issues/2820>) * kafka/network-load-balancer: Fixed a crash when an argument was set to empty. (#3002<https://github.com/syslog-ng/syslog-ng/pull/3002>) * python-source: Fixed a memory corruption during reload. (#3014<https://github.com/syslog-ng/syslog-ng/pull/3014>) * python-destination: Actually use return value of open method. (#2998<https://github.com/syslog-ng/syslog-ng/pull/2998>, #2513<https://github.com/syslog-ng/syslog-ng/issues/2513>) * python-fetcher: Fixed FETCH_NO_DATA and FETCH_TRY_AGAIN constants. (#3012<https://github.com/syslog-ng/syslog-ng/pull/3012>) * python: Fixed python Exception reporting when no Exception happened. (#2995<https://github.com/syslog-ng/syslog-ng/pull/2995>) * telegram: Fixed the syntax error of the use-system-cert-store() option. (#2977<https://github.com/syslog-ng/syslog-ng/pull/2977>) * config: Throw error to single dots, which were ignored before. (#3000<https://github.com/syslog-ng/syslog-ng/pull/3000>) * file-destination: Delay ACKs until messages are written to disk. This fixes message drop on I/O error and message lost in the LogProtoFileWriter in case of a crash, by retrying to send the message. (#2985<https://github.com/syslog-ng/syslog-ng/pull/2985>) * http-destination: Handle global template options values. (#3020<https://github.com/syslog-ng/syslog-ng/pull/3020>) * timeutils: Fixed month and day name parsing, when only the first 2 characters matched. (#3035<https://github.com/syslog-ng/syslog-ng/pull/3035>) * logmsg: Added default PRI value (LOG_USER | LOG_NOTICE) to log messages created without initial parsing. (#2974<https://github.com/syslog-ng/syslog-ng/pull/2974>) * packaging: Added ordering dependencies network.target and network-online.target to the service files. (#2994<https://github.com/syslog-ng/syslog-ng/pull/2994>, #2667<https://github.com/syslog-ng/syslog-ng/issues/2667>) * amqp: Support older (0.7.1) version (#2999<https://github.com/syslog-ng/syslog-ng/pull/2999>) * loggen: Set plugin path in installation time. (#3019<https://github.com/syslog-ng/syslog-ng/pull/3019>) * timeutils/patterndb: Fixed some undefined behaviours. (#2969<https://github.com/syslog-ng/syslog-ng/pull/2969>) * stomp: Fixed a buffer over-read on connection. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) * pseudofile: Fixed a crash, when template() option is not set. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) * wildcard-source: Fixed a crash, when max-files() was set to 0. (#2988<https://github.com/syslog-ng/syslog-ng/pull/2988>) Other changes * syslog-ng-debun: Various maintenance updates and small fixes. (#2993<https://github.com/syslog-ng/syslog-ng/pull/2993>) * scl: Avoid @requires loading the plugins themselves. (#2887<https://github.com/syslog-ng/syslog-ng/pull/2887>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Clément Besnier, Gabor Nagy, jadhavsumit98, Janos Szigetvari, Laszlo Budai, Laszlo Szemere, László Várady, MikeLim, Nikita Uvarov, Norbert Takacs, pabloli, Péter Kókai, Zoltan Pallagi.

1 0