January 2020 - syslog-ng - lists.balabit.hu

Load Balancing question
by Janczuk, George 13 Jan '20

13 Jan '20

I'm new to syslog-ng, so please excuse my ignorance if this a newbie or dumb question (I did do some googling first though - regrettably without success). Where I work we have deployed a syslog-ng OSE cluster (five members) fronted by an F5 LTM. One of our up-stream syslog event sources is a service provider's syslog-ng server acting as a relay and passing on all ingested syslog events to our cluster (via the F5 LTM VIP). The up-steam syslog-ng server aggregates events from hundreds of devices, and therefore the stream is very busy (i.e. never idle/quiet). Therefore, it turns out that the connection from the up-stream relay is NEVER dropped and remains in place almost permanently (certainly days at a time). Consequently, without any cycling of the connection the connection is bound to a single cluster member, and ALL events from the up-stream relay are delivered to that single cluster member and are NOT actually load balanced (with the remaining four members essentially being idle as the volume from the relay greatly eclipses the event volume from other even sources). I suppose the question is whether: a) Can we in some way configure the up-steam syslog-ng relay to cycle the outbound connection after some sort of threshold (be that time based - say every 15 seconds; or volume based - say every 100 events; or even both... - whichever occurs first)?, or b) Can the up-stream syslog-ng keep multiple connections open concurrently and can it then use round-robin or stochastic logic to determine which of the connections to send each relayed event to? I would very much like to hear from people about how to actually load balance an event stream from a syslog-ng relay. NOTE: People might ask the question: "If the up-stream relay is a single node, then why do even need a down-stream cluster, if it's not required upstream"? Two reasons really: 1. The up-steam service provider has deployed dedicated chunky tin (i.e. they've used a vertical scale model), whilst for our syslog-ng cluster we're using a private-cloud with much smaller VMs combined with a horizontal scale model, and 2. The relay is doing a fairly minimal collect and forward, whilst our syslog-ng cluster will be doing more event transformation and enrichment, which is ultimately more CPU intensive. ************** IMPORTANT MESSAGE ***************************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Unsubscribe in the subject line. **************************************************************

3 2

Disk-buffer in kafka-destinations
by Ronald Fenner 10 Jan '20

10 Jan '20

I can't see to me get syslog-ng to start if I add the disk-buffer to the destination. From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block. Is this tied to a specific destination type? We are trying to make out messages going to kafka more reliable when the cluster is unavailable. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 4

Increase number of records per file
by William Luiz Ribeiro Vasconcelos Da Silva 10 Jan '20

10 Jan '20

Good morning everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

1 0

customized rewrite to mysql in syslog-ng
by JAHANZAIB SYED 09 Jan '20

09 Jan '20

I have syslog-ng version 3.25.1 on Ubuntu 18.04 server. All logs from remote mikrotik router device is stored in local file and in mysql DB as well. I want to know if there is any method in which we can customize the incoming message and write it to mysql table in different columns. Incoming message example: Jan 8 13:50:24 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49466->172.217.19.10:443, NAT (172.16.0.2:49466->101.11.11.252:2223)->172.217.19.10:443, len 1023 Jan 8 13:53:09 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49482->58.27.130.12:443, NAT (172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443, len 1152 The relevant part that I want to store is HOSTNAME, DATETIME, AND message part 172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443 * into following tables hostname datatime username userip userport wanip wanport dst-ip dst-port 101.11.11.252 xxxx pppoe-zaib 172.16.0.2 49466 101.11.11.252 2223 172.217.19.10 443 [cid:48adc02d-e37b-47af-aa2d-67e59b25f8d6] Regards, SYED JAHANZAIB <http:///>

2 6

Syslog-NG mysql log with auto increment not working
by JAHANZAIB SYED 08 Jan '20

08 Jan '20

I have Syslog-ng (ver 3.25.1) on Ubuntu 18.4 server. I am using following code to dynamically create date wise table and insert records accordingly. ************************ destination d_mysql { sql(type(mysql) host("localhost") username("root") password("XXXXXXXX") database("syslog") table("${R_YEAR}_${R_MONTH}_${R_DAY}") columns( "id int(20) unsigned not null auto_increment primary key", "host varchar(40) not null", "facility varchar(20)", "priority varchar(10)", "level varchar(10)", "program text", "date date not null", "time time not null", "message text not null") values("", "$FULLHOST", "$FACILITY", "$PRIORITY", "$LEVEL", "$PROGRAM", "$R_YEAR-$R_MONTH-$R_DAY", "$R_HOUR:$R_MIN:$R_SEC", "$MSG",) indexes("host","priority")); }; ************************ Above code auto creates table like this ************************ | Field | Type | Null | Key | Default | Extra | +----------+------------------+------+-----+---------+----------------+ | id | int(11) unsigned | NO | PRI | NULL | auto_increment | | host | varchar(40) | NO | MUL | NULL | | | facility | varchar(20) | YES | | NULL | | | priority | varchar(10) | YES | MUL | NULL | | | level | varchar(10) | YES | | NULL | | | program | text | YES | | NULL | | | date | date | NO | | NULL | | | time | time | NO | | NULL | | | message | text | NO | | NULL | | +----------+------------------+------+-----+---------+----------------+ 9 rows in set (0.01 sec) ************************ Problem is that its not inserting record in the table. I enabled mysql log and I can see only one entry ************************ 2020-01-08T05:26:29.016553Z 167 Query INSERT INTO 2020_01_08 (id, host, facility, priority, level, program, date, time, message) VALUES ('NULL', '101.11.11.252', 'user', 'notice', 'notice', 'script,warning', '2020-01-08', '10:23:30', '10:23:30 jan/08/2020') ************************ but this entry is not recorded in the mysql table, When I remove the ID columns/values from the syslog code, DROP the table, & restart syslog-ng service, then table again recates, and all records inserts fine without problem. When I insert record using phpmyadmin, the record enter fine with this code INSERT INTO `2020_01_08` (`id`, `host`, `facility`, `priority`, `level`, `program`, `date`, `time`, `message`) VALUES (NULL, '2', '2', '2', '2', '2', '2020-01-01', '42:00:00', '2'); I tried to add NULL in the values for ID in the syslog-ng code , but no use, as soon as i remove ID DROP the table, & restart syslog-ng service, all goes fine. How can I configure the auto increment column ? Regards, SYED JAHANZAIB Web : http://aacable.wordpress.com LinkedIn: http://pk.linkedin.com/pub/syed-jahanzaib/24/3b/407 <http:///>

2 2