- syslog-ng - lists.balabit.hu

facility override
by Alexandre Santos 20 Jan '20

20 Jan '20

Hi, Is there a way of overriding the facility of the messages on the destination, besides using a template like: { template("$(if ('${LEVEL_NUM}' == '2') '<186>1' $(if ('${LEVEL_NUM}' == '3') '<187>1' $(if ('${LEVEL_NUM}' == '4') '<188>1' ${HOST}\\@`HOSTIPV4` ${PROGRAM} ${PID} ${SDATA} ${MESSAGE}\n"); }; Thanks in advance, Alex

2 1

Rewrite set var value with tag?
by Faine, Mark R. (MSFC-IS40)[NICS] 20 Jan '20

20 Jan '20

Is it possible to do something like: rewrite { set-tag("bar"); }; And then use a rewrite to set the value of a variable to the value of the tag rewrite { set("${TAGS}", value("foo")); }; I need to set a value that I've determined in a log path at the top of the file in several log paths that occur after. It looks the scope for variables is at the log path level so that doesn't help me. I'm also trying to avoid nesting all of the other log paths inside of the top log path where I determine the value. I'm trying to avoid: log{ ... filter{ ... tags("bar"); rewrite{ set("bar", "foo"); }; }; }; log{... log{... Thanks, -Mark

3 3

Go away
by johns spam 16 Jan '20

16 Jan '20

Hello, I am wondering if there is any facility to tell a client to go away? i.e. https://http2.github.io/http2-spec/#GOAWAY I know syslog-ng is not http2 but is there any side-channel mechanism a central server can use to make a client go away? The use case I am thinking is doing a rolling restart of a pool of syslog-ng central server. I think this could also be useful to cycle connections periodically, i.e. to rebalance connections to a central server. Thanks

2 1

librdkafka and messages dropped
by John Skopis 15 Jan '20

15 Jan '20

Hello, We have a syslog-ng aggregator that accepts messages in a few different formats and logs the message format to kafka. The mapping looks something like: format1 -> topic1 format2 -> topic2 ... Each agent will emit messages in ~4 different formats and send to the aggregator. The aggregator is configured with a kafka_c destination with 8 threads/destination. We were previously using the java kafka client without issue. We recently upgraded to 3.24.1 + librdkafka 1.22 After upgrading we are seeing an issue where (seemingly randomly) syslog-ng stops producing messages into kafka. Checking syslog-ng-ctl stats I can see d_kafka_format1 has dropped message counter increasing d_kafka_format1 queued messages counter increases to 80000 (10000*8 threads iw-size) d_kafka_format1 processed messages does not increase There are messages printed to the log that the destination queue is full for all of the rdkafka threads, There are spurious failures to a kafka broker but it seems rdkafka reconnects. Seems like once the destination queue becomes full syslog-ng never recovers and the instance must be restarted. It also seems like rdkafka stops trying to reconnect to kafka (possibly?). Is this expected behavior or is there some bug around reconnecting to kafka after a spurious timeout? Just speculating here but does rdkafka stop trying to reconnect after N times? Are there any examples of destroy/create destination client after failure thresholds? Thanks

2 5

3.25 and my RPM stable repos
by Peter Czanik (pczanik) 13 Jan '20

13 Jan '20

Hi, Version 3.25 of syslog-ng was released a week ago, and it seems to be rock solid. Still, as today is my last working day this year, I feel more comfortable to push it into my stable repos only on the first week of January, when I am back. If you still want to experiment with 3.25 on openSUSE/SLES or fedora/RHEL/CentOS you can do that in the versioned repositories. Note that RHEL 8 / CentOS 8 support is now merged with the rest, it's not in a separate repository any more. suse: https://build.opensuse.org/project/show/home:czanik:syslog-ng325 red hat: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng325/ Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Lines per file + Syslog-ng
by William Luiz Ribeiro Vasconcelos Da Silva 13 Jan '20

13 Jan '20

Hello everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1

Load Balancing question
by Janczuk, George 13 Jan '20

13 Jan '20

I'm new to syslog-ng, so please excuse my ignorance if this a newbie or dumb question (I did do some googling first though - regrettably without success). Where I work we have deployed a syslog-ng OSE cluster (five members) fronted by an F5 LTM. One of our up-stream syslog event sources is a service provider's syslog-ng server acting as a relay and passing on all ingested syslog events to our cluster (via the F5 LTM VIP). The up-steam syslog-ng server aggregates events from hundreds of devices, and therefore the stream is very busy (i.e. never idle/quiet). Therefore, it turns out that the connection from the up-stream relay is NEVER dropped and remains in place almost permanently (certainly days at a time). Consequently, without any cycling of the connection the connection is bound to a single cluster member, and ALL events from the up-stream relay are delivered to that single cluster member and are NOT actually load balanced (with the remaining four members essentially being idle as the volume from the relay greatly eclipses the event volume from other even sources). I suppose the question is whether: a) Can we in some way configure the up-steam syslog-ng relay to cycle the outbound connection after some sort of threshold (be that time based - say every 15 seconds; or volume based - say every 100 events; or even both... - whichever occurs first)?, or b) Can the up-stream syslog-ng keep multiple connections open concurrently and can it then use round-robin or stochastic logic to determine which of the connections to send each relayed event to? I would very much like to hear from people about how to actually load balance an event stream from a syslog-ng relay. NOTE: People might ask the question: "If the up-stream relay is a single node, then why do even need a down-stream cluster, if it's not required upstream"? Two reasons really: 1. The up-steam service provider has deployed dedicated chunky tin (i.e. they've used a vertical scale model), whilst for our syslog-ng cluster we're using a private-cloud with much smaller VMs combined with a horizontal scale model, and 2. The relay is doing a fairly minimal collect and forward, whilst our syslog-ng cluster will be doing more event transformation and enrichment, which is ultimately more CPU intensive. ************** IMPORTANT MESSAGE ***************************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Unsubscribe in the subject line. **************************************************************

3 2

Disk-buffer in kafka-destinations
by Ronald Fenner 10 Jan '20

10 Jan '20

I can't see to me get syslog-ng to start if I add the disk-buffer to the destination. From the docs page it looks like it should be added to the kafka section but it doesn't work and doesn't work if I put it with in the destination block. Is this tied to a specific destination type? We are trying to make out messages going to kafka more reliable when the cluster is unavailable. Ronald Fenner Network Architect Game Circus LLC. rfenner(a)gamecircus.com

2 4

Increase number of records per file
by William Luiz Ribeiro Vasconcelos Da Silva 10 Jan '20

10 Jan '20

Good morning everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

1 0

customized rewrite to mysql in syslog-ng
by JAHANZAIB SYED 09 Jan '20

09 Jan '20

I have syslog-ng version 3.25.1 on Ubuntu 18.04 server. All logs from remote mikrotik router device is stored in local file and in mysql DB as well. I want to know if there is any method in which we can customize the incoming message and write it to mysql table in different columns. Incoming message example: Jan 8 13:50:24 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49466->172.217.19.10:443, NAT (172.16.0.2:49466->101.11.11.252:2223)->172.217.19.10:443, len 1023 Jan 8 13:53:09 101.11.11.252 firewall,info forward: in:<pppoe-zaib> out:ether1-agp-wan, src-mac d0:bf:9c:f7:88:76, proto UDP, 172.16.0.2:49482->58.27.130.12:443, NAT (172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443, len 1152 The relevant part that I want to store is HOSTNAME, DATETIME, AND message part 172.16.0.2:49482->101.11.11.252:2224)->58.27.130.12:443 * into following tables hostname datatime username userip userport wanip wanport dst-ip dst-port 101.11.11.252 xxxx pppoe-zaib 172.16.0.2 49466 101.11.11.252 2223 172.217.19.10 443 [cid:48adc02d-e37b-47af-aa2d-67e59b25f8d6] Regards, SYED JAHANZAIB <http:///>

2 6