January 2020 - syslog-ng - lists.balabit.hu

Set log directory based on the source
by Frank DiGennaro 18 Feb '20

18 Feb '20

Hello; I'm somewhat of a newbie when it comes to syslog-ng so forgive me if the question is somewhat basic. I am running syslog-ng 3.25 on both server and clients and would like to set a variable based on the source. This is what I have: @define server-logdir "/var/log" @define client-logdir "/var/log/CLIENTS/$YEAR/$MONTH_ABBREV/$DAY/$HOST" @define logdir "" source s_local { logdir `server-logdir`; *No! ... ... }; source s_network { logdir `client-logdir`; *No! ... ... }; destination d_local0 { file( "`logdir`/local/local0.log" ); }; The goal is to set 'logdir' to either 'server-logdir' or 'client-logdir' and use it in the destination. I know there are probably several ways that this can be accomplished but I think this was will cut down on the number of lines in the configuration. So my question Is this. How do I set 'logdir' to either 'server-logdir' or 'client-logdir' in the source definition? Thanks; DiGennaro Frank DiGennaro 301-676-8193 (C) <mailto:frank@digennaro.com> frank(a)digennaro.com

3 5

C_* macros with UTC time?
by Matus UHLAR - fantomas 04 Feb '20

04 Feb '20

Hello, I have syslog-ng OSE 3.8 on debian 9 server (soon to be upgraded). when I receive syslog lines using the syslog protocol, I found out that C_* macros seem to contain date in UTC format (one hour than our current zone). I have tried to use following template: "${C_DATE} ${R_DATE} HOST ${HOST} FULLHOST ${FULLHOST} HOST_FROM ${HOST_FROM} FULLHOST_FROM ${FULLHOST_FROM} ${MESSAGE}\n" with following result: Jan 31 10:42:06 Jan 31 11:42:07 HOST fantomas FULLHOST fantomas HOST_FROM 195.80.174.185 FULLHOST_FROM 195.80.174.185 Configuration reload request received, reloading configuration; Is there any reason why C_DATE should be in different time zone than R_DATE? timezones are not configured on either of servers, both servers are in CET (DST+1) thank you -- Matus UHLAR - fantomas, uhlar(a)fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Fighting for peace is like fucking for virginity...

2 4

Memory not freeing up in syslog-ng-3.6.2 during reload
by Nitish Saboo 29 Jan '20

29 Jan '20

Hi, I am using syslog-ng version 3.6.2: syslog-ng --version syslog-ng 3.6.2 Installer-Version: 3.6.2 Revision: Compile-Date: May 2 2019 01:20:32 Available-Modules: afmongodb,afstomp,afuser,csvparser,afprog,afamqp,cryptofuncs,afsocket,graphite,dbparser,affile,syslogformat,afsocket-notls,system-source,confgen,pseudofile,linux-kmsg-format,basicfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off I initialised syslog-ng with the following configuration: int initialize_engine(const gchar* filename, const gchar* _module_path) { module_path = _module_path; app_startup(); msg_init(TRUE); configuration = cfg_new(0x0302); plugin_load_module("basicfuncs", configuration, NULL); plugin_load_module("syslogformat", configuration, NULL); pattern_db_global_init(); return 0; } int reload_pattern_db(const gchar* filename, key_value_cb cb) { if(patterndb!= NULL){ pattern_db_free(patterndb); } patterndb = pattern_db_new(); pattern_db_reload_ruleset(patterndb, configuration, filename); pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb); return 0; } When I reload the syslog-ng, the memory keeps on increasing.Looks like syslog-ng is not freeing up the memory.It continues to occupy the memory unless the process is restarted. Just want to understand if it is a sign of memory leak or default behavior of syslog-ng to not free-up memory during reload ? Thanks, Nitish

2 15

which platform do you use for syslog-ng?
by Peter Czanik (pczanik) 28 Jan '20

28 Jan '20

Hi, I am often asked on which platforms people are running syslog-ng. Based on mailing list questions, github issues and so on I have some guesses, but every once in a while I run a Twitter poll: https://twitter.com/PCzanik/status/1222089870782484480 Please answer it and share it, so more people can answer it! Thanks, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 0

facility override
by Alexandre Santos 20 Jan '20

20 Jan '20

Hi, Is there a way of overriding the facility of the messages on the destination, besides using a template like: { template("$(if ('${LEVEL_NUM}' == '2') '<186>1' $(if ('${LEVEL_NUM}' == '3') '<187>1' $(if ('${LEVEL_NUM}' == '4') '<188>1' ${HOST}\\@`HOSTIPV4` ${PROGRAM} ${PID} ${SDATA} ${MESSAGE}\n"); }; Thanks in advance, Alex

2 1

Rewrite set var value with tag?
by Faine, Mark R. (MSFC-IS40)[NICS] 20 Jan '20

20 Jan '20

Is it possible to do something like: rewrite { set-tag("bar"); }; And then use a rewrite to set the value of a variable to the value of the tag rewrite { set("${TAGS}", value("foo")); }; I need to set a value that I've determined in a log path at the top of the file in several log paths that occur after. It looks the scope for variables is at the log path level so that doesn't help me. I'm also trying to avoid nesting all of the other log paths inside of the top log path where I determine the value. I'm trying to avoid: log{ ... filter{ ... tags("bar"); rewrite{ set("bar", "foo"); }; }; }; log{... log{... Thanks, -Mark

3 3

Go away
by johns spam 16 Jan '20

16 Jan '20

Hello, I am wondering if there is any facility to tell a client to go away? i.e. https://http2.github.io/http2-spec/#GOAWAY I know syslog-ng is not http2 but is there any side-channel mechanism a central server can use to make a client go away? The use case I am thinking is doing a rolling restart of a pool of syslog-ng central server. I think this could also be useful to cycle connections periodically, i.e. to rebalance connections to a central server. Thanks

2 1

librdkafka and messages dropped
by John Skopis 15 Jan '20

15 Jan '20

Hello, We have a syslog-ng aggregator that accepts messages in a few different formats and logs the message format to kafka. The mapping looks something like: format1 -> topic1 format2 -> topic2 ... Each agent will emit messages in ~4 different formats and send to the aggregator. The aggregator is configured with a kafka_c destination with 8 threads/destination. We were previously using the java kafka client without issue. We recently upgraded to 3.24.1 + librdkafka 1.22 After upgrading we are seeing an issue where (seemingly randomly) syslog-ng stops producing messages into kafka. Checking syslog-ng-ctl stats I can see d_kafka_format1 has dropped message counter increasing d_kafka_format1 queued messages counter increases to 80000 (10000*8 threads iw-size) d_kafka_format1 processed messages does not increase There are messages printed to the log that the destination queue is full for all of the rdkafka threads, There are spurious failures to a kafka broker but it seems rdkafka reconnects. Seems like once the destination queue becomes full syslog-ng never recovers and the instance must be restarted. It also seems like rdkafka stops trying to reconnect to kafka (possibly?). Is this expected behavior or is there some bug around reconnecting to kafka after a spurious timeout? Just speculating here but does rdkafka stop trying to reconnect after N times? Are there any examples of destroy/create destination client after failure thresholds? Thanks

2 5

3.25 and my RPM stable repos
by Peter Czanik (pczanik) 13 Jan '20

13 Jan '20

Hi, Version 3.25 of syslog-ng was released a week ago, and it seems to be rock solid. Still, as today is my last working day this year, I feel more comfortable to push it into my stable repos only on the first week of January, when I am back. If you still want to experiment with 3.25 on openSUSE/SLES or fedora/RHEL/CentOS you can do that in the versioned repositories. Note that RHEL 8 / CentOS 8 support is now merged with the rest, it's not in a separate repository any more. suse: https://build.opensuse.org/project/show/home:czanik:syslog-ng325 red hat: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng325/ Bye, Peter Czanik (CzP) <peter.czanik(a)oneidentity.com> Balabit (a OneIdentity company) / syslog-ng upstream https://syslog-ng.com/community/ https://twitter.com/PCzanik

1 1

Lines per file + Syslog-ng
by William Luiz Ribeiro Vasconcelos Da Silva 13 Jan '20

13 Jan '20

Hello everyone, Is it possible to configure syslog-ng to generate 1 file per minute or 1 file with at least 100,000 lines? Could you help by showing some examples please. This is the configuration we have in our destination function: file("/syslog_out/udp_files/${HOST}_${YEAR}${MONTH}${DAY}${HOUR}${MIN}_GMT${TZ}.log" template("${PROGRAM};${MESSAGE};\n") ); Regards, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1