- syslog-ng - lists.balabit.hu

Install Syslog-ng OSE into Virtual Machine - RHEL OS
by William Luiz Ribeiro Vasconcelos Da Silva 17 Oct '19

17 Oct '19

Good aftenoon ! I will install the syslog-ng OSE into a virtual machine with RHEL 7. Is there anything I should worry about when installing on a virtual machine? Is there something to configurated on OS, before or after the install syslog-ng? Tks, William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

3 4

Syslog-ng + graylog2 destination
by László Pál 14 Oct '19

14 Oct '19

Hi, I’m playing with syslog-ng OSE and Graylog. In this project I try to find the best method to deliver logs from my central log server to Graylog. So far, it seems graylog2 destination looks the best, however I have a strange issue and I can’t find a solution It seems the other side permanenty drops the connections like this Oct 14 13:53:20 lumberjack syslog-ng[2174]: Casting error; value='', type-hint='int32' Oct 14 13:53:20 lumberjack syslog-ng[2174]: I/O error occurred while writing; fd='481', error='Connection reset by peer (104)' Oct 14 13:53:20 lumberjack syslog-ng[2174]: Syslog connection broken; fd='481', server='AF_INET(10.72.0.137:12201)', time_reopen='10' Oct 14 13:53:30 lumberjack syslog-ng[2174]: Syslog connection established; fd='501', server='AF_INET(10.72.0.137:12201)', local='AF_INET(0.0.0.0:0)’ My destination config is quite simple #Graylog native GELF destination destination d_graylog_gelf { graylog2( log_fifo_size(500000) host("10.72.0.137") transport (tcp) ); }; I’ve tried this with flow-control and w/o flow-control, so I think it must be some tuning issue either on Graylog side or at Syslog. What is interesting, is I don’t have such an issue if I use raw-tcp or syslog destinations toward Graylog Thx L:

2 3

"Reinsert" message into syslog-ng via pipe?
by Matthias Gruber 14 Oct '19

14 Oct '19

Hi Perhaps my idea is absolutely nuts and there is a better way, but I would like to do the following: I receive weblogs in apache-format from my F5 Loadbalancer via syslog-tcp now I would like to use the apache-accesslog-parser on the message to extract some values more easy. I would try to setup following (not a syslog-ng syntax in that description) source: F5 -- via 5140/tcp into syslog-ng syslog-ng: Destination pipe(my-f5-logs) syslog-ng: source(pipe(my-f5-logs)) -> parser(apache-accesslog-parser) -> extract some fields -> write to file The whole runs on the same syslog-ng Is that too complex or a normal way? We are running a syslog-ng OSE 3.23.1 Any clues or a simple "nod thats okay" would be nice cheers Matthias ------------------------------------------------------------------------------------ METZLER Informationstechnologie Matthias Gruber IT-Infrastruktur & -Betrieb B. Metzler seel. Sohn & Co. Kommanditgesellschaft auf Aktien Untermainanlage 1 60329 Frankfurt am Main Telefon (0 69) 21 04 - 43 30 Telefax (0 69) 21 04 - 40 40 MGruber(a)metzler.com www.metzler.com Persönlich haftende Gesellschafter: Harald Illy, Michael Klaus, Friedrich von Metzler, Emmerich Müller, Gerhard Wiesheu Vorsitzender des Aufsichtsrats: Dr. Christoph Schücking Sitz der Gesellschaft: Frankfurt am Main, Handelsregister-Nr. HRB 27 515 Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfänger sein, so bitten wir Sie höflich, dies unverzüglich dem Absender mitzuteilen und die Nachricht zu löschen. Es ist unzulässig, die Nachricht unbefugt weiterzuleiten oder zu kopieren. Da wir nicht die Echtheit oder Vollständigkeit der in dieser Nachricht enthaltenen Informationen garantieren oder zusichern können, sind die vorstehenden Ausführungen rechtlich nicht bindend. Eine Haftung hierfür wird ausgeschlossen. This message is confidential. If you are not the intended recipient, we kindly ask you to inform the sender and delete the information. Any unauthorised dissemination or copying hereof is prohibited. As we cannot guarantee or assure the genuineness or completeness of the information contained in this message, the statements set forth above are not legally binding. Accordingly we cannot accept any liability for their contents.

3 2

unix-stream: ignore non-existent pipe
by Sergey 10 Oct '19

10 Oct '19

Hello. Can I configure syslog-ng to ignore non-existing unix-stream? -- Regards, Sergey

4 6

Adding new fields to existing logs
by Allen Olivas 09 Oct '19

09 Oct '19

Hello, I need to add some new fields to an existing log before it gets sent to its destination. I'm trying to decode these logs with Wazuh and will need some unique fields in the beginning of the message so I can successfully prematch my decoder. I was looking through previous cases and saw the comments on Regular Expressions in rewrite. After studying those posts to this mailing list (https://lists.balabit.hu/pipermail/syslog-ng/2019-August/025392.html) and reviewing the documentation I'm pretty sure this can be done, just not sure how to start, to use a macro or no macro, template or no template. Here are the specifics: Raw log file snippet: "Oct 1 13:48:01 SOME-DEVICE-HOSTNAME Core: Last message...." I would like to add something like "Ruckus VirtualSmartZone" to the log before SOME-DEVICE-HOSTNAME. So the new log entry that goes to the destination looks like "Oct 1 13:48:01 Ruckus VirtualSmartZone SOME-DEVICE-HOSTNAME Core: Last message..." Do I need to write a filter (with regex) to screen for the time stamp and hostname, then rewrite it to add that entry "Ruckus VirtualSmartZone"? Or if not how best to approach this? Thanks, Allen Olivas

2 2

Unofficial Debian(including Debian 10)/Ubuntu packages for syslog-ng 3.24.1
by Laszlo Budai 08 Oct '19

08 Oct '19

Hi, syslog-ng 3.24.1[1] packages are available in OBS repo[2, 3]. List of supported OSs: * Debian 8.0 * Debian 9.0 [including armv7l] * Raspbian 9.0 * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 18.04 * Ubuntu 18.10 newly added distributions: * Rasbian 10 * Debian 10 (without java modules) * Ubuntu 19.04 (without java modules) Please note that newly added distributions are not tested yet. Any feedback are welcome. Install ------- example: Debian 9.0 1. get release key wget -qO - http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… | sudo apt-key add - 2. add repo to APT sources eg.: /etc/apt/sources.list.d/syslog-ng-obs.list deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/Deb… ./ Then `apt-get update` and `apt-get install syslog-ng-core` Links -------- [1] https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.24.1<https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.23.1> [2] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-…<https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-…> [3] https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-… regards, Laszlo Budai

1 0

syslog-ng-3.24.1
by Laszlo Budai 08 Oct '19

08 Oct '19

3.24.1 Highlights * Add a new template function called $(format-flat-json), which generates flattened json output. This is useful for destinations, where the json parsing does not handle nested json format. (#2890<https://github.com/syslog-ng/syslog-ng/pull/2890>) * Add ISO 8601 compliant week numbering. Use it with the ${ISOWEEK} macro and and all its variants: S_ISOWEEK, R_ISOWEEK and C_ISOWEEK. (#2878<https://github.com/syslog-ng/syslog-ng/pull/2878>) * Add add-contextual-data() glob selector. It matches the message with shell style globbing. Enable it by setting selector(glob("$my_template") in the add-contextual-data() block. (#2936<https://github.com/syslog-ng/syslog-ng/pull/2936>) * Add new rewrite operations to manipulate the timezone portion of timestamps have been added. set-timezone() to set the timezone value to a specific value, fix-timezone() to fix up an incorrectly recognized timezone and guess-timezone() to automatically deduce the timezone value on the assumption that the message is received in near real time. (#2818<https://github.com/syslog-ng/syslog-ng/pull/2818>) * Send Server Name Identification (SNI) information with transport(tls). Enable it by setting the sni(yes) option in the tls block in your destination. (#2930<https://github.com/syslog-ng/syslog-ng/pull/2930>) Features * templates: change the $LOGHOST macro to honour use-fqdn() (#2894<https://github.com/syslog-ng/syslog-ng/pull/2894>) * Define syslog-ng-sysconfdir (#2932<https://github.com/syslog-ng/syslog-ng/pull/2932>) * dqtool: add assign dqfile to persist file feature (#2872<https://github.com/syslog-ng/syslog-ng/pull/2872>) Bugfixes * Fix backtick subsitution of defines/environment variables in the main configuration file. (#2906<https://github.com/syslog-ng/syslog-ng/issues/2906>, #2909<https://github.com/syslog-ng/syslog-ng/pull/2909>) * Fix SCL block parameter substitution of quoted escaped newline (#2901<https://github.com/syslog-ng/syslog-ng/pull/2901>) * python, diskq, random-generator source: crash after failed reload (#2907<https://github.com/syslog-ng/syslog-ng/pull/2907>) * Fix crash at shutdown on 32bit systems (#2893<https://github.com/syslog-ng/syslog-ng/issues/2893>, #2895<https://github.com/syslog-ng/syslog-ng/pull/2895>) * Invalidate the value of the LEGACY_MSGHDR macro in case either the PID or the PROGRAM macros are unset() using a rewrite rule. Previously LEGACY_MSGHDR would retain the old values. (#2896<https://github.com/syslog-ng/syslog-ng/pull/2896>) * on 32bit platform diskq ftruncate could fail due to size 32/64 interface (#2892<https://github.com/syslog-ng/syslog-ng/pull/2892>) * Support new tzdata format, starting from version 2009.XXX, in tzinfo parser. (#2898<https://github.com/syslog-ng/syslog-ng/pull/2898>) * udp, udp6, tcp, tcp6, syslog, network destination: Correctly detect and set IP_MULTICAST_TTL in case of multicast ip address (#2905<https://github.com/syslog-ng/syslog-ng/pull/2905>) * Fix hostname resolve on systems with only the loopback network interface configured (#2933<https://github.com/syslog-ng/syslog-ng/pull/2933>) * wildcard-file(): Add multi-line(), pad_size() and multi-line-mode() option validation. (#2922<https://github.com/syslog-ng/syslog-ng/pull/2922>) * kafka-c: Fix multiple memleaks (#2944<https://github.com/syslog-ng/syslog-ng/pull/2944>) Other changes * geoip: remove deprecated module, geoip2 database location detection (#2780<https://github.com/syslog-ng/syslog-ng/pull/2780>) * various refactor, build issue fixes (#2902<https://github.com/syslog-ng/syslog-ng/pull/2902>) Notes to the developers * LightRunWithStrace: Run syslog-ng behind strace (#2921<https://github.com/syslog-ng/syslog-ng/pull/2921>) * LightVerboseLogOnError: Increase default pytest verbosity on error (#2919<https://github.com/syslog-ng/syslog-ng/pull/2919>) * Dbld image caching (#2858<https://github.com/syslog-ng/syslog-ng/pull/2858>) * Dbld gradle caching (#2857<https://github.com/syslog-ng/syslog-ng/pull/2857>) * logreader,logsource: move scratch-buffer mark and reclaim into LogSource (#2903<https://github.com/syslog-ng/syslog-ng/pull/2903>) Credits syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng. Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute. We would like to thank the following people for their contribution: Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Bertrand Jacquin, Gabor Nagy, Henrik Grindal Bakken, Kerin Millar, kjhee43, Laszlo Budai, Laszlo Szemere, László Várady, Péter Kókai, Raghunath Adhyapak, Zoltan Pallagi.

1 0

group-by() send all messages to destination() ?
by Jason Cooper 06 Oct '19

06 Oct '19

All, I have Cloudflare Workers sending log messages to my syslog-ng server. There is an intermediary step where nginx receives a POST with a JSON body containing all the messages for a given run (try{}catch(e){} ensures it sends the accumulated messages at the end of each run). Some Lua in the nginx server parses the JSON and separates the array of messages into individual writes to the unix domain socket syslog-ng is listening on. This whole setup is tested, and working great. I even get the stack trace (JS, not a core dump) when I screw up. :-) Now, I'd like to auto-fire an email on certain events. e.g. when .json.level == "CRIT" (script threw an error). The trick is, I'd like to dump *all* the log messages for the matching run (only a single line from a run might match) to the email / process destination. See the commented out alert_parser{}, below. The last message of every run (equivalent to a unique .json.rayid, which I treat as a process id) always starts with 'BAIL'. So I use that as a trigger. I've been using $searchengine[*] and not been able to figure out how to send /all/ of the messages in a group to aggregate(). I even dug into the source a bit and saw that I can reference messages by '@2' for the second message back. But I see no way to a) get the number of messages, b) loop through the messages, or c) reference all of them, e.g. '@*'. Has anyone solved this problem? tia, Jason. ----------->8--------------------------------------------------------- @version: 3.22 # common parser nginx-lua-parser { json-parser (prefix(".json.")); }; #parser alert_parser { # grouping-by( # key("${json.rayid}") # scope("process") # timeout(5) # having("CRIT") # trigger("BAIL") # aggregate( # value("MESSAGE" "\n\n") # inherit-mode("context") # ) # inject-mode("pass-through") # ); #}; # template nginx-lua-template "${.json.timestamp} ${.json.colo} ${.json.script}[${.json.rayid}]: ${.json.level} ${.json.message}\n"; source worker-src { unix-stream("/var/run/nginx-lua/worker.sock", group(nginx) flags(no-parse)); }; # development logs filter worker-dev-filter {match("-dev" value (".json.script"));}; destination worker-dev-dest { file("/var/log/worker/development.log" template(nginx-lua-template)); }; log { source(worker-src); parser(nginx-lua-parser); filter(worker-dev-filter); destination(worker-dev-dest); }; # production logs filter worker-prod-filter {match("-prod" value (".json.script"));}; destination worker-prod-dest { file("/var/log/worker/production.log" template(nginx-lua-template)); }; log { source(worker-src); parser(nginx-lua-parser); filter(worker-prod-filter); destination(worker-prod-dest); };

4 10

In-list problem
by László Pál 04 Oct '19

04 Oct '19

Hi, I have a strange (at least to me :) issue. In my new system I’m heavily using in-list in my filters to tell which logs should be forwarded. In my lists it is sometimes hostnames, sometimes programs sometimes source IP, but it is very erratic. What I think because I’m using it wrong. What exact pattern matching is used in this feature? Is this posix regex or simply glob? I have lines like this in my files 1.2..3.4 Host.domain.com ssh.* host.* The problem is, sometimes it is matches sometimes not…. So, my question is how to use it to work properly? Unfortunately the admin guide is quite poor around this subject Thanks Laszlo

1 1

Remove Part of message
by William Luiz Ribeiro Vasconcelos Da Silva 03 Oct '19

03 Oct '19

Good afternoon every one!, This is a example of message collected using syslo-ng: RELEASE: 100.80.93.85 -> 189.40.107.74:55200-55215 0x5d8f3e5a And I need remove from message this part: 0x5d8f3e5a. My question is, how I can remove it ? Because the message should be send to destination without this part. Thanks for your help! William Luiz Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas e/ou de caráter confidencial, não podendo ser retransmitida sem autorização do remetente. Se você não é o destinatário ou pessoa autorizada para recebê-la, informamos que o seu uso, divulgação, cópia ou arquivamento são proibidos. Portanto, se você recebeu esta mensagem por engano, por favor nos informe respondendo imediatamente a este e-mail e delete o seu conteúdo. This message, including its attachments, may contain privileged or confidential information, and it must not be fowarded without the express authorization of the sender. If you are not the intended recipient, we hereby inform you that the use, disclosure, copy or filing are forbidden. So, if you received this message as a mistake, please inform us by answering this e-mail and deleting its contents Questo messaggio, inclusi gli allegati, potrebbe contenere informazioni privilegiate e/o riservate, e non deve essere ritrasmesse senza l'autorizzazione del mittente. Se non siete il destinatario o la persona autorizzata a riceverlo, informiamo che il suo utilizzo, diffusione, copia o archiviazione sono proibite. Quindi, se avete ricevuto questo messaggio per errore, per cortesia ci informi rispondendo immediatamente a questa email e cancelli il suo contenuto

2 1