[syslog-ng] Multi-line messages across socket

[SZIGETVÁRI János]

Hi Alex,

You might want to look at the multi-line-prefix() option. Unfortunately
though, that doesn't seem to be available for unix-dgram() and
unix-socket() sources.
Your remaining options include using a regular file (with using
multi-line-prefix()) for passing on the logs between the two syslog-ng
instances, or you could also try to bind one to some port on 127.0.0.1, and
have the other send the logs to that port using the syslog() driver.

Also, it may be important to know whether your logs are read from the
original file/source properly (not as separate log messages on every
newline). If the messages are improperly broken up there, there is little
one can do about them in case of the second syslog-ng.

Best regards,
János
--
Janos SZIGETVARI
RHCE, License no. 150-053-692
<https://www.redhat.com/rhtapps/verify/?certId=150-053-692>

LinkedIn: linkedin.com/in/janosszigetvari

__ at __˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp


Alexandre Santos <alexandre.rosas.santos at gmail.com> ezt írta (időpont:
2020. szept. 7., H, 18:17):

> Hi,
>
> I have some multi-line messages that are being broken, into single log
> messages, when they are sent over some remote host. In my configuration, I
> have one syslog-ng sending messages to other syslog-ng over
> unix-domain-socket. (check configurations in attachment).
> My goal is to have content of  "/tmp/test1_udp_file.log" equal to the
> content of  "/var/log/netconf-commands.log".
>
> Do know what am I missing?
> Any help is appreciated. Thanks in advance.
> Alex
>
> *</usr/sbin/syslog-ng -Fvde>*
> [2020-09-07T15:57:17.711583] Incoming log entry from journal;
> message='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00 \x0awith newline
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB'
> [2020-09-07T15:57:17.711760] json-parser(): no marker at the beginning of
> the message, skipping JSON parsing ;
> input='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00 \x0awith newline
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB',
> marker='@cim:'
> [2020-09-07T15:57:17.712016] Reliable disk-buffer state saved;
> filename='/tmp/syslog-ng-00002.rqf', qdisk_length='0'
> [2020-09-07T15:57:17.712046] Initializing destination file writer;
> template='/var/log/netconf-command.log',
> filename='/var/log/netconf-command.log'
> [2020-09-07T15:57:17.712150] Outgoing message; message='<158>1
> 2020-09-07T15:57:17.449+00:00 localhost root 25947 - -
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00 \x0awith newline
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB\x0a'
> [2020-09-07T15:57:17.712299] Outgoing message; message='<158>1
> 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - -
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00 \x0awith newline
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB\x0a'
>
> *</var/log/netconf-commands.log>*
> <158>1 2020-09-07T15:57:17.449+00:00 localhost root 25947 - -
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00
> with newline
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
>
>
>
>
> *<ip vrf exec MGMT /usr/sbin/syslog-ng -Fvde
> --cfgfile=/etc/syslog-ng/mgmt-syslog-ng.conf
> --pidfile=/var/lib/syslog-ng/mgmt-syslog-ng.pid
> --persist-file=/var/lib/syslog-ng/mgmt-syslog-ng.persist
> --control=/var/lib/syslog-ng/mgmt-syslog-ng.ctl>*
> [2020-09-07T15:57:17.712514] Incoming log entry; line='<158>1
> 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - -
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00 '
> [2020-09-07T15:57:17.712762] Incoming log entry; line='with newline
> BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB'
> [2020-09-07T15:57:17.712997] Outgoing message; message='<158>1
> 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - [meta sequenceId="2"]
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00 \x0a'
> [2020-09-07T15:57:17.713250] Outgoing message; message='<158>1
> 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - [meta sequenceId="2"]
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00 \x0a'
>
> *</tmp/test1_udp_file.log>*
> <158>1 2020-09-07T15:57:17.449+00:00 MYHOSTNAME root 25947 - [meta
> sequenceId="2"]
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> Test at 2020-09-07T15:57:17,448482284+00:00
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200907/74c5334d/attachment-0001.html>