[syslog-ng] Syslog messages not stored in separate lines
Balazs Scheidler
bazsi77 at gmail.com
Fri Feb 3 22:44:02 UTC 2023
Evan seems to be right, if the sending device is not adding line ends, we
can't know where the log record ends.
This format on TCP is pretty useless as TCP is line oriented, so the
application doesn't know where the records terminate.
To split this input one would need to split az the header (e.g. starting
the <pri> field in the front), which should only fail is messages would
contain similar sequences of characters.
Maybe this is easier to process using UDP (if the device allows that), if
the app sends these messages in different datagrams at least, that would
work.
The only other workaround I can think of is to write a frontend to
syslog-ng that splits these messages and resend them to syslog-ng with good
framing.
Bazsi
On Fri, Feb 3, 2023, 19:53 Dragan Zecevic <dragan.zecevic at live.com> wrote:
> Hi Evan,
> thank you for your reply.
>
> I copied a part of the stored log file below. There is no line break
> between a lot of syslog messages and then just before (hashtags) this
> performance output it starts separating lines.
> And each performance stat is a new line. After that (right away after
> hashtags) it starts storing again without line breaks.
> This is mostly going on and on like this.
>
> I didn't change windows size so it should be default. Currently there is
> only one source host sending syslog messages.
>
> ...<189>Jan 13 2023 13:29:58+02:00 hostname
> %%01SSH/5/SSHS_REKEY_STATUS(s):CID=xxx;SSH server key renegotiation with
> client. (SessionID=1, RekeyReason=Rekey timer timed out, Status=Success,
> UserAddress=x.y.w.z, LocalAddress=x.y.w.z, VPNInstanceName=_xxx_)<189>Jan
> 13 2023 13:29:59+02:00 hostname %%01SSH/5/SSHS_REKEY_STATUS(s):CID=xxx;SSH
> server key renegotiation with client. (SessionID=2, RekeyReason=Rekey timer
> timed out, Status=Begin, UserAddress=x.y.w.z, LocalAddress=x.y.w.z,
> VPNInstanceName=_xxx_)<189>Jan 13 2023 13:30:00+02:00 hostname
> %%01SSH/5/SSHS_REKEY_STATUS(s):CID=xxx;SSH server key renegotiation with
> client. (SessionID=2, RekeyReason=Rekey timer timed out, Status=Success,
> UserAddress=x.y.w.z, LocalAddress=x.y.w.z, VPNInstanceName=_xxx_)<190>Jan
> 13 2023 13:48:30+02:00 hostname
> %%01DEBUG/6/DBG_HEALTH(l):CID=0x80cc000d;Automatic record:
> ###########################################
> #Automatic record log end,current health information as follows:
> <190>Jan 13 2023 13:48:30+02:00 hostname
> %%01DEBUG/6/DBG_HEALTH(l):CID=xxx;Automatic record:
> Slot CPU Memory(Used/Total) Physical Memory
> Usage(Free/Total/Cache)
>
> --------------------------------------------------------------------------------------
> 1 IPU(Master) 12% 26% 4006MB/15394MB 34%
> 10123MB/15396MB/2287MB
> CPU0 17%
> CPU1 12%
> CPU2 13%
> CPU3 14%
> CPU4 7%
> CPU5 11%
> ProcessId CPU
> 1019 1%
> 1001 3%
> 1005 0%
> 1015 41%
> 1012 0%
> 3 1%
> 10001 2%
> 1006 1%
> 1018 0%
> 1000 0%
> 1013 2%
> 1003 1%
> 1007 0%
> 1010 1%
> 1008 1%
> 1016 201%
> 1017 1%
> 1014 0%
> 1011 0%
> .<190>Jan 13 2023 13:48:30+02:00 hostname
> %%01DEBUG/6/DBG_HEALTH(l):CID=xxxx;Automatic record:
> #DateTime Stamp:2023-01-13 13:48:30.673
> ###########################################<188>Jan 13 2023 13:49:16+02:00
> hostname %%01SNMP/4/SNMP_MIB_SET(s):CID=xxx;MIB node set. (UserName=xxx,
> SourceIP=x.y.w.z, DestIP=x.y.w.z, Version=v3, RequestId=xxx,
> hwCfgOperateType.89005=6,hwCfgOperateProtocol.89005=3,hwCfgOperateFileName.89005=[xxx(hex)],hwCfgOperateServerAddress.89005=x.y.w.z,hwCfgOperateUserName.89005=[xxx(hex)],hwCfgOperateUserPassword.89005=******,hwCfgOperateServerPort.89005=xxx,hwCfgOperateRowStatus.89005=xxx,
> VPN=xxx)<188>Jan 13 2023 13:49:17+02:00 hostname
> %%01CONFIGURATION/4/CONFIGMIB_FILE_OPERATE_FINISH(l):CID=xxx;Configuration
> was copied. (OperationType=6, OptTime=94, OptState=2,
> OptEndTime=24674900)...
>
>
>
>
> Br,
> Dragan
> ------------------------------
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Evan
> Rempel <erempel at uvic.ca>
> *Sent:* Thursday, February 2, 2023 2:07 PM
> *To:* syslog-ng at lists.balabit.hu <syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Syslog messages not stored in separate lines
>
>
> Is there a line break anywhere in the log file?
>
> If yes
>
> 1. is the line break in the middle of a syslog line from this device?
> 2. is the line break at the maximum message size?
> 3. is the line break at the end of a log message from a different device
> that is logging to the same s_xxx_xxx source?
>
>
> What I am wondering is that the source of the log messages is logging the
> exact same stream of data to the TCP port that it would have over the UDP
> port (which would be an error). UDP messages are not terminated with a new
> line, while the TCP messages are. If that were the case then syslog-ng
> would never see multiple messages, and would write a continuous stream on a
> single line until it reached the maximum message length, or it logged a
> correctly terminated message from a different device.
>
> Evan.
>
>
> On 2023-02-02 05:55, Dragan Zecevic wrote:
>
>
> Hi Balazs,
>
> this is how the whole config snippet looks like:
>
> source s_xxx_xxx {
> network(
> ip(0.0.0.0)
> transport(tcp)
> port(xxxx)
> flags(store-raw-message)
> );
> };
>
>
>
> filter filter_xxx_xxx {
>
> host("xxx") or host("xxx") ...;
> };
>
>
> destination folder_xxx_xxx {
>
> file(
>
> "/xxx/.../xxx/${R_YEAR}${R_MONTH}${R_DAY}/${SOURCEIP}_${HOST}_${R_HOUR}.log"
> template("${RAWMSG}\n")
> dir-group(xxx)
> dir-perm(xxx)
> group(xxx)
>
> );
> };
>
>
> log {
> source(s_xxx_xxx); filter(filter_xxx_xxx);
> destination(folder_xxx_xxx);
> };
>
> Sorry for the xxx but I can't export real parameters in conversation like
> this. Also, I can't provide some pcap or tcpdump.
>
> I restarted syslog-ng multiple times because I also added some other
> sources and there were no error messages. And this is the only part of the
> configuration where either this destination or folder are used.
>
> I hope this info is helpful.
>
> Thanks,
> Dragan
>
> ------------------------------
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu>
> <syslog-ng-bounces at lists.balabit.hu> on behalf of Balazs Scheidler
> <bazsi77 at gmail.com> <bazsi77 at gmail.com>
> *Sent:* Tuesday, January 31, 2023 9:18 PM
> *To:* Syslog-ng users' and developers' mailing list
> <syslog-ng at lists.balabit.hu> <syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Syslog messages not stored in separate lines
>
> If there's an initialization error with a config at reloading, syslog-ng
> can fall back to the old one. Can this happen? Or two destinations writing
> the same file?
>
> If the problem persists, can you create minimal example with complete with
> config and sample message that you send and which reproduces the issue?
>
> Thanks
>
> On Tue, Jan 31, 2023, 20:44 Dragan Zecevic <dragan.zecevic at live.com>
> wrote:
>
>
> Hi Balazs,
> thank you for your reply.
> Yes, I used config like this for other sources as well and restarted
> syslog-ng service.
> I don't get it why it doesn't work in this case. I was thinking it is up
> to the input.
>
> ------------------------------
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of
> Balazs Scheidler <bazsi77 at gmail.com>
> *Sent:* Monday, January 30, 2023 6:48 AM
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] Syslog messages not stored in separate lines
>
> This would be very strange indeed as the template of your file destination
> includes a newline character at the end of every message, so it should not
> depend on the input.
>
> You sure that this is the destination config that you quote here? Did you
> reload syslog-ng to use that config?
>
> On Sun, Jan 29, 2023, 13:55 Dragan Zecevic <dragan.zecevic at live.com>
> wrote:
>
>
> Hi,
> I am collecting logs from a network device. They configured syslog format
> on their source side to be RFC3164.
>
> On syslog-ng side I am using source and destination like this:
>
> source s_xxx {
> network(
> ip(0.0.0.0)
> transport(tcp)
> port(xxx)
> flags(store-raw-message)
> );
> };
>
>
> destination folder_xxx {
>
> file(
>
> "/xxx/${R_YEAR}${R_MONTH}${R_DAY}/${SOURCEIP}_${HOST}_${R_HOUR}.log"
> template("${RAWMSG}\n")
> );
> };
>
> syslog-ng version 3.34
> CentOS Linux release 7.9.2009
>
> The problem is that syslog messages are stored in raw format but not
> separated in different line. Parity bit of new message starts imidiatelly
> after previous line -without space or enter.
>
> I have the same config for some other hosts and there log files are
> created with separate lines. Vendor says they can't change anything on
> source side.
>
> Do you have any idea what is the cause of this?
>
> Thank you.
>
> Br,
> Dragan
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20230203/b6cff0a7/attachment-0001.htm>
More information about the syslog-ng
mailing list