a log message is output in the two line
Hello, all, I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6. I got it from eple epel repository. I am troubled in how to use the source file driver. The touble is that a log message is output in the two lines on remote syslog server. How to reproduce is followings. 1. Prepare two hosts running syslog-ng. Host_A configuration is followings. ---------------------------------------- source test { file("/tmp/a.log"); }; destination d_remote { udp("192.168.0.2"); }; log { source(test); destination(d_remote); }; ---------------------------------------- Host_B(192.168.0.2) configuration is defaults. 2.Execute following command on Host_A. $ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log 3.Check the log on Host_B. As the result, a log message is output in the two lines on Host_B. One line is following. The num of "a" is 8192. aaaaaaaaaaaaa....... The other line is following. The num of "a" is 1. a I hope a log message is output in the one line on Host_B. Is my hope readily achievable? Please advise me.
Hi, On 06/24/2016 04:10 PM, Hirose, Shinsaku wrote:
Hello, all,
I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6. I got it from eple epel repository.
I am troubled in how to use the source file driver. The touble is that a log message is output in the two lines on remote syslog server.
How to reproduce is followings.
1. Prepare two hosts running syslog-ng.
Host_A configuration is followings. ---------------------------------------- source test { file("/tmp/a.log"); }; destination d_remote { udp("192.168.0.2"); }; log { source(test); destination(d_remote); }; ----------------------------------------
UDP transport limits datagram size to 64k
Host_B(192.168.0.2) configuration is defaults.
2.Execute following command on Host_A.
$ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log
3.Check the log on Host_B.
As the result, a log message is output in the two lines on Host_B.
One line is following. The num of "a" is 8192. aaaaaaaaaaaaa.......
The other line is following. The num of "a" is 1. a
I hope a log message is output in the one line on Host_B. Is my hope readily achievable?
syslog isn't rsync so the syslog standards must get taken into account. Original syslog was designed to work with single-line, small messages (up to 1k) fitting into a single UDP packet to avoid fragmentating the datagram. IIRC the maximal message size in syslog-ng defaults to 8192 bytes, so from stream-like sources input will be splitted when this length is reached.
Please advise me.
You can raise log_msg_size to higer value but don't expect that values larger than 64k will work with the udp() transport. You should switch to another transport driver like tcp() or even better to syslog() Regards, Sandor
Dear Sandor Thank you for your quick responce.
UDP transport limits datagram size to 64k
OK. Thanks! I raised log_msg_size to higer value. But the behavior did not changed. I added ' log_msg_size (16384);' in options on the two hosts. ------------------------------------------------------------ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_msg_size (16384); }; ------------------------------------------------------------ And I did restarting service. But the behavior did not changed. Do you know this reason? Best regards, -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Sandor Geller Sent: Friday, June 24, 2016 11:32 PM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] a log message is output in the two line Hi, On 06/24/2016 04:10 PM, Hirose, Shinsaku wrote:
Hello, all,
I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6. I got it from eple epel repository.
I am troubled in how to use the source file driver. The touble is that a log message is output in the two lines on remote syslog server.
How to reproduce is followings.
1. Prepare two hosts running syslog-ng.
Host_A configuration is followings. ---------------------------------------- source test { file("/tmp/a.log"); }; destination d_remote { udp("192.168.0.2"); }; log { source(test); destination(d_remote); }; ----------------------------------------
UDP transport limits datagram size to 64k
Host_B(192.168.0.2) configuration is defaults.
2.Execute following command on Host_A.
$ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log
3.Check the log on Host_B.
As the result, a log message is output in the two lines on Host_B.
One line is following. The num of "a" is 8192. aaaaaaaaaaaaa.......
The other line is following. The num of "a" is 1. a
I hope a log message is output in the one line on Host_B. Is my hope readily achievable?
syslog isn't rsync so the syslog standards must get taken into account. Original syslog was designed to work with single-line, small messages (up to 1k) fitting into a single UDP packet to avoid fragmentating the datagram. IIRC the maximal message size in syslog-ng defaults to 8192 bytes, so from stream-like sources input will be splitted when this length is reached.
Please advise me.
You can raise log_msg_size to higer value but don't expect that values larger than 64k will work with the udp() transport. You should switch to another transport driver like tcp() or even better to syslog() Regards, Sandor ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
For some reason the log-msg-size() setting wasn't effective. Clearly syslog-ng split the line at 8192 bytes. It could either be the client or the server, but I would vote for the client being the culprit. syslog-ng emits a debug message that starts like: "Incoming message", there you should see what was received as a single entry. My guess is that you'll see the line split there. This way you can pinpoint which one is the culprit (the client or the server). On Jun 24, 2016 11:41 AM, "Hirose, Shinsaku" <hirose.shinsaku@jp.fujitsu.com> wrote:
Dear Sandor
Thank you for your quick responce.
UDP transport limits datagram size to 64k
OK. Thanks!
I raised log_msg_size to higer value. But the behavior did not changed.
I added ' log_msg_size (16384);' in options on the two hosts. ------------------------------------------------------------ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_msg_size (16384); }; ------------------------------------------------------------
And I did restarting service. But the behavior did not changed.
Do you know this reason?
Best regards,
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] On Behalf Of Sandor Geller Sent: Friday, June 24, 2016 11:32 PM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] a log message is output in the two line
Hi,
On 06/24/2016 04:10 PM, Hirose, Shinsaku wrote:
Hello, all,
I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6. I got it from eple epel repository.
I am troubled in how to use the source file driver. The touble is that a log message is output in the two lines on remote syslog server.
How to reproduce is followings.
1. Prepare two hosts running syslog-ng.
Host_A configuration is followings. ---------------------------------------- source test { file("/tmp/a.log"); }; destination d_remote { udp("192.168.0.2"); }; log { source(test); destination(d_remote); }; ----------------------------------------
UDP transport limits datagram size to 64k
Host_B(192.168.0.2) configuration is defaults.
2.Execute following command on Host_A.
$ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log
3.Check the log on Host_B.
As the result, a log message is output in the two lines on Host_B.
One line is following. The num of "a" is 8192. aaaaaaaaaaaaa.......
The other line is following. The num of "a" is 1. a
I hope a log message is output in the one line on Host_B. Is my hope readily achievable?
syslog isn't rsync so the syslog standards must get taken into account. Original syslog was designed to work with single-line, small messages (up to 1k) fitting into a single UDP packet to avoid fragmentating the datagram.
IIRC the maximal message size in syslog-ng defaults to 8192 bytes, so from stream-like sources input will be splitted when this length is reached.
Please advise me.
You can raise log_msg_size to higer value but don't expect that values larger than 64k will work with the udp() transport. You should switch to another transport driver like tcp() or even better to syslog()
Regards,
Sandor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Dear Scheidler Thanks for your response! I tried debug mode. I saw that client received two "Incoming message". As you said, the client is looked like being the culprit. I think my hope is difficult for my skill..... So I will try to find the another solutoion. I am graceful for your advices! Best regards, -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Scheidler, Balazs Sent: Saturday, June 25, 2016 5:06 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] a log message is output in the two line For some reason the log-msg-size() setting wasn't effective. Clearly syslog-ng split the line at 8192 bytes. It could either be the client or the server, but I would vote for the client being the culprit. syslog-ng emits a debug message that starts like: "Incoming message", there you should see what was received as a single entry. My guess is that you'll see the line split there. This way you can pinpoint which one is the culprit (the client or the server). On Jun 24, 2016 11:41 AM, "Hirose, Shinsaku" <hirose.shinsaku@jp.fujitsu.com <mailto:hirose.shinsaku@jp.fujitsu.com> > wrote: Dear Sandor Thank you for your quick responce. > UDP transport limits datagram size to 64k OK. Thanks! I raised log_msg_size to higer value. But the behavior did not changed. I added ' log_msg_size (16384);' in options on the two hosts. ------------------------------------------------------------ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_msg_size (16384); }; ------------------------------------------------------------ And I did restarting service. But the behavior did not changed. Do you know this reason? Best regards, -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu <mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu <mailto:syslog-ng-bounces@lists.balabit.hu> ] On Behalf Of Sandor Geller Sent: Friday, June 24, 2016 11:32 PM To: syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] a log message is output in the two line Hi, On 06/24/2016 04:10 PM, Hirose, Shinsaku wrote: > Hello, all, > > I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6. > I got it from eple epel repository. > > I am troubled in how to use the source file driver. > The touble is that a log message is output in the two lines on remote syslog server. > > How to reproduce is followings. > > 1. Prepare two hosts running syslog-ng. > > Host_A configuration is followings. > ---------------------------------------- > source test { > file("/tmp/a.log"); > }; > destination d_remote { udp("192.168.0.2"); }; > log { source(test); destination(d_remote); }; > ---------------------------------------- UDP transport limits datagram size to 64k > Host_B(192.168.0.2) configuration is defaults. > > 2.Execute following command on Host_A. > > $ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log > > 3.Check the log on Host_B. > > As the result, a log message is output in the two lines on Host_B. > > One line is following. The num of "a" is 8192. > aaaaaaaaaaaaa....... > > The other line is following. The num of "a" is 1. > a > > I hope a log message is output in the one line on Host_B. > Is my hope readily achievable? syslog isn't rsync so the syslog standards must get taken into account. Original syslog was designed to work with single-line, small messages (up to 1k) fitting into a single UDP packet to avoid fragmentating the datagram. IIRC the maximal message size in syslog-ng defaults to 8192 bytes, so from stream-like sources input will be splitted when this length is reached. > Please advise me. You can raise log_msg_size to higer value but don't expect that values larger than 64k will work with the udp() transport. You should switch to another transport driver like tcp() or even better to syslog() Regards, Sandor ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Are you absolutely sure, that syslog-ng is reading the config file where you set log-msg-size() to 8192? Maybe you should try to generate a syntax error on purpose and see if it fails to start up. Sometimes people have multiple configs on their system. Or just supply the -f option to syslog-ng that tells syslog-ng where to find its configuration explicitly. You can also set log-msg-size() on a per source basis, if I remember correctly. -- Bazsi On Mon, Jun 27, 2016 at 12:26 PM, Hirose, Shinsaku < hirose.shinsaku@jp.fujitsu.com> wrote:
Dear Scheidler
Thanks for your response!
I tried debug mode.
I saw that client received two "Incoming message". As you said, the client is looked like being the culprit.
I think my hope is difficult for my skill..... So I will try to find the another solutoion.
I am graceful for your advices!
Best regards,
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] On Behalf Of Scheidler, Balazs Sent: Saturday, June 25, 2016 5:06 AM To: Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] a log message is output in the two line
For some reason the log-msg-size() setting wasn't effective. Clearly syslog-ng split the line at 8192 bytes. It could either be the client or the server, but I would vote for the client being the culprit.
syslog-ng emits a debug message that starts like: "Incoming message", there you should see what was received as a single entry. My guess is that you'll see the line split there.
This way you can pinpoint which one is the culprit (the client or the server).
On Jun 24, 2016 11:41 AM, "Hirose, Shinsaku" < hirose.shinsaku@jp.fujitsu.com <mailto:hirose.shinsaku@jp.fujitsu.com> > wrote:
Dear Sandor
Thank you for your quick responce.
> UDP transport limits datagram size to 64k
OK. Thanks!
I raised log_msg_size to higer value. But the behavior did not changed.
I added ' log_msg_size (16384);' in options on the two hosts. ------------------------------------------------------------ options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_msg_size (16384); }; ------------------------------------------------------------
And I did restarting service. But the behavior did not changed.
Do you know this reason?
Best regards,
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu <mailto: syslog-ng-bounces@lists.balabit.hu> [mailto: syslog-ng-bounces@lists.balabit.hu <mailto: syslog-ng-bounces@lists.balabit.hu> ] On Behalf Of Sandor Geller Sent: Friday, June 24, 2016 11:32 PM To: syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] a log message is output in the two line
Hi,
On 06/24/2016 04:10 PM, Hirose, Shinsaku wrote: > Hello, all, > > I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6. > I got it from eple epel repository. > > I am troubled in how to use the source file driver. > The touble is that a log message is output in the two lines on remote syslog server. > > How to reproduce is followings. > > 1. Prepare two hosts running syslog-ng. > > Host_A configuration is followings. > ---------------------------------------- > source test { > file("/tmp/a.log"); > }; > destination d_remote { udp("192.168.0.2"); }; > log { source(test); destination(d_remote); }; > ----------------------------------------
UDP transport limits datagram size to 64k
> Host_B(192.168.0.2) configuration is defaults. > > 2.Execute following command on Host_A. > > $ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log > > 3.Check the log on Host_B. > > As the result, a log message is output in the two lines on Host_B. > > One line is following. The num of "a" is 8192. > aaaaaaaaaaaaa....... > > The other line is following. The num of "a" is 1. > a > > I hope a log message is output in the one line on Host_B. > Is my hope readily achievable?
syslog isn't rsync so the syslog standards must get taken into account. Original syslog was designed to work with single-line, small messages (up to 1k) fitting into a single UDP packet to avoid fragmentating the datagram.
IIRC the maximal message size in syslog-ng defaults to 8192 bytes, so from stream-like sources input will be splitted when this length is reached.
> Please advise me.
You can raise log_msg_size to higer value but don't expect that values larger than 64k will work with the udp() transport. You should switch to another transport driver like tcp() or even better to syslog()
Regards,
Sandor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, sorry for the dumb question. Is Gradle needed on syslog servers after the build ?
Hi, No, it's needed only for building the Java based destinations. You can run syslog-ng without it. Best, Tibor 2016-06-30 14:23 GMT+02:00 Scot Needy <scotrn@gmail.com>:
Hi, sorry for the dumb question.
Is Gradle needed on syslog servers after the build ?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Hirose, Shinsaku
-
Sandor Geller
-
Scheidler, Balázs
-
Scot Needy
-
Tibor Benke