I'm seeing an error in my log indicating that messages aren't being sent to a queue in RabbitMQ. This happens a few times a week. Here's the error: Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10' I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren't sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that's limiting the messages?
Nothing intended limits syslog-ng here. That error message happens when rabbitmq returns an error condition. It definitely should include a more detailed error cause. Syslog-ng suspends this destination for a few seconds (controlled by time-reopen ()) And then reconnects to the amqp server. I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ. This happens a few times a week. Here’s the error: Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10' I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that’s limiting the messages? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, I have just created a new pull request: https://github.com/balabit/syslog-ng/pull/811 This one adds an "error" tag to the message above which hopefully identifies what went wrong. At the same time, I've verified that all error handlers report this information, so it becomes easier to troubleshoot amqp. Unfortunately though I don't have an amqp service running, so I am unable to test, thus I'd appreciate some testing. Thanks Bazsi -- Bazsi On Thu, Nov 26, 2015 at 8:46 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Nothing intended limits syslog-ng here. That error message happens when rabbitmq returns an error condition. It definitely should include a more detailed error cause.
Syslog-ng suspends this destination for a few seconds (controlled by time-reopen ()) And then reconnects to the amqp server.
I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ. This happens a few times a week. Here’s the error:
Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10'
I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that’s limiting the messages?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
So are messages lost when this happens, or are they resent when after reconnection? From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Scheidler, Balázs Sent: Thursday, November 26, 2015 2:47 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] rabbitmq errors Nothing intended limits syslog-ng here. That error message happens when rabbitmq returns an error condition. It definitely should include a more detailed error cause. Syslog-ng suspends this destination for a few seconds (controlled by time-reopen ()) And then reconnects to the amqp server. I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ. This happens a few times a week. Here’s the error: Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10' I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that’s limiting the messages? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
They should be resent. On Nov 30, 2015 4:50 PM, "Al Itchon" <AItchon@boardreader.com> wrote:
So are messages lost when this happens, or are they resent when after reconnection?
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Thursday, November 26, 2015 2:47 AM *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] rabbitmq errors
Nothing intended limits syslog-ng here. That error message happens when rabbitmq returns an error condition. It definitely should include a more detailed error cause.
Syslog-ng suspends this destination for a few seconds (controlled by time-reopen ()) And then reconnects to the amqp server.
I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ. This happens a few times a week. Here’s the error:
Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10'
I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that’s limiting the messages?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, With a quick test I can verify that there is a new "error" tag in the error message: [2015-12-02T14:28:42.828127] Initializing AMQP destination; vhost='/almafa', host='172.17.0.3', port='5672', exchange='applogs', exchange_type='header' [New Thread 0x7ffff7fdd700 (LWP 40)] [2015-12-02T14:28:42.831050] Worker thread started; driver='d_signupws_amqp#0' [2015-12-02T14:28:42.831158] Running application hooks; hook='1' [2015-12-02T14:28:42.831174] Running application hooks; hook='3' [2015-12-02T14:28:42.831213] syslog-ng starting up; version='3.8.0alpha0' [New Thread 0x7ffff315f700 (LWP 41)] [2015-12-02T14:28:43.833637] Incoming log entry; line='ads asda sda sdas dsa dsa' [2015-12-02T14:28:45.833349] Error during AMQP login; driver='d_signupws_amqp#0', error='a socket error occurred', time_reopen='60' [2015-12-02T14:28:48.837345] Error during AMQP login; driver='d_signupws_amqp#0', error='a socket error occurred', time_reopen='60' Br, Micek On Mon, Nov 30, 2015 at 5:19 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
They should be resent. On Nov 30, 2015 4:50 PM, "Al Itchon" <AItchon@boardreader.com> wrote:
So are messages lost when this happens, or are they resent when after reconnection?
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Thursday, November 26, 2015 2:47 AM *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] rabbitmq errors
Nothing intended limits syslog-ng here. That error message happens when rabbitmq returns an error condition. It definitely should include a more detailed error cause.
Syslog-ng suspends this destination for a few seconds (controlled by time-reopen ()) And then reconnects to the amqp server.
I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ. This happens a few times a week. Here’s the error:
Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10'
I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that’s limiting the messages?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Well, the kind of socket error should also be reported, this still doesnt say much. :( On Dec 2, 2015 3:30 PM, "Mitzki, András" <andras.mitzki@balabit.com> wrote:
Hi,
With a quick test I can verify that there is a new "error" tag in the error message: [2015-12-02T14:28:42.828127] Initializing AMQP destination; vhost='/almafa', host='172.17.0.3', port='5672', exchange='applogs', exchange_type='header' [New Thread 0x7ffff7fdd700 (LWP 40)] [2015-12-02T14:28:42.831050] Worker thread started; driver='d_signupws_amqp#0' [2015-12-02T14:28:42.831158] Running application hooks; hook='1' [2015-12-02T14:28:42.831174] Running application hooks; hook='3' [2015-12-02T14:28:42.831213] syslog-ng starting up; version='3.8.0alpha0' [New Thread 0x7ffff315f700 (LWP 41)] [2015-12-02T14:28:43.833637] Incoming log entry; line='ads asda sda sdas dsa dsa' [2015-12-02T14:28:45.833349] Error during AMQP login; driver='d_signupws_amqp#0', error='a socket error occurred', time_reopen='60' [2015-12-02T14:28:48.837345] Error during AMQP login; driver='d_signupws_amqp#0', error='a socket error occurred', time_reopen='60'
Br, Micek
On Mon, Nov 30, 2015 at 5:19 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
They should be resent. On Nov 30, 2015 4:50 PM, "Al Itchon" <AItchon@boardreader.com> wrote:
So are messages lost when this happens, or are they resent when after reconnection?
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Thursday, November 26, 2015 2:47 AM *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] rabbitmq errors
Nothing intended limits syslog-ng here. That error message happens when rabbitmq returns an error condition. It definitely should include a more detailed error cause.
Syslog-ng suspends this destination for a few seconds (controlled by time-reopen ()) And then reconnects to the amqp server.
I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ. This happens a few times a week. Here’s the error:
Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10'
I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that’s limiting the messages?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I do see a similar error when trying to send a long message (318525 characters): Dec 15 12:50:35 rabbit-to-kafka01 syslog-ng[14073]: Network error while inserting into AMQP server; driver='#anon-destination17#0', time_reopen='10' Is there a limit on the length of a message going to rabbitmq from syslog-ng? From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Wednesday, December 02, 2015 2:12 PM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] rabbitmq errors Well, the kind of socket error should also be reported, this still doesnt say much. :( On Dec 2, 2015 3:30 PM, "Mitzki, András" <andras.mitzki@balabit.com<mailto:andras.mitzki@balabit.com>> wrote: Hi, With a quick test I can verify that there is a new "error" tag in the error message: [2015-12-02T14:28:42.828127] Initializing AMQP destination; vhost='/almafa', host='172.17.0.3', port='5672', exchange='applogs', exchange_type='header' [New Thread 0x7ffff7fdd700 (LWP 40)] [2015-12-02T14:28:42.831050] Worker thread started; driver='d_signupws_amqp#0' [2015-12-02T14:28:42.831158] Running application hooks; hook='1' [2015-12-02T14:28:42.831174] Running application hooks; hook='3' [2015-12-02T14:28:42.831213] syslog-ng starting up; version='3.8.0alpha0' [New Thread 0x7ffff315f700 (LWP 41)] [2015-12-02T14:28:43.833637] Incoming log entry; line='ads asda sda sdas dsa dsa' [2015-12-02T14:28:45.833349] Error during AMQP login; driver='d_signupws_amqp#0', error='a socket error occurred', time_reopen='60' [2015-12-02T14:28:48.837345] Error during AMQP login; driver='d_signupws_amqp#0', error='a socket error occurred', time_reopen='60' Br, Micek On Mon, Nov 30, 2015 at 5:19 PM, Scheidler, Balázs <balazs.scheidler@balabit.com<mailto:balazs.scheidler@balabit.com>> wrote: They should be resent. On Nov 30, 2015 4:50 PM, "Al Itchon" <AItchon@boardreader.com<mailto:AItchon@boardreader.com>> wrote: So are messages lost when this happens, or are they resent when after reconnection? From: syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu> [mailto:syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>] On Behalf Of Scheidler, Balázs Sent: Thursday, November 26, 2015 2:47 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: Re: [syslog-ng] rabbitmq errors Nothing intended limits syslog-ng here. That error message happens when rabbitmq returns an error condition. It definitely should include a more detailed error cause. Syslog-ng suspends this destination for a few seconds (controlled by time-reopen ()) And then reconnects to the amqp server. I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ. This happens a few times a week. Here’s the error: Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10' I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that’s limiting the messages? ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Not by syslog-ng. If you apply the patch which adds the error tag to this log message you'd have slightly more information. On Dec 15, 2015 7:07 PM, "Al Itchon" <AItchon@boardreader.com> wrote:
I do see a similar error when trying to send a long message (318525 characters):
Dec 15 12:50:35 rabbit-to-kafka01 syslog-ng[14073]: Network error while inserting into AMQP server; driver='#anon-destination17#0', time_reopen='10'
Is there a limit on the length of a message going to rabbitmq from syslog-ng?
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Balazs Scheidler *Sent:* Wednesday, December 02, 2015 2:12 PM *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] rabbitmq errors
Well, the kind of socket error should also be reported, this still doesnt say much. :(
On Dec 2, 2015 3:30 PM, "Mitzki, András" <andras.mitzki@balabit.com> wrote:
Hi,
With a quick test I can verify that there is a new "error" tag in the error message: [2015-12-02T14:28:42.828127] Initializing AMQP destination; vhost='/almafa', host='172.17.0.3', port='5672', exchange='applogs', exchange_type='header' [New Thread 0x7ffff7fdd700 (LWP 40)] [2015-12-02T14:28:42.831050] Worker thread started; driver='d_signupws_amqp#0' [2015-12-02T14:28:42.831158] Running application hooks; hook='1' [2015-12-02T14:28:42.831174] Running application hooks; hook='3' [2015-12-02T14:28:42.831213] syslog-ng starting up; version='3.8.0alpha0' [New Thread 0x7ffff315f700 (LWP 41)] [2015-12-02T14:28:43.833637] Incoming log entry; line='ads asda sda sdas dsa dsa' [2015-12-02T14:28:45.833349] Error during AMQP login; driver='d_signupws_amqp#0', error='a socket error occurred', time_reopen='60' [2015-12-02T14:28:48.837345] Error during AMQP login; driver='d_signupws_amqp#0', error='a socket error occurred', time_reopen='60'
Br,
Micek
On Mon, Nov 30, 2015 at 5:19 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
They should be resent.
On Nov 30, 2015 4:50 PM, "Al Itchon" <AItchon@boardreader.com> wrote:
So are messages lost when this happens, or are they resent when after reconnection?
*From:* syslog-ng-bounces@lists.balabit.hu [mailto: syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Scheidler, Balázs *Sent:* Thursday, November 26, 2015 2:47 AM *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] rabbitmq errors
Nothing intended limits syslog-ng here. That error message happens when rabbitmq returns an error condition. It definitely should include a more detailed error cause.
Syslog-ng suspends this destination for a few seconds (controlled by time-reopen ()) And then reconnects to the amqp server.
I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ. This happens a few times a week. Here’s the error:
Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10'
I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server. It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error. Could this be a network issue or something within syslog-ng that’s limiting the messages?
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Al Itchon
-
Balazs Scheidler
-
Mitzki, András
-
Scheidler, Balázs