Packet fragmentation issue
Hi all, I'm running syslog-ng 2.4.1, log sources send to a log server which beside keeping the original data as is in files, forwards them in real time to a SIEM, spoofing source IP. My problem comes after some logs are too long to fit in a single frame, log server fragments those packets when sending them to SIEM and spoofing is not performed for them. Enviado desde mi iPhone
Sorry, my previous message went out unfinished and I see I've placed it as an answer to someone else's question. I just wanted to add that I was posting because I had not seen this issue in the list; Sorry if I'm wrong. Thanks very much in advance. Kind regards. Enviado desde mi iPhone El 02/03/2012, a las 14:40, José Moreno <jmorenoa@gmail.com> escribió:
Hi all,
I'm running syslog-ng 2.4.1, log sources send to a log server which beside keeping the original data as is in files, forwards them in real time to a SIEM, spoofing source IP.
My problem comes after some logs are too long to fit in a single frame, log server fragments those packets when sending them to SIEM and spoofing is not performed for them.
Enviado desde mi iPhone
Hi, There is no syslog-ng 2.4.1 version, the last 2.x version was 2.1.4 which is pretty much obsolete. Anyway, syslog-ng and any syslog daemon in general isn't a transport mechanism for arbitrary content so some limitations are in place. You're using spoofing which means UDP. The 64k size limitation of a single UDP datagram is definitely a limiting factor. What is log_msg_size in your config? How long are the lines in the logfiles which end up splitted into multiple messages on the other end? Regards, Sandor 2012/3/2 José Moreno <jmorenoa@gmail.com>:
Sorry, my previous message went out unfinished and I see I've placed it as an answer to someone else's question.
I just wanted to add that I was posting because I had not seen this issue in the list; Sorry if I'm wrong.
Thanks very much in advance. Kind regards.
Enviado desde mi iPhone
El 02/03/2012, a las 14:40, José Moreno <jmorenoa@gmail.com> escribió:
Hi all,
I'm running syslog-ng 2.4.1, log sources send to a log server which beside keeping the original data as is in files, forwards them in real time to a SIEM, spoofing source IP.
My problem comes after some logs are too long to fit in a single frame, log server fragments those packets when sending them to SIEM and spoofing is not performed for them.
Enviado desde mi iPhone
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Thanks very much for your help. Version is 2.1.4, Sorry for the mistake. Actually, log_msg_size is not present in the configuration file, so I guess it defaults to 8192 bytes, after reading documentation, though I don't know if that figure applies to that version too. Nevertheless, the long logs are no more than 5Kbyte, they are Windows events. I'll try setting log_msg_size to some value higher than my longest logs and will try to upgrade syslog-ng. Meanwhile your comments would be much appreciated. Thanks very much again and kind regards. José Moreno El 06/03/2012, a las 11:42, Sandor Geller <Sandor.Geller@morganstanley.com> escribió:
Hi,
There is no syslog-ng 2.4.1 version, the last 2.x version was 2.1.4 which is pretty much obsolete. Anyway, syslog-ng and any syslog daemon in general isn't a transport mechanism for arbitrary content so some limitations are in place. You're using spoofing which means UDP. The 64k size limitation of a single UDP datagram is definitely a limiting factor. What is log_msg_size in your config? How long are the lines in the logfiles which end up splitted into multiple messages on the other end?
Regards,
Sandor
2012/3/2 José Moreno <jmorenoa@gmail.com>:
Sorry, my previous message went out unfinished and I see I've placed it as an answer to someone else's question.
I just wanted to add that I was posting because I had not seen this issue in the list; Sorry if I'm wrong.
Thanks very much in advance. Kind regards.
Enviado desde mi iPhone
El 02/03/2012, a las 14:40, José Moreno <jmorenoa@gmail.com> escribió:
Hi all,
I'm running syslog-ng 2.4.1, log sources send to a log server which beside keeping the original data as is in files, forwards them in real time to a SIEM, spoofing source IP.
My problem comes after some logs are too long to fit in a single frame, log server fragments those packets when sending them to SIEM and spoofing is not performed for them.
Enviado desde mi iPhone
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Tue, 2012-03-06 at 11:42 +0100, Sandor Geller wrote:
Hi,
There is no syslog-ng 2.4.1 version, the last 2.x version was 2.1.4 which is pretty much obsolete. Anyway, syslog-ng and any syslog daemon in general isn't a transport mechanism for arbitrary content so some limitations are in place. You're using spoofing which means UDP. The 64k size limitation of a single UDP datagram is definitely a limiting factor. What is log_msg_size in your config? How long are the lines in the logfiles which end up splitted into multiple messages on the other end?
Yup, checking the code in question, it prepares a single UDP datagram, and sends it off fire-and-forget, without thinking a little bit about MTU settings. I'm not sure how libnet/kernel processes these packets, it might simply truncate them or drop it altogether. If the kernel chooses to refragment such packets (which might easily happen if you are using connection tracking on Linux, even if the core kernel doesn't do it), it should properly produce well correct IP addresses in the 2nd and subsequent fragments. -- Bazsi
Thanks very much Bazsi, I'll check connection tracking and will let the list know about any progress. Kind regards. José Moreno El 18/03/2012, a las 13:15, Balazs Scheidler <bazsi@balabit.hu> escribió:
On Tue, 2012-03-06 at 11:42 +0100, Sandor Geller wrote:
Hi,
There is no syslog-ng 2.4.1 version, the last 2.x version was 2.1.4 which is pretty much obsolete. Anyway, syslog-ng and any syslog daemon in general isn't a transport mechanism for arbitrary content so some limitations are in place. You're using spoofing which means UDP. The 64k size limitation of a single UDP datagram is definitely a limiting factor. What is log_msg_size in your config? How long are the lines in the logfiles which end up splitted into multiple messages on the other end?
Yup, checking the code in question, it prepares a single UDP datagram, and sends it off fire-and-forget, without thinking a little bit about MTU settings.
I'm not sure how libnet/kernel processes these packets, it might simply truncate them or drop it altogether.
If the kernel chooses to refragment such packets (which might easily happen if you are using connection tracking on Linux, even if the core kernel doesn't do it), it should properly produce well correct IP addresses in the 2nd and subsequent fragments.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Balazs Scheidler
-
Jose Moreno
-
José Moreno
-
Sandor Geller