Syslog-NG forwarding the messages to a Remote TCP port with more than one message in a single packet
I am using syslog-ng *version * [root@Cypher-210 ~]# /usr/local/sbin/syslog-ng -V syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Mar 6 2012 13:06:17 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,parser,syslogformat Available-Modules: afprog,convertfuncs,affile,afsocket-tls,confgen,csvparser,syslogformat,dbparser,basicfuncs,afmongodb,dummy,afuser,afsocket Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on Problem: ------------ Syslog-ng is forwarding the TCP packets to a Remote TCP port with More than one message in a single TCP Packet.But, in between the messages inside a packet, It doesn't includes any delimiter like "\n" or some other.But, the receiving Program listens on TCP requires a newline as a delimiter b/w messages so as to parse the messages individually. But, As Syslog-ng doesn't includes a separator b/w messages in a packet, The Receiver simply considering whole messages in a packet as a single Message which leads to false and unacceptable parsing. Please look into the following messages exists in a single TCP packet.The one I colored red is a start of new message. root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shadow *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/head -n 4 /etc/inittab *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/tail -n 1 *<13>*mar 6 17:22:15 cypher-210 sshd[5583]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:15 cypher-210 sshd[5765]: accepted password for root from 10.0.15.218 port 44258 ssh2 <13>mar 6 17:22:15 cypher-210 sshd[5765]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:16 cypher-210 sshd[5278]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 pam_timestamp_check: pam `/' permissions are lax <13>mar 6 17:22:17 cypher-210 sshd[5281]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shad These are the settings i made in the syslog-ng.conf file: source s_test { internal(); unix-stream("/dev/log"); udp(); file("/var/log/syslog-ng_local"); }; destination d_test { tcp("10.0.15.18" port(9500)); #My Program listening on tcp port 9500 requires newline as a separator for each message. }; log { source(s_test); destination(d_test); }; Can anyone kindly help me, 1. How can i limit one tcp packet to hold only one message? [or] 2. How can I add newline as a delimiter b/w each message before the packet(s) are forwarding to a destination so that it could parse properly? Your Help is really appreciate.. Thanks&Regards Anjaneyulu P
Hi Anji, at first glance it seems as if you are using a template in your destination (because of the small m in mar) and are missing the "\n" from the end of the template. Are you sure that you have pasted the right config snippet? Balint On 03/07/2012 05:38 AM, anji prassana wrote:
I am using syslog-ng *version * [root@Cypher-210 ~]# /usr/local/sbin/syslog-ng -V syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Mar 6 2012 13:06:17 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,parser,syslogformat Available-Modules: afprog,convertfuncs,affile,afsocket-tls,confgen,csvparser,syslogformat,dbparser,basicfuncs,afmongodb,dummy,afuser,afsocket Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
Problem: ------------ Syslog-ng is forwarding the TCP packets to a Remote TCP port with More than one message in a single TCP Packet.But, in between the messages inside a packet, It doesn't includes any delimiter like "\n" or some other.But, the receiving Program listens on TCP requires a newline as a delimiter b/w messages so as to parse the messages individually. But, As Syslog-ng doesn't includes a separator b/w messages in a packet, The Receiver simply considering whole messages in a packet as a single Message which leads to false and unacceptable parsing.
Please look into the following messages exists in a single TCP packet.The one I colored red is a start of new message.
root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shadow *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/head -n 4 /etc/inittab *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/tail -n 1 *<13>*mar 6 17:22:15 cypher-210 sshd[5583]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:15 cypher-210 sshd[5765]: accepted password for root from 10.0.15.218 port 44258 ssh2 <13>mar 6 17:22:15 cypher-210 sshd[5765]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:16 cypher-210 sshd[5278]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 pam_timestamp_check: pam `/' permissions are lax <13>mar 6 17:22:17 cypher-210 sshd[5281]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shad
These are the settings i made in the syslog-ng.conf file:
source s_test { internal(); unix-stream("/dev/log"); udp(); file("/var/log/syslog-ng_local"); };
destination d_test { tcp("10.0.15.18" port(9500)); #My Program listening on tcp port 9500 requires newline as a separator for each message. }; log { source(s_test); destination(d_test); };
Can anyone kindly help me, 1. How can i limit one tcp packet to hold only one message? [or] 2. How can I add newline as a delimiter b/w each message before the packet(s) are forwarding to a destination so that it could parse properly?
Your Help is really appreciate..
Thanks&Regards Anjaneyulu P
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Balint, Thanks for your kind reply and sorry for the delay.As i was on leave till today, My reply is delayed.However, I didn't use any template.The Packet data which i had pasted in my previous message is what i received from the application running at the destination end "10.0.15.18" with port 9500;Might be this application is displaying as small letter 'm'. But,Syslog -ng is forwarding with 'M' only to the Destination with multiple messages in a single TCP packet.This is been confirmed through WireShark.Please look into the attached image file captured from Packet Analyzer tool wireshark. I have attached a file which was captured through wireshark to show you the multiple events forwarded by syslog-ng to the Destination.But, unfortunately it was rejected as it's not under the size limits of Syslog-ng pipermail. Kindly assist me on How can i configure syslog-ng in order to send only one message per single tcp packet or else having a new line character '\n' at the end of each message. Please let me know if you need any further information. Thanks & Regards Anjaneyulu P On Wed, Mar 7, 2012 at 2:04 PM, Balint Kovacs <balint.kovacs@balabit.com>wrote:
Hi Anji,
at first glance it seems as if you are using a template in your destination (because of the small m in mar) and are missing the "\n" from the end of the template. Are you sure that you have pasted the right config snippet?
Balint
On 03/07/2012 05:38 AM, anji prassana wrote:
I am using syslog-ng *version * [root@Cypher-210 ~]# /usr/local/sbin/syslog-ng -V syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c<ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master%23d199a1980be6b23fe24189e86a882812288e292c> Compile-Date: Mar 6 2012 13:06:17 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,parser,syslogformat Available-Modules: afprog,convertfuncs,affile,afsocket-tls,confgen,csvparser,syslogformat,dbparser,basicfuncs,afmongodb,dummy,afuser,afsocket Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
Problem: ------------ Syslog-ng is forwarding the TCP packets to a Remote TCP port with More than one message in a single TCP Packet.But, in between the messages inside a packet, It doesn't includes any delimiter like "\n" or some other.But, the receiving Program listens on TCP requires a newline as a delimiter b/w messages so as to parse the messages individually. But, As Syslog-ng doesn't includes a separator b/w messages in a packet, The Receiver simply considering whole messages in a packet as a single Message which leads to false and unacceptable parsing.
Please look into the following messages exists in a single TCP packet.The one I colored red is a start of new message.
root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shadow *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/head -n 4 /etc/inittab *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/tail -n 1 *<13>*mar 6 17:22:15 cypher-210 sshd[5583]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:15 cypher-210 sshd[5765]: accepted password for root from 10.0.15.218 port 44258 ssh2 <13>mar 6 17:22:15 cypher-210 sshd[5765]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:16 cypher-210 sshd[5278]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 pam_timestamp_check: pam `/' permissions are lax <13>mar 6 17:22:17 cypher-210 sshd[5281]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shad
These are the settings i made in the syslog-ng.conf file:
source s_test { internal(); unix-stream("/dev/log"); udp(); file("/var/log/syslog-ng_local"); };
destination d_test { tcp("10.0.15.18" port(9500)); #My Program listening on tcp port 9500 requires newline as a separator for each message. }; log { source(s_test); destination(d_test); };
Can anyone kindly help me, 1. How can i limit one tcp packet to hold only one message? [or] 2. How can I add newline as a delimiter b/w each message before the packet(s) are forwarding to a destination so that it could parse properly?
Your Help is really appreciate..
Thanks&Regards Anjaneyulu P
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Sat, 2012-03-17 at 19:48 +0530, anji prassana wrote:
Hi Balint,
Thanks for your kind reply and sorry for the delay.As i was on leave till today, My reply is delayed.However, I didn't use any template.The Packet data which i had pasted in my previous message is what i received from the application running at the destination end "10.0.15.18" with port 9500;Might be this application is displaying as small letter 'm'. But,Syslog-ng is forwarding with 'M' only to the Destination with multiple messages in a single TCP packet.This is been confirmed through WireShark.Please look into the attached image file captured from Packet Analyzer tool wireshark.
I have attached a file which was captured through wireshark to show you the multiple events forwarded by syslog-ng to the Destination.But, unfortunately it was rejected as it's not under the size limits of Syslog-ng pipermail.
Kindly assist me on How can i configure syslog-ng in order to send only one message per single tcp packet or else having a new line character '\n' at the end of each message.
Please let me know if you need any further information.
syslog-ng doesn't really care about packet boundaries when using TCP, as it is against the spirit of TCP itself, however it must correctly delimit messages using \n, especially if you are not using templates. Are you sure you correctly diagnosed the problem and the '\n' are not there? It might very well happen that the same packet holds two (or more messages), but that's not an issue in itself. The configuration file didn't have a template specification, so this code should apply to your case (quoted from logwriter.c, log_writer_format_log function): const gchar *p; gssize len; if (self->flags & LW_FORMAT_FILE) { log_stamp_format(stamp, result, self->options->template_options.ts_format, time_zone_info_get_offset(self->options->template_options.time_zone_info[LTZ_SEND], stamp->tv_sec), self->options->template_options.frac_digits); } else if (self->flags & LW_FORMAT_PROTO) { g_string_append_c(result, '<'); format_uint32_padded(result, 0, 0, 10, lm->pri); g_string_append_c(result, '>'); /* always use BSD timestamp by default, the use can override this using a custom template */ log_stamp_append_format(stamp, result, TS_FMT_BSD, time_zone_info_get_offset(self->options->template_options.time_zone_info[LTZ_SEND], stamp->tv_sec), self->options->template_options.frac_digits); } g_string_append_c(result, ' '); p = log_msg_get_value(lm, LM_V_HOST, &len); g_string_append_len(result, p, len); g_string_append_c(result, ' '); if ((lm->flags & LF_LEGACY_MSGHDR)) { p = log_msg_get_value(lm, LM_V_LEGACY_MSGHDR, &len); g_string_append_len(result, p, len); } else { p = log_msg_get_value(lm, LM_V_PROGRAM, &len); if (len > 0) { g_string_append_len(result, p, len); p = log_msg_get_value(lm, LM_V_PID, &len); if (len > 0) { g_string_append_c(result, '['); g_string_append_len(result, p, len); g_string_append_c(result, ']'); } g_string_append_len(result, ": ", 2); } } p = log_msg_get_value(lm, LM_V_MESSAGE, &len); g_string_append_len(result, p, len); g_string_append_c(result, '\n'); log_writer_do_padding(self, result); As you can see, the '\n' marker is unconditionally appended at the end of the function. -- Bazsi
participants (3)
-
anji prassana
-
Balazs Scheidler
-
Balint Kovacs