Transparent TCP forward: what am I doing wrong?
I need to be able to accept TCP/514 and transparently forward it to UDP/514 on the same box. I have a config that almost works, except I'm getting an "extra" time stamp and hostname on the TCP messages. I'm using Netcat to send: echo "<182>loop_test[$$]: LOOP testing 514 UDP, one ping only..." | nc -u -w1 192.168.1.151 514 echo "<182>loop_test[$$]: LOOP testing 514 TCP, one ping only..." | nc 192.168.1.151 514 I'm getting: <182>loop_test[11061]: LOOP testing 514 UDP, one ping only... <182>Jun 30 01:48:36 192.168.1.10 loop_test[11061]: LOOP testing 514 TCP, one ping only... I want: <182>loop_test[11061]: LOOP testing 514 UDP, one ping only... <182>loop_test[11061]: LOOP testing 514 TCP, one ping only... If I send the messages into syslog-ng or rsyslog, I get the correct data with no "extra" fields in the log file. But unfortunately, I need to send messages into another proprietary syslog listener, and in that one I'm getting the extra fields. I'd blame the whole mess on that, except that when I do a packet dump I do see the 2 extra fields in there. So I *think* that syslog-ng and rsyslog are smart enough to handle them and my other thing isn't. I'm already using keep_timestamp(no) and keep_hostname(no) but they have no effect I've been able to detect. I've tried commented out, set to yes and set to no, but no matter what I get the 2 extra fields. I've also tried syslog-ng 3.0.8 and 3.2.4, same effect. This is the relevant part of the syslog-ng config I'm using: ####### BEGIN: Forward TCP syslog source s_tcpincoming { tcp( ip(0.0.0.0) port(514) max_connections(1000) keep_timestamp(no) keep_hostname(no) ); }; destination d_fe_tcp { udp( "127.0.0.2" port(514) spoof_source(yes) ); }; log { source(s_tcpincoming); # With no "filter" we get everything, which is what we want destination(d_fe_tcp); }; ####### END: Forward TCP syslog As I said, my goal is to receive TCP/514 and **transparently** forward logs with no changes, as if they came in via UDP, to the localhost via UDP/514. In other words, I'm using syslog-ng as a shim to feed syslog over TCP to a listener which only listens on UDP. Why doesn't it work? What totally obvious thing am I missing? Am I doing anything else dumb? TIA, JP ----------------------------|:::======|------------------------------- JP Vossen, CISSP |:::======| http://bashcookbook.com/ My Account, My Opinions |=========| http://www.jpsdomain.org/ ----------------------------|=========|------------------------------- "Microsoft Tax" = the additional hardware & yearly fees for the add-on software required to protect Windows from its own poorly designed and implemented self, while the overhead incidentally flattens Moore's Law.
Hi, On Thu, Jun 30, 2011 at 10:12 AM, JP Vossen <jp@jpsdomain.org> wrote:
I need to be able to accept TCP/514 and transparently forward it to UDP/514 on the same box. I have a config that almost works, except I'm getting an "extra" time stamp and hostname on the TCP messages.
I'm using Netcat to send: echo "<182>loop_test[$$]: LOOP testing 514 UDP, one ping only..." | nc -u -w1 192.168.1.151 514 echo "<182>loop_test[$$]: LOOP testing 514 TCP, one ping only..." | nc 192.168.1.151 514
I'm getting: <182>loop_test[11061]: LOOP testing 514 UDP, one ping only... <182>Jun 30 01:48:36 192.168.1.10 loop_test[11061]: LOOP testing 514 TCP, one ping only...
this is expected behaviour
I want: <182>loop_test[11061]: LOOP testing 514 UDP, one ping only... <182>loop_test[11061]: LOOP testing 514 TCP, one ping only...
syslog-ng could beoverkill for such a purpose, writing a few lines of perl code could be easier...
If I send the messages into syslog-ng or rsyslog, I get the correct data with no "extra" fields in the log file. But unfortunately, I need to send messages into another proprietary syslog listener, and in that one I'm getting the extra fields. I'd blame the whole mess on that, except that when I do a packet dump I do see the 2 extra fields in there. So I *think* that syslog-ng and rsyslog are smart enough to handle them and my other thing isn't.
syslog-ng adds the timestamp and hostname fields by default. Many people think that syslog daemons should accept random junk, but the syslog daemon shouldn't be treated like a plain transport mechanism. There are RFCs describing the syslog formats and the daemons should enforce the standards.
As I said, my goal is to receive TCP/514 and **transparently** forward logs with no changes, as if they came in via UDP, to the localhost via UDP/514. In other words, I'm using syslog-ng as a shim to feed syslog over TCP to a listener which only listens on UDP.
Why doesn't it work? What totally obvious thing am I missing? Am I doing anything else dumb?
Your incoming logs aren't properly formatted syslog messages so syslog-ng has to guess which fields are present and which are missing and adds the required fields. If you don't want this then tell syslog-ng to don't parse the logs and then just use a custom template, something like this: source s_tcpincoming { tcp( ip(0.0.0.0) port(514) max_connections(1000) keep_timestamp(no) keep_hostname(no) flags(no-parse) ); }; destination d_fe_tcp { udp( "127.0.0.2" port(514) spoof_source(yes) template("${MESSAGE}\n") ); }; log { source(s_tcpincoming); # With no "filter" we get everything, which is what we want destination(d_fe_tcp); }; hth, Sandor
On 06/30/2011 04:47 AM, Sandor Geller wrote: <snip>
syslog-ng could beoverkill for such a purpose, writing a few lines of perl code could be easier...
True, but I'm already using syslog-ng and the fewer things I have to build, deploy and maintain the better.
If I send the messages into syslog-ng or rsyslog, I get the correct data with no "extra" fields in the log file. But unfortunately, I need to send messages into another proprietary syslog listener, and in that one I'm getting the extra fields. I'd blame the whole mess on that, except that when I do a packet dump I do see the 2 extra fields in there. So I *think* that syslog-ng and rsyslog are smart enough to handle them and my other thing isn't.
syslog-ng adds the timestamp and hostname fields by default. Many people think that syslog daemons should accept random junk, but the syslog daemon shouldn't be treated like a plain transport mechanism. There are RFCs describing the syslog formats and the daemons should enforce the standards.
Very true!!! Unfortunately, the environments I have to handle do have quite a lot of random junk that pretends to be "syslog" and I have to deal with it. That's partly why I was using that cruddy netcat command, it's actuality quite a good test case for my needs. :-/ But if *everything* was RFC-compliant my (and I suspect a lot of folks on this list) life would be a lot easier. :-)
As I said, my goal is to receive TCP/514 and **transparently** forward logs with no changes, as if they came in via UDP, to the localhost via UDP/514. In other words, I'm using syslog-ng as a shim to feed syslog over TCP to a listener which only listens on UDP.
Why doesn't it work? What totally obvious thing am I missing? Am I doing anything else dumb?
Your incoming logs aren't properly formatted syslog messages so syslog-ng has to guess which fields are present and which are missing and adds the required fields. If you don't want this then tell syslog-ng to don't parse the logs and then just use a custom template, something like this:
source s_tcpincoming { tcp( ip(0.0.0.0) port(514) max_connections(1000) keep_timestamp(no) keep_hostname(no) flags(no-parse) ); }; destination d_fe_tcp { udp( "127.0.0.2" port(514) spoof_source(yes) template("${MESSAGE}\n") ); }; <snip> hth, Sandor
Yes! Thank you! The no-parse and template lines did the trick. Thanks also to Gergely Nagy for the same solution. This is part of why I love open source, random people all over the world are just so helpful. Thanks also to Jose Pedro Oliveira and others on the list for lots of help with other (build on CentOS-5) issues. Later, JP ----------------------------|:::======|------------------------------- JP Vossen, CISSP |:::======| http://bashcookbook.com/ My Account, My Opinions |=========| http://www.jpsdomain.org/ ----------------------------|=========|------------------------------- "Microsoft Tax" = the additional hardware & yearly fees for the add-on software required to protect Windows from its own poorly designed and implemented self, while the overhead incidentally flattens Moore's Law.
The syslog-ng Agent for Windows 4.0 manual states "Similarly to the Linux version, the agent now sends MARK messages to the server to indicate that the client host is alive but there are no log messages to send. A MARK message is sent every ten minutes." but running version 4.0.1 does not send these MARK messages. We see lots of other messages from the host in our syslog server, but not the MARK messages. Any suggestions? Evan.
JP Vossen <jp@jpsdomain.org> writes:
As I said, my goal is to receive TCP/514 and **transparently** forward logs with no changes, as if they came in via UDP, to the localhost via UDP/514. In other words, I'm using syslog-ng as a shim to feed syslog over TCP to a listener which only listens on UDP.
While it's early in the morning and I didn't have my dose of coffee yet: wouldn't flags(no-parse) on the source side, and template("$MSG") on the destination side work? As far as I understand, no-parse would put the whole original message in the $MSG macro, and then you override the destination template to skip everything, and just send the $MSG. But I haven't tested this, and I'm not even sure if it would work, but I have a hunch that it should. Let me know if it doesn't! -- |8]
participants (4)
-
Evan Rempel
-
Gergely Nagy
-
JP Vossen
-
Sandor Geller