ts_format(iso) bug or misunderstanding?
So I'm using iso timezone format for my syslog clients: options { … ts_format(iso); … }; I expected this to use the iso format for all syslog entries. However, I've found that it only uses iso format for local syslog entries. syslog entries sent to the syslog-ng collector are in old rfc3164 format.
chiestand@host:~$ logger -t my-test asdlfkjasdfasdf
which sends (3 local entries, 1 remote entry):
sudo strace -ff -p 13658 ... write(10, "2012-04-05T18:14:37-07:00 host "..., 64) = 64 write(11, "2012-04-05T18:14:37-07:00 host "..., 64) = 64 write(9, "<13>Apr 5 18:25:09 host my-test"..., 58) = 58 write(14, "2012-04-05T18:14:37-07:00 host "..., 64) = 64
or viewing with tcpdump:
chiestand@host:/var/log$ sudo tcpdump -A host syslog.server.salk.edu … .va<13>Apr 5 18:20:31 host my-test: asdlfkjasdfasdf
I would expect the iso format to be sent to the syslog-ng collector as well. Is this a bug or expected behavior? I checked bugzilla and debian bug tracker and didn't see anything. I'm running syslog-ng v3.1 on Debian squeeze. Thanks, Chris
Somewhere in between bug and misunderstanding. The bug would be in documentation, but the behavior is deliberate. The reason is that when sending over the network to a syslog server, the server expects the message in a certain format. When you change the timestamp, that format is now invalid and the remote end might not be able to parse it. Now you could put `ts_format(iso)` in the `tcp()` destination driver. But if your remote server is looking for a timestamp in ISO format, it probably supports the syslog message protocol <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guide-admin-en.html/concepts_message_ietfsyslog.html>, which uses ISO timestamps. Syslog-ng uses the syslog <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guide-admin-en.html/reference_destination_syslog.html> destination driver for sending in this format. The syslog message protocol looks like this: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8 The forementioned bug in the documentation is that it says the tcp() destination driver ts_format uses the global ts_format setting. It doesnt. -Patrick Sent: Thu Apr 05 2012 21:31:54 GMT-0400 (EDT) From: Chris Hiestand <chiestand@salk.edu> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] ts_format(iso) bug or misunderstanding?
So I'm using iso timezone format for my syslog clients: options { ... ts_format(iso); ... };
I expected this to use the iso format for all syslog entries. However, I've found that it only uses iso format for local syslog entries. syslog entries sent to the syslog-ng collector are in old rfc3164 format.
chiestand@host:~$ logger -t my-test asdlfkjasdfasdf
which sends (3 local entries, 1 remote entry):
sudo strace -ff -p 13658 ... write(10, "2012-04-05T18:14:37-07:00 host "..., 64) = 64 write(11, "2012-04-05T18:14:37-07:00 host "..., 64) = 64 write(9, "<13>Apr 5 18:25:09 host my-test"..., 58) = 58 write(14, "2012-04-05T18:14:37-07:00 host "..., 64) = 64
or viewing with tcpdump:
chiestand@host:/var/log$ sudo tcpdump -A host syslog.server.salk.edu ... .va<13>Apr 5 18:20:31 host my-test: asdlfkjasdfasdf
I would expect the iso format to be sent to the syslog-ng collector as well. Is this a bug or expected behavior? I checked bugzilla and debian bug tracker and didn't see anything.
I'm running syslog-ng v3.1 on Debian squeeze.
Thanks, Chris=
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thank you very much for your reply Patrick, that was very helpful. I have downloaded syslog-ng v3.3 (Debian Wheezy) just to get the latest, and I'm still having a problem. Based in your advice, I was able to successfully get the iso ts_format if I use the syslog() destination driver. However, if I use the tcp() destination driver, I still cannot get iso ts_format. syslog-ng ignores my parameter and sends old style timestamps. my driver:
destination My_Syslog { tcp("syslog.server.salk.edu" port(514) ts_format(iso) ); }; log { source(s_src); destination(My_Syslog); };
tcpdump:
@.m..<hw<86>Apr 6 01:24:01 host CRON[1923]: pam_unix(cron:session): session closed for user root
In fact, I have tried all variations of ts_format (rfc3164, bsd, rfc3339, iso) and I always get the same result. Eventually I will switch to the syslog message protocol, so this is not a show-stopper. But not getting something to work as advertised is still troubling. Could I be missing something else? Or might we be in bug/documentation bug territory? Thanks, Chris On Apr 5, 2012, at 7:10 PM, Patrick Hemmer wrote:
Somewhere in between bug and misunderstanding. The bug would be in documentation, but the behavior is deliberate. The reason is that when sending over the network to a syslog server, the server expects the message in a certain format. When you change the timestamp, that format is now invalid and the remote end might not be able to parse it.
Now you could put `ts_format(iso)` in the `tcp()` destination driver. But if your remote server is looking for a timestamp in ISO format, it probably supports the syslog message protocol, which uses ISO timestamps. Syslog-ng uses the syslog destination driver for sending in this format.
The syslog message protocol looks like this: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
The forementioned bug in the documentation is that it says the tcp() destination driver ts_format uses the global ts_format setting. It doesnt.
-Patrick
If syslog-ng is letting you set ts_format on the tcp destination driver (not throwing a syntax error), but isnt using it, then I'd definitely think bug (though this is something the balabit folks should confirm). An alternate method would be to use a template on the tcp destination driver and explicitly build a format which uses the ISO timestamp. For example: template t_tcp { template("<$BSDTAG> $ISODATE $HOST $MSGHDR$MESSAGE\n") }; destination d_tcp { tcp('1.2.3.4' template(t_tcp)); }; Note the lack of space between $MSGHDR and $MESSAGE, thats deliberate. -Patrick Sent: Fri Apr 06 2012 04:40:07 GMT-0400 (EDT) From: Chris Hiestand <chiestand@salk.edu> To: Patrick Hemmer <syslogng@stormcloud9.net>, Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] ts_format(iso) bug or misunderstanding?
Thank you very much for your reply Patrick, that was very helpful.
I have downloaded syslog-ng v3.3 (Debian Wheezy) just to get the latest, and I'm still having a problem.
Based in your advice, I was able to successfully get the iso ts_format if I use the syslog() destination driver. However, if I use the tcp() destination driver, I still cannot get iso ts_format. syslog-ng ignores my parameter and sends old style timestamps.
my driver:
destination My_Syslog { tcp("syslog.server.salk.edu <http://syslog.server.salk.edu>" port(514) ts_format(iso) ); }; log { source(s_src); destination(My_Syslog); };
tcpdump:
@.m..<hw<86>Apr 6 01:24:01 host CRON[1923]: pam_unix(cron:session): session closed for user root
In fact, I have tried all variations of ts_format (rfc3164, bsd, rfc3339, iso) and I always get the same result.
Eventually I will switch to the syslog message protocol, so this is not a show-stopper. But not getting something to work as advertised is still troubling.
Could I be missing something else? Or might we be in bug/documentation bug territory?
Thanks, Chris
On Apr 5, 2012, at 7:10 PM, Patrick Hemmer wrote:
Somewhere in between bug and misunderstanding. The bug would be in documentation, but the behavior is deliberate. The reason is that when sending over the network to a syslog server, the server expects the message in a certain format. When you change the timestamp, that format is now invalid and the remote end might not be able to parse it.
Now you could put `ts_format(iso)` in the `tcp()` destination driver. But if your remote server is looking for a timestamp in ISO format, it probably supports the syslog message protocol <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guide-admin-en.html/concepts_message_ietfsyslog.html>, which uses ISO timestamps. Syslog-ng uses the syslog <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guide-admin-en.html/reference_destination_syslog.html> destination driver for sending in this format.
The syslog message protocol looks like this: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com <http://mymachine.example.com> su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
The forementioned bug in the documentation is that it says the tcp() destination driver ts_format uses the global ts_format setting. It doesnt.
-Patrick
On Fri, 2012-04-06 at 07:08 -0400, Patrick Hemmer wrote:
If syslog-ng is letting you set ts_format on the tcp destination driver (not throwing a syntax error), but isnt using it, then I'd definitely think bug (though this is something the balabit folks should confirm).
Yes, this is a bug (the question is whether the bug is that it is possible to specify this option or that it doesn't work :) I was discussing this with Algernon the other day, and the conclusion was that we should probably implement ts-format() this way: 1) there'd be a global option, just like now 2) for file destinations the default ts-format() is to use the global one 3) for non-file destinations, the default ts-format() is to use the BSD one, ignoring the global setting. That way there'd be a couple of ways to specify the ts-format for non-file destinations: 1) use ts-format(global) to specify that we want the global setting to be applied 2) use ts-format(iso) to use a specific format for the given destination 3) use a custom template I think the current behaviour of ts-format() worked like this since 2.0, so I wouldn't declare this a blocker issue, although it'd be best to fix it. I'd appreciate patches though :) -- Bazsi
On Fri, 2012-04-06 at 01:40 -0700, Chris Hiestand wrote:
Thank you very much for your reply Patrick, that was very helpful.
I have downloaded syslog-ng v3.3 (Debian Wheezy) just to get the latest, and I'm still having a problem.
Based in your advice, I was able to successfully get the iso ts_format if I use the syslog() destination driver. However, if I use the tcp() destination driver, I still cannot get iso ts_format. syslog-ng ignores my parameter and sends old style timestamps.
yup, you need to specify a custom template to change the formatting on the tcp channel. template("$ISODATE $HOST $MSGHDR$MSG\n") should do the trick. ts-format() is only used for local file destinations.
my driver:
destination My_Syslog { tcp("syslog.server.salk.edu" port(514) ts_format(iso) ); }; log { source(s_src); destination(My_Syslog); };
tcpdump:
@.m..<hw<86>Apr 6 01:24:01 host CRON[1923]: pam_unix(cron:session): session closed for user root
In fact, I have tried all variations of ts_format (rfc3164, bsd, rfc3339, iso) and I always get the same result.
Eventually I will switch to the syslog message protocol, so this is not a show-stopper. But not getting something to work as advertised is still troubling.
Could I be missing something else? Or might we be in bug/documentation bug territory?
Thanks, Chris
On Apr 5, 2012, at 7:10 PM, Patrick Hemmer wrote:
Somewhere in between bug and misunderstanding. The bug would be in documentation, but the behavior is deliberate. The reason is that when sending over the network to a syslog server, the server expects the message in a certain format. When you change the timestamp, that format is now invalid and the remote end might not be able to parse it.
Now you could put `ts_format(iso)` in the `tcp()` destination driver. But if your remote server is looking for a timestamp in ISO format, it probably supports the syslog message protocol, which uses ISO timestamps. Syslog-ng uses the syslog destination driver for sending in this format.
The syslog message protocol looks like this: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
The forementioned bug in the documentation is that it says the tcp() destination driver ts_format uses the global ts_format setting. It doesnt.
-Patrick
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi-- Bazsi
participants (3)
-
Balazs Scheidler
-
Chris Hiestand
-
Patrick Hemmer