Recently I use the syslog-ng for log collection environment encountered a time zone problem,hope somebody can help me.Thanks The environment is like this: The apache server(PDT timezone,UTC-8) forword apache log through syslog to the syslog-ng server（CST timezone，UTC＋8）. The server with same time zones no problem between the forward. Syslog-ng test configuration is as follows: destination int_apache (file ( "/ disk3/cookie_logs/pid1 / $ YEAR-$ MONTH-$ DAY / $ (HOST). apache.log" time_zone (-08:00) template ( "$ DAY-$ HOUR $ S_DAY-$ S_HOUR $ R_DAY-$ R_HOUR $ $ ISODATE $ MSGONLY \ n ") template_escape (no));); The original apache log is: 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / www.example.com/HTTP/1.1" 200 2028 "-" "Mozilla/4.0 ( compatible; MSIE 7.0; Windows NT 5.1) " The syslog-ng server generated log is: 14-22 14-22 14-22 2009-01-14T22 :27:47-08: 00 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / www.example. com / HTTP/1.1 "200 2028" - "" Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) " Originally generated documents should be kept in /disk3/cookie_logs/pid1/2009-01-14 However, the actual file is stored in the /disk3/cookie_logs/pid1/2009-01-15 It can be seen, in fact, syslog-ng configuration item file() use the $DAY and template() use the $DAY not with a identical Macros. Is this the syslog-ng a bug? or I configuration errors. Who can help me answer you?Thanks!
On Thu, 2009-01-15 at 15:03 +0800, Charles Chan wrote:
Recently I use the syslog-ng for log collection environment encountered a time zone problem,hope somebody can help me.Thanks The environment is like this: The apache server(PDT timezone,UTC-8) forword apache log through syslog to the syslog-ng server（CST timezone，UTC＋8）. The server with same time zones no problem between the forward. Syslog-ng test configuration is as follows: destination int_apache (file ( "/ disk3/cookie_logs/pid1 / $ YEAR-$ MONTH-$ DAY / $ (HOST). apache.log" time_zone (-08:00) template ( "$ DAY-$ HOUR $ S_DAY-$ S_HOUR $ R_DAY-$ R_HOUR $ $ ISODATE $ MSGONLY \ n ") template_escape (no)););
The original apache log is: 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / www.example.com/ HTTP/1.1" 200 2028 "-" "Mozilla/4.0 ( compatible; MSIE 7.0; Windows NT 5.1) "
The syslog-ng server generated log is: 14-22 14-22 14-22 2009-01-14T22 :27:47-08: 00 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / www.example. com / HTTP/1.1 "200 2028" - "" Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) "
Originally generated documents should be kept in /disk3/cookie_logs/pid1/2009-01-14 However, the actual file is stored in the /disk3/cookie_logs/pid1/2009-01-15 It can be seen, in fact, syslog-ng configuration item file() use the $DAY and template() use the $DAY not with a identical Macros. Is this the syslog-ng a bug? or I configuration errors. Who can help me answer you?Thanks!
Well, as I researched this area, there's some version dependance here. In syslog-ng 2.1 (and probably 2.0, but I haven't checked) if you use the time related macros in a filename, then the timestamps are converted to use the local timezone of the syslog-ng process. In the examples above, the time_zone() option only affects the _contents_ of the file, and not the filenames. In 3.0 this "behaviour" is fixed and the timezone of the timestamp is used, without alteration. (and NOTE that in 3.0 the timezone handling was expanded significantly, e.g. you can specify a location _by name_ instead of using plain timezone offsets which do not support day-light-saving) -- Bazsi
We update syslog-ng to 3.0.1,configuration same with 2.1,but the problem still exists,In 3.0 just have two parameter about timezone("time_zone" and "timezone"),not find specify a location by name instead of using plain timezone offsets which do not support day-light-saving as you say.The Doc not include "timezone" parameter deail example， Can you help me fix the problem and give me some config examples? 2009/1/17 Balazs Scheidler <bazsi@balabit.hu>
On Thu, 2009-01-15 at 15:03 +0800, Charles Chan wrote:
Recently I use the syslog-ng for log collection environment encountered a time zone problem,hope somebody can help me.Thanks The environment is like this: The apache server(PDT timezone,UTC-8) forword apache log through syslog to the syslog-ng server（CST timezone，UTC＋8）. The server with same time zones no problem between the forward. Syslog-ng test configuration is as follows: destination int_apache (file ( "/ disk3/cookie_logs/pid1 / $ YEAR-$ MONTH-$ DAY / $ (HOST). apache.log" time_zone (-08:00) template ( "$ DAY-$ HOUR $ S_DAY-$ S_HOUR $ R_DAY-$ R_HOUR $ $ ISODATE $ MSGONLY \ n ") template_escape (no)););
The original apache log is: 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / www.example.com/ HTTP/1.1" 200 2028 "-" "Mozilla/4.0 ( compatible; MSIE 7.0; Windows NT 5.1) "
The syslog-ng server generated log is: 14-22 14-22 14-22 2009-01-14T22 :27:47-08: 00 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / www.example. com / HTTP/1.1 "200 2028" - "" Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) "
Originally generated documents should be kept in /disk3/cookie_logs/pid1/2009-01-14 However, the actual file is stored in the /disk3/cookie_logs/pid1/2009-01-15 It can be seen, in fact, syslog-ng configuration item file() use the $DAY and template() use the $DAY not with a identical Macros. Is this the syslog-ng a bug? or I configuration errors. Who can help me answer you?Thanks!
Well, as I researched this area, there's some version dependance here. In syslog-ng 2.1 (and probably 2.0, but I haven't checked) if you use the time related macros in a filename, then the timestamps are converted to use the local timezone of the syslog-ng process. In the examples above, the time_zone() option only affects the _contents_ of the file, and not the filenames.
In 3.0 this "behaviour" is fixed and the timezone of the timestamp is used, without alteration.
(and NOTE that in 3.0 the timezone handling was expanded significantly, e.g. you can specify a location _by name_ instead of using plain timezone offsets which do not support day-light-saving)
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Tue, 2009-02-10 at 11:47 +0800, Charles Chan wrote:
We update syslog-ng to 3.0.1,configuration same with 2.1,but the problem still exists,In 3.0 just have two parameter about timezone("time_zone" and "timezone"),not find specify a location by name instead of using plain timezone offsets which do not support day-light-saving as you say.The Doc not include "timezone" parameter deail example， Can you help me fix the problem and give me some config examples?
When using timezone() in 3.0, you can reference the name of the timezone, e.g. timezone("Europe/Budapest"); The name of the timezone can either be a direct timezone offset reference as earlier (e.g. +01:00), or the name of a timezone info file under /usr/share/zoneinfo. I'm Ccing the documentation team, to clarify the documentation about this.
2009/1/17 Balazs Scheidler <bazsi@balabit.hu>
On Thu, 2009-01-15 at 15:03 +0800, Charles Chan wrote: > Recently I use the syslog-ng for log collection environment > encountered a time zone problem,hope somebody can help me.Thanks > The environment is like this: > The apache server(PDT timezone,UTC-8) forword apache log through > syslog to the syslog-ng server（CST timezone，UTC＋8）. > The server with same time zones no problem between the forward. > Syslog-ng test configuration is as follows: > destination int_apache (file ( "/ disk3/cookie_logs/pid1 / $ YEAR-$ > MONTH-$ DAY / $ (HOST). apache.log" time_zone (-08:00) template ( "$ > DAY-$ HOUR $ S_DAY-$ S_HOUR $ R_DAY-$ R_HOUR $ $ ISODATE $ MSGONLY \ n > ") template_escape (no));); > > The original apache log is: > 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / > www.example.com/ HTTP/1.1" 200 2028 "-" "Mozilla/4.0 ( compatible; > MSIE 7.0; Windows NT 5.1) " > > The syslog-ng server generated log is: > 14-22 14-22 14-22 2009-01-14T22 :27:47-08: 00 58.240.69.161 - - - > [14/Jan/2009: 22:27:47 -0800] "GET / www.example. com / HTTP/1.1 "200 > 2028" - "" Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) " > > Originally generated documents should be kept in > /disk3/cookie_logs/pid1/2009-01-14 > However, the actual file is stored in the > /disk3/cookie_logs/pid1/2009-01-15 > It can be seen, in fact, syslog-ng configuration item file() use the > $DAY and template() use the $DAY not with a identical Macros. Is this > the syslog-ng a bug? or I configuration errors. Who can help me answer > you?Thanks!
Well, as I researched this area, there's some version dependance here. In syslog-ng 2.1 (and probably 2.0, but I haven't checked) if you use the time related macros in a filename, then the timestamps are converted to use the local timezone of the syslog-ng process. In the examples above, the time_zone() option only affects the _contents_ of the file, and not the filenames.
In 3.0 this "behaviour" is fixed and the timezone of the timestamp is used, without alteration.
(and NOTE that in 3.0 the timezone handling was expanded significantly, e.g. you can specify a location _by name_ instead of using plain timezone offsets which do not support day-light-saving)
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
I use follow configuration still not work,display "syntax error". destination us_int_apache { file("/disk2/apache_logs/$YEAR-$MONTH-$DAY/$HOST.apache.log" timezone("PST8PDT") template(apache_template)); }; syslog-ng version as follows: [root@wikid ~]# /opt/syslog-ng/sbin/syslog-ng -V syslog-ng 3.0.1 Revision: Compile-Date: Feb 9 2009 09:58:34 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: off 2009/2/10 Balazs Scheidler <bazsi@balabit.hu>
On Tue, 2009-02-10 at 11:47 +0800, Charles Chan wrote:
We update syslog-ng to 3.0.1,configuration same with 2.1,but the problem still exists,In 3.0 just have two parameter about timezone("time_zone" and "timezone"),not find specify a location by name instead of using plain timezone offsets which do not support day-light-saving as you say.The Doc not include "timezone" parameter deail example， Can you help me fix the problem and give me some config examples?
When using timezone() in 3.0, you can reference the name of the timezone, e.g.
timezone("Europe/Budapest");
The name of the timezone can either be a direct timezone offset reference as earlier (e.g. +01:00), or the name of a timezone info file under /usr/share/zoneinfo.
I'm Ccing the documentation team, to clarify the documentation about this.
2009/1/17 Balazs Scheidler <bazsi@balabit.hu>
On Thu, 2009-01-15 at 15:03 +0800, Charles Chan wrote: > Recently I use the syslog-ng for log collection environment > encountered a time zone problem,hope somebody can help me.Thanks > The environment is like this: > The apache server(PDT timezone,UTC-8) forword apache log through > syslog to the syslog-ng server（CST timezone，UTC＋8）. > The server with same time zones no problem between the forward. > Syslog-ng test configuration is as follows: > destination int_apache (file ( "/ disk3/cookie_logs/pid1 / $ YEAR-$ > MONTH-$ DAY / $ (HOST). apache.log" time_zone (-08:00) template ( "$ > DAY-$ HOUR $ S_DAY-$ S_HOUR $ R_DAY-$ R_HOUR $ $ ISODATE $ MSGONLY \ n > ") template_escape (no));); > > The original apache log is: > 58.240.69.161 - - - [14/Jan/2009: 22:27:47 -0800] "GET / > www.example.com/ HTTP/1.1" 200 2028 "-" "Mozilla/4.0 ( compatible; > MSIE 7.0; Windows NT 5.1) " > > The syslog-ng server generated log is: > 14-22 14-22 14-22 2009-01-14T22 :27:47-08: 00 58.240.69.161 - - - > [14/Jan/2009: 22:27:47 -0800] "GET / www.example. com / HTTP/1.1 "200 > 2028" - "" Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) " > > Originally generated documents should be kept in > /disk3/cookie_logs/pid1/2009-01-14 > However, the actual file is stored in the > /disk3/cookie_logs/pid1/2009-01-15 > It can be seen, in fact, syslog-ng configuration item file() use the > $DAY and template() use the $DAY not with a identical Macros. Is this > the syslog-ng a bug? or I configuration errors. Who can help me answer > you?Thanks!
Well, as I researched this area, there's some version dependance here. In syslog-ng 2.1 (and probably 2.0, but I haven't checked) if you use the time related macros in a filename, then the timestamps are converted to use the local timezone of the syslog-ng process. In the examples above, the time_zone() option only affects the _contents_ of the file, and not the filenames.
In 3.0 this "behaviour" is fixed and the timezone of the timestamp is used, without alteration.
(and NOTE that in 3.0 the timezone handling was expanded significantly, e.g. you can specify a location _by name_ instead of using plain timezone offsets which do not support day-light-saving)
-- Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Wed, 2009-02-11 at 09:43 +0800, Charles Chan wrote:
I use follow configuration still not work,display "syntax error". destination us_int_apache { file("/disk2/apache_logs/$YEAR-$MONTH-$DAY/$HOST.apache.log" timezone("PST8PDT") template(apache_template)); };
sorry, the keyword is called 'time_zone'. notice the underscore. -- Bazsi
I use time_zone instead timezone,configuration like this =========================================================== template apache_template { template("$DAY-$HOUR $S_DAY-$S_HOUR $R_DAY-$R_HOUR $MSGONLY\n"); template_escape(no); }; destination us_int_apache { file("/disk2/apache_logs/$YEAR-$MONTH-$DAY/$HOST.apache.log" timezone("PST8PDT") template(apache_template)); }; ============================================================ service can start,but the problem still exists. the file is stored in the /disk2/apache_logs/2009-02-12 generated log is: 11-17 11-17 11-17 222.69.201.23 - - - [11/Feb/2009:17:37:03 -0800] "GET /atmgateway.client.html ................... two date string still difference. I use syslog-ng 3.0.1 OSE version,is this OSE version problem? 2009/2/11 Balazs Scheidler <bazsi@balabit.hu>
On Wed, 2009-02-11 at 09:43 +0800, Charles Chan wrote:
I use follow configuration still not work,display "syntax error". destination us_int_apache { file("/disk2/apache_logs/$YEAR-$MONTH-$DAY/$HOST.apache.log" timezone("PST8PDT") template(apache_template)); };
sorry, the keyword is called 'time_zone'. notice the underscore.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, 2009-02-12 at 09:52 +0800, Charles Chan wrote:
I use time_zone instead timezone,configuration like this =========================================================== template apache_template { template("$DAY-$HOUR $S_DAY-$S_HOUR $R_DAY-$R_HOUR $MSGONLY\n"); template_escape(no); }; destination us_int_apache { file("/disk2/apache_logs/$YEAR-$MONTH-$DAY/$HOST.apache.log" timezone("PST8PDT") template(apache_template)); }; ============================================================ service can start,but the problem still exists. the file is stored in the /disk2/apache_logs/2009-02-12 generated log is: 11-17 11-17 11-17 222.69.201.23 - - - [11/Feb/2009:17:37:03 -0800] "GET /atmgateway.client.html ................... two date string still difference. I use syslog-ng 3.0.1 OSE version,is this OSE version problem?
The PE and OSE versions are the same in this regard. Apart from the concrete differences listed on our website, I work hard to keep the two versions in sync. So let me try to understand your problem a bit deeper... Sorry for not having done so previously. Sometimes I just reply without really thinking through problems in the question. hmm.. as I see your problem seems to be that the name of the file and the _content_ of the file contains a different timestamp. Right? This is intentional. When expanding the macros in the name of the files, syslog-ng _always_ uses the server-side, local timezone. The timezone used in the content of the file can be controlled using the time_zone() option. Currently there's no way to control the filename zone offset. The reasoning behind that was that I already found the timezone related configuration options too complex, and I didn't see the need so far. I felt that agregating logs based on the _received timestamp_ is more trustworthy (because clients are free to specify a timestamp they feel-like), and that is in local timezone anyway. Also, noone else has complained previously :) That said, I'm willing to add another option to control the timezone used when expanding the filename, if you really think that solves your problem. -- Bazsi
yes,I really think that solves the problem，and this feature is very useful, especially in large sites across time zones to centralize collect log.I really hope you will be able to in the old and new versions are added to this option,deeply thanks. 2009/2/15 Balazs Scheidler <bazsi@balabit.hu>
On Thu, 2009-02-12 at 09:52 +0800, Charles Chan wrote:
I use time_zone instead timezone,configuration like this =========================================================== template apache_template { template("$DAY-$HOUR $S_DAY-$S_HOUR $R_DAY-$R_HOUR $MSGONLY\n"); template_escape(no); }; destination us_int_apache { file("/disk2/apache_logs/$YEAR-$MONTH-$DAY/$HOST.apache.log" timezone("PST8PDT") template(apache_template)); }; ============================================================ service can start,but the problem still exists. the file is stored in the /disk2/apache_logs/2009-02-12 generated log is: 11-17 11-17 11-17 222.69.201.23 - - - [11/Feb/2009:17:37:03 -0800] "GET /atmgateway.client.html ................... two date string still difference. I use syslog-ng 3.0.1 OSE version,is this OSE version problem?
The PE and OSE versions are the same in this regard. Apart from the concrete differences listed on our website, I work hard to keep the two versions in sync.
So let me try to understand your problem a bit deeper... Sorry for not having done so previously. Sometimes I just reply without really thinking through problems in the question.
hmm.. as I see your problem seems to be that the name of the file and the _content_ of the file contains a different timestamp. Right?
This is intentional.
When expanding the macros in the name of the files, syslog-ng _always_ uses the server-side, local timezone. The timezone used in the content of the file can be controlled using the time_zone() option.
Currently there's no way to control the filename zone offset. The reasoning behind that was that I already found the timezone related configuration options too complex, and I didn't see the need so far. I felt that agregating logs based on the _received timestamp_ is more trustworthy (because clients are free to specify a timestamp they feel-like), and that is in local timezone anyway. Also, noone else has complained previously :)
That said, I'm willing to add another option to control the timezone used when expanding the filename, if you really think that solves your problem.
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
participants (2)
-
Balazs Scheidler
-
Charles Chan