syslog-ng logging with wrong year
Hello, We have a log collector receiving logs from several nodes and it is logging in the wrong $YEAR directory every 1st of the month for some host for about 2 minutes... We have 75 nodes sending logs, and we have a directory created under /var/log/hosts/2007/*/01/hostname for about 3-4 host. As you can see below, the folder 01 (January 1st) was created at 23:59 on December 31st... Same thing for Feb. 1st to now... /var/log/hosts/2007/01# ls -l total 4 drwxr-xr-x 2 root root 4096 2007-12-31 23:59 01 Here are my destination and log rules destination d_local { file("/var/log/hosts/$YEAR/$MONTH/$DAY/$HOST"); }; log { source(s_tcp); source(s_udp); source(s_local); destination(d_local); }; Any one has any idea why this is happening? NOTICE: This email contains privileged and confidential information and is intended only for the individual to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this transmission by mistake and delete this communication from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. AVIS: Le pr�sent courriel contient des renseignements de nature privil�gi�e et confidentielle et n�est destin� qu'� la personne � qui il est adress�. Si vous n��tes pas le destinataire pr�vu, vous �tes par les pr�sentes avis�s que toute diffusion, distribution ou reproduction de cette communication est strictement interdite.� Si vous avez re�u ce courriel par erreur, veuillez en aviser imm�diatement l�exp�diteur et le supprimer de votre syst�me. Notez que la transmission de courriel ne peut en aucun cas �tre consid�r� comme inviolable ou exempt d�erreur puisque les informations qu�il contient pourraient �tre intercept�s, corrompues, perdues, d�truites, arriv�es en retard ou incompl�tes ou contenir un virus. �
Hi,
Hello,
We have a log collector receiving logs from several nodes and it is logging in the wrong $YEAR directory every 1st of the month for some host for about 2 minutes...
My first guess is that the system clock of the affected hosts might be skewed.
We have 75 nodes sending logs, and we have a directory created under /var/log/hosts/2007/*/01/hostname for about 3-4 host.
Could you check the system clock of these?
destination d_local { file("/var/log/hosts/$YEAR/$MONTH/$DAY/$HOST"); };
These macros use the timestamp of the log message. Macros prefixed with R_ use the timestamp of the receiver (the syslog-ng server). So either the system clock of the hosts should be adjusted or the $R_YEAR, $R_MONTH, ... macros should be used. I'd check the system clock first. Regards, Sandor -------------------------------------------------------- NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.
On Wed, 2008-04-09 at 07:05 +0100, Geller, Sandor (IT) wrote:
Hi,
Hello,
We have a log collector receiving logs from several nodes and it is logging in the wrong $YEAR directory every 1st of the month for some host for about 2 minutes...
My first guess is that the system clock of the affected hosts might be skewed.
We have 75 nodes sending logs, and we have a directory created under /var/log/hosts/2007/*/01/hostname for about 3-4 host.
Could you check the system clock of these?
destination d_local { file("/var/log/hosts/$YEAR/$MONTH/$DAY/$HOST"); };
These macros use the timestamp of the log message. Macros prefixed with R_ use the timestamp of the receiver (the syslog-ng server). So either the system clock of the hosts should be adjusted or the $R_YEAR, $R_MONTH, ... macros should be used. I'd check the system clock first.
Regards,
And in addition, the timestamp of the message does not contain a year, there's a heuristic in syslog-ng to determine that. Here's the heuristic used: tm.tm_year = nowtm.tm_year; if (tm.tm_mon > nowtm.tm_mon) tm.tm_year--; E.g. if the current month is smaller than the month in the timestamp, syslog-ng assumes that it comes from the previous year. Hmm... Maybe this heuristic would be better: tm.tm_year = nowtm.tm_year; if (tm.tm_mon == 11 && nowtm.tm_mon == 0) tm.tm_year--; E.g. the year is decreased only if the receiver's time is in January, and the sender came in as December. This would not handle really skewed timestamps, but your case would be covered. I'm reluctant to change this in 2.0 (the current algorithm has been in place for about a decade now), however I can commit a patch to 2.1. What do others think? And a side-note: the best solution is to use a timestamp that actually includes the year information, like ISODATE. -- Bazsi
And in addition, the timestamp of the message does not contain a year, there's a heuristic in syslog-ng to determine that.
Here's the heuristic used:
tm.tm_year = nowtm.tm_year; if (tm.tm_mon > nowtm.tm_mon) tm.tm_year--;
E.g. if the current month is smaller than the month in the timestamp, syslog-ng assumes that it comes from the previous year. Hmm... Maybe this heuristic would be better:
tm.tm_year = nowtm.tm_year; if (tm.tm_mon == 11 && nowtm.tm_mon == 0) tm.tm_year--;
E.g. the year is decreased only if the receiver's time is in January, and the sender came in as December. This would not handle really
skewed
timestamps, but your case would be covered.
I'm reluctant to change this in 2.0 (the current algorithm has been in place for about a decade now), however I can commit a patch to 2.1. What do others think?
And a side-note: the best solution is to use a timestamp that actually includes the year information, like ISODATE.
How can I set ISODATE? NOTICE: This email contains privileged and confidential information and is intended only for the individual to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this transmission by mistake and delete this communication from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. AVIS: Le pr�sent courriel contient des renseignements de nature privil�gi�e et confidentielle et n�est destin� qu'� la personne � qui il est adress�. Si vous n��tes pas le destinataire pr�vu, vous �tes par les pr�sentes avis�s que toute diffusion, distribution ou reproduction de cette communication est strictement interdite.� Si vous avez re�u ce courriel par erreur, veuillez en aviser imm�diatement l�exp�diteur et le supprimer de votre syst�me. Notez que la transmission de courriel ne peut en aucun cas �tre consid�r� comme inviolable ou exempt d�erreur puisque les informations qu�il contient pourraient �tre intercept�s, corrompues, perdues, d�truites, arriv�es en retard ou incompl�tes ou contenir un virus. �
On Wed, 2008-04-09 at 09:20 -0400, Jean-Sebastien Pilon wrote:
And in addition, the timestamp of the message does not contain a year, there's a heuristic in syslog-ng to determine that.
Here's the heuristic used:
tm.tm_year = nowtm.tm_year; if (tm.tm_mon > nowtm.tm_mon) tm.tm_year--;
E.g. if the current month is smaller than the month in the timestamp, syslog-ng assumes that it comes from the previous year. Hmm... Maybe this heuristic would be better:
tm.tm_year = nowtm.tm_year; if (tm.tm_mon == 11 && nowtm.tm_mon == 0) tm.tm_year--;
E.g. the year is decreased only if the receiver's time is in January, and the sender came in as December. This would not handle really
skewed
timestamps, but your case would be covered.
I'm reluctant to change this in 2.0 (the current algorithm has been in place for about a decade now), however I can commit a patch to 2.1. What do others think?
And a side-note: the best solution is to use a timestamp that actually includes the year information, like ISODATE.
How can I set ISODATE?
You should use a template to override the format that syslog-ng uses by default, like: destination d1 { tcp("host" template("<$PRI>$ISODATE $HOST $MSG\n"); }; On the client. On the server no changes are necessary it will autodetect the timestamp as received from the network. -- Bazsi
Ok thank you I will try this out, I prefer this over the R_ timestamps. Thank you all for your input.
You should use a template to override the format that syslog-ng uses by default, like:
destination d1 { tcp("host" template("<$PRI>$ISODATE $HOST $MSG\n"); };
On the client. On the server no changes are necessary it will autodetect the timestamp as received from the network.
-- Bazsi NOTICE: This email contains privileged and confidential information and is intended only for the individual to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this transmission by mistake and delete this communication from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
AVIS: Le présent courriel contient des renseignements de nature privilégiée et confidentielle et nest destiné qu'à la personne à qui il est adressé. Si vous nêtes pas le destinataire prévu, vous êtes par les présentes avisés que toute diffusion, distribution ou reproduction de cette communication est strictement interdite. Si vous avez reçu ce courriel par erreur, veuillez en aviser immédiatement lexpéditeur et le supprimer de votre système. Notez que la transmission de courriel ne peut en aucun cas être considéré comme inviolable ou exempt derreur puisque les informations quil contient pourraient être interceptés, corrompues, perdues, détruites, arrivées en retard ou incomplètes ou contenir un virus.
Balazs Scheidler wrote:
On Wed, 2008-04-09 at 09:20 -0400, Jean-Sebastien Pilon wrote:
And in addition, the timestamp of the message does not contain a year, there's a heuristic in syslog-ng to determine that.
Here's the heuristic used:
tm.tm_year = nowtm.tm_year; if (tm.tm_mon > nowtm.tm_mon) tm.tm_year--;
E.g. if the current month is smaller than the month in the timestamp, syslog-ng assumes that it comes from the previous year. Hmm... Maybe this heuristic would be better:
tm.tm_year = nowtm.tm_year; if (tm.tm_mon == 11 && nowtm.tm_mon == 0) tm.tm_year--;
E.g. the year is decreased only if the receiver's time is in January, and the sender came in as December. This would not handle really skewed timestamps, but your case would be covered.
I'm reluctant to change this in 2.0 (the current algorithm has been in place for about a decade now), however I can commit a patch to 2.1. What do others think?
And a side-note: the best solution is to use a timestamp that actually includes the year information, like ISODATE.
How can I set ISODATE?
You should use a template to override the format that syslog-ng uses by default, like:
destination d1 { tcp("host" template("<$PRI>$ISODATE $HOST $MSG\n"); };
On the client. On the server no changes are necessary it will autodetect the timestamp as received from the network.
Except that when I try to use the $ISODATE in the "on the wire" tcp connection in syslog-ng 2.0.8 (both ends) syslog-ng didn't parse the message correctly at all. It behaved just like there was no time or host in the message at all. template accounting { template("<030>$R_ISODATE $HOST dsmacct: UVIC_ADM1: $MSG\n"); template_escape(no); }; options { sync(0); log_fifo_size(100000); use_fqdn(yes); keep_hostname(no); chain_hostnames(no); time_reap(60); time_reopen(5); }; I just switched to $R_DATE and all works. I assumed that the ISODATE was just not handled by syslog-ng on an incoming connection as it is well outside the RFC. Evan. -- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
On Wed, 2008-04-09 at 12:32 -0700, Evan Rempel wrote:
Balazs Scheidler wrote:
On Wed, 2008-04-09 at 09:20 -0400, Jean-Sebastien Pilon wrote:
And in addition, the timestamp of the message does not contain a year, there's a heuristic in syslog-ng to determine that.
Here's the heuristic used:
tm.tm_year = nowtm.tm_year; if (tm.tm_mon > nowtm.tm_mon) tm.tm_year--;
E.g. if the current month is smaller than the month in the timestamp, syslog-ng assumes that it comes from the previous year. Hmm... Maybe this heuristic would be better:
tm.tm_year = nowtm.tm_year; if (tm.tm_mon == 11 && nowtm.tm_mon == 0) tm.tm_year--;
E.g. the year is decreased only if the receiver's time is in January, and the sender came in as December. This would not handle really skewed timestamps, but your case would be covered.
I'm reluctant to change this in 2.0 (the current algorithm has been in place for about a decade now), however I can commit a patch to 2.1. What do others think?
And a side-note: the best solution is to use a timestamp that actually includes the year information, like ISODATE.
How can I set ISODATE?
You should use a template to override the format that syslog-ng uses by default, like:
destination d1 { tcp("host" template("<$PRI>$ISODATE $HOST $MSG\n"); };
On the client. On the server no changes are necessary it will autodetect the timestamp as received from the network.
Except that when I try to use the $ISODATE in the "on the wire" tcp connection in syslog-ng 2.0.8 (both ends) syslog-ng didn't parse the message correctly at all. It behaved just like there was no time or host in the message at all.
template accounting { template("<030>$R_ISODATE $HOST dsmacct: UVIC_ADM1: $MSG\n"); template_escape(no); };
options { sync(0); log_fifo_size(100000); use_fqdn(yes); keep_hostname(no); chain_hostnames(no); time_reap(60); time_reopen(5); };
I just switched to $R_DATE and all works. I assumed that the ISODATE was just not handled by syslog-ng on an incoming connection as it is well outside the RFC.
Hm... there's code for it in the message parsing, there are even unit tests to test it. Let me see... I've just checked it, it works just fine. -- Bazsi
My first guess is that the system clock of the affected hosts might be skewed.
We use ntp, and the time appears ok on the syslog clients.
Could you check the system clock of these?
Note that the 3-4 hosts changes each month... looks like they may be the only ones that sent logs around 00:00.
These macros use the timestamp of the log message. Macros prefixed with R_ use the timestamp of the receiver (the syslog-ng server). So either the system clock of the hosts should be adjusted or the $R_YEAR, $R_MONTH, ... macros should be used. I'd check the system clock first.
That could be a solution, but I prefer keeping timestamp from host. This will be even more important if we move to the premium edition of Syslog-ng with disk buffering. NOTICE: This email contains privileged and confidential information and is intended only for the individual to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this transmission by mistake and delete this communication from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. AVIS: Le présent courriel contient des renseignements de nature privilégiée et confidentielle et nest destiné qu'à la personne à qui il est adressé. Si vous nêtes pas le destinataire prévu, vous êtes par les présentes avisés que toute diffusion, distribution ou reproduction de cette communication est strictement interdite. Si vous avez reçu ce courriel par erreur, veuillez en aviser immédiatement lexpéditeur et le supprimer de votre système. Notez que la transmission de courriel ne peut en aucun cas être considéré comme inviolable ou exempt derreur puisque les informations quil contient pourraient être interceptés, corrompues, perdues, détruites, arrivées en retard ou incomplètes ou contenir un virus.
participants (4)
-
Balazs Scheidler
-
Evan Rempel
-
Geller, Sandor (IT)
-
Jean-Sebastien Pilon