file written in the wrong place
Hi I've installed syslog-ng 1.6.8 on a Linux FC4 system to act as a centeralized logging server. For a while everything was fine, until I tried to add another statement to one of the filters. After the addition syslog-ng started writing the destination files in the wrong places, writing the same file in 2 locations and ignoring messages sent from certain origins. Here are the relevant parts of my syslog-ng.conf: # Global options options { sync (0); time_reopen (1); log_fifo_size (999999); long_hostnames (yes); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); }; # source source s_net { udp(ip("192.168.1.100") port(514)); }; # destinations destination d_switch { file("/var/log/company/switches/$HOST.log" perm(0644)); }; destination d_edge { file("/var/log/company/edge_devices/$HOST.log" perm(0644)); }; destination d_fw { file("/var/log/company/firewalls/$HOST.log" perm(0644)); }; destination d_router { file("/var/log/company/routers/$HOST.log" perm(0644)); }; # filters filter f_edge { host("edge*") or host("10.10.*"); }; filter f_router { host("3600-primary") or host("backbone-3550") or host("secondary-3550"); }; filter f_switch { host("sw*") or host("backbone-5510") or host(pp8600*); }; filter f_fw { host("fw*"); }; # wrap everything up log { source(s_net); filter(f_switch); destination(d_switch); }; log { source(s_net); filter(f_router); destination(d_router); }; log { source(s_net); filter(f_edge); destination(d_edge); }; log { source(s_net); filter(f_fw); destination(d_fw); }; # end of configuration file when I run the command `find /var/log/company -type f -print` I get the following output: ./firewalls/fw-backup.log ./firewalls/edge-location1.log ./firewalls/fwmgmt.log ./edge_devices/edge-location2.log ./edge_devices/edge-location3.log ./edge_devices/edge-location4.log ./edge_devices/edge-location5.log ./edge_devices/edge-location6.log ./edge_devices/edge-location7.log ./edge_devices/edge-location8.log ./edge_devices/edge-location9.log ./edge_devices/edge-location1.log ./edge_devices/edge-location10.log ./edge_devices/edge-location11.log ./switches/edge-location3.log ./switches/fw-backup.log ./switches/edge-location6.log ./switches/edge-location7.log ./switches/edge-location11.log ./switches/edge-location12.log I tried to upgrade syslog-ng to 1.6.9 but it didn't resolve the problems. Any idea on why this happnes and how to fix this will be greatly appriciated. TIA Paolo PS - the systems sending the syslog messages are nortel switches, Checkpoint firewalls and edge devices. __________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
Paolo Supino wrote:
Hi
I've installed syslog-ng 1.6.8 on a Linux FC4 system to act as a centeralized logging server. For a while everything was fine, until I tried to add another statement to one of the filters. After the addition syslog-ng started writing the destination files in the wrong places, writing the same file in 2 locations and ignoring messages sent from certain origins.
Which filter is causing your problems?
# destinations destination d_switch { file("/var/log/company/switches/$HOST.log" perm(0644)); }; destination d_edge { file("/var/log/company/edge_devices/$HOST.log" perm(0644)); }; destination d_fw { file("/var/log/company/firewalls/$HOST.log" perm(0644)); }; destination d_router { file("/var/log/company/routers/$HOST.log" perm(0644)); };
Use the $FULLHOST macro, not just $HOST
# filters filter f_edge { host("edge*") or host("10.10.*"); }; filter f_router { host("3600-primary") or host("backbone-3550") or host("secondary-3550"); }; filter f_switch { host("sw*") or host("backbone-5510") or host(pp8600*); }; filter f_fw { host("fw*"); };
You should check your hostnames, whether they contain your filter patterns. I suggest using "^edge.*"-style regexp, this is more precise than "edge*". I suspect your fully qualified hostnames contain somewhere the "fw" or the "sw" strings.
# wrap everything up log { source(s_net); filter(f_switch); destination(d_switch); }; log { source(s_net); filter(f_router); destination(d_router); }; log { source(s_net); filter(f_edge); destination(d_edge); }; log { source(s_net); filter(f_fw); destination(d_fw); };
Consider using the "final" flag in your log sections, this can improve the performance of syslog-ng, but be aware that if sou use final, then the order of your log sections does matter! -- Sandor Geller wildy@balabit.hu
--- Sandor Geller <wildy@balabit.hu> wrote:
Paolo Supino wrote:
Hi
I've installed syslog-ng 1.6.8 on a Linux FC4 system to act as a centeralized logging server. For a while everything was fine, until I tried to add another statement to one of the filters. After the addition syslog-ng started writing the destination files in the wrong places, writing the same file in 2 locations and ignoring messages sent from certain origins.
Which filter is causing your problems?
# destinations destination d_switch { file("/var/log/company/switches/$HOST.log" perm(0644)); }; destination d_edge { file("/var/log/company/edge_devices/$HOST.log" perm(0644)); }; destination d_fw { file("/var/log/company/firewalls/$HOST.log" perm(0644)); }; destination d_router { file("/var/log/company/routers/$HOST.log" perm(0644)); };
Use the $FULLHOST macro, not just $HOST
# filters filter f_edge { host("edge*") or host("10.10.*"); }; filter f_router { host("3600-primary") or host("backbone-3550") or host("secondary-3550"); }; filter f_switch { host("sw*") or host("backbone-5510") or host(pp8600*); }; filter f_fw { host("fw*"); };
You should check your hostnames, whether they contain your filter patterns. I suggest using "^edge.*"-style regexp, this is more precise than "edge*". I suspect your fully qualified hostnames contain somewhere the "fw" or the "sw" strings.
The names of the hostnames only appear in the DNS server. The system themselves don't know the name.
# wrap everything up log { source(s_net); filter(f_switch); destination(d_switch); }; log { source(s_net); filter(f_router); destination(d_router); }; log { source(s_net); filter(f_edge); destination(d_edge); }; log { source(s_net); filter(f_fw); destination(d_fw); };
Consider using the "final" flag in your log sections, this can improve the performance of syslog-ng, but be aware that if sou use final, then the order of your log sections does matter!
I will try this and see what happens.
-- Sandor Geller wildy@balabit.hu _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
Hi Thanx Sandor. Your suggestions solved the problem. I think I misunderstand something: When a message is received and the filter statement is host(something). Does the filter parse the message to look for the string or matches the message based on the server that sent the message? TIA Paolo --- Sandor Geller <wildy@balabit.hu> wrote:
Paolo Supino wrote:
Hi
I've installed syslog-ng 1.6.8 on a Linux FC4 system to act as a centeralized logging server. For a while everything was fine, until I tried to add another statement to one of the filters. After the addition syslog-ng started writing the destination files in the wrong places, writing the same file in 2 locations and ignoring messages sent from certain origins.
Which filter is causing your problems?
# destinations destination d_switch { file("/var/log/company/switches/$HOST.log" perm(0644)); }; destination d_edge { file("/var/log/company/edge_devices/$HOST.log" perm(0644)); }; destination d_fw { file("/var/log/company/firewalls/$HOST.log" perm(0644)); }; destination d_router { file("/var/log/company/routers/$HOST.log" perm(0644)); };
Use the $FULLHOST macro, not just $HOST
# filters filter f_edge { host("edge*") or host("10.10.*"); }; filter f_router { host("3600-primary") or host("backbone-3550") or host("secondary-3550"); }; filter f_switch { host("sw*") or host("backbone-5510") or host(pp8600*); }; filter f_fw { host("fw*"); };
You should check your hostnames, whether they contain your filter patterns. I suggest using "^edge.*"-style regexp, this is more precise than "edge*". I suspect your fully qualified hostnames contain somewhere the "fw" or the "sw" strings.
# wrap everything up log { source(s_net); filter(f_switch); destination(d_switch); }; log { source(s_net); filter(f_router); destination(d_router); }; log { source(s_net); filter(f_edge); destination(d_edge); }; log { source(s_net); filter(f_fw); destination(d_fw); };
Consider using the "final" flag in your log sections, this can improve the performance of syslog-ng, but be aware that if sou use final, then the order of your log sections does matter!
-- Sandor Geller wildy@balabit.hu _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
Hi How do I control (and eliminate) the writing delay between the time that syslog-ng receives the message and the time it writes the message to the file? TIA Paolo __________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
On Sun, Dec 25, 2005 at 11:03:36PM -0800, Paolo Supino wrote:
How do I control (and eliminate) the writing delay between the time that syslog-ng receives the message and the time it writes the message to the file?
http://www.campin.net/syslog-ng/faq.html#sync -- Nate In German "invent-a-new-word-where-a-perfectly-good-one-already-exists" is probably a word.
Hi Thanx Nate. But for some reason sync doesn't do it in my configuration. I have the following global section in my configuration: options { sync (0); time_reopen (1); log_fifo_size (999999); long_hostnames (yes); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); }; and still messages are written in delay to the files: TIA Paolo --- Nate Campi <nate@campin.net> wrote:
On Sun, Dec 25, 2005 at 11:03:36PM -0800, Paolo Supino wrote:
How do I control (and eliminate) the writing delay between the time that syslog-ng receives the message and the time it writes the message to the file?
http://www.campin.net/syslog-ng/faq.html#sync
-- Nate
In German "invent-a-new-word-where-a-perfectly-good-one-already-exists" is probably a word.
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________________ Yahoo! DSL Something to write home about. Just $16.99/mo. or less. dsl.yahoo.com
On Sun, Dec 25, 2005 at 11:15:18PM -0800, Paolo Supino wrote:
Thanx Nate. But for some reason sync doesn't do it in my configuration. I have the following global section in my configuration: options { sync (0); time_reopen (1); log_fifo_size (999999); long_hostnames (yes); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); }; and still messages are written in delay to the files:
What kind of delay are we talking about here? I'd think that the only remaining cause of delay isn't syslog-ng but the OS and/or your C libraries, but maybe something weird is going on. -- Nate "If you torture the data enough, it will confess." ~ Ronald Coase.
Hi Nate Here is the tcpdump output: 09:18:34.047104 IP switch-01.1024 > syslog-server.syslog: UDP, length 73 the current clock on the syslog server is: 09:27:09 (as printed by date). Here is the output destination of file `ls -l` output: -rw-r--r-- 1 root root 6234 Dec 26 08:03 switch-01.log This server is also the syslog server for some very chatty checkpoint firewalls that send all their logs to it and I feel the syslog-ng fails to be up to speed because of them (didn't try to turn them off yet). TIA Paolo PS - The time now is alerady 09:31:29 and the file wasn't written yet (though new messages were sent to it). --- Nate Campi <nate@campin.net> wrote:
On Sun, Dec 25, 2005 at 11:15:18PM -0800, Paolo Supino wrote:
Thanx Nate. But for some reason sync doesn't do it in my configuration. I have the following global section in my
configuration:
options { sync (0); time_reopen (1); log_fifo_size (999999); long_hostnames (yes); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); }; and still messages are written in delay to the files:
What kind of delay are we talking about here? I'd think that the only remaining cause of delay isn't syslog-ng but the OS and/or your C libraries, but maybe something weird is going on. -- Nate
"If you torture the data enough, it will confess." ~ Ronald Coase.
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
Hi Here is an update on the interval in writing to the log file: the output of ls -l on the file: -rw-r--r-- 1 root root 6234 Dec 26 08:03 switch-01.log and the last messsege coming in on the NIC (captured by tcpdump): 12:56:57.564348 IP switch-01.1024 > syslog.hot.net.il.syslog: UDP, length 75 As you can see the time interval is almost 5 hours... :-( TIA Paolo --- Nate Campi <nate@campin.net> wrote:
On Sun, Dec 25, 2005 at 11:15:18PM -0800, Paolo Supino wrote:
Thanx Nate. But for some reason sync doesn't do it in my configuration. I have the following global section in my
configuration:
options { sync (0); time_reopen (1); log_fifo_size (999999); long_hostnames (yes); use_dns (yes); dns_cache(yes); dns_cache_size(1000); dns_cache_expire(604800); use_fqdn (yes); create_dirs (yes); keep_hostname (yes); }; and still messages are written in delay to the files:
What kind of delay are we talking about here? I'd think that the only remaining cause of delay isn't syslog-ng but the OS and/or your C libraries, but maybe something weird is going on. -- Nate
"If you torture the data enough, it will confess." ~ Ronald Coase.
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________________ Yahoo! DSL Something to write home about. Just $16.99/mo. or less. dsl.yahoo.com
On Mon, Dec 26, 2005 at 03:14:21AM -0800, Paolo Supino wrote:
Here is an update on the interval in writing to the log file: the output of ls -l on the file: -rw-r--r-- 1 root root 6234 Dec 26 08:03 switch-01.log and the last messsege coming in on the NIC (captured by tcpdump): 12:56:57.564348 IP switch-01.1024 > syslog.hot.net.il.syslog: UDP, length 75 As you can see the time interval is almost 5 hours... :-(
Wow, I can see how you'd be frustrated. Do you have any filters that might be preventing particular messages from being written to the file? Put in a catchall destination to see if it differs from the logging to your filtered files. http://www.campin.net/syslog-ng/faq.html#logall Hope this helps. -- Nate "Java is, in many ways, C++--." - Michael Feldman.
Nothing but what I already sent the mailing list. I think that the problem might be 1 if 2: 1. the directive: log_fifo_size (999999); 2. a problem in the way that syslog-ng writes to files. I will try to decrease the log_fifo_size directive and see what happens. TIA Paolo --- Nate Campi <nate@campin.net> wrote:
On Mon, Dec 26, 2005 at 03:14:21AM -0800, Paolo Supino wrote:
Here is an update on the interval in writing to the log file: the output of ls -l on the file: -rw-r--r-- 1 root root 6234 Dec 26 08:03 switch-01.log and the last messsege coming in on the NIC (captured by tcpdump): 12:56:57.564348 IP switch-01.1024 > syslog.hot.net.il.syslog: UDP, length 75 As you can see the time interval is almost 5 hours... :-(
Wow, I can see how you'd be frustrated. Do you have any filters that might be preventing particular messages from being written to the file? Put in a catchall destination to see if it differs from the logging to your filtered files.
http://www.campin.net/syslog-ng/faq.html#logall
Hope this helps. -- Nate
"Java is, in many ways, C++--." - Michael Feldman.
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
On Mon, 2005-12-26 at 03:14 -0800, Paolo Supino wrote:
Hi
Here is an update on the interval in writing to the log file: the output of ls -l on the file: -rw-r--r-- 1 root root 6234 Dec 26 08:03 switch-01.log and the last messsege coming in on the NIC (captured by tcpdump): 12:56:57.564348 IP switch-01.1024 > syslog.hot.net.il.syslog: UDP, length 75 As you can see the time interval is almost 5 hours... :-(
I can't see how this could happen, log_fifo_size only specifies the maximum size of the buffer and does not increase write latency on its own. Does the message arrive to the destination at the end? Because if it is not then probably it is an unrelated issue. The only thing that affects write latency is sync() -- Bazsi
Hi I'm not sure weather the message gets written or not to the file eventually (there is no specific message I was able to track) so I cannot answer your question with absolute certainty. If I had to guess than I'd say it does get written to the file eventually. I think that the problem is in the way that syslog write the messages to the files. From what I read then on every message that is received a single write() is done. This means that each destination gets opened and closed repeatedly (and this was confirmed by running `fuser` on each of the files). On my server there are a few destinations (the ones that belong to the firewalls) that are so busy that they are constantly open or are opened/closed in such a pace that it's impossible to track manually. Might it be that in such a scenario messages will 1st be written to a destination that is already open and messages to destinations that are closed will wait an undefined amount of time? Is there a way I can track what happens with a specific message in syslog-ng? TIA Paolo --- Balazs Scheidler <bazsi@balabit.hu> wrote:
On Mon, 2005-12-26 at 03:14 -0800, Paolo Supino wrote:
Hi
Here is an update on the interval in writing to the log file: the output of ls -l on the file: -rw-r--r-- 1 root root 6234 Dec 26 08:03 switch-01.log and the last messsege coming in on the NIC (captured by tcpdump): 12:56:57.564348 IP switch-01.1024 > syslog.hot.net.il.syslog: UDP, length 75 As you can see the time interval is almost 5 hours... :-(
I can't see how this could happen, log_fifo_size only specifies the maximum size of the buffer and does not increase write latency on its own.
Does the message arrive to the destination at the end? Because if it is not then probably it is an unrelated issue. The only thing that affects write latency is sync()
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________________ Yahoo! DSL Something to write home about. Just $16.99/mo. or less. dsl.yahoo.com
On Tue, 2005-12-27 at 06:07 -0800, Paolo Supino wrote:
Hi
I'm not sure weather the message gets written or not to the file eventually (there is no specific message I was able to track) so I cannot answer your question with absolute certainty. If I had to guess than I'd say it does get written to the file eventually. I think that the problem is in the way that syslog write the messages to the files. From what I read then on every message that is received a single write() is done. This means that each destination gets opened and closed repeatedly (and this was confirmed by running `fuser` on each of the files). On my server there are a few destinations (the ones that belong to the firewalls) that are so busy that they are constantly open or are opened/closed in such a pace that it's impossible to track manually. Might it be that in such a scenario messages will 1st be written to a destination that is already open and messages to destinations that are closed will wait an undefined amount of time?
syslog-ng does not reopen destination files at such a pace. It basically opens a destination whenever there's a message to write and keeps the file open up to the value specified by time_reap(), which is 60 seconds by default. So busy destinations should be kept opened indefinitely, destinations where incoming messages are rare are closed and reopened whenever they are needed.
Is there a way I can track what happens with a specific message in syslog-ng?
No, sorry. My suspicion is that syslog-ng blocks for some reason on external sources, the most common cases are: 1) two processes reading /proc/kmsg 2) DNS -- Bazsi
Hi 1. I checked weather /proc/kmsg is being read by 2 processes. It isn't. The only process reading the file is syslog-ng (and there is only 1 instance of syslog-ng running). 2. All systems that report to the syslog server have forward and backward resolving setup. Here is the output: forward lookup: # nslookup switch-01 Server: 192.168.200.101 Address: 192.168.200.101#53 Name: switch-01.company.net Address: 192.168.63.1 backward lookup: # nslookup 192.168.63.1 Server: 192.168.200.101 Address: 192.168.200.101#53 1.63.168.192.in-addr.arpa name = switch-01.company.net. Everything looks OK ... TIA Paolo --- Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2005-12-27 at 06:07 -0800, Paolo Supino wrote:
Hi
I'm not sure weather the message gets written or not to the file eventually (there is no specific message I was able to track) so I cannot answer your question with absolute certainty. If I had to guess than I'd say it does get written to the file eventually. I think that the problem is in the way that syslog write the messages to the files. From what I read then on every message that is received a single write() is done. This means that each destination gets opened and closed repeatedly (and this was confirmed by running `fuser` on each of the files). On my server there are a few destinations (the ones that belong to the firewalls) that are so busy that they are constantly open or are opened/closed in such a pace that it's impossible to track manually. Might it be that in such a scenario messages will 1st be written to a destination that is already open and messages to destinations that are closed will wait an undefined amount of time?
syslog-ng does not reopen destination files at such a pace. It basically opens a destination whenever there's a message to write and keeps the file open up to the value specified by time_reap(), which is 60 seconds by default. So busy destinations should be kept opened indefinitely, destinations where incoming messages are rare are closed and reopened whenever they are needed.
Is there a way I can track what happens with a specific message in syslog-ng?
No, sorry.
My suspicion is that syslog-ng blocks for some reason on external sources, the most common cases are:
1) two processes reading /proc/kmsg 2) DNS
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
On Tue, 2005-12-27 at 08:59 -0800, Paolo Supino wrote:
Hi
1. I checked weather /proc/kmsg is being read by 2 processes. It isn't. The only process reading the file is syslog-ng (and there is only 1 instance of syslog-ng running). 2. All systems that report to the syslog server have forward and backward resolving setup. Here is the output: forward lookup: # nslookup switch-01 Server: 192.168.200.101 Address: 192.168.200.101#53
Name: switch-01.company.net Address: 192.168.63.1
backward lookup: # nslookup 192.168.63.1 Server: 192.168.200.101 Address: 192.168.200.101#53
1.63.168.192.in-addr.arpa name = switch-01.company.net.
Everything looks OK ...
I understand that your DNS is set up correctly I was only wondering whether syslog-ng might block on DNS queries for some reason. I'm sure syslog-ng is doing something, either it is buffering data (because of sync) or is blocking on something. -- Bazsi
Hi It's not necesserly a problem with syslog-ng... In trying to debug the problem I ran 2 concurrent captures of the offending devices. 1 to STDOUT and one to a file. after I saw a few lines were captured in the STDOUT capture I wanted to look at the content of the file and see what was the content of the syslog messages and there I encountered a problem: The capture file was still empty. Only after I stopped the capture to the file, linux released the file cache and wrote the lines into the file itself. This happened to me on /tmp FS which is a 2GB shared memory (tmpfs) filesystem. The log files (syslog-ng writes to) are located on ext3 filesystem that is 45GB in size. Might it be that there is something tuned in the filesystem driver that tells it to buffer the data before its flushed to the hard drive and is the fault in the long delays of writing to log files? Another thing I noticed is that the offending devices have their clocks set to UTC while the syslog server is set to a different timezone.... TIA Paolo PS - to counter what I wrote above: There are other devices that syslog-ng does delay writing to their respective log files. --- Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2005-12-27 at 08:59 -0800, Paolo Supino wrote:
Hi
1. I checked weather /proc/kmsg is being read by 2 processes. It isn't. The only process reading the file is syslog-ng (and there is only 1 instance of syslog-ng running). 2. All systems that report to the syslog server have forward and backward resolving setup. Here is the output: forward lookup: # nslookup switch-01 Server: 192.168.200.101 Address: 192.168.200.101#53
Name: switch-01.company.net Address: 192.168.63.1
backward lookup: # nslookup 192.168.63.1 Server: 192.168.200.101 Address: 192.168.200.101#53
1.63.168.192.in-addr.arpa name = switch-01.company.net.
Everything looks OK ...
I understand that your DNS is set up correctly I was only wondering whether syslog-ng might block on DNS queries for some reason. I'm sure syslog-ng is doing something, either it is buffering data (because of sync) or is blocking on something.
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
Hi I solved the problem of the delay: syslog-ng wasn't delaying messages to the files. He was ignoring them alltogether. Syslog-ng did this because the name in the DNS (and the filter) was different than the hostname displayed in the syslog message itself. After I changed the DNS names to match the name appearing in the syslog message the problem vanished. Now messages are written according the sync drictive (i.e immediately :-)). I apologize for the head aches and wasted brain waves I caused because being a green newbie (bad newbie, bad, bad .... ;-)) TIA Paolo --- Balazs Scheidler <bazsi@balabit.hu> wrote:
On Tue, 2005-12-27 at 08:59 -0800, Paolo Supino wrote:
Hi
1. I checked weather /proc/kmsg is being read by 2 processes. It isn't. The only process reading the file is syslog-ng (and there is only 1 instance of syslog-ng running). 2. All systems that report to the syslog server have forward and backward resolving setup. Here is the output: forward lookup: # nslookup switch-01 Server: 192.168.200.101 Address: 192.168.200.101#53
Name: switch-01.company.net Address: 192.168.63.1
backward lookup: # nslookup 192.168.63.1 Server: 192.168.200.101 Address: 192.168.200.101#53
1.63.168.192.in-addr.arpa name = switch-01.company.net.
Everything looks OK ...
I understand that your DNS is set up correctly I was only wondering whether syslog-ng might block on DNS queries for some reason. I'm sure syslog-ng is doing something, either it is buffering data (because of sync) or is blocking on something.
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________ Yahoo! for Good - Make a difference this year. http://brand.yahoo.com/cybergivingweek2005/
On Wed, 2005-12-28 at 06:21 -0800, Paolo Supino wrote:
Hi
I solved the problem of the delay: syslog-ng wasn't delaying messages to the files. He was ignoring them alltogether. Syslog-ng did this because the name in the DNS (and the filter) was different than the hostname displayed in the syslog message itself. After I changed the DNS names to match the name appearing in the syslog message the problem vanished. Now messages are written according the sync drictive (i.e immediately :-)).
I apologize for the head aches and wasted brain waves I caused because being a green newbie (bad newbie, bad, bad .... ;-))
Thanks for letting us know and no problem being a newbie it happens at least once to everyone. :) -- Bazsi
Hi Even thoguh thi never happened to anyone (acheiving uniqueness is hard work ;-)) I think that it's worth mentioning somewhere that when using host directive in a filter the name used must be the same as the name of te sending server.... Paolo --- Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2005-12-28 at 06:21 -0800, Paolo Supino wrote:
Hi
I solved the problem of the delay: syslog-ng wasn't delaying messages to the files. He was ignoring them alltogether. Syslog-ng did this because the name in the DNS (and the filter) was different than the hostname displayed in the syslog message itself. After I changed the DNS names to match the name appearing in the syslog message the problem vanished. Now messages are written according the sync drictive (i.e immediately :-)).
I apologize for the head aches and wasted brain waves I caused because being a green newbie (bad newbie, bad, bad .... ;-))
Thanks for letting us know and no problem being a newbie it happens at least once to everyone. :)
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________________ Yahoo! DSL Something to write home about. Just $16.99/mo. or less. dsl.yahoo.com
On Wed, 2005-12-28 at 08:57 -0800, Paolo Supino wrote:
Hi
Even thoguh thi never happened to anyone (acheiving uniqueness is hard work ;-)) I think that it's worth mentioning somewhere that when using host directive in a filter the name used must be the same as the name of te sending server....
it depends. the host() directive matches the host part in the message which might have been rewritten by syslog-ng according to various options keep_hostname(), use_dns() and of course similar options by the sender server. If the whole chain of syslog relays keep the hostname part intact then you might be able to filter based on the original host name, but of course this requires some trust in the relays as nothing authenticates the hostname there. -- Bazsi
Hi I went to acheck what is the situation in the firewall log file and here is what I found: Dec 27 16:22:27 fwmgmt/fwmgmt logger: 10:00:19 drop .... The date on fwmgmt is set correctly and shows the current time. TIA Paolo PS - I will try to remove the firewall logging and see if it will make a difference. --- Balazs Scheidler <bazsi@balabit.hu> wrote:
On Mon, 2005-12-26 at 03:14 -0800, Paolo Supino wrote:
Hi
Here is an update on the interval in writing to the log file: the output of ls -l on the file: -rw-r--r-- 1 root root 6234 Dec 26 08:03 switch-01.log and the last messsege coming in on the NIC (captured by tcpdump): 12:56:57.564348 IP switch-01.1024 > syslog.hot.net.il.syslog: UDP, length 75 As you can see the time interval is almost 5 hours... :-(
I can't see how this could happen, log_fifo_size only specifies the maximum size of the buffer and does not increase write latency on its own.
Does the message arrive to the destination at the end? Because if it is not then probably it is an unrelated issue. The only thing that affects write latency is sync()
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
__________________________________________ Yahoo! DSL Something to write home about. Just $16.99/mo. or less. dsl.yahoo.com
participants (4)
-
Balazs Scheidler
-
Nate Campi
-
Paolo Supino
-
Sandor Geller