Hi, What packages does it need to compile syslog-ng? Is it the same package when to make RPM? (bison, flex, gcc-c++, pkgconfig, glib2-devel, libevtlog-devel) (It needs only libc to compile eventlog, doesn't it?) And if it's not necessary to connect database, does it need to install libdbi8? Thanks hiro _________________________________________________________________ 【MSNビデオ】超貴重！驚きの大物対談が実現。作家 村上龍が話題のあの人に迫る http://video.msn.co.jp/rvr/default.htm
On Mon, 2007-10-01 at 19:29 +0900, hi ro wrote:
Hi,
What packages does it need to compile syslog-ng? Is it the same package when to make RPM? (bison, flex, gcc-c++, pkgconfig, glib2-devel, libevtlog-devel)
you might need libnet if you intend to compile with spoof-source support and libwrap if you need tcpwrappers.
(It needs only libc to compile eventlog, doesn't it?)
And if it's not necessary to connect database, does it need to install libdbi8?
Only the PE version of syslog-ng has direct database connectivity in which case you need libdbi8. -- Bazsi
Balazs Scheidler wrote:
Only the PE version of syslog-ng has direct database connectivity in which case you need libdbi8.
Was there consideration of database access using ODBC? If there was, could you provide a couple of reasons (rational or otherwise) why ODBC was not implimented? -- Evan Rempel
On Mon, 2007-10-01 at 11:27 -0700, Evan Rempel wrote:
Balazs Scheidler wrote:
Only the PE version of syslog-ng has direct database connectivity in which case you need libdbi8.
Was there consideration of database access using ODBC? If there was, could you provide a couple of reasons (rational or otherwise) why ODBC was not implimented?
Not really. I did not even consider ODBC as I felt it was too complicated for my case, and my perception was that ODBC is not that well supported on Linux/UNIX. Because of your email I've googled a bit, and I've found unixodbc, which seems to do the trick, however it seems inherently more complex than libdbi. (and some of the drivers cost extra money) Do you have any specific advantage in mind regarding libdbi vs. ODBC? -- Bazsi
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes. The problem we seem to have encountered is that our source section and destination definitions are; source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); }; destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); }; It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca until some maximum length was reached and the hostname field became truncated. This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng. Comments? -- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
On Wed, 2007-10-17 at 10:42 -0700, Evan Rempel wrote:
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes.
The problem we seem to have encountered is that our source section and destination definitions are;
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); };
destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); };
It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of
local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca
until some maximum length was reached and the hostname field became truncated.
This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng.
I was already thinking about the idea of dropping internal messages generated while another internal message is being delivered to prevent such loops. I added this to my todo list. -- Bazsi
On 10/18/07, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2007-10-17 at 10:42 -0700, Evan Rempel wrote:
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes.
The problem we seem to have encountered is that our source section and destination definitions are;
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); };
destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); };
It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of
local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca
until some maximum length was reached and the hostname field became truncated.
This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng.
I was already thinking about the idea of dropping internal messages generated while another internal message is being delivered to prevent such loops.
Just out of curiosity: What is the intention of the line 'tcp( localip(127.0.0.1) port(514) );' ? Which local logging messages are not processed if just using: source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); internal(); }; Thanks, Andreas
Andreas Kurz wrote:
On 10/18/07, Balazs Scheidler <bazsi@balabit.hu> wrote:
On Wed, 2007-10-17 at 10:42 -0700, Evan Rempel wrote:
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes.
The problem we seem to have encountered is that our source section and destination definitions are;
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); };
destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); };
It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of
local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca
until some maximum length was reached and the hostname field became truncated.
This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng. I was already thinking about the idea of dropping internal messages generated while another internal message is being delivered to prevent such loops.
Just out of curiosity:
What is the intention of the line 'tcp( localip(127.0.0.1) port(514) );' ?
Which local logging messages are not processed if just using:
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); internal(); };
We use multiple instances of syslog-ng on our systems to get logs from applications that can not syslog. We moved away from having the "system syslog" tail files and read pipes and placed this functionality into a separate syslog-ng that is started with and shutdown with the applications init scripts. These other instances of syslog-ng have NO WAY to "syslog" files as the author has declined the suggestion of having a "syslog" destination in syslog-ng (I know, I can write a patch :-). Anyhow, the only way to get data back into the "system syslog" which is where the internal messages need to go, is to have some supported destination. We have been using a syslog port, but I think we will be moving to a pipe for security reasons. I hope this is at least a little clear :-) -- Evan Rempel
....
Just out of curiosity:
What is the intention of the line 'tcp( localip(127.0.0.1) port(514) );' ?
Which local logging messages are not processed if just using:
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); internal(); };
We use multiple instances of syslog-ng on our systems to get logs from applications that can not syslog. We moved away from having the "system syslog" tail files and read pipes and placed this functionality into a separate syslog-ng that is started with and shutdown with the applications init scripts.
These other instances of syslog-ng have NO WAY to "syslog" files as the author has declined the suggestion of having a "syslog" destination in syslog-ng (I know, I can write a patch :-). Anyhow, the only way to get data back into the "system syslog" which is where the internal messages need to go, is to have some supported destination. We have been using a syslog port, but I think we will be moving to a pipe for security reasons.
I hope this is at least a little clear :-)
OK, I see .... We are using syslog-ng 2.0.5 and the file() driver to feed MySQL errors to syslog-ng. I think it's quite handy since syslog-ng preserves the last logfile position on syslog-ng restarts. Looks like this: source s_tail { file("/var/log/mysql.err" follow_freq(1) flags(no-parse)); }; Works very well for us. Regards, Andreas
-- Evan Rempel _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
OK, I see .... We are using syslog-ng 2.0.5 and the file() driver to feed MySQL errors to syslog-ng. I think it's quite handy since syslog-ng preserves the last logfile position on syslog-ng restarts. Looks like this:
source s_tail { file("/var/log/mysql.err" follow_freq(1) flags(no-parse)); };
Works very well for us.
Yes, this works well, however, if I need to make a change to the logging for the application (mysql), or merely want to shutdown the mysql server to move the logs somewhere else due to space, or need to unmount the filesystem to add more space or a bunch of other things that are specific to the application, I DO NOT want to have to stop/start the system logging. We can then provide sudo access to application administrators to start/stop their own logging along with the application without giving them the ability to stop/start the system logging. For these reasons we use a second instance of syslog-ng to do exactly what you are saying, but to get it into the system syslog, which will route copies of messages to primary and backup central syslog servers etc, I need a way to get one syslog-ng instance to send messages to a second syslog-ng instance ON THE SAME HOST. That's why I listen on localhost:514, and am looking at using a pipe (permissions are an issue here). Evan.
Balazs Scheidler wrote:
On Wed, 2007-10-17 at 10:42 -0700, Evan Rempel wrote:
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes.
The problem we seem to have encountered is that our source section and destination definitions are;
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); };
destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); };
It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of
local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca
until some maximum length was reached and the hostname field became truncated.
This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng.
I was already thinking about the idea of dropping internal messages generated while another internal message is being delivered to prevent such loops.
The messages were not exclusively internal messages. A message from the kernel would be logged and sent to 127.0.0.1 rather than being logged and sent to syslog.uvic.ca syslog-ng would receive this message from 127.0.0.1 and process it as usual, by logging it to disk and sending it to 127.0.0.1 which would repeat the process. -- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
On Wed, 2007-10-17 at 10:42 -0700, Evan Rempel wrote:
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes.
The problem we seem to have encountered is that our source section and destination definitions are;
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); };
destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); };
It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of
local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca
until some maximum length was reached and the hostname field became truncated.
This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng.
While looking through my pile of syslog-ng todo items I reread this message, and I missed something last time. Currently, syslog-ng falls back to 0.0.0.0 when resolving the DNS name fails, this in turn is interpreted by the kernel as 127.0.0.1 when used as a destination address. So that's the exact cause of the breakage. I have a half-baked patch that should fix this behaviour, however it does not sit too well with optional(yes) setting. (e.g. will fail even if optional is yes). Anyhow, here's the patch as it stands: http://git.balabit.hu/?p=bazsi/syslog-ng-2.0.git;a=commit;h=6cd4fdaea7d77f1f... -- Bazsi
Hi, This is an old message, but I thought I'd let you know that I've committed a patch to syslog-ng OSE 2.1 that should retry TCP connections when a resolution error occurs. This is the patch in question: http://git.balabit.hu/?p=bazsi/syslog-ng-2.1.git;a=commit;h=56f6115c623f793a... Some testing would be appreciated. Thanks in advance. On Tue, 2007-10-23 at 18:26 +0200, Balazs Scheidler wrote:
On Wed, 2007-10-17 at 10:42 -0700, Evan Rempel wrote:
We recently has an unscheduled power outage in our data center. Our servers came back prior to our DNS being available (actually, prior to the network coming back up). All hosts running syslog-ng consumed their log filespace as fast as the disks would allow writing which took about 2 minutes.
The problem we seem to have encountered is that our source section and destination definitions are;
source local { unix-stream("/dev/log" max-connections(200)); file("/proc/kmsg" log_prefix("kernel: ")); tcp( localip(127.0.0.1) port(514) ); internal(); };
destination syslogServer1 { tcp("syslog.uvic.ca" log_fifo_size(50000) ); };
It seems that if syslog.uvic.ca could not be resolved, syslog-ng took it upon itself to use 127.0.0.1 as its destination and started logging to itself. Chaining of hostnames is on, which means that we could see a message path of
local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca/local@myhost.uvic.ca
until some maximum length was reached and the hostname field became truncated.
This should be easy to repeat if you use a source like the one above, disconnect the network and start syslog-ng.
While looking through my pile of syslog-ng todo items I reread this message, and I missed something last time.
Currently, syslog-ng falls back to 0.0.0.0 when resolving the DNS name fails, this in turn is interpreted by the kernel as 127.0.0.1 when used as a destination address.
So that's the exact cause of the breakage. I have a half-baked patch that should fix this behaviour, however it does not sit too well with optional(yes) setting. (e.g. will fail even if optional is yes).
Anyhow, here's the patch as it stands:
http://git.balabit.hu/?p=bazsi/syslog-ng-2.0.git;a=commit;h=6cd4fdaea7d77f1f...
-- Bazsi
Balazs Scheidler wrote:
On Mon, 2007-10-01 at 11:27 -0700, Evan Rempel wrote:
Balazs Scheidler wrote:
Only the PE version of syslog-ng has direct database connectivity in which case you need libdbi8. Was there consideration of database access using ODBC? If there was, could you provide a couple of reasons (rational or otherwise) why ODBC was not implimented?
Not really. I did not even consider ODBC as I felt it was too complicated for my case, and my perception was that ODBC is not that well supported on Linux/UNIX.
Because of your email I've googled a bit, and I've found unixodbc, which seems to do the trick, however it seems inherently more complex than libdbi. (and some of the drivers cost extra money)
Do you have any specific advantage in mind regarding libdbi vs. ODBC?
From my perspective I like the odbc approach better because the odbc drivers are generally written by the database vendor/author rather than an third party. Someone else would have to correct me, but I don't think any database group writes their own dbi drivers. This seems like it would result in a more stable environment, and that's what I want in my logging. The whole things comes to a confusing mess when you think of the dbiodbc which is an odbc driver for the dbi API which would then allow you to use any odbc driver you wish. Of course the same is true in reverse, you can get an odbcdbi which provides and dbi driver with an odbc interface. Just my $0.02 -- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
On Fri, 2007-11-16 at 10:38 -0800, Evan Rempel wrote:
Balazs Scheidler wrote:
On Mon, 2007-10-01 at 11:27 -0700, Evan Rempel wrote:
Balazs Scheidler wrote:
Only the PE version of syslog-ng has direct database connectivity in which case you need libdbi8. Was there consideration of database access using ODBC? If there was, could you provide a couple of reasons (rational or otherwise) why ODBC was not implimented?
Not really. I did not even consider ODBC as I felt it was too complicated for my case, and my perception was that ODBC is not that well supported on Linux/UNIX.
Because of your email I've googled a bit, and I've found unixodbc, which seems to do the trick, however it seems inherently more complex than libdbi. (and some of the drivers cost extra money)
Do you have any specific advantage in mind regarding libdbi vs. ODBC?
From my perspective I like the odbc approach better because the odbc drivers are generally written by the database vendor/author rather than an third party. Someone else would have to correct me, but I don't think any database group writes their own dbi drivers. This seems like it would result in a more stable environment, and that's what I want in my logging.
The whole things comes to a confusing mess when you think of the dbiodbc which is an odbc driver for the dbi API which would then allow you to use any odbc driver you wish. Of course the same is true in reverse, you can get an odbcdbi which provides and dbi driver with an odbc interface.
True enough, I've already ran into problems with the Oracle DBI driver. For now it works, we support our own libdbi/dbd packages, but the end result might be that we migrate to using ODBC in the future. -- Bazsi
participants (4)
-
Andreas Kurz
-
Balazs Scheidler
-
Evan Rempel
-
hi ro