Firstly, THANKS for such a great replacement to syslog! I'm quite happy with the ease of configurability and the use of the tcp() source and destinations -- very nice for tunnelling over OpenSSL. Next thing to ask for is a feature request that would make our use of syslog infintely more flexible: a string formatter module. Take this hypothetical syslog-ng.conf entry as an example: --------- START EXAMPLE ----------- # First, set some global options. options { long_hostnames(off); sync(0); }; source remotehost { tcp( 192.168.1.1, 514 ); }; # sql batch file destination sql_output { file ("/var/log/log.sql" owner ("root") group("adm") perm(0640)); }; destination xml_output { file ("/var/log/log.xml" owner ("root") group("adm") perm(0640)); }; # Now, the format string using printf style placeholders where %h == # host, %d == date, %t == time, %f == facility, %l == level, and %m == # message format o_sql { printf( "INSERT mytable ( host, date, time, facility, level, message )\nVALUES ( '%h', '%d', '%t', '%f', '%l', '%m' );\n" ); }; # Or, we could specify template files for the printf driver # # Where the file would contain character literals rather than escapes # (for things such as tab and newline): # # INSERT mytable ( host, date, time, facility, level, message ) # VALUES ( '%h', '%d', '%t', '%f', '%l', '%m' );\n" ); # format o_sql { printf(template("/usr/share/syslog-ng/sql.tmpl")); }; format o_xml { printf(template("/usr/share/syslog-ng/xml.tmpl")); }; # Now, out to the log log { source(remotehost); format(o_sql); destination(sql_output); }; log { source(remotehost); format(o_xml); destination(xml_output); }; --------- START EXAMPLE ----------- The reason I suggest having format as a separate module instead of a file option is for instance you want to have a real-time process to log INSERT statements into a database. You could specificy your destination as a named pipe source s_host{ tcp( 192.168.1.1, 514 ); }; format o_sql{ printf(template("/etc/syslog-ng/custom_sql.tmpl")); }; destination d_sqlpipe{ file("/dev/log_sql" owner ("root") group("adm") perm(0640)); }; log{ source(s_host); format(o_sql); destination(d_sqlpipe); }; to which your database client is listening, waiting for input. Let's hypothise for a moment that psql, the PostgreSQL command line client, can open up a named pipe as an input file -- as I'm not sure if it really can. A command like the following is all you'd need for real-time database logging: bash$ whoami syslogd bash$ nohup psql -h localhost -d sysylogdb /dev/log_sql& bash$ We preserve the UNIX philosophy in that you let your tools do only one job, but one job done VERY well. Why reimplement the wheel and embed an SQL client into syslog-ng? No reason at all. What do we have to do to change the output format? Change the output template in the log rule. Simple, clean, efficient. What do you think? -- Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie http://www.wookimus.net/ | s.k.a. gunnarr Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD
quick question: why not just output to a program? and have that program deal with the input in any way you choose... you can do this pretty easily with perl #!/usr/bin/perl -w use strict; use DBI; use DBD::Pg; &connect_to_db(); # make this function obviously while ($line=<>) { # mangle data for input # insert into db } On Tue, Mar 27, 2001 at 04:11:54PM -0600, Chad C. Walstrom wrote: | Firstly, THANKS for such a great replacement to syslog! I'm quite | happy with the ease of configurability and the use of the tcp() source | and destinations -- very nice for tunnelling over OpenSSL. | | Next thing to ask for is a feature request that would make our use of | syslog infintely more flexible: a string formatter module. Take this | hypothetical syslog-ng.conf entry as an example: | | --------- START EXAMPLE ----------- | # First, set some global options. | options { long_hostnames(off); sync(0); }; | source remotehost { tcp( 192.168.1.1, 514 ); }; | | # sql batch file | destination sql_output { file ("/var/log/log.sql" owner ("root") | group("adm") perm(0640)); }; | destination xml_output { file ("/var/log/log.xml" owner ("root") | group("adm") perm(0640)); }; | | # Now, the format string using printf style placeholders where %h == | # host, %d == date, %t == time, %f == facility, %l == level, and %m == | # message | format o_sql { printf( "INSERT mytable ( host, date, time, facility, | level, message )\nVALUES ( '%h', '%d', '%t', '%f', '%l', '%m' );\n" ); }; | | # Or, we could specify template files for the printf driver | # | # Where the file would contain character literals rather than escapes | # (for things such as tab and newline): | # | # INSERT mytable ( host, date, time, facility, level, message ) | # VALUES ( '%h', '%d', '%t', '%f', '%l', '%m' );\n" ); | # | format o_sql { printf(template("/usr/share/syslog-ng/sql.tmpl")); }; | format o_xml { printf(template("/usr/share/syslog-ng/xml.tmpl")); }; | | # Now, out to the log | log { source(remotehost); format(o_sql); destination(sql_output); }; | log { source(remotehost); format(o_xml); destination(xml_output); }; | --------- START EXAMPLE ----------- | | The reason I suggest having format as a separate module instead of a | file option is for instance you want to have a real-time process to | log INSERT statements into a database. You could specificy your | destination as a named pipe | | source s_host{ tcp( 192.168.1.1, 514 ); }; | format o_sql{ printf(template("/etc/syslog-ng/custom_sql.tmpl")); }; | destination d_sqlpipe{ file("/dev/log_sql" owner ("root") | group("adm") perm(0640)); }; | log{ source(s_host); format(o_sql); destination(d_sqlpipe); }; | | to which your database client is listening, waiting for input. Let's | hypothise for a moment that psql, the PostgreSQL command line client, | can open up a named pipe as an input file -- as I'm not sure if it | really can. A command like the following is all you'd need for | real-time database logging: | | bash$ whoami | syslogd | bash$ nohup psql -h localhost -d sysylogdb /dev/log_sql& | bash$ | | We preserve the UNIX philosophy in that you let your tools do only one | job, but one job done VERY well. Why reimplement the wheel and embed | an SQL client into syslog-ng? No reason at all. What do we have to | do to change the output format? Change the output template in the log | rule. Simple, clean, efficient. | | What do you think? | | -- | Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie | http://www.wookimus.net/ | s.k.a. gunnarr | Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD |
On Tue, Mar 27, 2001 at 05:56:39PM -0500, jon@dumbo.pobox.com wrote:
quick question: why not just output to a program? and have that program deal with the input in any way you choose...
Because the syslog entry that is being output does not contain the facility, level, or host that the message was received from. The only thing that an external program can do is parse the actual syslog entry in hopes that enough detail is there to parse out error severities and hostname. Not all syslog clients actually report their own hostname in the syslog message; case in point, the Cisco 675's. By allowing the administrator to specify the output format of the syslog string, or by adding things like a prepend driver, we can get more valuable data. I don't believe daemons like syslogd and klogd should be talking to database engines directly or even managing databases, but they should allow us to retrieve all information that may be considered valuable. In that case, I agree with being able to pipe out to an external program. Still, the syslog entry in its current state is not as valuable as it could be. The filters module does help out in that sense. Another reason I don't want to pipe the output directly to a program is the overhead of program startup. If I want to connect to a database, I have to start the program (and any virtual machine it relies upon, such as perl), initialize its class structures, make the connection to the database, process the input from file/stdin, process, close connection, exit. Better is to pipe to a special device file, such as a named pipe/fifo file. destination( file("/dev/syslog_output"); ); A daemon perl program, which by its nature only needs to be initialized once, can process log entries as they are output to the fifo by syslogd. Using placeholders in SQL statements with the DBI perl module is a good example of optimizing code: Without using placeholders, the insert statement shown previously would have to contain the literal values to be inserted and would have to be re-prepared and re-executed for each row. With placeholders, the insert statement only needs to be prepared once. The bind values for each row can be given to the `execute' method each time it's called. By avoiding the need to re-prepare the statement for each row, the application typically runs many times faster. Here's an example: my $sth = $dbh->prepare(q{ INSERT INTO sales (product_code, qty, price) VALUES (?, ?, ?) }) || die $dbh->errstr; while (<>) { chop; my ($product_code, $qty, $price) = split /,/; $sth->execute($product_code, $qty, $price) || die $dbh->errstr; } $dbh->commit || die $dbh->errstr; See the executeand bind_param entries elsewhere in this document for more details. Granted, we wouldn't have to worry about this particular statement if we could output the syslogd entry as a configurable SQL statement and pipe it to a fifo and let the database client handle all the details about reading the statement and executing the query. An added advantage to this very generic idea of a string formatter for syslog-ng is that if you want your messages to look like Windoze NT logs, you can do so with a simple template. -- Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie http://www.wookimus.net/ | s.k.a. gunnarr Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD
participants (2)
-
Chad C. Walstrom
-
jon＠dumbo.pobox.com