Next thing to ask for is a feature request that would make our use of syslog infintely more flexible: a string formatter module. Take this hypothetical syslog-ng.conf entry as an example:
the 1.5 development releases allow you to specify what fields to go the output for some destinations. i've attached a previous post of mine with the field list inside, and quoting Balazs' original feature announcement. you may be able to use this in conjunction with other facilities, a log watcher, the program destination, whatever, to achieve your goal. -afort
On Wed, Mar 28, 2001 at 09:23:35AM +1000, Andrew Fort wrote:
the 1.5 development releases allow you to specify what fields to go the output for some destinations. i've attached a previous post of mine with the field list inside, and quoting Balazs' original feature announcement. you may be able to use this in conjunction with other facilities, a log watcher, the program destination, whatever, to achieve your goal.
SWEEEET! Thanks for the reference! I'll try it out! -- Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie http://www.wookimus.net/ | s.k.a. gunnarr Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD
On Tue, Mar 27, 2001 at 05:37:15PM -0600, Chad C. Walstrom wrote:
On Wed, Mar 28, 2001 at 09:23:35AM +1000, Andrew Fort wrote:
the 1.5 development releases allow you to specify what fields to go the output for some destinations. i've attached a previous post of mine with the field list inside, and quoting Balazs' original feature announcement. you may be able to use this in conjunction with other facilities, a log watcher, the program destination, whatever, to achieve your goal.
SWEEEET! Thanks for the reference! I'll try it out!
That's all I needed! Thanks! Here's my config and output to show she worked well: --------------------------------------------------------------------- destination sql_file { file("/tmp/syslog.sql" owner("root") group("adm") perm(0640) template("INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg) VALUES( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n")); }; log{ source(src); destination(sql_file); }; --------------------------------------------------------------------- Its output was beautiful SQL. ;-) --------------------------------------------------------------------- INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg ) VALUES( 'skuld', 'syslog', 'notice', 'notice', '2d', '2001-03-27', '18:27:01', 'syslog-ng', 'syslog-ng[16935]: syslog-ng version 1.5.4 starting' ); INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg ) VALUES( 'skuld', 'user', 'notice', 'notice', '0d', '2001-03-27', '18:27:28', 'chewie', 'chewie: this is a another test' ); --------------------------------------------------------------------- Thank you VERY much for your help. ;-) -- Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie http://www.wookimus.net/ | s.k.a. gunnarr Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD
How stable is 1.5? This looks like a very cool function that I would like to be able to use on our logserver. Is it stable enough for a production environment? sim "Chad C. Walstrom" wrote:
On Tue, Mar 27, 2001 at 05:37:15PM -0600, Chad C. Walstrom wrote:
On Wed, Mar 28, 2001 at 09:23:35AM +1000, Andrew Fort wrote:
the 1.5 development releases allow you to specify what fields to go the output for some destinations. i've attached a previous post of mine with the field list inside, and quoting Balazs' original feature announcement. you may be able to use this in conjunction with other facilities, a log watcher, the program destination, whatever, to achieve your goal.
SWEEEET! Thanks for the reference! I'll try it out!
That's all I needed! Thanks! Here's my config and output to show she worked well:
--------------------------------------------------------------------- destination sql_file { file("/tmp/syslog.sql" owner("root") group("adm") perm(0640) template("INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg) VALUES( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n")); }; log{ source(src); destination(sql_file); }; ---------------------------------------------------------------------
Its output was beautiful SQL. ;-)
--------------------------------------------------------------------- INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg ) VALUES( 'skuld', 'syslog', 'notice', 'notice', '2d', '2001-03-27', '18:27:01', 'syslog-ng', 'syslog-ng[16935]: syslog-ng version 1.5.4 starting' ); INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg ) VALUES( 'skuld', 'user', 'notice', 'notice', '0d', '2001-03-27', '18:27:28', 'chewie', 'chewie: this is a another test' ); ---------------------------------------------------------------------
Thank you VERY much for your help. ;-)
-- Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie http://www.wookimus.net/ | s.k.a. gunnarr Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD
------------------------------------------------------------------------ Part 1.2Type: application/pgp-signature
On Tue, Mar 27, 2001 at 06:37:30PM -0600, Chad C. Walstrom wrote:
template("INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg) VALUES( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n"));
NB: from a security perspective, this may not be a good idea. What if $MSG is created by a hostile host and includes a single quote followed by some SQL statement? This is the standard "mixed code + externally supplied data" problem. - Morty
On Wed, Mar 28, 2001 at 01:01:30AM -0500, Mordechai T. Abzug wrote:
On Tue, Mar 27, 2001 at 06:37:30PM -0600, Chad C. Walstrom wrote:
template("INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg) VALUES( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n"));
NB: from a security perspective, this may not be a good idea. What if $MSG is created by a hostile host and includes a single quote followed by some SQL statement? This is the standard "mixed code + externally supplied data" problem.
Thanks for the tip. That's a good thing to point out. Hostile or not, messages could have characters that need to be escaped. In terms of robust design, my suggestion probably falls along the lines of a hack. ;-) Also, in terms of portable SQL, my use of INTO is inappropriate. ;-) -- Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie http://www.wookimus.net/ | s.k.a. gunnarr Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD
Chad C. Walstrom wrote:
template("INSERT INTO mytable ( host, facility, priority, level, tag, date, time, program, msg) VALUES( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG');\n"));
Mordechai T. Abzug wrote:
NB: from a security perspective, this may not be a good idea. What if $MSG is created by a hostile host and includes a single quote followed by some SQL statement? This is the standard "mixed code + externally supplied data" problem.
It looks like I'll have to go with raw data output, probably pipe-delimited with the $MSG text as the last field. I could see the next iteration of the template() option to be an escaped version, where you can specify what your escape character should be and which characters it should apply to. Something like: destination{ file("/tmp/blah" template( "$MSG" escape("\") to-escape("'\"\\") ) ); }; Nasty grammar to escape ', ", and \, but necessary if you think about it. Ideas, flames, suggestions? BTW, I'm willing to code and send in patches; I just need to figure out this funky use of scheme... -- Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie http://www.wookimus.net/ | s.k.a. gunnarr Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD
Ideas, flames, suggestions? BTW, I'm willing to code and send in patches; I just need to figure out this funky use of scheme...
scheme is just used to create the .x files out of the specially formatted comments in .c and .h files. This special formatting is documented in lsh's source package (where this make_class stuff came from). Lsh is available at ftp://ftp.lysator.liu.se/pub/security/lsh. The .x files contain the C structure definitions corresponding to the class definitions in comments, and some automatically generated functions to help garbage collection. -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
On Wed, Mar 28, 2001 at 12:04:25PM -0600, Chad C. Walstrom wrote:
It looks like I'll have to go with raw data output, probably pipe-delimited with the $MSG text as the last field. I could see the next iteration of the template() option to be an escaped version, where you can specify what your escape character should be and which characters it should apply to. Something like:
destination{ file("/tmp/blah" template( "$MSG" escape("\") to-escape("'\"\\") ) ); };
That seems like overkill. You need an external program to catch the data/SQL statements anyway and insert the data/run the SQL statements. Why not output the data as data, and have the external program deal with generating your SQL statements while escaping any characters you need escaped? - Morty
where you can specify what your escape character should be and which characters it should apply to. Something like:
destination{ file("/tmp/blah" template( "$MSG" escape("\") to-escape("'\"\\") ) ); };
That seems like overkill. You need an external program to catch the data/SQL statements anyway and insert the data/run the SQL statements. Why not output the data as data, and have the external program deal with generating your SQL statements while escaping any characters you need escaped?
Step away from the idea that I'll be making SQL statements for a moment, and think in terms of passing data from one program to another. Syslog and syslog-ng are wonderful for collecting log entries from various sources and redirecting them to new destinations -- and we're familiar with syslog-ng's flexibility in these terms. So, the question is not "can I 'escape' the string data in another program," rather "should I have to 'escape' the string data in another program?" I would argue that there are times when syslog-ng is the best place to tie in the output templates with actual output format, including escaped characters in message text. It ties in very well logically. syslog-ng knows what the data is. It has control of each atomic part. It's job is to output that data to a number of destinations, and now, in customizable formats. It makes sense to finish the job of data presentation at this layer. hmm... You know. I've got a lot of ideas to mull over a couple dozen times and to test out. I do think this very simple and very useful functionality to the output format is not an unnecessary feature, but I do agree that you can get by without it. -- Chad Walstrom <chewie@wookimus.net> | a.k.a. ^chewie http://www.wookimus.net/ | s.k.a. gunnarr Key fingerprint = B4AB D627 9CBD 687E 7A31 1950 0CC7 0B18 206C 5AFD
participants (5)
-
Andrew Fort
-
Balazs Scheidler
-
Chad C. Walstrom
-
Mordechai T. Abzug
-
Simeon Johnston