I must be doing something wrong or the syntax is really dependant on the program name now. Orginally under the v2.1 build, the intent of the filter was to strip any incoming message going to the syslog-ng server from the data being sent from the syslog-ng client. There was no one specific program attached to the incoming data. It could come from the client's syslog-ng side via logger command, apache data, kernel, etc. Server-side syslog-ng.conf filter section: filter M_audit { not match("Auditing" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_repeat { not match("Repeating" value("last message repeated") flags(ignore-case)); }; filter M_stats { not match("Stats" value("Log statistics") flags(ignore-case)); }; On the client side, I would just do 'logger daemon' to send over the message to the syslog-ng server. On the server side, running syslog-ng -d reveals this output: Incoming log entry; line='<13>Sep 20 17:16:31 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='Audit daemon rotating log files' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_audit' Filter rule evaluation begins; filter_rule='M_repeat' No such value known; value='last message repeated' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_repeat' Filter rule evaluation begins; filter_rule='M_stats' No such value known; value='Log statistics' Filter node evaluation result; filter_result='match' Filter rule evaluation result; filter_result='match', filter_rule='M_stats' Server-side syslog output being seen in the data file: Sep 20 17:16:31 drupal root: daemon ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
On Mon, Sep 20, 2010 at 05:23:28PM -0400, Worsham, Michael wrote:
No such value known; value='Audit daemon rotating log files' No such value known; value='last message repeated' No such value known; value='Log statistics'
I believe this output indicates you have the incorrect information in the value argument. The value argument is supposed to be used to indicate which message macro should be checked for the string or regex in question. So you probably want the value argument to be one of these: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guid... Matthew. The most interesting ones for your application would be the ones below. Consider using an output template which outputs the value in each macro, so you can see which macro you should be matching for each of your filter rules. For example, if you output messages with this template, you would see the value in the MSGONLY macro. You could use a longer version of this to print out all the macros and figure out which should be used for the different matches you are trying to perform. template t_raw { template("${MSGONLY}\n"); }; MSG or MESSAGE Description: Text contents of the log message without the program name and pid. Note that this has changed in syslog-ng version 3.0; in earlier versions this macro included the program name and the pid. In syslog-ng 3.0, the MSG macro became equivalent with the MSGONLY macro. The program name and the pid together are available in the MSGHDR macro. MSGHDR Description: The name and the pid of the program that sent the log message in PROGRAM: PID format. Includes a trailing whitespace. Note that the macro returns an empty value if both the program and pid fields of the message are empty. MSGONLY Description: Message contents without the program name or pid. PROGRAM Description: The name of the program sending the message. Note that the content of the $PROGRAM variable may not be completely trusted as it is provided by the client program that constructed the message.
Well I tried the following two lines and neither one works: filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; Incoming log entry; line='<13>Sep 20 18:26:19 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='MSGONLY' filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; Incoming log entry; line='<13>Sep 20 18:16:15 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='Audit daemon rotating log files' WTF am I doing wrong and please quit quoting URLs to look at. -- M ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 6:12 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? On Mon, Sep 20, 2010 at 05:23:28PM -0400, Worsham, Michael wrote:
No such value known; value='Audit daemon rotating log files' No such value known; value='last message repeated' No such value known; value='Log statistics'
I believe this output indicates you have the incorrect information in the value argument. The value argument is supposed to be used to indicate which message macro should be checked for the string or regex in question. So you probably want the value argument to be one of these: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guid... Matthew. The most interesting ones for your application would be the ones below. Consider using an output template which outputs the value in each macro, so you can see which macro you should be matching for each of your filter rules. For example, if you output messages with this template, you would see the value in the MSGONLY macro. You could use a longer version of this to print out all the macros and figure out which should be used for the different matches you are trying to perform. template t_raw { template("${MSGONLY}\n"); }; MSG or MESSAGE Description: Text contents of the log message without the program name and pid. Note that this has changed in syslog-ng version 3.0; in earlier versions this macro included the program name and the pid. In syslog-ng 3.0, the MSG macro became equivalent with the MSGONLY macro. The program name and the pid together are available in the MSGHDR macro. MSGHDR Description: The name and the pid of the program that sent the log message in PROGRAM: PID format. Includes a trailing whitespace. Note that the macro returns an empty value if both the program and pid fields of the message are empty. MSGONLY Description: Message contents without the program name or pid. PROGRAM Description: The name of the program sending the message. Note that the content of the $PROGRAM variable may not be completely trusted as it is provided by the client program that constructed the message. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
Your "value" is wrong. It's a variable name, not a literal string, so you use it like this: value(MSGONLY) or the cleaner version value(${MSGONLY}) It works like a bash variable in this regard On Tuesday 21 September 2010 00:30:06 Worsham, Michael wrote:
Well I tried the following two lines and neither one works:
filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; Incoming log entry; line='<13>Sep 20 18:26:19 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='MSGONLY' filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; Incoming log entry; line='<13>Sep 20 18:16:15 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='Audit daemon rotating log files' WTF am I doing wrong and please quit quoting URLs to look at.
-- M
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 6:12 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
On Mon, Sep 20, 2010 at 05:23:28PM -0400, Worsham, Michael wrote:
No such value known; value='Audit daemon rotating log files' No such value known; value='last message repeated' No such value known; value='Log statistics'
I believe this output indicates you have the incorrect information in the value argument. The value argument is supposed to be used to indicate which message macro should be checked for the string or regex in question.
So you probably want the value argument to be one of these:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui de-admin-en.html/reference_macros.html
Matthew.
The most interesting ones for your application would be the ones below.
Consider using an output template which outputs the value in each macro, so you can see which macro you should be matching for each of your filter rules.
For example, if you output messages with this template, you would see the value in the MSGONLY macro. You could use a longer version of this to print out all the macros and figure out which should be used for the different matches you are trying to perform.
template t_raw { template("${MSGONLY}\n"); };
MSG or MESSAGE Description: Text contents of the log message without the program name and pid. Note that this has changed in syslog-ng version 3.0; in earlier versions this macro included the program name and the pid. In syslog-ng 3.0, the MSG macro became equivalent with the MSGONLY macro. The program name and the pid together are available in the MSGHDR macro.
MSGHDR Description: The name and the pid of the program that sent the log message in PROGRAM: PID format. Includes a trailing whitespace. Note that the macro returns an empty value if both the program and pid fields of the message are empty.
MSGONLY Description: Message contents without the program name or pid.
PROGRAM
Description: The name of the program sending the message. Note that the content of the $PROGRAM variable may not be completely trusted as it is provided by the client program that constructed the message.
___________________________________________________________________________ ___ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
-- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions +27 11 575 7585 Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you.
On Tue, Sep 21, 2010 at 12:52:17AM +0200, Alan McKinnon wrote:
Your "value" is wrong. It's a variable name, not a literal string, so you use it like this:
value(MSGONLY)
or the cleaner version
value(${MSGONLY})
Are you sure? If you are sure about it, there is one example in the documentation where it is not performed this way. http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guid... The following filter statement selects the messages that contain the word deny and come from the host example. filter demo_filter { host("example") and match("deny" value("MESSAGE")); }; So we better get somebody from Balabit to correct the documentation ASAP if it's wrong like you suspect it to be. Matthew.
On Tuesday 21 September 2010 01:24:44 Matthew Hall wrote:
On Tue, Sep 21, 2010 at 12:52:17AM +0200, Alan McKinnon wrote:
Your "value" is wrong. It's a variable name, not a literal string, so you use it like this:
value(MSGONLY)
or the cleaner version
value(${MSGONLY})
Are you sure? If you are sure about it, there is one example in the documentation where it is not performed this way.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui de-admin-en.html/configuring_filters.html
The following filter statement selects the messages that contain the word deny and come from the host example.
filter demo_filter { host("example") and match("deny" value("MESSAGE")); };
So we better get somebody from Balabit to correct the documentation ASAP if it's wrong like you suspect it to be.
I have run into this before, and it was only a lucky grep through the pdf doc that gave me the clue to the solution. But I haven't extensively researched all of it to be able to say something conclusive. It's entirely possible there are two code paths through the config parser and they give different results (hey, it's software - we all know how that works). -- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions +27 11 575 7585 Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you.
On Mon, 2010-09-20 at 16:24 -0700, Matthew Hall wrote:
On Tue, Sep 21, 2010 at 12:52:17AM +0200, Alan McKinnon wrote:
Your "value" is wrong. It's a variable name, not a literal string, so you use it like this:
value(MSGONLY)
or the cleaner version
value(${MSGONLY})
Are you sure? If you are sure about it, there is one example in the documentation where it is not performed this way.
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guid...
The following filter statement selects the messages that contain the word deny and come from the host example.
filter demo_filter { host("example") and match("deny" value("MESSAGE")); };
So we better get somebody from Balabit to correct the documentation ASAP if it's wrong like you suspect it to be.
match() is complicated as its meaning has changed slightly when changing from 2.1 to 3.0. In 2.1 (and in 2.1 compatibility mode of 3.0) match matches against the string "program[pid]: message". Starting with 3.0 (if @version: 3.0 is specified in the config file), match() becomes a is a generic filter to match against the contents of any name-value pair. The pattern is the first argument and the optional value() option specifies which name-value pair to look into. By default it works against the contents of $MESSAGE. match("almafa") Means to look for the string "almafa" inside $MESSAGE. You can also achieve the same functionality by using message("almafa") Also, to complicate things even further (but I hope we can get over the upgrade hassles quickly :) in 3.0 (but not in 3.1) the match() filter couldn't cope with macros, whereas it could work with name-value pairs. It is very difficult to decide what is a name-value pair and what is a macro without checking the source code. That's why this has changed in 3.1, where match() (and other similar functionality) will transparently use macro values just like name-value pairs. The difference between macros and Name-value pairs are that macros are read-only values, generated when a template is formatted. name-value pairs on the other hand are read-write values associated with messages. Hope this makes things clearer. Mathew, where did you see the erroneous example? I couldn't find any? -- Bazsi
Hi Bazsi, On Mon, Sep 27, 2010 at 03:07:31PM +0200, Balazs Scheidler wrote:
Mathew, where did you see the erroneous example? I couldn't find any?
Sorry for the confusion. Let me try to say it differently from before. I was not saying the example must be erroneous, just that what Alan said and what the example did conflicted with each other. If Alan is right that you must have special markers on the variables passed to match, then this example from the HTML is wrong: filter demo_filter { host("example") and match("deny" value("MESSAGE")); }; Because it does not put $ or ${} which Alan believed necessary for it to work. Alternatively, Alan made a mistake thinking these chars were mandatory. I think both the doc and Alan can't be right at the same time. So I was trying to get confirmation from Balabit about it.
Bazsi
Matthew.
On Mon, 2010-09-27 at 10:05 -0700, Matthew Hall wrote:
Hi Bazsi,
On Mon, Sep 27, 2010 at 03:07:31PM +0200, Balazs Scheidler wrote:
Mathew, where did you see the erroneous example? I couldn't find any?
Sorry for the confusion. Let me try to say it differently from before.
I was not saying the example must be erroneous, just that what Alan said and what the example did conflicted with each other. If Alan is right that you must have special markers on the variables passed to match, then this example from the HTML is wrong:
filter demo_filter { host("example") and match("deny" value("MESSAGE")); };
Because it does not put $ or ${} which Alan believed necessary for it to work. Alternatively, Alan made a mistake thinking these chars were mandatory. I think both the doc and Alan can't be right at the same time.
So I was trying to get confirmation from Balabit about it.
No need for the "$", it was intentionally not put there as value is not a template. e.g. you can't write this: match("deny" value("$MSGHDR$MSG")); syslog-ng 3.1 gives you a warning if you use '$' in the value but will work correctly. 3.0 is not this forgiving, it'll simply not work. -- Bazsi
Okay, maybe I am not making it clear enough on what it is I am trying to do... In the older syslong-ng v2.1, this line works perfectly: filter M_audit { not match("Audit daemon rotating log files"); }; Under 3.0.8, none of the following are working (if added one line at a time) and the daemon restarted: filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY) flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files")); }; What I am looking to do is if any incoming data has the following "Audit daemon rotating log files" message being detected, it should be filtered (i.e. dropped), and then not show up in the actual message log file. For example, if I go over on the syslog-ng client and do 'logger daemon': - The older v2.1 syslog-ng server drops the message. <-- This is correct - On the v3.0.8, the message passes through and found in the data file. <-- This is incorrect Since the above methods aren't working, I am asking here "what would be the best way of going about this as it seems my original filtering lines from v2.1 no longer work". -- Michael ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alan McKinnon [Alan.McKinnon@is.co.za] Sent: Monday, September 20, 2010 6:52 PM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? Your "value" is wrong. It's a variable name, not a literal string, so you use it like this: value(MSGONLY) or the cleaner version value(${MSGONLY}) It works like a bash variable in this regard On Tuesday 21 September 2010 00:30:06 Worsham, Michael wrote:
Well I tried the following two lines and neither one works:
filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; Incoming log entry; line='<13>Sep 20 18:26:19 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='MSGONLY' filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; Incoming log entry; line='<13>Sep 20 18:16:15 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='Audit daemon rotating log files' WTF am I doing wrong and please quit quoting URLs to look at.
-- M
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 6:12 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
On Mon, Sep 20, 2010 at 05:23:28PM -0400, Worsham, Michael wrote:
No such value known; value='Audit daemon rotating log files' No such value known; value='last message repeated' No such value known; value='Log statistics'
I believe this output indicates you have the incorrect information in the value argument. The value argument is supposed to be used to indicate which message macro should be checked for the string or regex in question.
So you probably want the value argument to be one of these:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui de-admin-en.html/reference_macros.html
Matthew.
The most interesting ones for your application would be the ones below.
Consider using an output template which outputs the value in each macro, so you can see which macro you should be matching for each of your filter rules.
For example, if you output messages with this template, you would see the value in the MSGONLY macro. You could use a longer version of this to print out all the macros and figure out which should be used for the different matches you are trying to perform.
template t_raw { template("${MSGONLY}\n"); };
MSG or MESSAGE Description: Text contents of the log message without the program name and pid. Note that this has changed in syslog-ng version 3.0; in earlier versions this macro included the program name and the pid. In syslog-ng 3.0, the MSG macro became equivalent with the MSGONLY macro. The program name and the pid together are available in the MSGHDR macro.
MSGHDR Description: The name and the pid of the program that sent the log message in PROGRAM: PID format. Includes a trailing whitespace. Note that the macro returns an empty value if both the program and pid fields of the message are empty.
MSGONLY Description: Message contents without the program name or pid.
PROGRAM
Description: The name of the program sending the message. Note that the content of the $PROGRAM variable may not be completely trusted as it is provided by the client program that constructed the message.
___________________________________________________________________________ ___ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
-- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions +27 11 575 7585 Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did. Sent: Mon Sep 20 2010 17:32:13 GMT-0600 (Mountain Daylight Time) From: Worsham, Michael <mworsham@SCIRES.COM> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Okay, maybe I am not making it clear enough on what it is I am trying to do...
In the older syslong-ng v2.1, this line works perfectly:
filter M_audit { not match("Audit daemon rotating log files"); };
Under 3.0.8, none of the following are working (if added one line at a time) and the daemon restarted:
filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY) flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files")); };
What I am looking to do is if any incoming data has the following "Audit daemon rotating log files" message being detected, it should be filtered (i.e. dropped), and then not show up in the actual message log file.
For example, if I go over on the syslog-ng client and do 'logger daemon': - The older v2.1 syslog-ng server drops the message. <-- This is correct - On the v3.0.8, the message passes through and found in the data file. <-- This is incorrect
Since the above methods aren't working, I am asking here "what would be the best way of going about this as it seems my original filtering lines from v2.1 no longer work".
-- Michael
------------------------------------------------------------------------ *From:* syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alan McKinnon [Alan.McKinnon@is.co.za] *Sent:* Monday, September 20, 2010 6:52 PM *To:* syslog-ng@lists.balabit.hu *Subject:* Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Your "value" is wrong. It's a variable name, not a literal string, so you use it like this:
value(MSGONLY)
or the cleaner version
value(${MSGONLY})
It works like a bash variable in this regard
On Tuesday 21 September 2010 00:30:06 Worsham, Michael wrote:
Well I tried the following two lines and neither one works:
filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; Incoming log entry; line='<13>Sep 20 18:26:19 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='MSGONLY' filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; Incoming log entry; line='<13>Sep 20 18:16:15 drupal root: daemon' Filter rule evaluation begins; filter_rule='M_audit' No such value known; value='Audit daemon rotating log files' WTF am I doing wrong and please quit quoting URLs to look at.
-- M
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 6:12 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
On Mon, Sep 20, 2010 at 05:23:28PM -0400, Worsham, Michael wrote:
No such value known; value='Audit daemon rotating log files' No such value known; value='last message repeated' No such value known; value='Log statistics'
I believe this output indicates you have the incorrect information in the value argument. The value argument is supposed to be used to indicate which message macro should be checked for the string or regex in question.
So you probably want the value argument to be one of these:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui
de-admin-en.html/reference_macros.html
Matthew.
The most interesting ones for your application would be the ones below.
Consider using an output template which outputs the value in each macro, so you can see which macro you should be matching for each of your filter rules.
For example, if you output messages with this template, you would see the value in the MSGONLY macro. You could use a longer version of this to print out all the macros and figure out which should be used for the different matches you are trying to perform.
template t_raw { template("${MSGONLY}\n"); };
MSG or MESSAGE Description: Text contents of the log message without the program name and pid. Note that this has changed in syslog-ng version 3.0; in earlier versions this macro included the program name and the pid. In syslog-ng 3.0, the MSG macro became equivalent with the MSGONLY macro. The program name and the pid together are available in the MSGHDR macro.
MSGHDR Description: The name and the pid of the program that sent the log message in PROGRAM: PID format. Includes a trailing whitespace. Note that the macro returns an empty value if both the program and pid fields of the message are empty.
MSGONLY Description: Message contents without the program name or pid.
PROGRAM
Description: The name of the program sending the message. Note that the content of the $PROGRAM variable may not be completely trusted as it is provided by the client program that constructed the message.
___________________________________________________________________________
___ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
-- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions
+27 11 575 7585
Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ------------------------------------------------------------------------
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Mon, Sep 20, 2010 at 05:44:10PM -0600, syslogng@feystorm.net wrote:
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did.
My only guess so far besides an outright bug: the message is formatted wrong inside the Syslog packet and the packet parser behavior changed from the old version to the new version in such a way that the macros are not being populated with the strings we expect. However I have set up several PCRE filters against message content using 3.1 and have not seen anything broken. So the bug possibility seems unlikely compared to an issue parsing the particular string. It would be helpful if we could get the tshark -V or full Wireshark payload of a message that fails to decode so we could see what was contained in the original packet. Matthew.
Wireshark is going to be a bit impossible as these are servers without front-end displays and without X installed. Strictly console-related VM server instances. Here's a link to my configuration just in case anyone wants to take a gander: http://www.murpe.com/syslog-ng-v3.conf.txt We are using TLS encryption (a requirement) and a destination breakdown (another requirement). Other than that, we just need some simple filtering for keywords that appear hundreds to thousands of times on our many RHEL servers that has SELinux and auditing enabled. -- M ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 7:53 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? On Mon, Sep 20, 2010 at 05:44:10PM -0600, syslogng@feystorm.net wrote:
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did.
My only guess so far besides an outright bug: the message is formatted wrong inside the Syslog packet and the packet parser behavior changed from the old version to the new version in such a way that the macros are not being populated with the strings we expect. However I have set up several PCRE filters against message content using 3.1 and have not seen anything broken. So the bug possibility seems unlikely compared to an issue parsing the particular string. It would be helpful if we could get the tshark -V or full Wireshark payload of a message that fails to decode so we could see what was contained in the original packet. Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
TShark output between the two syslog-ng servers (syslogsvr [192.168.0.80], syslogclt [192.168.0.81]): http://www.murpe.com/syslog-ng-v3.tshark.txt ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 8:06 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? Wireshark is going to be a bit impossible as these are servers without front-end displays and without X installed. Strictly console-related VM server instances. Here's a link to my configuration just in case anyone wants to take a gander: http://www.murpe.com/syslog-ng-v3.conf.txt We are using TLS encryption (a requirement) and a destination breakdown (another requirement). Other than that, we just need some simple filtering for keywords that appear hundreds to thousands of times on our many RHEL servers that has SELinux and auditing enabled. -- M ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 7:53 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? On Mon, Sep 20, 2010 at 05:44:10PM -0600, syslogng@feystorm.net wrote:
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did.
My only guess so far besides an outright bug: the message is formatted wrong inside the Syslog packet and the packet parser behavior changed from the old version to the new version in such a way that the macros are not being populated with the strings we expect. However I have set up several PCRE filters against message content using 3.1 and have not seen anything broken. So the bug possibility seems unlikely compared to an issue parsing the particular string. It would be helpful if we could get the tshark -V or full Wireshark payload of a message that fails to decode so we could see what was contained in the original packet. Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
Sorry to be a pain about it, but as stated in the original mail we'd do better with tshark -V or we can't see the payload of the packets. You also need to figure out an option to make sure the syslogs are decoded as syslogs so we get proper output, because right now they are coming out as RSH packets. You probably want to use this as shown in the manpage, to flag your custom Syslog ports as Syslog for them to decode. -d <layer type>==<selector>,<decode-as protocol> Hopefully we can see what's going on and get to the bottom of this soon for you. Matthew. On Mon, Sep 20, 2010 at 08:24:40PM -0400, Worsham, Michael wrote:
TShark output between the two syslog-ng servers (syslogsvr [192.168.0.80], syslogclt [192.168.0.81]):
http://www.murpe.com/syslog-ng-v3.tshark.txt
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 8:06 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Wireshark is going to be a bit impossible as these are servers without front-end displays and without X installed. Strictly console-related VM server instances.
Here's a link to my configuration just in case anyone wants to take a gander:
http://www.murpe.com/syslog-ng-v3.conf.txt
We are using TLS encryption (a requirement) and a destination breakdown (another requirement). Other than that, we just need some simple filtering for keywords that appear hundreds to thousands of times on our many RHEL servers that has SELinux and auditing enabled.
-- M
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 7:53 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
On Mon, Sep 20, 2010 at 05:44:10PM -0600, syslogng@feystorm.net wrote:
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did.
My only guess so far besides an outright bug: the message is formatted wrong inside the Syslog packet and the packet parser behavior changed from the old version to the new version in such a way that the macros are not being populated with the strings we expect.
However I have set up several PCRE filters against message content using 3.1 and have not seen anything broken. So the bug possibility seems unlikely compared to an issue parsing the particular string.
It would be helpful if we could get the tshark -V or full Wireshark payload of a message that fails to decode so we could see what was contained in the original packet.
Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Alot of what yuou just said went even over my head, so I went ahead and disabled TLS encryption for the stream. I enabled the syslog -d on both sides (server and client), then did a redirect of the tshark -V to a flat file. Then from the client side ran the 'logger daemon' command again to use as a marker to see that the data dump was actually being recorded. Dump is available here: http://www.murpe.com/syslog-ng.tshark-dump.txt As for the port, I am using TCP/514 -- which we are required to use going forward. We can't use upper 1024+ ports, even for this test. ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 9:15 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? Sorry to be a pain about it, but as stated in the original mail we'd do better with tshark -V or we can't see the payload of the packets. You also need to figure out an option to make sure the syslogs are decoded as syslogs so we get proper output, because right now they are coming out as RSH packets. You probably want to use this as shown in the manpage, to flag your custom Syslog ports as Syslog for them to decode. -d <layer type>==<selector>,<decode-as protocol> Hopefully we can see what's going on and get to the bottom of this soon for you. Matthew. On Mon, Sep 20, 2010 at 08:24:40PM -0400, Worsham, Michael wrote:
TShark output between the two syslog-ng servers (syslogsvr [192.168.0.80], syslogclt [192.168.0.81]):
http://www.murpe.com/syslog-ng-v3.tshark.txt
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 8:06 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Wireshark is going to be a bit impossible as these are servers without front-end displays and without X installed. Strictly console-related VM server instances.
Here's a link to my configuration just in case anyone wants to take a gander:
http://www.murpe.com/syslog-ng-v3.conf.txt
We are using TLS encryption (a requirement) and a destination breakdown (another requirement). Other than that, we just need some simple filtering for keywords that appear hundreds to thousands of times on our many RHEL servers that has SELinux and auditing enabled.
-- M
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 7:53 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
On Mon, Sep 20, 2010 at 05:44:10PM -0600, syslogng@feystorm.net wrote:
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did.
My only guess so far besides an outright bug: the message is formatted wrong inside the Syslog packet and the packet parser behavior changed from the old version to the new version in such a way that the macros are not being populated with the strings we expect.
However I have set up several PCRE filters against message content using 3.1 and have not seen anything broken. So the bug possibility seems unlikely compared to an issue parsing the particular string.
It would be helpful if we could get the tshark -V or full Wireshark payload of a message that fails to decode so we could see what was contained in the original packet.
Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
It seems tshark won't decode-as under tcp format for syslog, only udp, so I had to change my syslog-ng.conf on both the client and server sides. Then I figured out the redirect to use the decode-as protocol syslog: tshark -V -d udp.port==514,syslog >> dumpfile.txt Ran the 'logger daemon' command from the syslog-ng client to use as a marker. New dump file is available here: http://www.murpe.com/syslog-ng.tshark-dump.txt -- M ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 9:45 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? Alot of what yuou just said went even over my head, so I went ahead and disabled TLS encryption for the stream. I enabled the syslog -d on both sides (server and client), then did a redirect of the tshark -V to a flat file. Then from the client side ran the 'logger daemon' command again to use as a marker to see that the data dump was actually being recorded. Dump is available here: http://www.murpe.com/syslog-ng.tshark-dump.txt As for the port, I am using TCP/514 -- which we are required to use going forward. We can't use upper 1024+ ports, even for this test. ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 9:15 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? Sorry to be a pain about it, but as stated in the original mail we'd do better with tshark -V or we can't see the payload of the packets. You also need to figure out an option to make sure the syslogs are decoded as syslogs so we get proper output, because right now they are coming out as RSH packets. You probably want to use this as shown in the manpage, to flag your custom Syslog ports as Syslog for them to decode. -d <layer type>==<selector>,<decode-as protocol> Hopefully we can see what's going on and get to the bottom of this soon for you. Matthew. On Mon, Sep 20, 2010 at 08:24:40PM -0400, Worsham, Michael wrote:
TShark output between the two syslog-ng servers (syslogsvr [192.168.0.80], syslogclt [192.168.0.81]):
http://www.murpe.com/syslog-ng-v3.tshark.txt
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 8:06 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Wireshark is going to be a bit impossible as these are servers without front-end displays and without X installed. Strictly console-related VM server instances.
Here's a link to my configuration just in case anyone wants to take a gander:
http://www.murpe.com/syslog-ng-v3.conf.txt
We are using TLS encryption (a requirement) and a destination breakdown (another requirement). Other than that, we just need some simple filtering for keywords that appear hundreds to thousands of times on our many RHEL servers that has SELinux and auditing enabled.
-- M
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 7:53 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
On Mon, Sep 20, 2010 at 05:44:10PM -0600, syslogng@feystorm.net wrote:
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did.
My only guess so far besides an outright bug: the message is formatted wrong inside the Syslog packet and the packet parser behavior changed from the old version to the new version in such a way that the macros are not being populated with the strings we expect.
However I have set up several PCRE filters against message content using 3.1 and have not seen anything broken. So the bug possibility seems unlikely compared to an issue parsing the particular string.
It would be helpful if we could get the tshark -V or full Wireshark payload of a message that fails to decode so we could see what was contained in the original packet.
Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
On Mon, Sep 20, 2010 at 10:25:29PM -0400, Worsham, Michael wrote:
It seems tshark won't decode-as under tcp format for syslog, only udp, so I had to change my syslog-ng.conf on both the client and server sides. Then I figured out the redirect to use the decode-as protocol syslog:
If that tshark problem happened in a recent version it might be worth reporting a bug. As people come to depend more and more on TCP Syslog due to reliability and TLS issues this will be an important feature.
tshark -V -d udp.port==514,syslog >> dumpfile.txt
Ran the 'logger daemon' command from the syslog-ng client to use as a marker.
New dump file is available here: http://www.murpe.com/syslog-ng.tshark-dump.txt
-- M
The interesting part is here. Let's hope somebody can think this over tomorrow during Europe / East US time and help formulate a proper answer. To me the message appears to be proper RFC 3164 BSD syslog format which should work OK. http://www.ietf.org/rfc/rfc3164.txt Syslog message: USER.NOTICE: Sep 20 22:19:30 drupal root: daemon\n 0000 1... = Facility: USER - random user-level messages (1) .... .101 = Level: NOTICE - normal but significant condition (5) Message: Sep 20 22:19:30 drupal root: daemon\n Matthew.
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 9:45 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Alot of what yuou just said went even over my head, so I went ahead and disabled TLS encryption for the stream.
I enabled the syslog -d on both sides (server and client), then did a redirect of the tshark -V to a flat file. Then from the client side ran the 'logger daemon' command again to use as a marker to see that the data dump was actually being recorded.
Dump is available here: http://www.murpe.com/syslog-ng.tshark-dump.txt
As for the port, I am using TCP/514 -- which we are required to use going forward. We can't use upper 1024+ ports, even for this test. ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 9:15 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Sorry to be a pain about it, but as stated in the original mail we'd do better with tshark -V or we can't see the payload of the packets.
You also need to figure out an option to make sure the syslogs are decoded as syslogs so we get proper output, because right now they are coming out as RSH packets. You probably want to use this as shown in the manpage, to flag your custom Syslog ports as Syslog for them to decode.
-d <layer type>==<selector>,<decode-as protocol>
Hopefully we can see what's going on and get to the bottom of this soon for you.
Matthew.
On Mon, Sep 20, 2010 at 08:24:40PM -0400, Worsham, Michael wrote:
TShark output between the two syslog-ng servers (syslogsvr [192.168.0.80], syslogclt [192.168.0.81]):
http://www.murpe.com/syslog-ng-v3.tshark.txt
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 8:06 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Wireshark is going to be a bit impossible as these are servers without front-end displays and without X installed. Strictly console-related VM server instances.
Here's a link to my configuration just in case anyone wants to take a gander:
http://www.murpe.com/syslog-ng-v3.conf.txt
We are using TLS encryption (a requirement) and a destination breakdown (another requirement). Other than that, we just need some simple filtering for keywords that appear hundreds to thousands of times on our many RHEL servers that has SELinux and auditing enabled.
-- M
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 7:53 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
On Mon, Sep 20, 2010 at 05:44:10PM -0600, syslogng@feystorm.net wrote:
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did.
My only guess so far besides an outright bug: the message is formatted wrong inside the Syslog packet and the packet parser behavior changed from the old version to the new version in such a way that the macros are not being populated with the strings we expect.
However I have set up several PCRE filters against message content using 3.1 and have not seen anything broken. So the bug possibility seems unlikely compared to an issue parsing the particular string.
It would be helpful if we could get the tshark -V or full Wireshark payload of a message that fails to decode so we could see what was contained in the original packet.
Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Could the filtering be an issue with the v3.0.8 build and I should be looking at the v.3.1.x builds instead? -- M ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 10:58 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0? On Mon, Sep 20, 2010 at 10:25:29PM -0400, Worsham, Michael wrote:
It seems tshark won't decode-as under tcp format for syslog, only udp, so I had to change my syslog-ng.conf on both the client and server sides. Then I figured out the redirect to use the decode-as protocol syslog:
If that tshark problem happened in a recent version it might be worth reporting a bug. As people come to depend more and more on TCP Syslog due to reliability and TLS issues this will be an important feature.
tshark -V -d udp.port==514,syslog >> dumpfile.txt
Ran the 'logger daemon' command from the syslog-ng client to use as a marker.
New dump file is available here: http://www.murpe.com/syslog-ng.tshark-dump.txt
-- M
The interesting part is here. Let's hope somebody can think this over tomorrow during Europe / East US time and help formulate a proper answer. To me the message appears to be proper RFC 3164 BSD syslog format which should work OK. http://www.ietf.org/rfc/rfc3164.txt Syslog message: USER.NOTICE: Sep 20 22:19:30 drupal root: daemon\n 0000 1... = Facility: USER - random user-level messages (1) .... .101 = Level: NOTICE - normal but significant condition (5) Message: Sep 20 22:19:30 drupal root: daemon\n Matthew.
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 9:45 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Alot of what yuou just said went even over my head, so I went ahead and disabled TLS encryption for the stream.
I enabled the syslog -d on both sides (server and client), then did a redirect of the tshark -V to a flat file. Then from the client side ran the 'logger daemon' command again to use as a marker to see that the data dump was actually being recorded.
Dump is available here: http://www.murpe.com/syslog-ng.tshark-dump.txt
As for the port, I am using TCP/514 -- which we are required to use going forward. We can't use upper 1024+ ports, even for this test. ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 9:15 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Sorry to be a pain about it, but as stated in the original mail we'd do better with tshark -V or we can't see the payload of the packets.
You also need to figure out an option to make sure the syslogs are decoded as syslogs so we get proper output, because right now they are coming out as RSH packets. You probably want to use this as shown in the manpage, to flag your custom Syslog ports as Syslog for them to decode.
-d <layer type>==<selector>,<decode-as protocol>
Hopefully we can see what's going on and get to the bottom of this soon for you.
Matthew.
On Mon, Sep 20, 2010 at 08:24:40PM -0400, Worsham, Michael wrote:
TShark output between the two syslog-ng servers (syslogsvr [192.168.0.80], syslogclt [192.168.0.81]):
http://www.murpe.com/syslog-ng-v3.tshark.txt
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Monday, September 20, 2010 8:06 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
Wireshark is going to be a bit impossible as these are servers without front-end displays and without X installed. Strictly console-related VM server instances.
Here's a link to my configuration just in case anyone wants to take a gander:
http://www.murpe.com/syslog-ng-v3.conf.txt
We are using TLS encryption (a requirement) and a destination breakdown (another requirement). Other than that, we just need some simple filtering for keywords that appear hundreds to thousands of times on our many RHEL servers that has SELinux and auditing enabled.
-- M
________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Matthew Hall [mhall@mhcomputing.net] Sent: Monday, September 20, 2010 7:53 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Converting filtering from 2.1 to 3.0?
On Mon, Sep 20, 2010 at 05:44:10PM -0600, syslogng@feystorm.net wrote:
Your first line should be working. Not sure why it is not. However you can try using: not message('Audit daemon rotating log files' flags('ignore-case')) Simpler and does exactly what your old config did.
My only guess so far besides an outright bug: the message is formatted wrong inside the Syslog packet and the packet parser behavior changed from the old version to the new version in such a way that the macros are not being populated with the strings we expect.
However I have set up several PCRE filters against message content using 3.1 and have not seen anything broken. So the bug possibility seems unlikely compared to an issue parsing the particular string.
It would be helpful if we could get the tshark -V or full Wireshark payload of a message that fails to decode so we could see what was contained in the original packet.
Matthew. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
participants (5)
-
Alan McKinnon
-
Balazs Scheidler
-
Matthew Hall
-
syslogng＠feystorm.net
-
Worsham, Michael