Program() in destination driver not working for Macros defined in CSV-PARSER
Hello On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); }; I want to launch a script whenever the UDP messages are in.So I defined the destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; script is working fine with other destination drivers. Please help -- Kale
What do you get in your script? Empty fields? The program destination has to be on a direct log path subsequent to the parser. On Mar 4, 2014 7:27 PM, "Justin B" <justinkala@gmail.com> wrote:
Hello
On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); };
I want to launch a script whenever the UDP messages are in.So I defined the
destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); };
script is working fine with other destination drivers. Please help -- Kale
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Rereading my answer I may not have been completely clear. Make sure that your parser and the program destination are actually connected on the same log path. E.g. log { source(...); parser(p_apache); destination(d_program); }; If your apache parser is on an independent log path, it won't work: log { source(...); parser(p_apache); destination(...); }; log { source(...); destination(d_program); }; e.g. independent log paths will not inherit from the values set by the earlier log statements. changes are only propagated through the same pipeline. Hope this helps. On Fri, Mar 7, 2014 at 7:32 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
What do you get in your script? Empty fields?
The program destination has to be on a direct log path subsequent to the parser. On Mar 4, 2014 7:27 PM, "Justin B" <justinkala@gmail.com> wrote:
Hello
On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); };
I want to launch a script whenever the UDP messages are in.So I defined the
destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); };
script is working fine with other destination drivers. Please help -- Kale
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
My recollection is that macros are not expanded for program names. Whant you get is the environment variables from the shell that is used to start your program, so in most cases this will be empty. I think this is dangerous and did mantion it on the list previously. On 03/06/2014 10:32 PM, Balazs Scheidler wrote:
What do you get in your script? Empty fields?
The program destination has to be on a direct log path subsequent to the parser.
On Mar 4, 2014 7:27 PM, "Justin B" <justinkala@gmail.com <mailto:justinkala@gmail.com>> wrote:
Hello On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); }; I want to launch a script whenever the UDP messages are in.So I defined the destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; script is working fine with other destination drivers. Please help -- Kale
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
Yes Empty Fields. I am using this log path log { source (remote); filter (f_messages); parser (p_apache);destination (r_messages); }; Are you saying the I cannot pass the Macro values to a shell script through Program on destination driver?? On Fri, Mar 7, 2014 at 2:47 PM, Evan Rempel <erempel@uvic.ca> wrote:
My recollection is that macros are not expanded for program names. Whant you get is the environment variables from the shell that is used to start your program, so in most cases this will be empty.
I think this is dangerous and did mantion it on the list previously.
On 03/06/2014 10:32 PM, Balazs Scheidler wrote:
What do you get in your script? Empty fields?
The program destination has to be on a direct log path subsequent to the parser.
On Mar 4, 2014 7:27 PM, "Justin B" <justinkala@gmail.com <mailto: justinkala@gmail.com>> wrote:
Hello On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); }; I want to launch a script whenever the UDP messages are in.So I defined the destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; script is working fine with other destination drivers. Please help -- Kale
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
You can on stdin but not as arguments. On Mar 7, 2014 9:24 PM, "Justin B" <justinkala@gmail.com> wrote:
Yes Empty Fields.
I am using this log path log { source (remote); filter (f_messages); parser (p_apache);destination (r_messages); };
Are you saying the I cannot pass the Macro values to a shell script through Program on destination driver??
On Fri, Mar 7, 2014 at 2:47 PM, Evan Rempel <erempel@uvic.ca> wrote:
My recollection is that macros are not expanded for program names. Whant you get is the environment variables from the shell that is used to start your program, so in most cases this will be empty.
I think this is dangerous and did mantion it on the list previously.
On 03/06/2014 10:32 PM, Balazs Scheidler wrote:
What do you get in your script? Empty fields?
The program destination has to be on a direct log path subsequent to the parser.
On Mar 4, 2014 7:27 PM, "Justin B" <justinkala@gmail.com <mailto: justinkala@gmail.com>> wrote:
Hello On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); }; I want to launch a script whenever the UDP messages are in.So I defined the destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; script is working fine with other destination drivers. Please help -- Kale
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Can you share me a sample of the shell script that I can write to read the message lines that are being passed and extract those values into a different variables and output them into output log files?? On Fri, Mar 7, 2014 at 4:10 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
You can on stdin but not as arguments. On Mar 7, 2014 9:24 PM, "Justin B" <justinkala@gmail.com> wrote:
Yes Empty Fields.
I am using this log path log { source (remote); filter (f_messages); parser (p_apache);destination (r_messages); };
Are you saying the I cannot pass the Macro values to a shell script through Program on destination driver??
On Fri, Mar 7, 2014 at 2:47 PM, Evan Rempel <erempel@uvic.ca> wrote:
My recollection is that macros are not expanded for program names. Whant you get is the environment variables from the shell that is used to start your program, so in most cases this will be empty.
I think this is dangerous and did mantion it on the list previously.
On 03/06/2014 10:32 PM, Balazs Scheidler wrote:
What do you get in your script? Empty fields?
The program destination has to be on a direct log path subsequent to the parser.
On Mar 4, 2014 7:27 PM, "Justin B" <justinkala@gmail.com <mailto: justinkala@gmail.com>> wrote:
Hello On My Apache logs I applied csv_parser() and defined the Macros. parser p_apache { csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") delimiters("|") ); }; I want to launch a script whenever the UDP messages are in.So I defined the destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; script is working fine with other destination drivers. Please help -- Kale
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Kaladhar
I don't have anything quite as elaborate as what you have, but here is an example I have running in my production environment template t_rule_id { template("${.classifier.rule_id}\n"); template_escape(no); }; destination d_msgid_profiler { program("/opt/flare/bin/msgid_profiler" template(t_rule_id) ); }; I use a patterndb message parser to end up with the macro .classifier.rule_id and then send that to my program. Basically what you have done. One of the things that I do to troubleshoot things like this is to make a destination such as template t_json { template("$(format-json --scope everything )\n"); template_escape(no); }; destination d_syslog_json { file("/var/log/syslog-json.log" owner("root") group("syslogs") perm(0640) template(t_json)); }; and then add this destination right beside the one that you are having problems with. log { source(your_source); destination(d_mesg); destination(d_syslog_json); }; and then you can look at all of the macros that are present in the /var/log/syslog-json.log file and verify what you think is happening. To look at the json object in an easy to use manner, see http://jsonprettyprint.com/ Evan. On 03/09/2014 04:50 PM, Justin B wrote:
Can you share me a sample of the shell script that I can write to read the message lines that are being passed and extract those values into a different variables and output them into output log files??
On Fri, Mar 7, 2014 at 4:10 PM, Balazs Scheidler <bazsi77@gmail.com <mailto:bazsi77@gmail.com>> wrote:
You can on stdin but not as arguments.
On Mar 7, 2014 9:24 PM, "Justin B" <justinkala@gmail.com <mailto:justinkala@gmail.com>> wrote:
Yes Empty Fields. I am using this log path log { source (remote); filter (f_messages); parser (p_apache);destination (r_messages); }; Are you saying the I cannot pass the Macro values to a shell script through Program on destination driver??
On Fri, Mar 7, 2014 at 2:47 PM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
My recollection is that macros are not expanded for program names. Whant you get is the environment variables from the shell that is used to start your program, so in most cases this will be empty.
I think this is dangerous and did mantion it on the list previously.
On 03/06/2014 10:32 PM, Balazs Scheidler wrote: > What do you get in your script? Empty fields? > > The program destination has to be on a direct log path subsequent to the parser. > > On Mar 4, 2014 7:27 PM, "Justin B" <justinkala@gmail.com <mailto:justinkala@gmail.com> <mailto:justinkala@gmail.com <mailto:justinkala@gmail.com>>> wrote: > > > Hello > On My Apache logs I applied csv_parser() and defined the Macros. > parser p_apache { > csv-parser(columns("apache.ETSTAMP", "apache.TYPE", "apache.EHOSTNAME","apache.ESOURCE", "apache.EOUTCOME", "apache.EMSG","apache.EUSERID") > delimiters("|") ); > }; > I want to launch a script whenever the UDP messages are in.So I defined the > destination d_mesg (program("/tmp/test.sh" template("|${apache.ETSTAMP}|${apache.TYPE}|${apache.EHOSTNAME}|${apache.ESOURCE}|${apache.EOUTCOME}|${apache.EMSG}|${apache.EUSERID}\n")); }; > script is working fine with other destination drivers. Please help > -- > Kale
participants (3)
-
Balazs Scheidler
-
Evan Rempel
-
Justin B