[syslog-ng] RE syslog-ng OSE parsing

Tue Jun 19 11:05:09 UTC 2018

Hi Daniel!

> ... how to filter these logs before applying that parser to them?
It really depends on the format of your log messages, especially on the
non-compliant ones.

Of course if they have a well-defined structure you can use a specific
filter on these messages and then you can use the parser.
To process log messages differently in a log path you can use the new
if-elif branching in your config (or the previous junction-channel setup),
which is released in version 3.15.
So you can parse the non-complaint messages only, and the rest can go
through the usual path.
The if-else branching will be documented in the upcoming documentation,
until then you can read the PR about the feature which has a description.
https://github.com/balabit/syslog-ng/pull/1856#issuecomment-369049863

An example would be:
source{ ... flags(no-parse); };
if(filter(f_non_compliant)) { parser { kv-parser(); }; }
else {};
destination {....};

I know I have missed the definition of "f_non_compliant", but I don't know
what would be a good filter, hence not answering your original question. :)
I would like to improve our cisco parser if I prove that this is a Cisco
format:
> Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login
Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login
Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster
ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
What kind of device is sending this log message?

Regards,
Gabor

On Tue, Jun 19, 2018 at 8:13 AM, Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au>
wrote:

> Thanks Gabor,
>
>
>
> I’m wondering if you have any suggestion on how to filter these logs
> before applying that parser to them?
>
>
>
> I have numerous sources coming in on the udp 514 listener. Then logging
> them to files which Splunk reads.
>
>
>
> Below is my current config….:
>
> source s_network {
>
>    udp(port(514));
>
> };
>
>
>
> # Parser
>
>
>
> # Rewrite
>
>
>
> #Destinations
>
> destination d_files_splunk {
>
>    file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log"
> create_dirs(yes));
>
> };
>
> destination d_files_seamail {
>
>    file("/opt/splunk/var/lib/splunk/syslog-ng/seamail/$MONTH$DAY.log"
> create_dirs(yes));
>
> };
>
>
>
> # Filters
>
> filter seamail {
>
>    host("q=*" type(glob));
>
> };
>
> filter splunk {
>
>    not (filter(seamail));
>
> };
>
>
>
> # Log
>
> log {
>
>    source(s_network);
>
>    filter(splunk);
>
>    destination(d_files_splunk);
>
> };
>
> log {
>
>    source(s_network);
>
>    filter(seamail);
>
>    destination(d_files_seamail);
>
> };
>
>
>
> Kind Regards,
>
>
>
> *Daniel Ehrlich*
>
> ICT Security Officer
> Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au
>
>
>
> *From:* syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] *On Behalf
> Of *Nagy, Gábor
> *Sent:* Friday, 15 June 2018 6:20 PM
> *To:* Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> *Subject:* Re: [syslog-ng] RE syslog-ng OSE parsing
>
>
>
> Hi Daniel!
>
>
>
> Your log does not conform to either RFC-3164 or RFC-5424, it looks like
> kind-of cisco format to me.
>
> I've tried one or two ideas, using our `default-network-drivers()` which
> includes the cisco-parser() but it is not dealing with it properly.
>
>
>
> On second try, I've experimented with `kv-parser()` so the key-value parts
> in the message will be parsed in a structured format and after that you can
> use a rewrite rule to modify HOST field of the message.
>
> I had to adjust the pair-separator option to get an optimal result.
>
> `parser { kv-parser( prefix(".kv.") pair-separator("]")); };`
>
>
>
> After a successful parsing, you can use a rewrite rule to set the HOSTNAME.
>
> rewrite { set("${.kv.ID}" value("HOST")); };
>
> There is one catch: you need to know the name of the key: "Node ID" or
> "Hostname", which comes from the log message.
>
> So, if this expression is not fix, then your config will not work.
>
>
>
> You can also see that I've added the key "ID" rather than "Node ID", since
> your log message does not quote the keys in it and
>
> in this case kv-parser will only use the "ID" from "Node ID" part which
> leads to that similar key-values are overwritten: "App ID", "Cluster ID"
>
>
>
> I will think about it, but hopefully others will come up with a better
> idea. :)
>
>
>
> Regards,
>
> Gabor
>
>
>
> On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au>
> wrote:
>
> Hi,
>
>
>
> Hoping you can assist me, I haven’t really come across anything that makes
> full sense to me in my searching of various sites/forums.
>
>
>
> This is an example log.
>
> Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login
> Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login
> Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster
> ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
>
>
>
> Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as
> cucmsub-prd-t2 ? i.e. Node ID=
>
>
>
> Thanks you
>
>
>
> Kind Regards,
>
>
>
> *Daniel Ehrlich*
>
> *MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA*
>
> ICT Security Officer
> ICT Client Services|Infrastructure Services
> Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au
>
> Toowoomba | Queensland | 4350 | Australia
>
>
>
> _____________________________________________________________
>
> This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
>
>
>
> The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
>
>
>
> The University of Southern Queensland is a registered provider of education with the Australian Government.
>
> (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
>
> On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au>
> wrote:
>
> Hi,
>
>
>
> Hoping you can assist me, I haven’t really come across anything that makes
> full sense to me in my searching of various sites/forums.
>
>
>
> This is an example log.
>
> Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login
> Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login
> Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster
> ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
>
>
>
> Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as
> cucmsub-prd-t2 ? i.e. Node ID=
>
>
>
> Thanks you
>
>
>
> Kind Regards,
>
>
>
> *Daniel Ehrlich*
>
> *MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA*
>
> ICT Security Officer
> ICT Client Services|Infrastructure Services
> Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au
>
> Toowoomba | Queensland | 4350 | Australia
>
>
>
> _____________________________________________________________
>
> This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
>
>
>
> The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
>
>
>
> The University of Southern Queensland is a registered provider of education with the Australian Government.
>
> (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> _____________________________________________________________
> This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
>
> The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
>
> The University of Southern Queensland is a registered provider of education with the Australian Government.
> (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180619/d9413809/attachment-0001.html>