[syslog-ng] RE syslog-ng OSE parsing

Tue Jul 10 05:58:32 UTC 2018

Hey Bazsi,

Sorry for the delayed reply, have been too busy to run a capture.

I see you updated the cisco-parser() for the timestamp on 26th June.
Hopefully this helps for the host etc.

As it comes in:

35: Jul 10 2018 12:11:21 AM.371 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login Date/Time=07/10/2018 at 10:11:21][Login IP Address/Hostname=10.29.128.82][Login Interface=cucm-uds][Login UserID=U8005499][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.

Kind Regards,

Daniel Ehrlich
ICT Security Officer
Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>

From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Scheidler, Balázs
Sent: Wednesday, 20 June 2018 7:42 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] RE syslog-ng OSE parsing

Hi,

This format seems quite out of the ordinary, which the cisco parser wouldn't cope with at least today. There would be two level of support for this format:

  1.  at the very least, process date/host and message properly
  2.  parse the name-value pairs in brackets (e.g. Login Date/Time)
The first one should be doable by extending the cisco parser. But before going there, can you pls reproduce the log message character-by-character how syslog-ng receives it? In the original example, I think an extra space is inserted between the timestamp and the following colon. Cisco logs tend to be ": " separater (e.g. a colon and a space). If that assumption is true, then support for this can be added by changing cisco-timestamp-parser().

--
Bazsi

On Wed, Jun 20, 2018 at 7:45 AM, Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>> wrote:
Hi Bazsi and Gabor,

Bazsi, so maybe I should update syslog-ng and then try the cisco parser?

The logs are coming from Cisco Unified Call Manager logs, running v10.5.2.

Kind Regards,

Daniel Ehrlich
ICT Security Officer
Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>

From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>] On Behalf Of Nagy, Gábor
Sent: Tuesday, 19 June 2018 9:05 PM

To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] RE syslog-ng OSE parsing

Hi Daniel!

> ... how to filter these logs before applying that parser to them?
It really depends on the format of your log messages, especially on the non-compliant ones.

Of course if they have a well-defined structure you can use a specific filter on these messages and then you can use the parser.
To process log messages differently in a log path you can use the new if-elif branching in your config (or the previous junction-channel setup), which is released in version 3.15.
So you can parse the non-complaint messages only, and the rest can go through the usual path.
The if-else branching will be documented in the upcoming documentation, until then you can read the PR about the feature which has a description.
https://github.com/balabit/syslog-ng/pull/1856#issuecomment-369049863

An example would be:
source{ ... flags(no-parse); };
if(filter(f_non_compliant)) { parser { kv-parser(); }; }
else {};
destination {....};

I know I have missed the definition of "f_non_compliant", but I don't know what would be a good filter, hence not answering your original question. :)
I would like to improve our cisco parser if I prove that this is a Cisco format:
> Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
What kind of device is sending this log message?

Regards,
Gabor

On Tue, Jun 19, 2018 at 8:13 AM, Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>> wrote:
Thanks Gabor,

I’m wondering if you have any suggestion on how to filter these logs before applying that parser to them?

I have numerous sources coming in on the udp 514 listener. Then logging them to files which Splunk reads.

Below is my current config….:
source s_network {
   udp(port(514));
};

# Parser

# Rewrite

#Destinations
destination d_files_splunk {
   file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_dirs(yes));
};
destination d_files_seamail {
   file("/opt/splunk/var/lib/splunk/syslog-ng/seamail/$MONTH$DAY.log" create_dirs(yes));
};

# Filters
filter seamail {
   host("q=*" type(glob));
};
filter splunk {
   not (filter(seamail));
};

# Log
log {
   source(s_network);
   filter(splunk);
   destination(d_files_splunk);
};
log {
   source(s_network);
   filter(seamail);
   destination(d_files_seamail);
};

Kind Regards,

Daniel Ehrlich
ICT Security Officer
Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>

From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>] On Behalf Of Nagy, Gábor
Sent: Friday, 15 June 2018 6:20 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] RE syslog-ng OSE parsing

Hi Daniel!

Your log does not conform to either RFC-3164 or RFC-5424, it looks like kind-of cisco format to me.
I've tried one or two ideas, using our `default-network-drivers()` which includes the cisco-parser() but it is not dealing with it properly.

On second try, I've experimented with `kv-parser()` so the key-value parts in the message will be parsed in a structured format and after that you can use a rewrite rule to modify HOST field of the message.
I had to adjust the pair-separator option to get an optimal result.
`parser { kv-parser( prefix(".kv.") pair-separator("]")); };`

After a successful parsing, you can use a rewrite rule to set the HOSTNAME.
rewrite { set("${.kv.ID}" value("HOST")); };
There is one catch: you need to know the name of the key: "Node ID" or "Hostname", which comes from the log message.
So, if this expression is not fix, then your config will not work.

You can also see that I've added the key "ID" rather than "Node ID", since your log message does not quote the keys in it and
in this case kv-parser will only use the "ID" from "Node ID" part which leads to that similar key-values are overwritten: "App ID", "Cluster ID"

I will think about it, but hopefully others will come up with a better idea. :)

Regards,
Gabor

On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>> wrote:
Hi,

Hoping you can assist me, I haven’t really come across anything that makes full sense to me in my searching of various sites/forums.

This is an example log.
Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.

Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as cucmsub-prd-t2 ? i.e. Node ID=

Thanks you

Kind Regards,

Daniel Ehrlich
MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA
ICT Security Officer
ICT Client Services|Infrastructure Services
Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>
Toowoomba | Queensland | 4350 | Australia

_____________________________________________________________

This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government.

(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>> wrote:
Hi,

Hoping you can assist me, I haven’t really come across anything that makes full sense to me in my searching of various sites/forums.

This is an example log.
Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.

Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as cucmsub-prd-t2 ? i.e. Node ID=

Thanks you

Kind Regards,

Daniel Ehrlich
MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA
ICT Security Officer
ICT Client Services|Infrastructure Services
Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au<mailto:Daniel.Ehrlich at usq.edu.au>
Toowoomba | Queensland | 4350 | Australia

_____________________________________________________________

This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government.

(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

_____________________________________________________________

This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government.

(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

_____________________________________________________________

This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government.

(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

_____________________________________________________________
This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government.
(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180710/4ae74d22/attachment-0001.html>