[syslog-ng] RE syslog-ng OSE parsing

Fri Jun 22 13:18:15 UTC 2018

 I have added handling of this timestamp into our cisco-parser():

https://github.com/balabit/syslog-ng/pull/2134

-- 
Bazsi

On Wed, Jun 20, 2018 at 11:42 AM, Scheidler, Balázs <
balazs.scheidler at balabit.com> wrote:

> Hi,
>
> This format seems quite out of the ordinary, which the cisco parser
> wouldn't cope with at least today. There would be two level of support for
> this format:
>
>    1. at the very least, process date/host and message properly
>    2. parse the name-value pairs in brackets (e.g. Login Date/Time)
>
> The first one should be doable by extending the cisco parser. But before
> going there, can you pls reproduce the log message character-by-character
> how syslog-ng receives it? In the original example, I think an extra space
> is inserted between the timestamp and the following colon. Cisco logs tend
> to be ": " separater (e.g. a colon and a space). If that assumption is
> true, then support for this can be added by changing
> cisco-timestamp-parser().
>
>
>
> --
> Bazsi
>
> On Wed, Jun 20, 2018 at 7:45 AM, Daniel Ehrlich <Daniel.Ehrlich at usq.edu.au
> > wrote:
>
>> Hi Bazsi and Gabor,
>>
>>
>>
>> Bazsi, so maybe I should update syslog-ng and then try the cisco parser?
>>
>>
>>
>> The logs are coming from Cisco Unified Call Manager logs, running v10.5.2.
>>
>>
>>
>> Kind Regards,
>>
>>
>>
>> *Daniel Ehrlich*
>>
>> ICT Security Officer
>> Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au
>>
>>
>>
>> *From:* syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] *On Behalf
>> Of *Nagy, Gábor
>> *Sent:* Tuesday, 19 June 2018 9:05 PM
>>
>> *To:* Syslog-ng users' and developers' mailing list <
>> syslog-ng at lists.balabit.hu>
>> *Subject:* Re: [syslog-ng] RE syslog-ng OSE parsing
>>
>>
>>
>> Hi Daniel!
>>
>> > ... how to filter these logs before applying that parser to them?
>> It really depends on the format of your log messages, especially on the
>> non-compliant ones.
>>
>> Of course if they have a well-defined structure you can use a specific
>> filter on these messages and then you can use the parser.
>>
>> To process log messages differently in a log path you can use the new
>> if-elif branching in your config (or the previous junction-channel setup),
>> which is released in version 3.15.
>>
>> So you can parse the non-complaint messages only, and the rest can go
>> through the usual path.
>>
>> The if-else branching will be documented in the upcoming documentation,
>> until then you can read the PR about the feature which has a description.
>>
>> https://github.com/balabit/syslog-ng/pull/1856#issuecomment-369049863
>>
>>
>>
>> An example would be:
>>
>> source{ ... flags(no-parse); };
>>
>> if(filter(f_non_compliant)) { parser { kv-parser(); }; }
>>
>> else {};
>>
>> destination {....};
>>
>>
>>
>> I know I have missed the definition of "f_non_compliant", but I don't
>> know what would be a good filter, hence not answering your original
>> question. :)
>>
>> I would like to improve our cisco parser if I prove that this is a Cisco
>> format:
>>
>> > Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed:
>> %[Login Date/Time=06/15/2018 at 09:57:27][Login IP
>> Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login
>> UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]:
>> Login Authentication failed.
>>
>> What kind of device is sending this log message?
>>
>>
>>
>> Regards,
>>
>> Gabor
>>
>>
>>
>> On Tue, Jun 19, 2018 at 8:13 AM, Daniel Ehrlich <
>> Daniel.Ehrlich at usq.edu.au> wrote:
>>
>> Thanks Gabor,
>>
>>
>>
>> I’m wondering if you have any suggestion on how to filter these logs
>> before applying that parser to them?
>>
>>
>>
>> I have numerous sources coming in on the udp 514 listener. Then logging
>> them to files which Splunk reads.
>>
>>
>>
>> Below is my current config….:
>>
>> source s_network {
>>
>>    udp(port(514));
>>
>> };
>>
>>
>>
>> # Parser
>>
>>
>>
>> # Rewrite
>>
>>
>>
>> #Destinations
>>
>> destination d_files_splunk {
>>
>>    file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log"
>> create_dirs(yes));
>>
>> };
>>
>> destination d_files_seamail {
>>
>>    file("/opt/splunk/var/lib/splunk/syslog-ng/seamail/$MONTH$DAY.log"
>> create_dirs(yes));
>>
>> };
>>
>>
>>
>> # Filters
>>
>> filter seamail {
>>
>>    host("q=*" type(glob));
>>
>> };
>>
>> filter splunk {
>>
>>    not (filter(seamail));
>>
>> };
>>
>>
>>
>> # Log
>>
>> log {
>>
>>    source(s_network);
>>
>>    filter(splunk);
>>
>>    destination(d_files_splunk);
>>
>> };
>>
>> log {
>>
>>    source(s_network);
>>
>>    filter(seamail);
>>
>>    destination(d_files_seamail);
>>
>> };
>>
>>
>>
>> Kind Regards,
>>
>>
>>
>> *Daniel Ehrlich*
>>
>> ICT Security Officer
>> Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au
>>
>>
>>
>> *From:* syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] *On Behalf
>> Of *Nagy, Gábor
>> *Sent:* Friday, 15 June 2018 6:20 PM
>> *To:* Syslog-ng users' and developers' mailing list <
>> syslog-ng at lists.balabit.hu>
>> *Subject:* Re: [syslog-ng] RE syslog-ng OSE parsing
>>
>>
>>
>> Hi Daniel!
>>
>>
>>
>> Your log does not conform to either RFC-3164 or RFC-5424, it looks like
>> kind-of cisco format to me.
>>
>> I've tried one or two ideas, using our `default-network-drivers()` which
>> includes the cisco-parser() but it is not dealing with it properly.
>>
>>
>>
>> On second try, I've experimented with `kv-parser()` so the key-value
>> parts in the message will be parsed in a structured format and after that
>> you can use a rewrite rule to modify HOST field of the message.
>>
>> I had to adjust the pair-separator option to get an optimal result.
>>
>> `parser { kv-parser( prefix(".kv.") pair-separator("]")); };`
>>
>>
>>
>> After a successful parsing, you can use a rewrite rule to set the
>> HOSTNAME.
>>
>> rewrite { set("${.kv.ID}" value("HOST")); };
>>
>> There is one catch: you need to know the name of the key: "Node ID" or
>> "Hostname", which comes from the log message.
>>
>> So, if this expression is not fix, then your config will not work.
>>
>>
>>
>> You can also see that I've added the key "ID" rather than "Node ID",
>> since your log message does not quote the keys in it and
>>
>> in this case kv-parser will only use the "ID" from "Node ID" part which
>> leads to that similar key-values are overwritten: "App ID", "Cluster ID"
>>
>>
>>
>> I will think about it, but hopefully others will come up with a better
>> idea. :)
>>
>>
>>
>> Regards,
>>
>> Gabor
>>
>>
>>
>> On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <
>> Daniel.Ehrlich at usq.edu.au> wrote:
>>
>> Hi,
>>
>>
>>
>> Hoping you can assist me, I haven’t really come across anything that
>> makes full sense to me in my searching of various sites/forums.
>>
>>
>>
>> This is an example log.
>>
>> Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login
>> Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login
>> Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster
>> ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
>>
>>
>>
>> Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as
>> cucmsub-prd-t2 ? i.e. Node ID=
>>
>>
>>
>> Thanks you
>>
>>
>>
>> Kind Regards,
>>
>>
>>
>> *Daniel Ehrlich*
>>
>> *MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA*
>>
>> ICT Security Officer
>> ICT Client Services|Infrastructure Services
>> Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au
>>
>> Toowoomba | Queensland | 4350 | Australia
>>
>>
>>
>> _____________________________________________________________
>>
>> This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
>>
>>
>>
>> The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
>>
>>
>>
>> The University of Southern Queensland is a registered provider of education with the Australian Government.
>>
>> (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>>
>>
>> On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <
>> Daniel.Ehrlich at usq.edu.au> wrote:
>>
>> Hi,
>>
>>
>>
>> Hoping you can assist me, I haven’t really come across anything that
>> makes full sense to me in my searching of various sites/forums.
>>
>>
>>
>> This is an example log.
>>
>> Jun 14 11:57:27 PM.685 UTC :  %UC_LOGIN-4-AuthenticationFailed: %[Login
>> Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login
>> Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster
>> ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
>>
>>
>>
>> Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as
>> cucmsub-prd-t2 ? i.e. Node ID=
>>
>>
>>
>> Thanks you
>>
>>
>>
>> Kind Regards,
>>
>>
>>
>> *Daniel Ehrlich*
>>
>> *MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA*
>>
>> ICT Security Officer
>> ICT Client Services|Infrastructure Services
>> Phone: +61 7 4687 5600 Email: Daniel.Ehrlich at usq.edu.au
>>
>> Toowoomba | Queensland | 4350 | Australia
>>
>>
>>
>> _____________________________________________________________
>>
>> This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
>>
>>
>>
>> The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
>>
>>
>>
>> The University of Southern Queensland is a registered provider of education with the Australian Government.
>>
>> (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>> _____________________________________________________________
>>
>> This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
>>
>>
>>
>> The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
>>
>>
>>
>> The University of Southern Queensland is a registered provider of education with the Australian Government.
>>
>> (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>> _____________________________________________________________
>> This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
>>
>> The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
>>
>> The University of Southern Queensland is a registered provider of education with the Australian Government.
>> (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180622/a461284b/attachment-0001.html>