Hi Daniel!
... how to filter these logs before applying that parser to them? It really depends on the format of your log messages, especially on the non-compliant ones.
Of course if they have a well-defined structure you can use a specific filter on these messages and then you can use the parser. To process log messages differently in a log path you can use the new if-elif branching in your config (or the previous junction-channel setup), which is released in version 3.15. So you can parse the non-complaint messages only, and the rest can go through the usual path. The if-else branching will be documented in the upcoming documentation, until then you can read the PR about the feature which has a description. https://github.com/balabit/syslog-ng/pull/1856#issuecomment-369049863 An example would be: source{ ... flags(no-parse); }; if(filter(f_non_compliant)) { parser { kv-parser(); }; } else {}; destination {....}; I know I have missed the definition of "f_non_compliant", but I don't know what would be a good filter, hence not answering your original question. :) I would like to improve our cisco parser if I prove that this is a Cisco format:
Jun 14 11:57:27 PM.685 UTC : %UC_LOGIN-4-AuthenticationFailed: %[Login Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed. What kind of device is sending this log message?
Regards, Gabor On Tue, Jun 19, 2018 at 8:13 AM, Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au> wrote:
Thanks Gabor,
I’m wondering if you have any suggestion on how to filter these logs before applying that parser to them?
I have numerous sources coming in on the udp 514 listener. Then logging them to files which Splunk reads.
Below is my current config….:
source s_network {
udp(port(514));
};
# Parser
# Rewrite
#Destinations
destination d_files_splunk {
file("/opt/splunk/var/lib/splunk/syslog-ng/$HOST/$MONTH$DAY.log" create_dirs(yes));
};
destination d_files_seamail {
file("/opt/splunk/var/lib/splunk/syslog-ng/seamail/$MONTH$DAY.log" create_dirs(yes));
};
# Filters
filter seamail {
host("q=*" type(glob));
};
filter splunk {
not (filter(seamail));
};
# Log
log {
source(s_network);
filter(splunk);
destination(d_files_splunk);
};
log {
source(s_network);
filter(seamail);
destination(d_files_seamail);
};
Kind Regards,
*Daniel Ehrlich*
ICT Security Officer Phone: +61 7 4687 5600 Email: Daniel.Ehrlich@usq.edu.au
*From:* syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] *On Behalf Of *Nagy, Gábor *Sent:* Friday, 15 June 2018 6:20 PM *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] RE syslog-ng OSE parsing
Hi Daniel!
Your log does not conform to either RFC-3164 or RFC-5424, it looks like kind-of cisco format to me.
I've tried one or two ideas, using our `default-network-drivers()` which includes the cisco-parser() but it is not dealing with it properly.
On second try, I've experimented with `kv-parser()` so the key-value parts in the message will be parsed in a structured format and after that you can use a rewrite rule to modify HOST field of the message.
I had to adjust the pair-separator option to get an optimal result.
`parser { kv-parser( prefix(".kv.") pair-separator("]")); };`
After a successful parsing, you can use a rewrite rule to set the HOSTNAME.
rewrite { set("${.kv.ID}" value("HOST")); };
There is one catch: you need to know the name of the key: "Node ID" or "Hostname", which comes from the log message.
So, if this expression is not fix, then your config will not work.
You can also see that I've added the key "ID" rather than "Node ID", since your log message does not quote the keys in it and
in this case kv-parser will only use the "ID" from "Node ID" part which leads to that similar key-values are overwritten: "App ID", "Cluster ID"
I will think about it, but hopefully others will come up with a better idea. :)
Regards,
Gabor
On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au> wrote:
Hi,
Hoping you can assist me, I haven’t really come across anything that makes full sense to me in my searching of various sites/forums.
This is an example log.
Jun 14 11:57:27 PM.685 UTC : %UC_LOGIN-4-AuthenticationFailed: %[Login Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as cucmsub-prd-t2 ? i.e. Node ID=
Thanks you
Kind Regards,
*Daniel Ehrlich*
*MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA*
ICT Security Officer ICT Client Services|Infrastructure Services Phone: +61 7 4687 5600 Email: Daniel.Ehrlich@usq.edu.au
Toowoomba | Queensland | 4350 | Australia
_____________________________________________________________
This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
The University of Southern Queensland is a registered provider of education with the Australian Government.
(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, Jun 15, 2018 at 6:37 AM, Daniel Ehrlich <Daniel.Ehrlich@usq.edu.au> wrote:
Hi,
Hoping you can assist me, I haven’t really come across anything that makes full sense to me in my searching of various sites/forums.
This is an example log.
Jun 14 11:57:27 PM.685 UTC : %UC_LOGIN-4-AuthenticationFailed: %[Login Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed.
Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as cucmsub-prd-t2 ? i.e. Node ID=
Thanks you
Kind Regards,
*Daniel Ehrlich*
*MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA*
ICT Security Officer ICT Client Services|Infrastructure Services Phone: +61 7 4687 5600 Email: Daniel.Ehrlich@usq.edu.au
Toowoomba | Queensland | 4350 | Australia
_____________________________________________________________
This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
The University of Southern Queensland is a registered provider of education with the Australian Government.
(CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
_____________________________________________________________ This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.
The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.
The University of Southern Queensland is a registered provider of education with the Australian Government. (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq