Hi I am new to syslog-ng and logging in general so apologies but I am having trouble understanding timestamps. I am trying to extract the timestamp inside the message and create a field called 'timestamp' for example. The message looks like this: "MESSAGE":"06:17-16:21:21 fw-1-1 ulogd[9300]: id=\"2001\" severity=\"info\" sys=\"SecureNet\" sub=\"packetfilter\" name=\"Packet dropped\" action=\"drop\" fwrule=\"60002\" initf=\"eth0\" outitf=\"eth1\" srcmac=\"4c:00:82:8a:f4:11\" dstmac=\"00:1a:8c:f0:3c:40\" srcip=\"1.1.1.1\" dstip=\"2.2.2.2\" proto=\"17\" length=\"143\" tos=\"0x00\" prec=\"0x00\" ttl=\"125\" srcport=\"51941\" dstport=\"161\" How can I create a field called 'timestamp' which contains the timestamp in the message but also adds the year to it and puts it into a better format? I am also using the kv-parser and json output looks good I have pairs of everything from the message but I also need a pair called timestamp="Jun 16 2017 16:21:21' for example. Hope I'm making sense here. Thanks for any help.
Hi Andrew, If you have a recent enough syslog-ng version, you can use the date-parser to parse your date. Otherwise, I guess you could use the current year $YEAR and add it to the message using a rewrite rule. Cheers
I looked into it further and the firewall is sending the year in the message, I thought that it wasn't but it was getting chopped off in the json output. I rectified it by using ${MSGHDR}${MSG} in my template which now gives me the full timestamp in the message which is mainly what I needed. I will look into the date-parser thanks for the info. On Sun, Jun 18, 2017 at 7:14 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Andrew,
If you have a recent enough syslog-ng version, you can use the date-parser to parse your date. Otherwise, I guess you could use the current year $YEAR and add it to the message using a rewrite rule.
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Your firewall probably does not use rfc3164 properly and albeit syslog-ng does have a few heuristics to deal with differences, your fw may just get parsed incorrectly. You might want to disable parsing using flags(no-parse) and then deal with it accordingly. No-parse will put the entire message with headers to $MSG, which then can be broken down by various syslog-ng parsers, like the date-parser or regexp based ones. On Jun 18, 2017 02:40, "Andrew" <toranagtrx@gmail.com> wrote:
I looked into it further and the firewall is sending the year in the message, I thought that it wasn't but it was getting chopped off in the json output. I rectified it by using ${MSGHDR}${MSG} in my template which now gives me the full timestamp in the message which is mainly what I needed.
I will look into the date-parser thanks for the info.
On Sun, Jun 18, 2017 at 7:14 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Andrew,
If you have a recent enough syslog-ng version, you can use the date-parser to parse your date. Otherwise, I guess you could use the current year $YEAR and add it to the message using a rewrite rule.
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Or just use the time when syslog-ng has received the message. (keep-timestam(no)) On Sun, Jun 18, 2017 at 9:43 AM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Your firewall probably does not use rfc3164 properly and albeit syslog-ng does have a few heuristics to deal with differences, your fw may just get parsed incorrectly.
You might want to disable parsing using flags(no-parse) and then deal with it accordingly.
No-parse will put the entire message with headers to $MSG, which then can be broken down by various syslog-ng parsers, like the date-parser or regexp based ones.
On Jun 18, 2017 02:40, "Andrew" <toranagtrx@gmail.com> wrote:
I looked into it further and the firewall is sending the year in the message, I thought that it wasn't but it was getting chopped off in the json output. I rectified it by using ${MSGHDR}${MSG} in my template which now gives me the full timestamp in the message which is mainly what I needed.
I will look into the date-parser thanks for the info.
On Sun, Jun 18, 2017 at 7:14 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Andrew,
If you have a recent enough syslog-ng version, you can use the date-parser to parse your date. Otherwise, I guess you could use the current year $YEAR and add it to the message using a rewrite rule.
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Yes that was the exact problem, thanks for that!. Using no-parse and also the date-parser I have been able to get this working perfectly. I now have ISODATE matching to the same timestamp in the message from the firewall. Thanks for the help. On Sun, Jun 18, 2017 at 5:43 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Your firewall probably does not use rfc3164 properly and albeit syslog-ng does have a few heuristics to deal with differences, your fw may just get parsed incorrectly.
You might want to disable parsing using flags(no-parse) and then deal with it accordingly.
No-parse will put the entire message with headers to $MSG, which then can be broken down by various syslog-ng parsers, like the date-parser or regexp based ones.
On Jun 18, 2017 02:40, "Andrew" <toranagtrx@gmail.com> wrote:
I looked into it further and the firewall is sending the year in the message, I thought that it wasn't but it was getting chopped off in the json output. I rectified it by using ${MSGHDR}${MSG} in my template which now gives me the full timestamp in the message which is mainly what I needed.
I will look into the date-parser thanks for the info.
On Sun, Jun 18, 2017 at 7:14 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Andrew,
If you have a recent enough syslog-ng version, you can use the date-parser to parse your date. Otherwise, I guess you could use the current year $YEAR and add it to the message using a rewrite rule.
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Andrew
-
Fabien Wernli
-
Fekete, Róbert
-
Scheidler, Balázs