syslog-ng has add extra field
Hi all! I think I have a stupid question, but I really dont know how this make. Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming. Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp? message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal> mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1 <bla bla bla> I will be appreciate for any advice! P.s sorry for bad english it is not my native language best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68
Hello Ivan, Most commonly there may be two main formats of logs that you may encounter. One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164 The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424 In case of syslog-ng you would have to either use network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol)) respectively. The sample logs you included seem to resemble the IETF-style. What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?) Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari E-mail: janos@szigetvari.com, jszigetvari@gmail.com Web: janos.szigetvari.com __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 14:56):
Hi all!
I think I have a stupid question, but I really dont know how this make.
Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49*+03:00*", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming.
Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp?
message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal>
mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45*+03:00* HUAWEI-CORE-OFFICE-1 <bla bla bla>
I will be appreciate for any advice!
P.s sorry for bad english it is not my native language
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Thanks for your reply, János ! If you can explain me please what does that mean?
network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol))
====================================================================== my config: @version: 3.18 @include "scl.conf" source s_local { internal(); }; source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert")) ); }; destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); }; destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); }; log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); # destination(d_sorted); }; best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com> Sent: Thursday, March 25, 2021 6:24:09 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hello Ivan, Most commonly there may be two main formats of logs that you may encounter. One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164 The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424 In case of syslog-ng you would have to either use network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol)) respectively. The sample logs you included seem to resemble the IETF-style. What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?) Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692<https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari<http://linkedin.com/in/janosszigetvari> E-mail: janos@szigetvari.com<mailto:janos@szigetvari.com>, jszigetvari@gmail.com<mailto:jszigetvari@gmail.com> Web: janos.szigetvari.com<https://janos.szigetvari.com> __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 14:56): Hi all! I think I have a stupid question, but I really dont know how this make. Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming. Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp? message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal> mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1 <bla bla bla> I will be appreciate for any advice! P.s sorry for bad english it is not my native language best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Ivan, Okay, I see what you mean. The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions. The syslog(...) and network(...) statements I mentioned eariler were source definitions as well. In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is. You can either do this by * running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or * by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun <https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you) Thanks! Best Regards, János Szigetvári Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 16:38):
Thanks for your reply, János !
If you can explain me please what does that mean?
network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol))
======================================================================
my config:
@version: 3.18 @include "scl.conf"
source s_local { internal(); };
source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert")) ); };
destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); };
destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); };
log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); # destination(d_sorted); };
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com> *Sent:* Thursday, March 25, 2021 6:24:09 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng has add extra field
Hello Ivan,
Most commonly there may be two main formats of logs that you may encounter. One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164 The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424
In case of syslog-ng you would have to either use network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol)) respectively.
The sample logs you included seem to resemble the IETF-style. What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?)
Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
LinkedIn: linkedin.com/in/janosszigetvari E-mail: janos@szigetvari.com, jszigetvari@gmail.com Web: janos.szigetvari.com
__@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 14:56):
Hi all!
I think I have a stupid question, but I really dont know how this make.
Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49*+03:00*", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming.
Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp?
message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal>
mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45*+03:00* HUAWEI-CORE-OFFICE-1 <bla bla bla>
I will be appreciate for any advice!
P.s sorry for bad english it is not my native language
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I've made pcap file and did send to you. Piece of output (for history) ####### with +03:00 11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto UDP (17), length 297) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="system-view", LocalIp=192.168.102.1.) ####### without +03:00 11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto UDP (17), length 284) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.) Thank you for you kindness! best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com> Sent: Thursday, March 25, 2021 8:55:36 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hi Ivan, Okay, I see what you mean. The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions. The syslog(...) and network(...) statements I mentioned eariler were source definitions as well. In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is. You can either do this by * running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or * by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun<https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you) Thanks! Best Regards, János Szigetvári Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 16:38): Thanks for your reply, János ! If you can explain me please what does that mean?
network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol))
====================================================================== my config: @version: 3.18 @include "scl.conf" source s_local { internal(); }; source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert")) ); }; destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); }; destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); }; log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); # destination(d_sorted); }; best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com<mailto:jszigetvari@gmail.com>> Sent: Thursday, March 25, 2021 6:24:09 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hello Ivan, Most commonly there may be two main formats of logs that you may encounter. One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164 The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424 In case of syslog-ng you would have to either use network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol)) respectively. The sample logs you included seem to resemble the IETF-style. What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?) Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692<https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari<http://linkedin.com/in/janosszigetvari> E-mail: janos@szigetvari.com<mailto:janos@szigetvari.com>, jszigetvari@gmail.com<mailto:jszigetvari@gmail.com> Web: janos.szigetvari.com<https://janos.szigetvari.com> __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 14:56): Hi all! I think I have a stupid question, but I really dont know how this make. Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming. Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp? message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal> mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1 <bla bla bla> I will be appreciate for any advice! P.s sorry for bad english it is not my native language best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, This is a yet another date format, that our rfc3164 parser doesn't recognize. You will need to receive this via flags(no-parse), which is done by default-network-drivers (), but there's no application adapter for this kind of log to automatically recognize this yet. An adapter something along the like of: application huawei[syslog-raw] { filter { message("%%"); }; parser { ... }; }; The filter part should contain a filter that identifies huawei candidate messages, it doesn't need to be perfect (can even be absent), it just excludes messages that would definitely not be huawei messages, and is basically a performance optimization. The parser piece (which is dotted out for now), needs both identify the message and to extract fields that you want to make available within syslog-ng itself. If you look at the cisco-parser() source, it similarly recognizes a set of timestamps (cisco isn't very standards friendly either), and then extracts the cisco log triplet (severity, subsystem and mnemonic). If you come up with the parser expression, this would be a worthwhile addition to syslog-ng, so please submit it as a pull request to our scl directory. Thanks Bazsi On Fri, Mar 26, 2021, 09:42 Ivan Nepryahin - Bercut < Ivan.Nepryahin@bercut.com> wrote:
I've made pcap file and did send to you.
Piece of output (for history)
####### with +03:00 11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto UDP (17), length 297) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="system-view", LocalIp=192.168.102.1.)
####### without +03:00 11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto UDP (17), length 284) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.)
Thank you for you kindness!
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com> *Sent:* Thursday, March 25, 2021 8:55:36 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng has add extra field
Hi Ivan,
Okay, I see what you mean. The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions. The syslog(...) and network(...) statements I mentioned eariler were source definitions as well.
In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is.
You can either do this by * running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or * by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun <https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you)
Thanks!
Best Regards, János Szigetvári
Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 16:38):
Thanks for your reply, János !
If you can explain me please what does that mean?
network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol))
======================================================================
my config:
@version: 3.18 @include "scl.conf"
source s_local { internal(); };
source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert")) ); };
destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); };
destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); };
log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); # destination(d_sorted); };
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com> *Sent:* Thursday, March 25, 2021 6:24:09 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng has add extra field
Hello Ivan,
Most commonly there may be two main formats of logs that you may encounter. One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164 The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424
In case of syslog-ng you would have to either use network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol)) respectively.
The sample logs you included seem to resemble the IETF-style. What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?)
Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
LinkedIn: linkedin.com/in/janosszigetvari E-mail: janos@szigetvari.com, jszigetvari@gmail.com Web: janos.szigetvari.com
__@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 14:56):
Hi all!
I think I have a stupid question, but I really dont know how this make.
Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49*+03:00*", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming.
Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp?
message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal>
mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45*+03:00* HUAWEI-CORE-OFFICE-1 <bla bla bla>
I will be appreciate for any advice!
P.s sorry for bad english it is not my native language
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Got it. I will try do that. Thanks! best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77@gmail.com> Sent: Friday, March 26, 2021 12:11:03 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hi, This is a yet another date format, that our rfc3164 parser doesn't recognize. You will need to receive this via flags(no-parse), which is done by default-network-drivers (), but there's no application adapter for this kind of log to automatically recognize this yet. An adapter something along the like of: application huawei[syslog-raw] { filter { message("%%"); }; parser { ... }; }; The filter part should contain a filter that identifies huawei candidate messages, it doesn't need to be perfect (can even be absent), it just excludes messages that would definitely not be huawei messages, and is basically a performance optimization. The parser piece (which is dotted out for now), needs both identify the message and to extract fields that you want to make available within syslog-ng itself. If you look at the cisco-parser() source, it similarly recognizes a set of timestamps (cisco isn't very standards friendly either), and then extracts the cisco log triplet (severity, subsystem and mnemonic). If you come up with the parser expression, this would be a worthwhile addition to syslog-ng, so please submit it as a pull request to our scl directory. Thanks Bazsi On Fri, Mar 26, 2021, 09:42 Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> wrote: I've made pcap file and did send to you. Piece of output (for history) ####### with +03:00 11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto UDP (17), length 297) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="system-view", LocalIp=192.168.102.1.) ####### without +03:00 11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto UDP (17), length 284) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.) Thank you for you kindness! best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com<mailto:jszigetvari@gmail.com>> Sent: Thursday, March 25, 2021 8:55:36 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hi Ivan, Okay, I see what you mean. The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions. The syslog(...) and network(...) statements I mentioned eariler were source definitions as well. In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is. You can either do this by * running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or * by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun<https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you) Thanks! Best Regards, János Szigetvári Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 16:38): Thanks for your reply, János ! If you can explain me please what does that mean?
network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol))
====================================================================== my config: @version: 3.18 @include "scl.conf" source s_local { internal(); }; source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert")) ); }; destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); }; destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); }; log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); # destination(d_sorted); }; best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com<mailto:jszigetvari@gmail.com>> Sent: Thursday, March 25, 2021 6:24:09 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hello Ivan, Most commonly there may be two main formats of logs that you may encounter. One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164 The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424 In case of syslog-ng you would have to either use network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol)) respectively. The sample logs you included seem to resemble the IETF-style. What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?) Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692<https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari<http://linkedin.com/in/janosszigetvari> E-mail: janos@szigetvari.com<mailto:janos@szigetvari.com>, jszigetvari@gmail.com<mailto:jszigetvari@gmail.com> Web: janos.szigetvari.com<https://janos.szigetvari.com> __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 14:56): Hi all! I think I have a stupid question, but I really dont know how this make. Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming. Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp? message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal> mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1 <bla bla bla> I will be appreciate for any advice! P.s sorry for bad english it is not my native language best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Ivan, Sorry for the slight delay, but your emails ended up in my spam folder for some reason. I see Bazsi already answered to your last email, and I can confirm what he said. The logs are sent over UDP/514, and seem to follow the legacy/BSD-style log format, but the timestamp seems to be off. You see, syslog-ng would expect something like this: <34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8 While it gets something like this: <189>Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 ... This is different from the expected format in two aspects: - it includes the four digit year, which RFC3164 doesn't contain - it includes the timezone offset, like you mentioned I don't really know, and haven't used any Huawei switches, so I don't know how flexible they are from logging perspective, and what options they offer. These two fields are normally included as part of the IETF-syslog/RFC5424 format, but that log format looks slightly different, and the year is the first part of the timestamp: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8 Again I don't know whether Huawei devices allow you to select which format to use, but this second format seems to be closer to what you want. Either way, if the switch doesn't allow you to choose between these two formats, you would be best off with the flags(no-parse) option, like Bazsi mentioned. In that case I would recommend you to use the config below, if you just want to store the messages from the Huawei switch in their original format: @version: 3.18 @include "scl.conf" source s_local { internal(); }; source s_huawei-udp514-no-parse { network( transport(udp) port(514) flags(no-parse) ); }; source s_network { network( transport(tcp) port(514) ); syslog( transport(tcp) port(601) ); }; destination d_huawei-logs { file("/var/log/messages-huawei_${YEAR}-${MONTH}-${DAY}.log" template("${MSG}\n")); file("/var/log/messages-huawei_${HOST}.log" perm(0644)); }; destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); }; destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); }; log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); }; log { source(s_huawei-udp514-no-parse); destination(d_huawei-logs); destination(d_logstore); }; The only downside of my approach is that you won't be able to save the logs in key-value format, due to the parsing being turned off. If that was your main goal, then you are better off with Bazsi's recommendation of writing your own application adapter, but that's a bit more difficult than simply saving the messages in their original format. I hope I was able to help. Best Regards, János Szigetvári Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 26., P, 9:42):
I've made pcap file and did send to you.
Piece of output (for history)
####### with +03:00 11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto UDP (17), length 297) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="system-view", LocalIp=192.168.102.1.)
####### without +03:00 11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto UDP (17), length 284) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.)
Thank you for you kindness!
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com> *Sent:* Thursday, March 25, 2021 8:55:36 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng has add extra field
Hi Ivan,
Okay, I see what you mean. The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions. The syslog(...) and network(...) statements I mentioned eariler were source definitions as well.
In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is.
You can either do this by * running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or * by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun <https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you)
Thanks!
Best Regards, János Szigetvári
Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 16:38):
Thanks for your reply, János !
If you can explain me please what does that mean?
network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol))
======================================================================
my config:
@version: 3.18 @include "scl.conf"
source s_local { internal(); };
source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert")) ); };
destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); };
destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); };
log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); # destination(d_sorted); };
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com> *Sent:* Thursday, March 25, 2021 6:24:09 PM *To:* Syslog-ng users' and developers' mailing list *Subject:* Re: [syslog-ng] syslog-ng has add extra field
Hello Ivan,
Most commonly there may be two main formats of logs that you may encounter. One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164 The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424
In case of syslog-ng you would have to either use network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol)) respectively.
The sample logs you included seem to resemble the IETF-style. What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?)
Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
LinkedIn: linkedin.com/in/janosszigetvari E-mail: janos@szigetvari.com, jszigetvari@gmail.com Web: janos.szigetvari.com
__@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 14:56):
Hi all!
I think I have a stupid question, but I really dont know how this make.
Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49*+03:00*", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming.
Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp?
message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal>
mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45*+03:00* HUAWEI-CORE-OFFICE-1 <bla bla bla>
I will be appreciate for any advice!
P.s sorry for bad english it is not my native language
best regards, *Nepryahin Ivan* IT Department *Phone**: *+7 812 327 32 33 *Mobile: *+7 911 291 81 68
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I don't know what type of advice better for me I need some more time. Thanks for your kindness guys! best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com> Sent: Friday, March 26, 2021 4:01:26 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hi Ivan, Sorry for the slight delay, but your emails ended up in my spam folder for some reason. I see Bazsi already answered to your last email, and I can confirm what he said. The logs are sent over UDP/514, and seem to follow the legacy/BSD-style log format, but the timestamp seems to be off. You see, syslog-ng would expect something like this: <34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8 While it gets something like this: <189>Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 ... This is different from the expected format in two aspects: - it includes the four digit year, which RFC3164 doesn't contain - it includes the timezone offset, like you mentioned I don't really know, and haven't used any Huawei switches, so I don't know how flexible they are from logging perspective, and what options they offer. These two fields are normally included as part of the IETF-syslog/RFC5424 format, but that log format looks slightly different, and the year is the first part of the timestamp: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com<http://mymachine.example.com> su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8 Again I don't know whether Huawei devices allow you to select which format to use, but this second format seems to be closer to what you want. Either way, if the switch doesn't allow you to choose between these two formats, you would be best off with the flags(no-parse) option, like Bazsi mentioned. In that case I would recommend you to use the config below, if you just want to store the messages from the Huawei switch in their original format: @version: 3.18 @include "scl.conf" source s_local { internal(); }; source s_huawei-udp514-no-parse { network( transport(udp) port(514) flags(no-parse) ); }; source s_network { network( transport(tcp) port(514) ); syslog( transport(tcp) port(601) ); }; destination d_huawei-logs { file("/var/log/messages-huawei_${YEAR}-${MONTH}-${DAY}.log" template("${MSG}\n")); file("/var/log/messages-huawei_${HOST}.log" perm(0644)); }; destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); }; destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); }; log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); }; log { source(s_huawei-udp514-no-parse); destination(d_huawei-logs); destination(d_logstore); }; The only downside of my approach is that you won't be able to save the logs in key-value format, due to the parsing being turned off. If that was your main goal, then you are better off with Bazsi's recommendation of writing your own application adapter, but that's a bit more difficult than simply saving the messages in their original format. I hope I was able to help. Best Regards, János Szigetvári Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> ezt írta (időpont: 2021. márc. 26., P, 9:42): I've made pcap file and did send to you. Piece of output (for history) ####### with +03:00 11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto UDP (17), length 297) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="system-view", LocalIp=192.168.102.1.) ####### without +03:00 11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto UDP (17), length 284) 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256 Facility local7 (23), Severity notice (5) Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.) Thank you for you kindness! best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com<mailto:jszigetvari@gmail.com>> Sent: Thursday, March 25, 2021 8:55:36 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hi Ivan, Okay, I see what you mean. The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions. The syslog(...) and network(...) statements I mentioned eariler were source definitions as well. In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is. You can either do this by * running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or * by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun<https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you) Thanks! Best Regards, János Szigetvári Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 16:38): Thanks for your reply, János ! If you can explain me please what does that mean?
network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol))
====================================================================== my config: @version: 3.18 @include "scl.conf" source s_local { internal(); }; source s_network { default-network-drivers( # NOTE: TLS support # # the default-network-drivers() source driver opens the TLS # enabled ports as well, however without an actual key/cert # pair they will not operate and syslog-ng would display a # warning at startup. # #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert")) ); }; destination d_local { # file("/var/log/messages"); file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3)); file("/var/log/messages_${HOST}.log" perm(0644) ); }; destination d_logstore { file( "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) ); }; log { source(s_local); source(s_network); destination(d_local); destination(d_logstore); # destination(d_sorted); }; best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com<mailto:jszigetvari@gmail.com>> Sent: Thursday, March 25, 2021 6:24:09 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng has add extra field Hello Ivan, Most commonly there may be two main formats of logs that you may encounter. One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164 The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424 In case of syslog-ng you would have to either use network(transport(tcp|udp)) or syslog() or network(transport(tcp|udp) flags(syslog-protocol)) respectively. The sample logs you included seem to resemble the IETF-style. What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?) Best Regards, János -- Janos SZIGETVARI RHCE, License no. 150-053-692<https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari<http://linkedin.com/in/janosszigetvari> E-mail: janos@szigetvari.com<mailto:janos@szigetvari.com>, jszigetvari@gmail.com<mailto:jszigetvari@gmail.com> Web: janos.szigetvari.com<https://janos.szigetvari.com> __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com<mailto:Ivan.Nepryahin@bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 14:56): Hi all! I think I have a stupid question, but I really dont know how this make. Situation: When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming. Question: How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp? message without +03:00 Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal> mesage with +03:00 Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1 <bla bla bla> I will be appreciate for any advice! P.s sorry for bad english it is not my native language best regards, Nepryahin Ivan IT Department Phone: +7 812 327 32 33 Mobile: +7 911 291 81 68 ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Balazs Scheidler
-
Ivan Nepryahin - Bercut
-
SZIGETVÁRI János