[syslog-ng] syslog-ng has add extra field
Ivan Nepryahin - Bercut
Ivan.Nepryahin at bercut.com
Fri Mar 26 12:49:02 UTC 2021
Got it.
I will try do that.
Thanks!
best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77 at gmail.com>
Sent: Friday, March 26, 2021 12:11:03 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng has add extra field
Hi,
This is a yet another date format, that our rfc3164 parser doesn't recognize.
You will need to receive this via flags(no-parse), which is done by default-network-drivers (), but there's no application adapter for this kind of log to automatically recognize this yet.
An adapter something along the like of:
application huawei[syslog-raw] {
filter { message("%%"); };
parser { ... };
};
The filter part should contain a filter that identifies huawei candidate messages, it doesn't need to be perfect (can even be absent), it just excludes messages that would definitely not be huawei messages, and is basically a performance optimization.
The parser piece (which is dotted out for now), needs both identify the message and to extract fields that you want to make available within syslog-ng itself.
If you look at the cisco-parser() source, it similarly recognizes a set of timestamps (cisco isn't very standards friendly either), and then extracts the cisco log triplet (severity, subsystem and mnemonic).
If you come up with the parser expression, this would be a worthwhile addition to syslog-ng, so please submit it as a pull request to our scl directory.
Thanks
Bazsi
On Fri, Mar 26, 2021, 09:42 Ivan Nepryahin - Bercut <Ivan.Nepryahin at bercut.com<mailto:Ivan.Nepryahin at bercut.com>> wrote:
I've made pcap file and did send to you.
Piece of output (for history)
####### with +03:00
11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto UDP (17), length 297)
192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269
Facility local7 (23), Severity notice (5)
Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="system-view", LocalIp=192.168.102.1.)
####### without +03:00
11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto UDP (17), length 284)
192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256
Facility local7 (23), Severity notice (5)
Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.)
Thank you for you kindness!
best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of SZIGETVÁRI János <jszigetvari at gmail.com<mailto:jszigetvari at gmail.com>>
Sent: Thursday, March 25, 2021 8:55:36 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng has add extra field
Hi Ivan,
Okay, I see what you mean.
The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions.
The syslog(...) and network(...) statements I mentioned eariler were source definitions as well.
In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is.
You can either do this by
* running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or
* by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun<https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you)
Thanks!
Best Regards,
János Szigetvári
Ivan Nepryahin - Bercut <Ivan.Nepryahin at bercut.com<mailto:Ivan.Nepryahin at bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 16:38):
Thanks for your reply, János !
If you can explain me please what does that mean?
>network(transport(tcp|udp))
>or
>syslog() or network(transport(tcp|udp) flags(syslog-protocol))
======================================================================
my config:
@version: 3.18
@include "scl.conf"
source s_local {
internal();
};
source s_network {
default-network-drivers(
# NOTE: TLS support
#
# the default-network-drivers() source driver opens the TLS
# enabled ports as well, however without an actual key/cert
# pair they will not operate and syslog-ng would display a
# warning at startup.
#
#tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert"))
);
};
destination d_local {
# file("/var/log/messages");
file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
file("/var/log/messages_${HOST}.log"
perm(0644)
);
};
destination d_logstore {
file(
"/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log"
create-dirs(yes)
);
};
log {
source(s_local);
source(s_network);
destination(d_local);
destination(d_logstore);
# destination(d_sorted);
};
best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68
________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of SZIGETVÁRI János <jszigetvari at gmail.com<mailto:jszigetvari at gmail.com>>
Sent: Thursday, March 25, 2021 6:24:09 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng has add extra field
Hello Ivan,
Most commonly there may be two main formats of logs that you may encounter.
One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164
The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424
In case of syslog-ng you would have to either use
network(transport(tcp|udp))
or
syslog() or network(transport(tcp|udp) flags(syslog-protocol))
respectively.
The sample logs you included seem to resemble the IETF-style.
What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?)
Best Regards,
János
--
Janos SZIGETVARI
RHCE, License no. 150-053-692<https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
LinkedIn: linkedin.com/in/janosszigetvari<http://linkedin.com/in/janosszigetvari>
E-mail: janos at szigetvari.com<mailto:janos at szigetvari.com>, jszigetvari at gmail.com<mailto:jszigetvari at gmail.com>
Web: janos.szigetvari.com<https://janos.szigetvari.com>
__ at __˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
Ivan Nepryahin - Bercut <Ivan.Nepryahin at bercut.com<mailto:Ivan.Nepryahin at bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 14:56):
Hi all!
I think I have a stupid question, but I really dont know how this make.
Situation:
When I send syslog message with timestamp in format "1Mar 25 2021 16:35:49" everything works great, but when I send message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down file naming.
Question:
How can I say to syslog-ng server do not add extra fields when he get message with +03:00 in timestamp?
message without +03:00
Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal>
mesage with +03:00
Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1 <bla bla bla>
I will be appreciate for any advice!
P.s sorry for bad english it is not my native language
best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210326/0ea4201b/attachment-0001.html>
More information about the syslog-ng
mailing list