[syslog-ng] syslog-ng has add extra field
SZIGETVÁRI János
jszigetvari at gmail.com
Fri Mar 26 13:01:26 UTC 2021
Hi Ivan,
Sorry for the slight delay, but your emails ended up in my spam folder for
some reason.
I see Bazsi already answered to your last email, and I can confirm what he
said.
The logs are sent over UDP/514, and seem to follow the legacy/BSD-style log
format, but the timestamp seems to be off.
You see, syslog-ng would expect something like this:
<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8
While it gets something like this:
<189>Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 ...
This is different from the expected format in two aspects:
- it includes the four digit year, which RFC3164 doesn't contain
- it includes the timezone offset, like you mentioned
I don't really know, and haven't used any Huawei switches, so I don't know
how flexible they are from logging perspective, and what options they offer.
These two fields are normally included as part of the IETF-syslog/RFC5424
format, but that log format looks slightly different, and the year is the
first part of the timestamp:
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su
root' failed for lonvick on /dev/pts/8
Again I don't know whether Huawei devices allow you to select which format
to use, but this second format seems to be closer to what you want.
Either way, if the switch doesn't allow you to choose between these two
formats, you would be best off with the flags(no-parse) option, like Bazsi
mentioned.
In that case I would recommend you to use the config below, if you just
want to store the messages from the Huawei switch in their original format:
@version: 3.18
@include "scl.conf"
source s_local {
internal();
};
source s_huawei-udp514-no-parse {
network(
transport(udp)
port(514)
flags(no-parse)
);
};
source s_network {
network(
transport(tcp)
port(514)
);
syslog(
transport(tcp)
port(601)
);
};
destination d_huawei-logs {
file("/var/log/messages-huawei_${YEAR}-${MONTH}-${DAY}.log"
template("${MSG}\n"));
file("/var/log/messages-huawei_${HOST}.log" perm(0644));
};
destination d_local {
# file("/var/log/messages");
file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log"
template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n")
frac-digits(3));
file("/var/log/messages_${HOST}.log"
perm(0644)
);
};
destination d_logstore {
file(
"/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log"
create-dirs(yes)
);
};
log {
source(s_local);
source(s_network);
destination(d_local);
destination(d_logstore);
};
log {
source(s_huawei-udp514-no-parse);
destination(d_huawei-logs);
destination(d_logstore);
};
The only downside of my approach is that you won't be able to save the logs
in key-value format, due to the parsing being turned off.
If that was your main goal, then you are better off with Bazsi's
recommendation of writing your own application adapter, but that's a bit
more difficult than simply saving the messages in their original format.
I hope I was able to help.
Best Regards,
János Szigetvári
Ivan Nepryahin - Bercut <Ivan.Nepryahin at bercut.com> ezt írta (időpont:
2021. márc. 26., P, 9:42):
> I've made pcap file and did send to you.
>
>
> Piece of output (for history)
>
>
> ####### with +03:00
> 11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto
> UDP (17), length 297)
> 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269
> Facility local7 (23), Severity notice (5)
> Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1
> %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information.
> (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin,
> AuthenticationMethod="Tacacs", Command="system-view",
> LocalIp=192.168.102.1.)
>
>
> ####### without +03:00
> 11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto
> UDP (17), length 284)
> 192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256
> Facility local7 (23), Severity notice (5)
> Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1
> %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information.
> (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin,
> AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.)
>
>
>
> Thank you for you kindness!
>
>
> best regards,
> *Nepryahin Ivan*
> IT Department
> *Phone**: *+7 812 327 32 33
> *Mobile: *+7 911 291 81 68
>
> ------------------------------
> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of
> SZIGETVÁRI János <jszigetvari at gmail.com>
> *Sent:* Thursday, March 25, 2021 8:55:36 PM
> *To:* Syslog-ng users' and developers' mailing list
> *Subject:* Re: [syslog-ng] syslog-ng has add extra field
>
> Hi Ivan,
>
> Okay, I see what you mean.
> The thing is that the default-network-drivers() source in your config is
> automatically and seamlessly expanded to a series of syslog() and network()
> sources definitions.
> The syslog(...) and network(...) statements I mentioned eariler were
> source definitions as well.
>
> In order to get a more thorough look at your use-case, I would kindly
> request you to send (either to me privately, or to this mailing list) a
> packet dump of the Huawei device's logs so that I can identify which port
> the traffic is destined to (and through which protocol), and what the
> actual format of the logs is.
>
> You can either do this by
> * running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or
> port 601" ), or
> * by downloading (if the script wasn't already deployed along your
> syslog-ng installation) and running syslog-ng-debun
> <https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun>
> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering
> script for 5 minutes, and it will try to do the packet capture for you)
>
> Thanks!
>
> Best Regards,
> János Szigetvári
>
> Ivan Nepryahin - Bercut <Ivan.Nepryahin at bercut.com> ezt írta (időpont:
> 2021. márc. 25., Cs, 16:38):
>
>> Thanks for your reply, János !
>>
>>
>> If you can explain me please what does that mean?
>>
>>
>> >network(transport(tcp|udp))
>> >or
>> >syslog() or network(transport(tcp|udp) flags(syslog-protocol))
>>
>>
>>
>>
>> ======================================================================
>>
>>
>> my config:
>>
>>
>>
>> @version: 3.18
>> @include "scl.conf"
>>
>> source s_local {
>> internal();
>> };
>>
>> source s_network {
>> default-network-drivers(
>> # NOTE: TLS support
>> #
>> # the default-network-drivers() source driver opens the
>> TLS
>> # enabled ports as well, however without an actual
>> key/cert
>> # pair they will not operate and syslog-ng would display a
>> # warning at startup.
>> #
>> #tls(key-file("/path/to/ssl-private-key")
>> cert-file("/path/to/ssl-cert"))
>> );
>> };
>>
>> destination d_local {
>> # file("/var/log/messages");
>> file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log"
>> template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n")
>> frac-digits(3));
>> file("/var/log/messages_${HOST}.log"
>> perm(0644)
>> );
>> };
>>
>> destination d_logstore {
>> file(
>> "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log"
>> create-dirs(yes)
>> );
>> };
>>
>> log {
>> source(s_local);
>> source(s_network);
>> destination(d_local);
>> destination(d_logstore);
>> # destination(d_sorted);
>> };
>>
>>
>>
>> best regards,
>> *Nepryahin Ivan*
>> IT Department
>> *Phone**: *+7 812 327 32 33
>> *Mobile: *+7 911 291 81 68
>>
>> ------------------------------
>> *From:* syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of
>> SZIGETVÁRI János <jszigetvari at gmail.com>
>> *Sent:* Thursday, March 25, 2021 6:24:09 PM
>> *To:* Syslog-ng users' and developers' mailing list
>> *Subject:* Re: [syslog-ng] syslog-ng has add extra field
>>
>> Hello Ivan,
>>
>> Most commonly there may be two main formats of logs that you may
>> encounter.
>> One is the traditional BSD-style syslog, described in RFC 3164:
>> https://tools.ietf.org/html/rfc3164
>> The other is the IETF-style log format, described in RFC 5424:
>> https://tools.ietf.org/html/rfc5424
>>
>> In case of syslog-ng you would have to either use
>> network(transport(tcp|udp))
>> or
>> syslog() or network(transport(tcp|udp) flags(syslog-protocol))
>> respectively.
>>
>> The sample logs you included seem to resemble the IETF-style.
>> What type of source do you have configured in your syslog-ng setup?
>> (Could you please share your config file?)
>>
>> Best Regards,
>> János
>> --
>> Janos SZIGETVARI
>> RHCE, License no. 150-053-692
>> <https://www.redhat.com/rhtapps/verify/?certId=150-053-692>
>>
>> LinkedIn: linkedin.com/in/janosszigetvari
>> E-mail: janos at szigetvari.com, jszigetvari at gmail.com
>> Web: janos.szigetvari.com
>>
>> __ at __˚V˚
>> Make the switch to open (source) applications, protocols, formats now:
>> - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
>> - msn -> jabber protocol (Pidgin, Google Talk)
>> - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp
>>
>>
>> Ivan Nepryahin - Bercut <Ivan.Nepryahin at bercut.com> ezt írta (időpont:
>> 2021. márc. 25., Cs, 14:56):
>>
>>> Hi all!
>>>
>>>
>>>
>>> I think I have a stupid question, but I really dont know how this make.
>>>
>>> Situation:
>>> When I send syslog message with timestamp in format "1Mar 25 2021
>>> 16:35:49" everything works great, but when I send message with timestamp
>>> in format "1Mar 25 2021 16:35:49*+03:00*", syslog-ng adding two extra
>>> fields with timestamp and IP address and due that break down file naming.
>>>
>>> Question:
>>> How can I say to syslog-ng server do not add extra fields when he get
>>> message with +03:00 in timestamp?
>>>
>>> message without +03:00
>>> Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1 <bla bla bal>
>>>
>>> mesage with +03:00
>>> Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45*+03:00*
>>> HUAWEI-CORE-OFFICE-1 <bla bla bla>
>>>
>>>
>>>
>>> I will be appreciate for any advice!
>>>
>>>
>>>
>>> P.s sorry for bad english it is not my native language
>>>
>>>
>>>
>>> best regards,
>>> *Nepryahin Ivan*
>>> IT Department
>>> *Phone**: *+7 812 327 32 33
>>> *Mobile: *+7 911 291 81 68
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210326/749a30e3/attachment-0001.html>
More information about the syslog-ng
mailing list