[syslog-ng] syslog-ng has add extra field

Fri Mar 26 08:42:32 UTC 2021

I've made pcap file and did send to you.

Piece of output (for history)

####### with +03:00
11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto UDP (17), length 297)
    192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269
        Facility local7 (23), Severity notice (5)
        Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="system-view", LocalIp=192.168.102.1.)

####### without +03:00
11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto UDP (17), length 284)
    192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256
        Facility local7 (23), Severity notice (5)
        Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.)

Thank you for you kindness!

best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari at gmail.com>
Sent: Thursday, March 25, 2021 8:55:36 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng has add extra field

Hi Ivan,

Okay, I see what you mean.
The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions.
The syslog(...) and network(...) statements I mentioned eariler were source definitions as well.

In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is.

You can either do this by
* running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or
* by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun<https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-ng-debun> ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you)

Thanks!

Best Regards,
János Szigetvári

Ivan Nepryahin - Bercut <Ivan.Nepryahin at bercut.com<mailto:Ivan.Nepryahin at bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 16:38):

Thanks for your reply, János !

If you can explain me please what does that mean?

>network(transport(tcp|udp))
>or
>syslog() or network(transport(tcp|udp) flags(syslog-protocol))

======================================================================

my config:

@version: 3.18
@include "scl.conf"

source s_local {
        internal();
};

source s_network {
        default-network-drivers(
                # NOTE: TLS support
                #
                # the default-network-drivers() source driver opens the TLS
                # enabled ports as well, however without an actual key/cert
                # pair they will not operate and syslog-ng would display a
                # warning at startup.
                #
                #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert"))
        );
};

destination d_local {
#       file("/var/log/messages");
        file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
        file("/var/log/messages_${HOST}.log"
        perm(0644)
        );
};

destination d_logstore {
        file(
           "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log"
        create-dirs(yes)
    );
};

log {
        source(s_local);
        source(s_network);
        destination(d_local);
        destination(d_logstore);
#       destination(d_sorted);
};

best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu<mailto:syslog-ng-bounces at lists.balabit.hu>> on behalf of SZIGETVÁRI János <jszigetvari at gmail.com<mailto:jszigetvari at gmail.com>>
Sent: Thursday, March 25, 2021 6:24:09 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng has add extra field

Hello Ivan,

Most commonly there may be two main formats of logs that you may encounter.
One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164
The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424

In case of syslog-ng you would have to either use
network(transport(tcp|udp))
or
syslog() or network(transport(tcp|udp) flags(syslog-protocol))
respectively.

The sample logs you included seem to resemble the IETF-style.
What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?)

Best Regards,
János
--
Janos SZIGETVARI
RHCE, License no. 150-053-692<https://www.redhat.com/rhtapps/verify/?certId=150-053-692>

LinkedIn: linkedin.com/in/janosszigetvari<http://linkedin.com/in/janosszigetvari>
E-mail: janos at szigetvari.com<mailto:janos at szigetvari.com>, jszigetvari at gmail.com<mailto:jszigetvari at gmail.com>
Web: janos.szigetvari.com<https://janos.szigetvari.com>

__ at __˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp

Ivan Nepryahin - Bercut <Ivan.Nepryahin at bercut.com<mailto:Ivan.Nepryahin at bercut.com>> ezt írta (időpont: 2021. márc. 25., Cs, 14:56):

Hi all!

I think I have a stupid question, but I really dont know how this make.

Situation:
When I send syslog message with timestamp in  format "1Mar 25 2021 16:35:49" everything works great, but when  I send  message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down  file naming.

Question:
How can I say to syslog-ng server do not  add extra fields when he  get message with +03:00  in timestamp?

message without +03:00
Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1   <bla bla bal>

mesage with  +03:00
Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1  <bla bla bla>

I will be appreciate for any advice!

P.s sorry for bad english it is not my native language

best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20210326/9e686897/attachment-0001.html>