Hi Ivan,

Sorry for the slight delay, but your emails ended up in my spam folder for some reason.
I see Bazsi already answered to your last email, and I can confirm what he said.

The logs are sent over UDP/514, and seem to follow the legacy/BSD-style log format, but the timestamp seems to be off.
You see, syslog-ng would expect something like this:

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8
While it gets something like this:
<189>Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 ...
This is different from the expected format in two aspects:
- it includes the four digit year, which RFC3164 doesn't contain
- it includes the timezone offset, like you mentioned

I don't really know, and haven't used any Huawei switches, so I don't know how flexible they are from logging perspective, and what options they offer.
These two fields are normally included as part of the IETF-syslog/RFC5424 format, but that log format looks slightly different, and the year is the first part of the timestamp:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

Again I don't know whether Huawei devices allow you to select which format to use, but this second format seems to be closer to what you want.
Either way, if the switch doesn't allow you to choose between these two formats, you would be best off with the flags(no-parse) option, like Bazsi mentioned.
In that case I would recommend you to use the config below, if you just want to store the messages from the Huawei switch in their original format:

@version: 3.18
@include "scl.conf"
source s_local {
        internal();
};

source s_huawei-udp514-no-parse {
        network(
                transport(udp)
                port(514)
                flags(no-parse)
        );
};
source s_network {
        network(
                transport(tcp)
                port(514)
        );
        syslog(
                transport(tcp)
                port(601)
        );
};

destination d_huawei-logs {
        file("/var/log/messages-huawei_${YEAR}-${MONTH}-${DAY}.log" template("${MSG}\n"));
        file("/var/log/messages-huawei_${HOST}.log" perm(0644));
};
destination d_local {
#       file("/var/log/messages");
        file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
        file("/var/log/messages_${HOST}.log"
        perm(0644)
        );
};
destination d_logstore {
        file(
           "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log"
        create-dirs(yes)
    );
};
log {
        source(s_local);
        source(s_network);
        destination(d_local);
        destination(d_logstore);
};

log {
        source(s_huawei-udp514-no-parse);
        destination(d_huawei-logs);
        destination(d_logstore);
};

The only downside of my approach is that you won't be able to save the logs in key-value format, due to the parsing being turned off.
If that was your main goal, then you are better off with Bazsi's recommendation of writing your own application adapter, but that's a bit more difficult than simply saving the messages in their original format.

I hope I was able to help.

Best Regards,
János Szigetvári



Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 26., P, 9:42):

I've made pcap file and did send to you.


Piece of output (for history)


####### with +03:00
11:11:45.057726 IP (tos 0x0, ttl 239, id 17, offset 0, flags [none], proto UDP (17), length 297)
    192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 269
        Facility local7 (23), Severity notice (5)
        Msg: Mar 26 2021 11:11:44+03:00 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="system-view", LocalIp=192.168.102.1.)


####### without +03:00
11:23:37.882898 IP (tos 0x0, ttl 239, id 50, offset 0, flags [none], proto UDP (17), length 284)
    192.168.102.1.38514 > 172.18.0.2.syslog: SYSLOG, length: 256
        Facility local7 (23), Severity notice (5)
        Msg: Mar 26 2021 08:23:37 HUAWEI-CORE-IMAQLIQ-1 %%01CLI/5/CMDRECORD(s):CID=0x80ca2713;Recorded command information. (Task=VTY0, RemoteIp=192.168.55.41, VpnName=_public_, User=nepryahin, AuthenticationMethod="Tacacs", Command="quit", LocalIp=192.168.102.1.)



Thank you for you kindness!


best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68

From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com>
Sent: Thursday, March 25, 2021 8:55:36 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng has add extra field
 
Hi Ivan,

Okay, I see what you mean.
The thing is that the default-network-drivers() source in your config is automatically and seamlessly expanded to a series of syslog() and network() sources definitions.
The syslog(...) and network(...) statements I mentioned eariler were source definitions as well.

In order to get a more thorough look at your use-case, I would kindly request you to send (either to me privately, or to this mailing list) a packet dump of the Huawei device's logs so that I can identify which port the traffic is destined to (and through which protocol), and what the actual format of the logs is.

You can either do this by
* running tcpdump ( tcpdump -peni any -v -w /tmp/syslog.pcap "port 514 or port 601" ), or
* by downloading (if the script wasn't already deployed along your syslog-ng installation) and running syslog-ng-debun ( syslog-ng-debun -r -p -t 300 ) (this will run the information gathering script for 5 minutes, and it will try to do the packet capture for you)

Thanks!

Best Regards,
János Szigetvári

Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 16:38):

Thanks for your reply, János !


If you can explain me please what does that mean?


>network(transport(tcp|udp))
>or
>syslog() or network(transport(tcp|udp) flags(syslog-protocol))




======================================================================


my config:



@version: 3.18
@include "scl.conf"

source s_local {
        internal();
};

source s_network {
        default-network-drivers(
                # NOTE: TLS support
                #
                # the default-network-drivers() source driver opens the TLS
                # enabled ports as well, however without an actual key/cert
                # pair they will not operate and syslog-ng would display a
                # warning at startup.
                #
                #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert"))
        );
};

destination d_local {
#       file("/var/log/messages");
        file("/var/log/messages-kv_${YEAR}-${MONTH}-${DAY}.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
        file("/var/log/messages_${HOST}.log"
        perm(0644)
        );
};

destination d_logstore {
        file(
           "/var/log/remote/${HOST}/${HOST}_${YEAR}-${MONTH}-${DAY}.log"
        create-dirs(yes)
    );
};

log {
        source(s_local);
        source(s_network);
        destination(d_local);
        destination(d_logstore);
#       destination(d_sorted);
};



best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68

From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of SZIGETVÁRI János <jszigetvari@gmail.com>
Sent: Thursday, March 25, 2021 6:24:09 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] syslog-ng has add extra field
 
Hello Ivan,

Most commonly there may be two main formats of logs that you may encounter.
One is the traditional BSD-style syslog, described in RFC 3164: https://tools.ietf.org/html/rfc3164
The other is the IETF-style log format, described in RFC 5424: https://tools.ietf.org/html/rfc5424

In case of syslog-ng you would have to either use
network(transport(tcp|udp))
or
syslog() or network(transport(tcp|udp) flags(syslog-protocol))
respectively.

The sample logs you included seem to resemble the IETF-style.
What type of source do you have configured in your syslog-ng setup? (Could you please share your config file?)

Best Regards,
János
--
Janos SZIGETVARI
RHCE, License no. 150-053-692

Web: janos.szigetvari.com

__@__˚V˚
Make the switch to open (source) applications, protocols, formats now:
- windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice
- msn -> jabber protocol (Pidgin, Google Talk)
- mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp


Ivan Nepryahin - Bercut <Ivan.Nepryahin@bercut.com> ezt írta (időpont: 2021. márc. 25., Cs, 14:56):

Hi all!



I think I have a stupid question, but I really dont know how this make.

Situation: 
When I send syslog message with timestamp in  format "1Mar 25 2021 16:35:49" everything works great, but when  I send  message with timestamp in format "1Mar 25 2021 16:35:49+03:00", syslog-ng adding two extra fields with timestamp and IP address and due that break down  file naming.

Question:
How can I say to syslog-ng server do not  add extra fields when he  get message with +03:00  in timestamp?

message without +03:00
Mar 25 13:11:57 HUAWEI-CORE-OFFICE-1   <bla bla bal>

mesage with  +03:00 
Mar 25 13:46:45 192.168.100.34 Mar 25 2021 16:46:45+03:00 HUAWEI-CORE-OFFICE-1  <bla bla bla>



I will be appreciate for any advice!



P.s sorry for bad english it is not my native language 



best regards,
Nepryahin Ivan
IT Department
Phone: +7 812 327 32 33
Mobile: +7 911 291 81 68

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq