Hello,

Thanks for the update and a possible workaround for this issue. I just made a small test (I do not have cisco device) based on your logs.

I used syslog, cisco and app-parser(in a form of default-network-driver) and the result:
PARSER=syslog, PROGRAM=%SYS-6-LOGGINGHOST_STARTSTOP, MSG=Logging to host blahblah
PARSER=cisco, PROGRAM= MSG=%SYS-6-LOGGINGHOST_STARTSTOP: Logging to host blahblah
PARSER=app-parser, PROGRAM= MSG=%SYS-6-LOGGINGHOST_STARTSTOP: Logging to host blahblah

Maybe in the feature you could give a chance for the default-network-driver because it seems to handle the exact thing you aim to do (and more).

Btw the UTC time works, because now syslog parser things that UTC: is a program name, and it seeks for the message.

--
Kokan

On Mon, Sep 24, 2018 at 4:27 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Update - updating all of our (previously misbehaving) Cisco devices to log a time zone there have been no more incidents of syslog-ng parsing ${FULLHOST} as ":"



On Mon, Sep 17, 2018 at 3:40 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Yes, hostname was parsed properly in 3.9.  Three macros that I use changed after the upgrade:

${FULLHOST}
- old: "myhost.com"
- new: ":"

${PROGRAM}
- old: ""
- new: "%IP_VFR-4-FRAG_TABLE_OVERFLOW"

${MSG}
old: "%IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/1: the fragment table..."
new: "GigabitEthernet0/1: the fragment table..."



On Mon, Sep 17, 2018 at 3:26 PM Balazs Scheidler <bazsi77@gmail.com> wrote:
I think this would be handled by our cisco-parser() properly. Was this format properly parsed in 3.9?

On Mon, Sep 17, 2018 at 8:35 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Confirmed when a cisco device does not send a timezone syslog-ng fails to properly parse the message.  TImezone can be toggled on like this:

mydevice.com(config)#service timestamps log datetime show-timezone

 


On Mon, Sep 17, 2018 at 12:59 PM Nik Ambrosch <nik@ambrosch.com> wrote:
Well that's certainly easier, thanks :)

Patch did not succeed in fixing the issue - it may be because of the missing UTC that I mentioned in my earlier response?

Sep 11 12:14:51 1.1.1.1 <190>53: Sep 11 16:14:50.588: %SYS-6-LOGGINGHOST_STARTSTOP: broken device
Sep 11 13:17:39 2.2.2.2 <190>10474: Sep 11 17:17:38.447 UTC: %SYS-6-LOGGINGHOST_STARTSTOP: working device




On Mon, Sep 17, 2018 at 12:36 PM Budai, László <laszlo.budai@oneidentity.com> wrote:
the PR has been merged six days ago, and this build has been finished four days ago: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/build/797694/ which means, that these packages should contain that change.

L.


On Mon, Sep 17, 2018 at 6:29 PM, Nik Ambrosch <nik@ambrosch.com> wrote:
Having a hard time getting lib/ivykis to compile on a centos 7.4 system right now, haven't been able to figure it out yet.


On Mon, Sep 17, 2018 at 11:58 AM Budai, László <laszlo.budai@oneidentity.com> wrote:
Hi,

Do you need help in how to test the latest changes?

L.


On Mon, Sep 17, 2018 at 3:03 PM, Nik Ambrosch <nik@ambrosch.com> wrote:
Is there any other information I can provide in order to help resolve this?


On Tue, Sep 11, 2018 at 1:48 PM Nik Ambrosch <nik@ambrosch.com> wrote:
I setup a syslog-ng 3.9 device to capture a message using

network(transport(tcp) flags(no-parse));

Here's what was logged:

Sep 11 12:14:51 1.1.1.1 <190>53: Sep 11 16:14:50.588: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host blahblah

Followed up with a device that delivers a proper hostname, this is what was logged:

Sep 11 13:17:39 2.2.2.2 <190>10474: Sep 11 17:17:38.447 UTC: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host blahblah

Looks like the difference is the working device contains a timezone where as the non-working device does not.  Everything else is the same however neither contain a hostname like in your example.





On Tue, Sep 11, 2018 at 9:45 AM, Budai, László <laszlo.budai@oneidentity.com> wrote:
Hi,

instead of reverting the ipv6 heuristic, I propose another solution:

I think that when a timestamp is followed by a colon(':'), it is part of the timestamp and the (legacy) timestamp parser should 'eat' it.

I tested with the following log:
<0>91: *Oct 07 03:10:04: mydevice.com %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=150.1.1.1, prot=50, spi=0x72662541(1919296833), srcaddr=150.3.1.3

Could you validate that this is the same format that you have?

L.

On Mon, Sep 10, 2018 at 4:32 PM, Scheidler, Balázs <balazs.scheidler@oneidentity.com> wrote:
This branch has a patch to revert that specific commit, and I've confirmed that it resolves the issue for me, in exchange for not supporting IPV6 addresses in the hostname field.


On Mon, Sep 10, 2018 at 3:55 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
This patch broke it:

399d565e9857e7cb41253e9a714d5cc6ad4d50fb.

This patch can be reverted easily even on the latest master to resolve the issue.

On Mon, Sep 10, 2018 at 3:16 PM Scheidler, Balázs <balazs.scheidler@oneidentity.com> wrote:
This is probably not it, the syslog-parser() changed some behaviours that changed it.

On Mon, Sep 10, 2018, 13:45 Budai, László <laszlo.budai@oneidentity.com> wrote:
Hi,

in syslog-ng OSE 3.13 [1] we introduced a new feature, called app-parser [2] and the default network network driver is using it.
Maybe that could cause your issue.  If this is the case, then we have another PR [3] which makes it possible to disable the auto-parse (also part of 3.13).

Example:
source s_network {
  default-network-drivers(auto-parse(no));
};

If it not solves your problem then could you share the relevant part of your config?




regards,
Laszlo Budai


On Fri, Sep 7, 2018 at 6:00 PM, Nik Ambrosch <nik@ambrosch.com> wrote:
Recently I upgraded my centralized loghost from 3.9 -> 3.15 and I noticed that some of my cisco devices started being logged in an undesirable format... I don't want to enable the cisco parser because more than just cisco messages get delivered to this interface.  Here are the relevant fields that have changed before/after the upgrade:

syslog-ng 3.9, before upgrade ---
    ${FULLHOST}: "mydevice.com"
    ${PROGRAM}: ""
    message: "%CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for..."

syslog-ng 3.15, before upgrade ---
    ${FULLHOST}: ":"
    ${PROGRAM}: "%CRYPTO-4-RECVD_PKT_INV_SPI"
    ${MSG}: "decaps: rec'd IPSEC packet has invalid spi for..."


Is this unintended behavior or a bug?  This particular device is a Cisco 3845 running ios 12.4(22)T4.

Thanks in advance.

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



--
Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



--
Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq