[syslog-ng] Issue with processing syslog event

Raghunath Adhyapak funduraghu at gmail.com
Wed Apr 29 17:10:35 UTC 2020 

Hi,

I was planning to do the following
1. Receive events
2. Filter events matching format <134>1 1588062776.725141502 C0493 flows
allow src=10.0.31.145 dst=9.9.9.9 Mac=F1:37:59:38:BA:F8 protocol=udp
sport=50307 dport=53
3. Rewrite epoch timestamp to isodate format
4. Use syslog-parser with flags(syslog-protocol)
5. Finally, write to destination

source s_syslog {
        syslog(
                transport(udp)
                port(514)
        );
};

filter f_version_and_epoch {
        match ("(?:<[0-9]+>)(?:[0-9]+ )?(?:[0-9]+)(?:\.[0-9]+)? " value
("MESSAGE"));
};
rewrite epoch_to_isodate {
        subst ("(?:<[0-9]+>)(?:[0-9]+ )?(?<seconds>[0-9]+)(?:\.[0-9]+)? ",
"$(date --iso-8601=seconds -d @$seconds)", value("MESSAGE"));
};

log {
        source(s_syslog);
        filter(f_version_and_epoch);
        rewrite(epoch_to_isodate);
        parser {
                syslog-parser(flags(syslog-protocol));
        }
        destination(d_file);
};

Let me know if this will work.

Currently, I'm getting errors while trying to convert timestamp in seconds
to isodate.
Let me know if there is a better way.

Thanks
Raghu

On Wed, Apr 29, 2020 at 12:38 AM Balazs Scheidler <bazsi77 at gmail.com> wrote:

> Rant mode on.
>
> This sucks. RFC5424 finally attempted to make an orderly protocol, with
> prettty explicit format requirements, to fix the shortcomings of rfc3164.
> What do vendors do? They explicitly specify that they are using rfc5424 and
> then ignore the very format.
>
> Why do they add "1 "? If they left that out, syslog-ng would parse this
> properly. With that we kick in strict syntax checking, exactly to avoid
> deteriorating the syntax of rfc5424, exactly what happened to syslog prior
> to that. History is repeating itself.
>
> Rant mode off.
>
> I might look into attempting rfc3164 if rfc5424 fails that would possibly
> solve this. I can't remember if we allow unix timestamp as an incoming
> format though.
>
> On Tue, Apr 28, 2020, 17:52 Raghunath Adhyapak <funduraghu at gmail.com>
> wrote:
>
>> Hi,
>>
>> This log line was received from Cisco device.
>> I believe that the timestamp is not in acceptable format.
>> Is there any way we can configure syslog-ng to accept timestamp of this
>> form?
>>
>> Thanks
>> Raghu
>>
>> On Tue, Apr 28, 2020 at 6:26 PM Nagy Gábor <gabor.hl at gmail.com> wrote:
>>
>>> Hi!
>>>
>>> At first look it seems to me that your log message is not in RFC5424 [1]
>>> format, or in RFC3164 [2] format.
>>> You use syslog() source driver which expects these formats.
>>>
>>> Do you receive log messages from other clients too (I guess if you have
>>> max-connections(500))?
>>> What device is the log source where messages are coming form?
>>>
>>> Regards,
>>> Gabor
>>>
>>> [1] https://tools.ietf.org/html/rfc5424
>>> [2] https://tools.ietf.org/html/rfc3164
>>>
>>> Raghunath Adhyapak <funduraghu at gmail.com> ezt írta (időpont: 2020. ápr.
>>> 28., K, 13:00):
>>>
>>>> Hello all,
>>>>
>>>> I am receiving the following syslog line from one of devices.
>>>>
>>>> <134>1 1588062776.725141502 C0493 flows allow src=10.0.31.145
>>>> dst=9.9.9.9 Mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53
>>>>
>>>> This line contains a version filed immediately following the priority,
>>>> and then timestamp is in epoch format as against ISO8601 or other standard
>>>> format.
>>>>
>>>> I see the following error in syslog-ng log:
>>>> [2020-04-28T10:46:15.340911] Outgoing message; message='Apr 28 10:46:15
>>>> ip-172-31-240-95 syslog-ng[27873]: Error processing log message: <134>1>@<
>>>> 1588062776.725141502 C0493 flows allow src=10.0.31.145 dst=9.9.9.9
>>>> mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53'
>>>>
>>>> What could be the possible issue here?
>>>>
>>>> My config is as follows:
>>>>
>>>> ##========================================
>>>> ########################
>>>> # Global options
>>>> ########################
>>>> options {keep_hostname (yes);  use_dns (no); mark-freq(30);};
>>>> ########################
>>>> # Sources
>>>> ########################
>>>> source s_syslog {
>>>>         syslog(
>>>>                 transport(udp)
>>>>                 port(514)
>>>>                 max-connections(500)
>>>>         );
>>>> };
>>>> ########################
>>>> # Destinations
>>>> ########################
>>>> destination d_file {
>>>>     file("/var/log/dump.log");
>>>> };
>>>> ########################
>>>> # Log paths
>>>> ########################
>>>> log {
>>>>         source(s_syslog);
>>>>         destination(d_file);
>>>>         flags(flow-control);
>>>> };
>>>> ##========================================
>>>>
>>>> If I check my file /var/log/dump.log, I see that the error line is
>>>> getting written to it too.
>>>>
>>>> root at ip-172-31-240-95:~# tail -f /var/log/dump.log | grep "Error
>>>> processing"
>>>> Apr 28 10:44:46 ip-172-31-240-95 syslog-ng[27873]: Error processing log
>>>> message: <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145
>>>> dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53
>>>> Apr 28 10:46:15 ip-172-31-240-95 syslog-ng[27873]: Error processing log
>>>> message: <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145
>>>> dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53
>>>>
>>>> Thanks
>>>> Raghu
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200429/5e8052ba/attachment.html>

More information about the syslog-ng mailing list