[syslog-ng] Issue with processing syslog event
Balazs Scheidler
bazsi77 at gmail.com
Wed Apr 29 18:50:52 UTC 2020
Hi,
On Wed, Apr 29, 2020, 19:10 Raghunath Adhyapak <funduraghu at gmail.com> wrote:
> Hi,
>
> I was planning to do the following
> 1. Receive events
> 2. Filter events matching format <134>1 1588062776.725141502 C0493 flows
> allow src=10.0.31.145 dst=9.9.9.9 Mac=F1:37:59:38:BA:F8 protocol=udp
> sport=50307 dport=53
> 3. Rewrite epoch timestamp to isodate format
> 4. Use syslog-parser with flags(syslog-protocol)
> 5. Finally, write to destination
>
This sounds good in principle. Try to make the first filter as specific as
possible, to improve performance.
> source s_syslog {
> syslog(
> transport(udp)
> port(514)
> );
> };
>
> filter f_version_and_epoch {
> match ("(?:<[0-9]+>)(?:[0-9]+ )?(?:[0-9]+)(?:\.[0-9]+)? " value
> ("MESSAGE"));
> };
> rewrite epoch_to_isodate {
> subst ("(?:<[0-9]+>)(?:[0-9]+ )?(?<seconds>[0-9]+)(?:\.[0-9]+)? ",
> "$(date --iso-8601=seconds -d @$seconds)", value("MESSAGE"));
> };
>
$(date) doesn't exist, the template language of syslog-ng is not the same
as the shell language, so you can't freely use shell commands.
To use date related features of syslog-ng, you could parse the timestamp
using date-parser(), that can parse this format using %s (if i remember
correctly, supports strptime formats).
Then you can simply use any of syslog-ng's date/time macros.
> log {
> source(s_syslog);
> filter(f_version_and_epoch);
> rewrite(epoch_to_isodate);
> parser {
> syslog-parser(flags(syslog-protocol));
> }
> destination(d_file);
> };
>
> Let me know if this will work.
>
> Currently, I'm getting errors while trying to convert timestamp in seconds
> to isodate.
> Let me know if there is a better way.
>
> Thanks
> Raghu
>
> On Wed, Apr 29, 2020 at 12:38 AM Balazs Scheidler <bazsi77 at gmail.com>
> wrote:
>
>> Rant mode on.
>>
>> This sucks. RFC5424 finally attempted to make an orderly protocol, with
>> prettty explicit format requirements, to fix the shortcomings of rfc3164.
>> What do vendors do? They explicitly specify that they are using rfc5424 and
>> then ignore the very format.
>>
>> Why do they add "1 "? If they left that out, syslog-ng would parse this
>> properly. With that we kick in strict syntax checking, exactly to avoid
>> deteriorating the syntax of rfc5424, exactly what happened to syslog prior
>> to that. History is repeating itself.
>>
>> Rant mode off.
>>
>> I might look into attempting rfc3164 if rfc5424 fails that would possibly
>> solve this. I can't remember if we allow unix timestamp as an incoming
>> format though.
>>
>> On Tue, Apr 28, 2020, 17:52 Raghunath Adhyapak <funduraghu at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> This log line was received from Cisco device.
>>> I believe that the timestamp is not in acceptable format.
>>> Is there any way we can configure syslog-ng to accept timestamp of this
>>> form?
>>>
>>> Thanks
>>> Raghu
>>>
>>> On Tue, Apr 28, 2020 at 6:26 PM Nagy Gábor <gabor.hl at gmail.com> wrote:
>>>
>>>> Hi!
>>>>
>>>> At first look it seems to me that your log message is not in RFC5424
>>>> [1] format, or in RFC3164 [2] format.
>>>> You use syslog() source driver which expects these formats.
>>>>
>>>> Do you receive log messages from other clients too (I guess if you have
>>>> max-connections(500))?
>>>> What device is the log source where messages are coming form?
>>>>
>>>> Regards,
>>>> Gabor
>>>>
>>>> [1] https://tools.ietf.org/html/rfc5424
>>>> [2] https://tools.ietf.org/html/rfc3164
>>>>
>>>> Raghunath Adhyapak <funduraghu at gmail.com> ezt írta (időpont: 2020.
>>>> ápr. 28., K, 13:00):
>>>>
>>>>> Hello all,
>>>>>
>>>>> I am receiving the following syslog line from one of devices.
>>>>>
>>>>> <134>1 1588062776.725141502 C0493 flows allow src=10.0.31.145
>>>>> dst=9.9.9.9 Mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53
>>>>>
>>>>> This line contains a version filed immediately following the priority,
>>>>> and then timestamp is in epoch format as against ISO8601 or other standard
>>>>> format.
>>>>>
>>>>> I see the following error in syslog-ng log:
>>>>> [2020-04-28T10:46:15.340911] Outgoing message; message='Apr 28
>>>>> 10:46:15 ip-172-31-240-95 syslog-ng[27873]: Error processing log message:
>>>>> <134>1>@< 1588062776.725141502 C0493 flows allow src=10.0.31.145
>>>>> dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307 dport=53'
>>>>>
>>>>> What could be the possible issue here?
>>>>>
>>>>> My config is as follows:
>>>>>
>>>>> ##========================================
>>>>> ########################
>>>>> # Global options
>>>>> ########################
>>>>> options {keep_hostname (yes); use_dns (no); mark-freq(30);};
>>>>> ########################
>>>>> # Sources
>>>>> ########################
>>>>> source s_syslog {
>>>>> syslog(
>>>>> transport(udp)
>>>>> port(514)
>>>>> max-connections(500)
>>>>> );
>>>>> };
>>>>> ########################
>>>>> # Destinations
>>>>> ########################
>>>>> destination d_file {
>>>>> file("/var/log/dump.log");
>>>>> };
>>>>> ########################
>>>>> # Log paths
>>>>> ########################
>>>>> log {
>>>>> source(s_syslog);
>>>>> destination(d_file);
>>>>> flags(flow-control);
>>>>> };
>>>>> ##========================================
>>>>>
>>>>> If I check my file /var/log/dump.log, I see that the error line is
>>>>> getting written to it too.
>>>>>
>>>>> root at ip-172-31-240-95:~# tail -f /var/log/dump.log | grep "Error
>>>>> processing"
>>>>> Apr 28 10:44:46 ip-172-31-240-95 syslog-ng[27873]: Error processing
>>>>> log message: <134>1>@< 1588062776.725141502 C0493 flows allow
>>>>> src=10.0.31.145 dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307
>>>>> dport=53
>>>>> Apr 28 10:46:15 ip-172-31-240-95 syslog-ng[27873]: Error processing
>>>>> log message: <134>1>@< 1588062776.725141502 C0493 flows allow
>>>>> src=10.0.31.145 dst=9.9.9.9 mac=F1:37:59:38:BA:F8 protocol=udp sport=50307
>>>>> dport=53
>>>>>
>>>>> Thanks
>>>>> Raghu
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200429/011f7319/attachment-0001.html>
More information about the syslog-ng
mailing list