[syslog-ng] syslog-ng is skipping syslog events with no PRI

Tue Mar 20 23:48:15 UTC 2018

On Tue, Mar 20, 2018 at 10:23 AM, Asif Iqbal <vadud3 at gmail.com> wrote:

>
>
> On Tue, Mar 20, 2018 at 7:21 AM, Nagy, Gábor <gabor.nagy at balabit.com>
> wrote:
>
>> Hi Asif!
>>
>> I think the problem with the first message comes from the message
>> structure and the filter statement.
>> Its structure does not conform to either syslog RFC standards (RFC3164 or
>> RFC5424).
>> Syslog-ng still tries to parse the log message by its internal heuristics
>> and the "alarmLog" text is parsed as the 'program' field.
>> You can debug syslog-ng parsing with the format-json template in the
>> destination:
>>         `template("$(format-json -s syslog-proto)\n")`
>>
>> The output for this was:
>> {"PROGRAM":"alarmLog,","PRIORITY":"notice","MESSAGE":"applianceName=Branch-UC1,
>> tenantName=DEMO-CORP, alarmType=sla-not-met, alarmKey=INTERNET,
>> generateTime=1521503387, applianceId=1, vsnId=0, tenantId=2,
>> alarmCause=datapathState, alarmClearable=yes, alarmClass=cleared,
>> alarmKind=symptom, alarmEventType=equipmentAlarm, alarmSeverity=cleared,
>> alarmOwner=tenant, alarmSeqNo=1568, alarmText=delay:9 msec,
>> siteName=Branch-UC1","HOST":"+0000","FACILITY":"user","DATE":"Mar 13
>> 23:49:48"}
>>
>> The filter statement uses the `match()` filter which works on both the
>> header and message part of the log message and thus would match for the
>> first log message if the `value("MESSAGE")` part would not be there.
>> With that you restricted the filter to match only on the message part.
>> If you remove the value("MESSAGE") from the filter statement it will work.
>>
>
> That was it. It is working now!!
>

With global option keep_hostname(no) and use_dns(yes), I get the host A
record gets prepended to the log.

But I like to have the hostname of the client prepended instead, which is
not same as the dns A name of the client

I have about 40 clients in this group

If I use keep_hostname(yes), it provides no hostname at all, probably
because of these syslogs do not conform with syslog RFC standard?

    source s_alarm { udp( port(514) keep_hostname(yes) ); };
    destination d_alarm { file("/var/log/alarms.log"); };
    filter f_alarm { match("alarmLog" value("PROGRAM")); };
    log { source(s_alarm); filter(f_alarm); destination(d_alarm); };

faxmodem (from #syslog-ng) gave me few options like using local /etc/hosts
for lookup or using a metadata[1] from external file.
That feature, looks like, introduced on version 3.8. I am running latest
centos 7 with syslog-ng version 3.5.6.

Is there a easy way to prepend the actual hostname of the client?

Thanks for your help

[1]
https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/data-enrichment-add-contextual-data.html

>
> BTW, is there a way to generate a feed a pcap to some program on the
> terminal to generate a json formatted output like this short from modifying
> the syslog-ng config
> for destination to a template like you are showing?
>
> Thanks for your help and I see the logs being written to the file now!
>
>
>>
>> Regards,
>> Gabor
>>
>>
>> On Tue, Mar 20, 2018 at 1:52 AM, Asif Iqbal <vadud3 at gmail.com> wrote:
>>
>>> syslog-ng is *NOT* writing syslog like this to a file which has no <
>>> *PRI*>
>>>
>>> 23:49:48.306587 IP 192.168.1.100.39567 > 192.168.100.100.514: [|syslog]
>>> E..... at .>..U...g..........T.2018-03-19T23:49:48+0000 alarmLog,
>>> applianceName=Branch-UC1, tenantName=DEMO-CORP, alarmType=sla-not-met,
>>> alarmKey=INTERNET, generateTime=1521503387, applianceId=1, vsnId=0,
>>> tenantId=2, alarmCause=datapathState, alarmClearable=yes,
>>> alarmClass=cleared, alarmKind=symptom, alarmEventType=equipmentAlarm,
>>> alarmSeverity=cleared, alarmOwner=tenant, alarmSeqNo=1568,
>>> alarmText="delay:9 msec", siteName=Branch-UC1
>>> ................
>>>
>>>
>>> syslog-ng is writing syslog like this to a file *OK *
>>>
>>> 23:50:26.930023 IP 192.168.1.100.55078 > 192.168.100.100.514: SYSLOG
>>> mail.info, length: 76
>>> E..h.B at .>......g.....&...Tt.<22>Mar 19 23:50:26 SVL-remotehost-02 root:
>>> this is third test alarmLog................
>>>
>>>
>>> Here is my syslog-ng config
>>>     source s_udp { udp(ip(0.0.0.0) port(514)); };
>>>     destination d_alarm { file("/var/log/alarms.log"); };
>>>     filter f_alarm { match("alarmLog" value("MESSAGE")); };
>>>     log { source(s_udp); filter(f_alarm); destination(d_alarm); };
>>>
>>> I am using syslog-ng version 3.5.6 on centos 7
>>>
>>> Any idea why syslog-ng is writing the first log event into a file?
>>>
>>> Appreciate any help!
>>>
>>>
>>> --
>>> Asif Iqbal
>>> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
>>> A: Because it messes up the order in which people normally read text.
>>> Q: Why is top-posting such a bad thing?
>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
> Asif Iqbal
> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
>
>

-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180320/640d204d/attachment.html>