[syslog-ng] syslog-ng is skipping syslog events with no PRI

Wed Mar 21 13:46:32 UTC 2018

On Wed, Mar 21, 2018 at 8:26 AM, Nagy, Gábor <gabor.nagy at balabit.com> wrote:

> Hi Asif,
>
> If you need add-contextual-data can you upgrade using one of our
> unofficial binaries maybe?
> https://syslog-ng.com/blog/installing-latest-syslog-ng-on-
> rhel-and-other-rpm-distributions/#_ga=2.60852911.24278624.
> 1521459042-804758321.1501593964
>
> Can you describe your use case with examples, please?
>

My client hostname is svl-search-01 and its IP resolves to svl-remote-01.
Its syslogs do not have any PRI or hostname in HOST field.

I like to have svl-search-01 in the HOST field.

>
> Even if you set keep-hostname to 'yes' and message doesn't contain a
> hostname, syslog-ng will add a hostname/IP to the message depending on
> use-dns() option.
> See `keep-hostname` option in doc:
> https://syslog-ng.com/documents/html/syslog-ng-ose-
> latest-guides/en/syslog-ng-ose-guide-admin/html/reference-options.html
>
> Regards,
> Gabor
>
> On Wed, Mar 21, 2018 at 12:48 AM, Asif Iqbal <vadud3 at gmail.com> wrote:
>
>> On Tue, Mar 20, 2018 at 10:23 AM, Asif Iqbal <vadud3 at gmail.com> wrote:
>>
>>>
>>>
>>> On Tue, Mar 20, 2018 at 7:21 AM, Nagy, Gábor <gabor.nagy at balabit.com>
>>> wrote:
>>>
>>>> Hi Asif!
>>>>
>>>> I think the problem with the first message comes from the message
>>>> structure and the filter statement.
>>>> Its structure does not conform to either syslog RFC standards (RFC3164
>>>> or RFC5424).
>>>> Syslog-ng still tries to parse the log message by its internal
>>>> heuristics and the "alarmLog" text is parsed as the 'program' field.
>>>> You can debug syslog-ng parsing with the format-json template in the
>>>> destination:
>>>>         `template("$(format-json -s syslog-proto)\n")`
>>>>
>>>> The output for this was:
>>>> {"PROGRAM":"alarmLog,","PRIORITY":"notice","MESSAGE":"applianceName=Branch-UC1,
>>>> tenantName=DEMO-CORP, alarmType=sla-not-met, alarmKey=INTERNET,
>>>> generateTime=1521503387, applianceId=1, vsnId=0, tenantId=2,
>>>> alarmCause=datapathState, alarmClearable=yes, alarmClass=cleared,
>>>> alarmKind=symptom, alarmEventType=equipmentAlarm, alarmSeverity=cleared,
>>>> alarmOwner=tenant, alarmSeqNo=1568, alarmText=delay:9 msec,
>>>> siteName=Branch-UC1","HOST":"+0000","FACILITY":"user","DATE":"Mar 13
>>>> 23:49:48"}
>>>>
>>>> The filter statement uses the `match()` filter which works on both the
>>>> header and message part of the log message and thus would match for the
>>>> first log message if the `value("MESSAGE")` part would not be there.
>>>> With that you restricted the filter to match only on the message part.
>>>> If you remove the value("MESSAGE") from the filter statement it will
>>>> work.
>>>>
>>>
>>> That was it. It is working now!!
>>>
>>
>>
>> With global option keep_hostname(no) and use_dns(yes), I get the host A
>> record gets prepended to the log.
>>
>> But I like to have the hostname of the client prepended instead, which is
>> not same as the dns A name of the client
>>
>> I have about 40 clients in this group
>>
>> If I use keep_hostname(yes), it provides no hostname at all, probably
>> because of these syslogs do not conform with syslog RFC standard?
>>
>>     source s_alarm { udp( port(514) keep_hostname(yes) ); };
>>     destination d_alarm { file("/var/log/alarms.log"); };
>>     filter f_alarm { match("alarmLog" value("PROGRAM")); };
>>     log { source(s_alarm); filter(f_alarm); destination(d_alarm); };
>>
>> faxmodem (from #syslog-ng) gave me few options like using local
>> /etc/hosts for lookup or using a metadata[1] from external file.
>> That feature, looks like, introduced on version 3.8. I am running latest
>> centos 7 with syslog-ng version 3.5.6.
>>
>> Is there a easy way to prepend the actual hostname of the client?
>>
>> Thanks for your help
>>
>>
>>
>>
>>
>> [1] https://syslog-ng.com/documents/html/syslog-ng-ose-lates
>> t-guides/en/syslog-ng-ose-guide-admin/html/data-enrichme
>> nt-add-contextual-data.html
>>
>>
>>
>>
>>
>>
>>
>>
>>>
>>> BTW, is there a way to generate a feed a pcap to some program on the
>>> terminal to generate a json formatted output like this short from modifying
>>> the syslog-ng config
>>> for destination to a template like you are showing?
>>>
>>> Thanks for your help and I see the logs being written to the file now!
>>>
>>>
>>>>
>>>> Regards,
>>>> Gabor
>>>>
>>>>
>>>> On Tue, Mar 20, 2018 at 1:52 AM, Asif Iqbal <vadud3 at gmail.com> wrote:
>>>>
>>>>> syslog-ng is *NOT* writing syslog like this to a file which has no <
>>>>> *PRI*>
>>>>>
>>>>> 23:49:48.306587 IP 192.168.1.100.39567 > 192.168.100.100.514: [|syslog]
>>>>> E..... at .>..U...g..........T.2018-03-19T23:49:48+0000 alarmLog,
>>>>> applianceName=Branch-UC1, tenantName=DEMO-CORP, alarmType=sla-not-met,
>>>>> alarmKey=INTERNET, generateTime=1521503387, applianceId=1, vsnId=0,
>>>>> tenantId=2, alarmCause=datapathState, alarmClearable=yes,
>>>>> alarmClass=cleared, alarmKind=symptom, alarmEventType=equipmentAlarm,
>>>>> alarmSeverity=cleared, alarmOwner=tenant, alarmSeqNo=1568,
>>>>> alarmText="delay:9 msec", siteName=Branch-UC1
>>>>> ................
>>>>>
>>>>>
>>>>> syslog-ng is writing syslog like this to a file *OK *
>>>>>
>>>>> 23:50:26.930023 IP 192.168.1.100.55078 > 192.168.100.100.514: SYSLOG
>>>>> mail.info, length: 76
>>>>> E..h.B at .>......g.....&...Tt.<22>Mar 19 23:50:26 SVL-remotehost-02
>>>>> root: this is third test alarmLog................
>>>>>
>>>>>
>>>>> Here is my syslog-ng config
>>>>>     source s_udp { udp(ip(0.0.0.0) port(514)); };
>>>>>     destination d_alarm { file("/var/log/alarms.log"); };
>>>>>     filter f_alarm { match("alarmLog" value("MESSAGE")); };
>>>>>     log { source(s_udp); filter(f_alarm); destination(d_alarm); };
>>>>>
>>>>> I am using syslog-ng version 3.5.6 on centos 7
>>>>>
>>>>> Any idea why syslog-ng is writing the first log event into a file?
>>>>>
>>>>> Appreciate any help!
>>>>>
>>>>>
>>>>> --
>>>>> Asif Iqbal
>>>>> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
>>>>> A: Because it messes up the order in which people normally read text.
>>>>> Q: Why is top-posting such a bad thing?
>>>>>
>>>>>
>>>>> ____________________________________________________________
>>>>> __________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support
>>>>> /documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Asif Iqbal
>>> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
>>> A: Because it messes up the order in which people normally read text.
>>> Q: Why is top-posting such a bad thing?
>>>
>>>
>>
>>
>> --
>> Asif Iqbal
>> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>

-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180321/a2f19a7f/attachment-0001.html>