[syslog-ng] syslog-ng is skipping syslog events with no PRI

Wed Mar 21 12:26:31 UTC 2018

Hi Asif,

If you need add-contextual-data can you upgrade using one of our unofficial
binaries maybe?
https://syslog-ng.com/blog/installing-latest-syslog-ng-
on-rhel-and-other-rpm-distributions/#_ga=2.60852911.
24278624.1521459042-804758321.1501593964

Can you describe your use case with examples, please?

Even if you set keep-hostname to 'yes' and message doesn't contain a
hostname, syslog-ng will add a hostname/IP to the message depending on
use-dns() option.
See `keep-hostname` option in doc:
https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-options.html

Regards,
Gabor

On Wed, Mar 21, 2018 at 12:48 AM, Asif Iqbal <vadud3 at gmail.com> wrote:

> On Tue, Mar 20, 2018 at 10:23 AM, Asif Iqbal <vadud3 at gmail.com> wrote:
>
>>
>>
>> On Tue, Mar 20, 2018 at 7:21 AM, Nagy, Gábor <gabor.nagy at balabit.com>
>> wrote:
>>
>>> Hi Asif!
>>>
>>> I think the problem with the first message comes from the message
>>> structure and the filter statement.
>>> Its structure does not conform to either syslog RFC standards (RFC3164
>>> or RFC5424).
>>> Syslog-ng still tries to parse the log message by its internal
>>> heuristics and the "alarmLog" text is parsed as the 'program' field.
>>> You can debug syslog-ng parsing with the format-json template in the
>>> destination:
>>>         `template("$(format-json -s syslog-proto)\n")`
>>>
>>> The output for this was:
>>> {"PROGRAM":"alarmLog,","PRIORITY":"notice","MESSAGE":"applianceName=Branch-UC1,
>>> tenantName=DEMO-CORP, alarmType=sla-not-met, alarmKey=INTERNET,
>>> generateTime=1521503387, applianceId=1, vsnId=0, tenantId=2,
>>> alarmCause=datapathState, alarmClearable=yes, alarmClass=cleared,
>>> alarmKind=symptom, alarmEventType=equipmentAlarm, alarmSeverity=cleared,
>>> alarmOwner=tenant, alarmSeqNo=1568, alarmText=delay:9 msec,
>>> siteName=Branch-UC1","HOST":"+0000","FACILITY":"user","DATE":"Mar 13
>>> 23:49:48"}
>>>
>>> The filter statement uses the `match()` filter which works on both the
>>> header and message part of the log message and thus would match for the
>>> first log message if the `value("MESSAGE")` part would not be there.
>>> With that you restricted the filter to match only on the message part.
>>> If you remove the value("MESSAGE") from the filter statement it will
>>> work.
>>>
>>
>> That was it. It is working now!!
>>
>
>
> With global option keep_hostname(no) and use_dns(yes), I get the host A
> record gets prepended to the log.
>
> But I like to have the hostname of the client prepended instead, which is
> not same as the dns A name of the client
>
> I have about 40 clients in this group
>
> If I use keep_hostname(yes), it provides no hostname at all, probably
> because of these syslogs do not conform with syslog RFC standard?
>
>     source s_alarm { udp( port(514) keep_hostname(yes) ); };
>     destination d_alarm { file("/var/log/alarms.log"); };
>     filter f_alarm { match("alarmLog" value("PROGRAM")); };
>     log { source(s_alarm); filter(f_alarm); destination(d_alarm); };
>
> faxmodem (from #syslog-ng) gave me few options like using local /etc/hosts
> for lookup or using a metadata[1] from external file.
> That feature, looks like, introduced on version 3.8. I am running latest
> centos 7 with syslog-ng version 3.5.6.
>
> Is there a easy way to prepend the actual hostname of the client?
>
> Thanks for your help
>
>
>
>
>
> [1] https://syslog-ng.com/documents/html/syslog-ng-ose-
> latest-guides/en/syslog-ng-ose-guide-admin/html/data-
> enrichment-add-contextual-data.html
>
>
>
>
>
>
>
>
>>
>> BTW, is there a way to generate a feed a pcap to some program on the
>> terminal to generate a json formatted output like this short from modifying
>> the syslog-ng config
>> for destination to a template like you are showing?
>>
>> Thanks for your help and I see the logs being written to the file now!
>>
>>
>>>
>>> Regards,
>>> Gabor
>>>
>>>
>>> On Tue, Mar 20, 2018 at 1:52 AM, Asif Iqbal <vadud3 at gmail.com> wrote:
>>>
>>>> syslog-ng is *NOT* writing syslog like this to a file which has no <
>>>> *PRI*>
>>>>
>>>> 23:49:48.306587 IP 192.168.1.100.39567 > 192.168.100.100.514: [|syslog]
>>>> E..... at .>..U...g..........T.2018-03-19T23:49:48+0000 alarmLog,
>>>> applianceName=Branch-UC1, tenantName=DEMO-CORP, alarmType=sla-not-met,
>>>> alarmKey=INTERNET, generateTime=1521503387, applianceId=1, vsnId=0,
>>>> tenantId=2, alarmCause=datapathState, alarmClearable=yes,
>>>> alarmClass=cleared, alarmKind=symptom, alarmEventType=equipmentAlarm,
>>>> alarmSeverity=cleared, alarmOwner=tenant, alarmSeqNo=1568,
>>>> alarmText="delay:9 msec", siteName=Branch-UC1
>>>> ................
>>>>
>>>>
>>>> syslog-ng is writing syslog like this to a file *OK *
>>>>
>>>> 23:50:26.930023 IP 192.168.1.100.55078 > 192.168.100.100.514: SYSLOG
>>>> mail.info, length: 76
>>>> E..h.B at .>......g.....&...Tt.<22>Mar 19 23:50:26 SVL-remotehost-02
>>>> root: this is third test alarmLog................
>>>>
>>>>
>>>> Here is my syslog-ng config
>>>>     source s_udp { udp(ip(0.0.0.0) port(514)); };
>>>>     destination d_alarm { file("/var/log/alarms.log"); };
>>>>     filter f_alarm { match("alarmLog" value("MESSAGE")); };
>>>>     log { source(s_udp); filter(f_alarm); destination(d_alarm); };
>>>>
>>>> I am using syslog-ng version 3.5.6 on centos 7
>>>>
>>>> Any idea why syslog-ng is writing the first log event into a file?
>>>>
>>>> Appreciate any help!
>>>>
>>>>
>>>> --
>>>> Asif Iqbal
>>>> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
>>>> A: Because it messes up the order in which people normally read text.
>>>> Q: Why is top-posting such a bad thing?
>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> --
>> Asif Iqbal
>> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>>
>>
>
>
> --
> Asif Iqbal
> PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180321/cce91eb4/attachment.html>