On Wed, Mar 21, 2018 at 8:26 AM, Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hi Asif,


Can you describe your use case with examples, please?


My client hostname is svl-search-01 and its IP resolves to svl-remote-01. Its syslogs do not have any PRI or hostname in HOST field.

I like to have svl-search-01 in the HOST field.


 

Even if you set keep-hostname to 'yes' and message doesn't contain a hostname, syslog-ng will add a hostname/IP to the message depending on use-dns() option.
See `keep-hostname` option in doc:

Regards,
Gabor

On Wed, Mar 21, 2018 at 12:48 AM, Asif Iqbal <vadud3@gmail.com> wrote:
On Tue, Mar 20, 2018 at 10:23 AM, Asif Iqbal <vadud3@gmail.com> wrote:


On Tue, Mar 20, 2018 at 7:21 AM, Nagy, Gábor <gabor.nagy@balabit.com> wrote:
Hi Asif!

I think the problem with the first message comes from the message structure and the filter statement.
Its structure does not conform to either syslog RFC standards (RFC3164 or RFC5424).
Syslog-ng still tries to parse the log message by its internal heuristics and the "alarmLog" text is parsed as the 'program' field.
You can debug syslog-ng parsing with the format-json template in the destination:
        `template("$(format-json -s syslog-proto)\n")`

The output for this was:
{"PROGRAM":"alarmLog,","PRIORITY":"notice","MESSAGE":"applianceName=Branch-UC1, tenantName=DEMO-CORP, alarmType=sla-not-met, alarmKey=INTERNET, generateTime=1521503387, applianceId=1, vsnId=0, tenantId=2, alarmCause=datapathState, alarmClearable=yes, alarmClass=cleared, alarmKind=symptom, alarmEventType=equipmentAlarm, alarmSeverity=cleared, alarmOwner=tenant, alarmSeqNo=1568, alarmText=delay:9 msec, siteName=Branch-UC1","HOST":"+0000","FACILITY":"user","DATE":"Mar 13 23:49:48"}

The filter statement uses the `match()` filter which works on both the header and message part of the log message and thus would match for the first log message if the `value("MESSAGE")` part would not be there.
With that you restricted the filter to match only on the message part.
If you remove the value("MESSAGE") from the filter statement it will work.

That was it. It is working now!! 


With global option keep_hostname(no) and use_dns(yes), I get the host A record gets prepended to the log.

But I like to have the hostname of the client prepended instead, which is not same as the dns A name of the client

I have about 40 clients in this group

If I use keep_hostname(yes), it provides no hostname at all, probably because of these syslogs do not conform with syslog RFC standard?

    source s_alarm { udp( port(514) keep_hostname(yes) ); };
    destination d_alarm { file("/var/log/alarms.log"); };
    filter f_alarm { match("alarmLog" value("PROGRAM")); };
    log { source(s_alarm); filter(f_alarm); destination(d_alarm); };

faxmodem (from #syslog-ng) gave me few options like using local /etc/hosts for lookup or using a metadata[1] from external file. 
That feature, looks like, introduced on version 3.8. I am running latest centos 7 with syslog-ng version 3.5.6.

Is there a easy way to prepend the actual hostname of the client?

Thanks for your help






    
    
    


 

BTW, is there a way to generate a feed a pcap to some program on the terminal to generate a json formatted output like this short from modifying the syslog-ng config
for destination to a template like you are showing?

Thanks for your help and I see the logs being written to the file now!
 

Regards,
Gabor


On Tue, Mar 20, 2018 at 1:52 AM, Asif Iqbal <vadud3@gmail.com> wrote:
syslog-ng is NOT writing syslog like this to a file which has no <PRI>

23:49:48.306587 IP 192.168.1.100.39567 > 192.168.100.100.514: [|syslog]
E.....@.>..U...g..........T.2018-03-19T23:49:48+0000 alarmLog, applianceName=Branch-UC1, tenantName=DEMO-CORP, alarmType=sla-not-met, alarmKey=INTERNET, generateTime=1521503387, applianceId=1, vsnId=0, tenantId=2, alarmCause=datapathState, alarmClearable=yes, alarmClass=cleared, alarmKind=symptom, alarmEventType=equipmentAlarm, alarmSeverity=cleared, alarmOwner=tenant, alarmSeqNo=1568, alarmText="delay:9 msec", siteName=Branch-UC1
................


syslog-ng is writing syslog like this to a file OK 

23:50:26.930023 IP 192.168.1.100.55078 > 192.168.100.100.514: SYSLOG mail.info, length: 76
E..h.B@.>......g.....&...Tt.<22>Mar 19 23:50:26 SVL-remotehost-02 root: this is third test alarmLog................


Here is my syslog-ng config
    source s_udp { udp(ip(0.0.0.0) port(514)); };
    destination d_alarm { file("/var/log/alarms.log"); };
    filter f_alarm { match("alarmLog" value("MESSAGE")); };
    log { source(s_udp); filter(f_alarm); destination(d_alarm); };

I am using syslog-ng version 3.5.6 on centos 7

Any idea why syslog-ng is writing the first log event into a file?

Appreciate any help!


--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq





--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?




--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq




______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq





--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?