Forwarding flat files to a remote syslog-ng server
Hi, I'm trying to send various flat files to a central syslog-ng server. So, I have the following setups but somehow using the "local1" and "local2" facilities don't work, although no errors when restarting syslog services on both client & server. Any advice would be appreciated much! ======================================================================= My syslog-ng CLIENT (running syslog-ng 3.6.2): source s_file1 { file("/var/log/syslog/file1.log" follow-freq(1) flags(no-parse) program_override("audit")); }; filter f_file1 { facility(local1); }; destination d_file1 { syslog("10.10.10.10" transport("udp") port(514)); }; log { source(s_file1); filter(f_file1); destination(d_file1); }; source s_file2 { file("/var/log/syslog/file2.log" follow-freq(1) flags(no-parse) program_override("audit")); }; filter f_file2 { facility(local2); }; destination d_file2 { syslog("10.10.10.10" transport("udp") port(514)); }; log { source(s_file2); filter(f_file2); destination(d_file2); }; ======================================================================== My syslog-ng SERVER (10.10.10.10 - running syslog-ng 2.0.9): destination d_file1 { file ("/var/log/syslogs/file1-$YEAR$MONTH$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; filter f_file1 { facility(local1); }; log { source(remote); filter(f_file1); destination(d_file1); }; destination d_file2 { file ("/var/log/syslogs/file2-$YEAR$MONTH$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; filter f_file2 { facility(local2); }; log { source(remote); filter(f_file2); destination(d_file2); }; Thanks. - Young
Hi, Couple of notes: * When reading messages from a file, you are using flags(no-parse) which means that you will not have facility information. I dont see that you need the filter at all, as you are using dedicated destinations. I think that filter can be removed. * you dont need separate destinations to each file, just use one destination and connect both sources to it * If you run syslog-ng in foreground and enable debug mode it emits information about messages entering and leaving * 3.6 is ancient, not to mention 2.0.9, you should probably upgrade On May 22, 2018 04:08, "Song, Young" <young.song@sap.com> wrote:
Hi,
I’m trying to send various flat files to a central syslog-ng server.
So, I have the following setups but somehow using the “local1” and “local2” facilities don’t work, although no errors when restarting syslog services on both client & server.
Any advice would be appreciated much!
=======================================================================
*My syslog-ng CLIENT (running syslog-ng 3.6.2):*
source s_file1 { file("/var/log/syslog/file1.log" follow-freq(1) flags(no-parse) program_override("audit")); };
filter f_file1 { facility(local1); };
destination d_file1 { syslog("10.10.10.10" transport("udp") port(514)); };
log { source(s_file1); filter(f_file1); destination(d_file1); };
source s_file2 { file("/var/log/syslog/file2.log" follow-freq(1) flags(no-parse) program_override("audit")); };
filter f_file2 { facility(local2); };
destination d_file2 { syslog("10.10.10.10" transport("udp") port(514)); };
log { source(s_file2); filter(f_file2); destination(d_file2); };
========================================================================
*My syslog-ng SERVER (10.10.10.10 – running syslog-ng 2.0.9):*
destination d_file1 { file ("/var/log/syslogs/file1-$YEAR$MONTH$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); };
filter f_file1 { facility(local1); };
log { source(remote); filter(f_file1); destination(d_file1); };
destination d_file2 { file ("/var/log/syslogs/file2-$YEAR$MONTH$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); };
filter f_file2 { facility(local2); };
log { source(remote); filter(f_file2); destination(d_file2); };
Thanks.
- Young
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Balázs, In fact, I’m trying to forward the following 3 different types of logs: 1. Audit logs (from auditd on Solaris 10 using the “audit” facility) --> syslog-ng server cannot recognize “audit” facility, though, and thus not working 1. OS logs (from syslog facilities) --> Using various facilities and levels 1. Oracle logs (flat file) --> Using the “local1” facility, as in my first email, but not working How to separate them so that each logs get stored in different locations is my challenge. Thanks. - Young From: syslog-ng [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Scheidler, Balázs Sent: Monday, May 21, 2018 10:40 PM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Forwarding flat files to a remote syslog-ng server Hi, Couple of notes: * When reading messages from a file, you are using flags(no-parse) which means that you will not have facility information. I dont see that you need the filter at all, as you are using dedicated destinations. I think that filter can be removed. * you dont need separate destinations to each file, just use one destination and connect both sources to it * If you run syslog-ng in foreground and enable debug mode it emits information about messages entering and leaving * 3.6 is ancient, not to mention 2.0.9, you should probably upgrade On May 22, 2018 04:08, "Song, Young" <young.song@sap.com<mailto:young.song@sap.com>> wrote: Hi, I’m trying to send various flat files to a central syslog-ng server. So, I have the following setups but somehow using the “local1” and “local2” facilities don’t work, although no errors when restarting syslog services on both client & server. Any advice would be appreciated much! ======================================================================= My syslog-ng CLIENT (running syslog-ng 3.6.2): source s_file1 { file("/var/log/syslog/file1.log" follow-freq(1) flags(no-parse) program_override("audit")); }; filter f_file1 { facility(local1); }; destination d_file1 { syslog("10.10.10.10" transport("udp") port(514)); }; log { source(s_file1); filter(f_file1); destination(d_file1); }; source s_file2 { file("/var/log/syslog/file2.log" follow-freq(1) flags(no-parse) program_override("audit")); }; filter f_file2 { facility(local2); }; destination d_file2 { syslog("10.10.10.10" transport("udp") port(514)); }; log { source(s_file2); filter(f_file2); destination(d_file2); }; ======================================================================== My syslog-ng SERVER (10.10.10.10 – running syslog-ng 2.0.9): destination d_file1 { file ("/var/log/syslogs/file1-$YEAR$MONTH$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; filter f_file1 { facility(local1); }; log { source(remote); filter(f_file1); destination(d_file1); }; destination d_file2 { file ("/var/log/syslogs/file2-$YEAR$MONTH$DAY.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; filter f_file2 { facility(local2); }; log { source(remote); filter(f_file2); destination(d_file2); }; Thanks. - Young ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
"Song" == Song, Young <young.song@sap.com> writes:
Song> I'm trying to send various flat files to a central syslog-ng server. Song> So, I have the following setups but somehow using the "local1" Song> and "local2" facilities don't work, although no errors when Song> restarting syslog services on both client & server. Song> My syslog-ng CLIENT (running syslog-ng 3.6.2): Song> source s_file1 { Song> file("/var/log/syslog/file1.log" Song> follow-freq(1) flags(no-parse) Song> program_override("audit")); Song> }; Song> filter f_file1 { facility(local1); }; Song> destination d_file1 { Song> syslog("10.10.10.10" transport("udp") port(514)); Song> }; Song> log { Song> source(s_file1); filter(f_file1); destination(d_file1); Song> }; Both your `s_file1` and `s_file2` sources have `flags(no-parse)`, which tells syslog-ng to *not* parse the file. As such, it will not be able to figure out the facilities. Mind you, in a lot of cases, logs already on disk rarely contain the facilities they were originally sent with. In this case, even if you dropped the no-parse flag, the filter still wouldn't work, because that information is simply not in the file. Now, if you want to read `file1.log`, and *add* the local1 facility before forwarding, that'd be a different thing - but as far as I know, that's not something syslog-ng supports at the moment. I would suggest a different approach: instead of the facility, set some custom SDATA field on the sending side, use the RFC5424 protocol (in other words, `syslog()`, like you already do) for transfer over the network, and filter based on the SDATA field on the server side. For example: rewrite r_file1 { set("audit-log" value(".SDATA.custom@18372.4.source")); }; And on the server side: filter f_audit_source { match("audit-log" value(".SDATA.custom@18372.4.source")); }; And similarly for the others. -- |8]
participants (3)
-
Gergely Nagy
-
Scheidler, Balázs
-
Song, Young