Syslog-ng configuration for network source (udp)
Hello, I am trying to "read" a log stream from interface eth0 and to save it in a file (/var/log/messages_tcp). I am newbie on the use of syslog-ng but I try to learn reading the official manual and a lot of configuration examples. Although, I have try a lot of different configurations, the result is always the same : Syslog-ng cannot read the streaming data. I hope that you can help me. That's the version of syslog-ng that I use: *~# /usr/local/syslog-ng/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Sep 6 2012 14:36:19 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: on Enable-Pacct: off* There is an incoming log stream in eth0 (udp), as you can see next : *~# tcpdump -i eth0 udp 'port 514' -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 06:05:58.669992 IP (tos 0x0, ttl 254, id 20209, offset 0, flags [none], proto UDP (17), length 82) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 54 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host WAN:172.30.254.4\0x0a 06:05:58.670134 IP (tos 0x0, ttl 254, id 52, offset 0, flags [none], proto UDP (17), length 85) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 57 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host outside:172.26.10.4\0x0a ………..* The syslog-ng.conf that I use is the following: * @version: 3.2 options { chain_hostnames(off); long_hostnames(off); use_dns(no); # We don't want DNS names, IP's only. flush_lines(0); stats_freq(43200); time_reopen(10); }; source s_network { # tcp(ip("172.30.6.70") port(514)); # udp(ip("172.30.6.70") port(514)); # syslog( # flags(no-multi-line) # ip(172.30.6.70) # keep-alive(yes) # keep_hostname(yes) # transport(udp) # TLS Options # tls() # ); tcp(); udp(); }; destination d_tcp { file("/var/log/messages_tcp" owner("root") group("adm") perm(0640)); }; log { source(s_network); destination(d_tcp); }; * When I start syslog-ng in debug mode (syslog-ng -Fevd) the results are the followings: *~# /usr/local/syslog-ng/sbin/syslog-ng -Fevdt Trying to open module; module='syslogformat', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libsyslogformat.so' Trying to open module; module='basicfuncs', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libbasicfuncs.so' Trying to open module; module='afsocket', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafsocket.so' Trying to open module; module='affile', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libaffile.so' Trying to open module; module='afprog', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafprog.so' Trying to open module; module='afuser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafuser.so' Trying to open module; module='dbparser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libdbparser.so' Trying to open module; module='csvparser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libcsvparser.so' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.2.4' ^CTermination requested via signal, terminating; syslog-ng shutting down; version='3.2.4' Closing log transport fd; fd='6' Running application hooks; hook='4'* As you can see there is no "Incoming log entry..:" line. I pressed Ctrl+C for the termination of program. Also listening to the network interface seems to be right: *~# netstat -tanpu | grep syslog tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 25950/syslog-ng udp 0 0 0.0.0.0:514 0.0.0.0:* 25950/syslog-ng * Please, can anyone help me with this problem? I don't know what could be blocking the packets, if it is because of the OS or because of syslog-ng configuration.What could it be wrong? I am waiting for your response. PS. Sorry for my "bad" English. Thanks is advance, Alex
Hi, I don't see anything obvious. Could you check whether there are other apps listening on 172.30.6.70:514 ? In other words grep for 514 in the netstat output not for syslog. Are there packet filter rules dropping incoming traffic? Regards, Sandor On Wed, Sep 12, 2012 at 9:50 AM, Alex Clone <alexandros.clone@gmail.com> wrote:
Hello,
I am trying to "read" a log stream from interface eth0 and to save it in a file (/var/log/messages_tcp). I am newbie on the use of syslog-ng but I try to learn reading the official manual and a lot of configuration examples. Although, I have try a lot of different configurations, the result is always the same : Syslog-ng cannot read the streaming data. I hope that you can help me.
That's the version of syslog-ng that I use:
~# /usr/local/syslog-ng/sbin/syslog-ng -V syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Sep 6 2012 14:36:19 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: on Enable-Pcre: on Enable-Pacct: off
There is an incoming log stream in eth0 (udp), as you can see next :
~# tcpdump -i eth0 udp 'port 514' -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 06:05:58.669992 IP (tos 0x0, ttl 254, id 20209, offset 0, flags [none], proto UDP (17), length 82) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 54 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host WAN:172.30.254.4\0x0a 06:05:58.670134 IP (tos 0x0, ttl 254, id 52, offset 0, flags [none], proto UDP (17), length 85) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 57 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host outside:172.26.10.4\0x0a ………..
The syslog-ng.conf that I use is the following:
@version: 3.2
options { chain_hostnames(off); long_hostnames(off); use_dns(no); # We don't want DNS names, IP's only. flush_lines(0); stats_freq(43200); time_reopen(10); };
source s_network {
# tcp(ip("172.30.6.70") port(514)); # udp(ip("172.30.6.70") port(514)); # syslog( # flags(no-multi-line) # ip(172.30.6.70) # keep-alive(yes) # keep_hostname(yes) # transport(udp) # TLS Options # tls() # );
tcp(); udp(); };
destination d_tcp { file("/var/log/messages_tcp" owner("root") group("adm") perm(0640)); };
log { source(s_network); destination(d_tcp); };
When I start syslog-ng in debug mode (syslog-ng -Fevd) the results are the followings:
~# /usr/local/syslog-ng/sbin/syslog-ng -Fevdt Trying to open module; module='syslogformat', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libsyslogformat.so' Trying to open module; module='basicfuncs', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libbasicfuncs.so' Trying to open module; module='afsocket', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafsocket.so' Trying to open module; module='affile', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libaffile.so' Trying to open module; module='afprog', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafprog.so' Trying to open module; module='afuser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libafuser.so' Trying to open module; module='dbparser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libdbparser.so' Trying to open module; module='csvparser', filename='/usr/local/syslog-ng-3.2.4/lib/syslog-ng/libcsvparser.so' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.2.4' ^CTermination requested via signal, terminating; syslog-ng shutting down; version='3.2.4' Closing log transport fd; fd='6' Running application hooks; hook='4'
As you can see there is no "Incoming log entry..:" line. I pressed Ctrl+C for the termination of program.
Also listening to the network interface seems to be right:
~# netstat -tanpu | grep syslog tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 25950/syslog-ng udp 0 0 0.0.0.0:514 0.0.0.0:* 25950/syslog-ng
Please, can anyone help me with this problem? I don't know what could be blocking the packets, if it is because of the OS or because of syslog-ng configuration.What could it be wrong? I am waiting for your response.
PS. Sorry for my "bad" English.
Thanks is advance, Alex
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Wed, Sep 12, 2012 at 11:39 AM, Sandor Geller < Sandor.Geller@morganstanley.com> wrote:
Hi,
I don't see anything obvious. Could you check whether there are other apps listening on 172.30.6.70:514 ? In other words grep for 514 in the netstat output not for syslog. Are there packet filter rules dropping incoming traffic?
Regards,
Sandor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
@ Sandor Geller Thanks for your response. The port 514 is for syslog-ng as it is defined in the services file : *~# cat /etc/services | grep 514 shell 514/tcp cmd # no passwords used syslog-ng 514/udp* * ~# netstat -a --numeric-ports | grep 514 tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN udp 0 0 0.0.0.0:514 0.0.0.0:* unix 3 [ ] STREAM CONNECTED 1514 * As you can see there is only the syslog-ng running on that port. On Wed, Sep 12, 2012 at 11:45 AM, Gergely Nagy <algernon@balabit.hu> wrote:
Alex Clone <alexandros.clone@gmail.com> writes:
*~# tcpdump -i eth0 udp 'port 514' -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 06:05:58.669992 IP (tos 0x0, ttl 254, id 20209, offset 0, flags [none], proto UDP (17), length 82) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 54 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host WAN:172.30.254.4\0x0a 06:05:58.670134 IP (tos 0x0, ttl 254, id 52, offset 0, flags [none], proto UDP (17), length 85) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 57 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host outside:172.26.10.4\0x0a ………..*
These look like CISCO logs, which do not conform to the format syslog-ng expects, and as such, they're not recognised and not processed either.
If you just want to store them as-is, I'd recommend using the flags(no-parse) setting in the s_network source, like this:
source s_network { tcp(flags(no-parse)); udp(flags(no-parse)); };
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
@ Gergely Nagy Thanks for your response and your recommendation. I change the source block to the one that you mail me. The result is exactly the same. Syslog-ng cannot read the stream. By the way, when I tried to log local data using the internal(); option to the source block, syslog-ng worked perfectly. All the data logged to the given file. So, the problem, I have to solve, is why syslog-ng cannot read the interface eth0 (udp). Thanks in advance, Alex
Alex Clone <alexandros.clone@gmail.com> writes:
*~# tcpdump -i eth0 udp 'port 514' -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 06:05:58.669992 IP (tos 0x0, ttl 254, id 20209, offset 0, flags [none], proto UDP (17), length 82) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 54 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host WAN:172.30.254.4\0x0a 06:05:58.670134 IP (tos 0x0, ttl 254, id 52, offset 0, flags [none], proto UDP (17), length 85) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 57 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host outside:172.26.10.4\0x0a ………..*
These look like CISCO logs, which do not conform to the format syslog-ng expects, and as such, they're not recognised and not processed either. If you just want to store them as-is, I'd recommend using the flags(no-parse) setting in the s_network source, like this: source s_network { tcp(flags(no-parse)); udp(flags(no-parse)); }; -- |8]
On Wed, Sep 12, 2012 at 11:39 AM, Sandor Geller < Sandor.Geller@morganstanley.com> wrote:
Hi,
I don't see anything obvious. Could you check whether there are other apps listening on 172.30.6.70:514 ? In other words grep for 514 in the netstat output not for syslog. Are there packet filter rules dropping incoming traffic?
Regards,
Sandor
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
@ Sandor Geller
Thanks for your response. The port 514 is for syslog-ng as it is defined in the services file :
*~# cat /etc/services | grep 514 shell 514/tcp cmd # no passwords used syslog-ng 514/udp* * ~# netstat -a --numeric-ports | grep 514
tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN udp 0 0 0.0.0.0:514 0.0.0.0:*
unix 3 [ ] STREAM CONNECTED 1514 *
As you can see there is only the syslog-ng running on that port.
On Wed, Sep 12, 2012 at 11:45 AM, Gergely Nagy <algernon@balabit.hu>wrote:
Alex Clone <alexandros.clone@gmail.com> writes:
*~# tcpdump -i eth0 udp 'port 514' -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 06:05:58.669992 IP (tos 0x0, ttl 254, id 20209, offset 0, flags [none], proto UDP (17), length 82) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 54 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host WAN:172.30.254.4\0x0a 06:05:58.670134 IP (tos 0x0, ttl 254, id 52, offset 0, flags [none], proto UDP (17), length 85) 172.29.1.6.syslog-ng > 172.30.6.70.syslog-ng: SYSLOG, length: 57 Facility local4 (20), Severity debug (7) Msg: %ASA-7-609001: Built local-host outside:172.26.10.4\0x0a ………..*
These look like CISCO logs, which do not conform to the format syslog-ng expects, and as such, they're not recognised and not processed either.
If you just want to store them as-is, I'd recommend using the flags(no-parse) setting in the s_network source, like this:
source s_network { tcp(flags(no-parse)); udp(flags(no-parse)); };
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
@ Gergely Nagy
Thanks for your response and your recommendation. I change the source block to the one that you mail me. The result is exactly the same. Syslog-ng cannot read the stream.
By the way, when I tried to log local data using the internal(); option to the source block, syslog-ng worked perfectly. All the data logged to the given file. So, the problem, I have to solve, is why syslog-ng cannot read the interface eth0 (udp).
Thanks in advance, Alex
participants (3)
-
Alex Clone
-
Gergely Nagy
-
Sandor Geller