Re: [syslog-ng] odd behavior writing "binary" logfiles - followup data
and in the self-inflicted wounds category... I did some additional digging on the box and discovered what certainly *appears* to be the cause: [a0142566@vxpip-eeisl001 ~]$ grep host /etc/nsswitch.conf #hosts: db files nisplus nis dns hosts: files dns [a0142566@vxpip-eeisl001 ~]$ [a0142566@vxpip-eeisl001 ~]$ diff /etc/hosts /etc/hosts.03162014 2c2 < 10.177.74.55 vxpip-eeisl001 ---
10.177.74.55 . vxpip-eeisl001 [a0142566@vxpip-eeisl001 ~]$
I fixed this and then re-ran some tests using the attached syslog-ng.conf.DEBUG and the following invocation: sudo strace /usr/local/sbin/syslog-ng -vde -f /usr/local/etc/syslog-ng.conf.DEBUG > /tmp/syslog-ng-DEBUG-strace 2>&1 and things look MUCH better. [a0142566@vxpip-eeisl001 etc]$ file /var/log/debug* /var/log/debug.messages.vxpip-eeisl001.auth.info: ASCII text /var/log/debug.messages.vxpip-eeisl001.authpriv.notice: ASCII text /var/log/debug.messages.vxpip-eeisl001.cron.info: ASCII text /var/log/debug.network_messages.vddp13e-0d6b677: ASCII text /var/log/debug.template_messages: ASCII text [a0142566@vxpip-eeisl001 etc]$ Note: I also found another system that has the same config error: [a0142566@vxkip-eeisl001 ~]$ cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 10.181.73.32 . vxkip-eeisl001 and ran a quick test through a resolver perl script that does (basically) this: my $iaddr = inet_aton($thingy); if (my $name = gethostbyaddr($iaddr,AF_INET)) { print "$thingy,\t$name\n"; } [a0142566@vxkip-eeisl001 ~]$ echo 10.181.73.32 | ~/bin/resolve.pl 10.181.73.32, . Sorry again for bothering the list (I really should have caught that!!... And yes – I will be bringing this to the OS team’s attention :-) Although it does raise a point that (maybe) a default value like "please_fix_my_hostname" could be used in case this happens elsewhere? Just a thought. Thank you again!! Jim ---- Jim Hendrick <jrhendri@roadrunner.com> wrote:
I will take some time today and re-run this with various debugging flags.
(just realized I should have... )
On 03/15/2014 09:37 PM, jrhendri@roadrunner.com wrote:
Odd - I do not see any gethost calls at all in the strace
this is from "sudo strace /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf > syslog-strace 2>&1"
I include the config.status and config.log also to see if that may help.
Jim
---- Balazs Scheidler <bazsi77@gmail.com> wrote:
During startup syslog-ng queries the hostname using gethostname. Can you strace syslog-ng and look for this call to see what it returns?
We are also resolving this using the dns which is also interesting.
Thanks On Mar 13, 2014 7:08 PM, <jrhendri@roadrunner.com> wrote:
OK - after much poking around (including testing this on a different host with this morning's latest source) here's what I found
I believe that syslog-ng is (for some reason) unable to come up with a correct hostname for the local system. I would really like to know why this is happening I would also really like to know if any of the source options I tried should have worked (I was kind of desperate with some of the combinations...)
That said - here's my summary:
Examining the "binary" files, It was always writing 256 bytes of "0" where ${HOST} should have been.
It *never* happened with a network source (tried with Kiwi sysloggen using RCF and non RCF messages) It *always* happened with any local source including all the ones commented out (I tested each separately)
source s_local { unix-dgram("/dev/log"); # standard Linux log source (this is the default place for the syslog() function to send logs to) #!# unix-stream("/dev/log"); # standard Linux log source (this is the default place for the syslog() function to send logs to) #!# system(); #!# internal(); };
Once I identified this behavior, I tried a number of options / combinations to work around this including (each separately) source s_local { system(); #!# system( keep_hostname(no) host_override("localhost") ); #!# system( keep_hostname(yes) host_override("localhost") ); #!# system( flags(no-parse) host_override("localhost") ); #!# system( flags(no-hostname) host_override("localhost") ); #!# system( flags(no-parse) ); #!# system( flags(no-hostname) ); #!# internal( ); };
And nothing worked.
Finally I kinda punted... I am now doing something really kludgy as a work around: source s_local { system(); internal( ); };
source s_network { udp(); };
destination d_local_template {
file("/data/syslog-ng/$YEAR/$MONTH/$DAY/localhost/localhost.$FACILITY.$PRIORITY.$DATE" template("${ISODATE} localhost ${PROGRAM} ${MESSAGE}\n") ); };
destination d_network {
file("/data/syslog-ng/$YEAR/$MONTH/$DAY/$HOST_FROM/$HOST_FROM.$FACILITY.$PRIORITY.$DATE"); };
log { source(s_local); destination(d_local_template); };
log { source(s_network); destination(d_network); };
---- jrhendri@roadrunner.com wrote:
I have a really odd problem that I hope someone can assist with.
My install is inserting a number of zeroes into the logfile between the date and the rest of the message. Here are some details:
System is RHEL:
$ uname -a Linux vxpip-eeisl001 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux $
Syslog-ng was built on the box yesterday with these sources: eventlog-0.2.12+20120504+1700 syslog-ng-3.5.3
$ /usr/local/sbin/syslog-ng --version syslog-ng 3.5.3 Installer-Version: 3.5.3 Revision: ssh+git://algernon@git.balabit /var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#ccb05a22408ba4c837d998b2538854d994f845a5 Compile-Date: Mar 12 2014 11:37:32 Available-Modules: afmongodb,afstomp,syslogformat,affile,basicfuncs,csvparser,confgen,system-source,afamqp,linux-kmsg-format,afprog,afuser,afsocket,dbparser,cryptofuncs,afsocket-notls Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: off
This is the default config file that came with it:
#############################################################################
# Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. #
@version: 3.5 @include "scl.conf"
source s_local { system(); internal(); };
source s_network { udp(); };
destination d_local { file("/var/log/messages"); };
log { source(s_local);
# uncomment this line to open port 514 to receive messages #source(s_network); destination(d_local); };
Here is the file:
$ sudo file /var/log/messages /var/log/messages: data $ sudo cat /var/log/messages Mar 13 10:43:00 syslog-ng[18451]: syslog-ng starting up; version='3.5.3' Mar 13 10:43:01 CROND[18454]: (root) CMD (/util/avail/get_avail.sh 1>/util/avail/logs/get_avail.out 2>&1) Mar 13 10:43:16 sudo: a0142566 : TTY=pts/2 ; PWD=/home/a0142566 ; USER=root ; COMMAND=/usr/bin/file /var/log/messages And this is a capture from vi in hex mode on that file.
0000000: 4d61 7220 3133 2031 303a 3433 3a30 3020 Mar 13 10:43:00 0000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000100: 0000 0000 0000 0000 0000 0000 0000 0020 ............... 0000110: 7379 736c 6f67 2d6e 675b 3138 3435 315d syslog-ng[18451] 0000120: 3a20 7379 736c 6f67 2d6e 6720 7374 6172 : syslog-ng star 0000130: 7469 6e67 2075 703b 2076 6572 7369 6f6e ting up; version 0000140: 3d27 332e 352e 3327 0a4d 6172 2031 3320 ='3.5.3'.Mar 13 0000150: 3130 3a34 333a 3031 2000 0000 0000 0000 10:43:01 ....... 0000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000200: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000210: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000250: 0000 0000 0000 0000 2043 524f 4e44 5b31 ........ CROND[1 0000260: 3834 3534 5d3a 2028 726f 6f74 2920 434d 8454]: (root) CM 0000270: 4420 282f 7574 696c 2f61 7661 696c 2f67 D (/util/avail/g 0000280: 6574 5f61 7661 696c 2e73 6820 313e 2f75 et_avail.sh 1>/u 0000290: 7469 6c2f 6176 6169 6c2f 6c6f 6773 2f67 til/avail/logs/g 00002a0: 6574 5f61 7661 696c 2e6f 7574 2032 3e26 et_avail.out 2>& 00002b0: 3129 0a4d 6172 2031 3320 3130 3a34 333a 1).
Any ideas? Thanks in advance folks!! Jim
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Great to hear you could finally nail it. As for the validation idea, I'll give it a thought. Thanks. On Mar 17, 2014 4:20 PM, <jrhendri@roadrunner.com> wrote:
and in the self-inflicted wounds category...
I did some additional digging on the box and discovered what certainly *appears* to be the cause:
[a0142566@vxpip-eeisl001 ~]$ grep host /etc/nsswitch.conf #hosts: db files nisplus nis dns hosts: files dns [a0142566@vxpip-eeisl001 ~]$
[a0142566@vxpip-eeisl001 ~]$ diff /etc/hosts /etc/hosts.03162014 2c2 < 10.177.74.55 vxpip-eeisl001 ---
10.177.74.55 . vxpip-eeisl001 [a0142566@vxpip-eeisl001 ~]$
I fixed this and then re-ran some tests using the attached syslog-ng.conf.DEBUG and the following invocation:
sudo strace /usr/local/sbin/syslog-ng -vde -f /usr/local/etc/syslog-ng.conf.DEBUG > /tmp/syslog-ng-DEBUG-strace 2>&1
and things look MUCH better.
[a0142566@vxpip-eeisl001 etc]$ file /var/log/debug* /var/log/debug.messages.vxpip-eeisl001.auth.info: ASCII text /var/log/debug.messages.vxpip-eeisl001.authpriv.notice: ASCII text /var/log/debug.messages.vxpip-eeisl001.cron.info: ASCII text /var/log/debug.network_messages.vddp13e-0d6b677: ASCII text /var/log/debug.template_messages: ASCII text [a0142566@vxpip-eeisl001 etc]$
Note: I also found another system that has the same config error:
[a0142566@vxkip-eeisl001 ~]$ cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 10.181.73.32 . vxkip-eeisl001
and ran a quick test through a resolver perl script that does (basically) this: my $iaddr = inet_aton($thingy); if (my $name = gethostbyaddr($iaddr,AF_INET)) { print "$thingy,\t$name\n"; }
[a0142566@vxkip-eeisl001 ~]$ echo 10.181.73.32 | ~/bin/resolve.pl 10.181.73.32, .
Sorry again for bothering the list (I really should have caught that!!... And yes - I will be bringing this to the OS team's attention :-)
Although it does raise a point that (maybe) a default value like "please_fix_my_hostname" could be used in case this happens elsewhere?
Just a thought.
Thank you again!! Jim
---- Jim Hendrick <jrhendri@roadrunner.com> wrote:
I will take some time today and re-run this with various debugging flags.
(just realized I should have... )
On 03/15/2014 09:37 PM, jrhendri@roadrunner.com wrote:
Odd - I do not see any gethost calls at all in the strace
this is from "sudo strace /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf > syslog-strace 2>&1"
I include the config.status and config.log also to see if that may help.
Jim
---- Balazs Scheidler <bazsi77@gmail.com> wrote:
During startup syslog-ng queries the hostname using gethostname. Can you strace syslog-ng and look for this call to see what it returns?
We are also resolving this using the dns which is also interesting.
Thanks On Mar 13, 2014 7:08 PM, <jrhendri@roadrunner.com> wrote:
OK - after much poking around (including testing this on a different host with this morning's latest source) here's what I found
I believe that syslog-ng is (for some reason) unable to come up with a correct hostname for the local system. I would really like to know why this is happening I would also really like to know if any of the source options I tried should have worked (I was kind of desperate with some of the combinations...)
That said - here's my summary:
Examining the "binary" files, It was always writing 256 bytes of "0" where ${HOST} should have been.
It *never* happened with a network source (tried with Kiwi sysloggen using RCF and non RCF messages) It *always* happened with any local source including all the ones commented out (I tested each separately)
source s_local { unix-dgram("/dev/log"); # standard Linux log source (this is the default place for the syslog() function to send logs to) #!# unix-stream("/dev/log"); # standard Linux log source (this is the default place for the syslog() function to send logs to) #!# system(); #!# internal(); };
Once I identified this behavior, I tried a number of options / combinations to work around this including (each separately) source s_local { system(); #!# system( keep_hostname(no) host_override("localhost") ); #!# system( keep_hostname(yes) host_override("localhost") ); #!# system( flags(no-parse) host_override("localhost") ); #!# system( flags(no-hostname) host_override("localhost") ); #!# system( flags(no-parse) ); #!# system( flags(no-hostname) ); #!# internal( ); };
And nothing worked.
Finally I kinda punted... I am now doing something really kludgy as a work around: source s_local { system(); internal( ); };
source s_network { udp(); };
destination d_local_template {
file("/data/syslog-ng/$YEAR/$MONTH/$DAY/localhost/localhost.$FACILITY.$PRIORITY.$DATE"
template("${ISODATE} localhost ${PROGRAM} ${MESSAGE}\n") ); };
destination d_network {
file("/data/syslog-ng/$YEAR/$MONTH/$DAY/$HOST_FROM/$HOST_FROM.$FACILITY.$PRIORITY.$DATE");
};
log { source(s_local); destination(d_local_template); };
log { source(s_network); destination(d_network); };
---- jrhendri@roadrunner.com wrote:
I have a really odd problem that I hope someone can assist with.
My install is inserting a number of zeroes into the logfile between the date and the rest of the message. Here are some details:
System is RHEL:
$ uname -a Linux vxpip-eeisl001 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux $
Syslog-ng was built on the box yesterday with these sources: eventlog-0.2.12+20120504+1700 syslog-ng-3.5.3
$ /usr/local/sbin/syslog-ng --version syslog-ng 3.5.3 Installer-Version: 3.5.3 Revision: ssh+git://algernon@git.balabit
/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#ccb05a22408ba4c837d998b2538854d994f845a5
Compile-Date: Mar 12 2014 11:37:32 Available-Modules:
afmongodb,afstomp,syslogformat,affile,basicfuncs,csvparser,confgen,system-source,afamqp,linux-kmsg-format,afprog,afuser,afsocket,dbparser,cryptofuncs,afsocket-notls
Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: off
This is the default config file that came with it:
#############################################################################
# Default syslog-ng.conf file which collects all local logs into a # single file called /var/log/messages. #
@version: 3.5 @include "scl.conf"
source s_local { system(); internal(); };
source s_network { udp(); };
destination d_local { file("/var/log/messages"); };
log { source(s_local);
# uncomment this line to open port 514 to receive messages #source(s_network); destination(d_local); };
Here is the file:
$ sudo file /var/log/messages /var/log/messages: data $ sudo cat /var/log/messages Mar 13 10:43:00 syslog-ng[18451]: syslog-ng starting up; version='3.5.3' Mar 13 10:43:01 CROND[18454]: (root) CMD (/util/avail/get_avail.sh 1>/util/avail/logs/get_avail.out 2>&1) Mar 13 10:43:16 sudo: a0142566 : TTY=pts/2 ; PWD=/home/a0142566 ; USER=root ; COMMAND=/usr/bin/file /var/log/messages And this is a capture from vi in hex mode on that file.
0000000: 4d61 7220 3133 2031 303a 3433 3a30 3020 Mar 13 10:43:00 0000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000100: 0000 0000 0000 0000 0000 0000 0000 0020 ............... 0000110: 7379 736c 6f67 2d6e 675b 3138 3435 315d syslog-ng[18451] 0000120: 3a20 7379 736c 6f67 2d6e 6720 7374 6172 : syslog-ng star 0000130: 7469 6e67 2075 703b 2076 6572 7369 6f6e ting up; version 0000140: 3d27 332e 352e 3327 0a4d 6172 2031 3320 ='3.5.3'.Mar 13 0000150: 3130 3a34 333a 3031 2000 0000 0000 0000 10:43:01 ....... 0000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000200: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000210: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000250: 0000 0000 0000 0000 2043 524f 4e44 5b31 ........ CROND[1 0000260: 3834 3534 5d3a 2028 726f 6f74 2920 434d 8454]: (root) CM 0000270: 4420 282f 7574 696c 2f61 7661 696c 2f67 D (/util/avail/g 0000280: 6574 5f61 7661 696c 2e73 6820 313e 2f75 et_avail.sh 1>/u 0000290: 7469 6c2f 6176 6169 6c2f 6c6f 6773 2f67 til/avail/logs/g 00002a0: 6574 5f61 7661 696c 2e6f 7574 2032 3e26 et_avail.out 2>& 00002b0: 3129 0a4d 6172 2031 3320 3130 3a34 333a 1).
Any ideas? Thanks in advance folks!! Jim
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Balazs Scheidler
-
jrhendri＠roadrunner.com