<p dir="ltr">Great to hear you could finally nail it. As for the validation idea, I'll give it a thought.</p>
<p dir="ltr">Thanks.</p>
<div class="gmail_quote">On Mar 17, 2014 4:20 PM, <<a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
and in the self-inflicted wounds category...<br>
<br>
I did some additional digging on the box and discovered what certainly *appears* to be the cause:<br>
<br>
[a0142566@vxpip-eeisl001 ~]$ grep host /etc/nsswitch.conf<br>
#hosts: db files nisplus nis dns<br>
hosts: files dns<br>
[a0142566@vxpip-eeisl001 ~]$<br>
<br>
[a0142566@vxpip-eeisl001 ~]$ diff /etc/hosts /etc/hosts.03162014<br>
2c2<br>
< 10.177.74.55 vxpip-eeisl001<br>
---<br>
> 10.177.74.55 . vxpip-eeisl001<br>
[a0142566@vxpip-eeisl001 ~]$<br>
<br>
I fixed this and then re-ran some tests using the attached syslog-ng.conf.DEBUG and the following invocation:<br>
<br>
sudo strace /usr/local/sbin/syslog-ng -vde -f /usr/local/etc/syslog-ng.conf.DEBUG > /tmp/syslog-ng-DEBUG-strace 2>&1<br>
<br>
<br>
and things look MUCH better.<br>
<br>
<br>
[a0142566@vxpip-eeisl001 etc]$ file /var/log/debug*<br>
/var/log/<a href="http://debug.messages.vxpip-eeisl001.auth.info" target="_blank">debug.messages.vxpip-eeisl001.auth.info</a>: ASCII text<br>
/var/log/debug.messages.vxpip-eeisl001.authpriv.notice: ASCII text<br>
/var/log/<a href="http://debug.messages.vxpip-eeisl001.cron.info" target="_blank">debug.messages.vxpip-eeisl001.cron.info</a>: ASCII text<br>
/var/log/debug.network_messages.vddp13e-0d6b677: ASCII text<br>
/var/log/debug.template_messages: ASCII text<br>
[a0142566@vxpip-eeisl001 etc]$<br>
<br>
Note: I also found another system that has the same config error:<br>
<br>
[a0142566@vxkip-eeisl001 ~]$ cat /etc/hosts<br>
127.0.0.1 localhost.localdomain localhost<br>
10.181.73.32 . vxkip-eeisl001<br>
<br>
and ran a quick test through a resolver perl script that does (basically) this:<br>
my $iaddr = inet_aton($thingy);<br>
if (my $name = gethostbyaddr($iaddr,AF_INET)) {<br>
print "$thingy,\t$name\n";<br>
}<br>
<br>
[a0142566@vxkip-eeisl001 ~]$ echo 10.181.73.32 | ~/bin/<a href="http://resolve.pl" target="_blank">resolve.pl</a><br>
10.181.73.32, .<br>
<br>
Sorry again for bothering the list (I really should have caught that!!... And yes – I will be bringing this to the OS team’s attention :-)<br>
<br>
Although it does raise a point that (maybe) a default value like "please_fix_my_hostname" could be used in case this happens elsewhere?<br>
<br>
Just a thought.<br>
<br>
Thank you again!!<br>
Jim<br>
<br>
<br>
<br>
<br>
---- Jim Hendrick <<a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a>> wrote:<br>
> I will take some time today and re-run this with various debugging flags.<br>
><br>
> (just realized I should have... )<br>
><br>
> On 03/15/2014 09:37 PM, <a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a> wrote:<br>
> > Odd - I do not see any gethost calls at all in the strace<br>
> ><br>
> > this is from "sudo strace /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf > syslog-strace 2>&1"<br>
> ><br>
> ><br>
> > I include the config.status and config.log also to see if that may help.<br>
> ><br>
> > Jim<br>
> ><br>
> ><br>
> > ---- Balazs Scheidler <<a href="mailto:bazsi77@gmail.com">bazsi77@gmail.com</a>> wrote:<br>
> >> During startup syslog-ng queries the hostname using gethostname. Can you<br>
> >> strace syslog-ng and look for this call to see what it returns?<br>
> >><br>
> >> We are also resolving this using the dns which is also interesting.<br>
> >><br>
> >> Thanks<br>
> >> On Mar 13, 2014 7:08 PM, <<a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a>> wrote:<br>
> >><br>
> >>> OK - after much poking around (including testing this on a different host<br>
> >>> with this morning's latest source) here's what I found<br>
> >>><br>
> >>> I believe that syslog-ng is (for some reason) unable to come up with a<br>
> >>> correct hostname for the local system.<br>
> >>> I would really like to know why this is happening<br>
> >>> I would also really like to know if any of the source options I tried<br>
> >>> should have worked (I was kind of desperate with some of the combinations...)<br>
> >>><br>
> >>> That said - here's my summary:<br>
> >>><br>
> >>> Examining the "binary" files, It was always writing 256 bytes of "0" where<br>
> >>> ${HOST} should have been.<br>
> >>><br>
> >>> It *never* happened with a network source (tried with Kiwi sysloggen using<br>
> >>> RCF and non RCF messages)<br>
> >>> It *always* happened with any local source including all the ones<br>
> >>> commented out (I tested each separately)<br>
> >>><br>
> >>> source s_local {<br>
> >>> unix-dgram("/dev/log"); # standard Linux log source (this is the default<br>
> >>> place for the syslog() function to send logs to)<br>
> >>> #!# unix-stream("/dev/log"); # standard Linux log source (this is the<br>
> >>> default place for the syslog() function to send logs to)<br>
> >>> #!# system();<br>
> >>> #!# internal();<br>
> >>> };<br>
> >>><br>
> >>> Once I identified this behavior, I tried a number of options /<br>
> >>> combinations to work around this including (each separately)<br>
> >>> source s_local {<br>
> >>> system();<br>
> >>> #!# system( keep_hostname(no) host_override("localhost") );<br>
> >>> #!# system( keep_hostname(yes) host_override("localhost") );<br>
> >>> #!# system( flags(no-parse) host_override("localhost") );<br>
> >>> #!# system( flags(no-hostname) host_override("localhost") );<br>
> >>> #!# system( flags(no-parse) );<br>
> >>> #!# system( flags(no-hostname) );<br>
> >>> #!# internal( );<br>
> >>> };<br>
> >>><br>
> >>> And nothing worked.<br>
> >>><br>
> >>> Finally I kinda punted... I am now doing something really kludgy as a work<br>
> >>> around:<br>
> >>> source s_local {<br>
> >>> system();<br>
> >>> internal( );<br>
> >>> };<br>
> >>><br>
> >>> source s_network {<br>
> >>> udp();<br>
> >>> };<br>
> >>><br>
> >>> destination d_local_template {<br>
> >>><br>
> >>> file("/data/syslog-ng/$YEAR/$MONTH/$DAY/localhost/localhost.$FACILITY.$PRIORITY.$DATE"<br>
> >>> template("${ISODATE} localhost ${PROGRAM} ${MESSAGE}\n") );<br>
> >>> };<br>
> >>><br>
> >>><br>
> >>> destination d_network {<br>
> >>><br>
> >>> file("/data/syslog-ng/$YEAR/$MONTH/$DAY/$HOST_FROM/$HOST_FROM.$FACILITY.$PRIORITY.$DATE");<br>
> >>> };<br>
> >>><br>
> >>> log {<br>
> >>> source(s_local);<br>
> >>> destination(d_local_template);<br>
> >>> };<br>
> >>><br>
> >>> log {<br>
> >>> source(s_network);<br>
> >>> destination(d_network);<br>
> >>> };<br>
> >>><br>
> >>><br>
> >>> ---- <a href="mailto:jrhendri@roadrunner.com">jrhendri@roadrunner.com</a> wrote:<br>
> >>>> I have a really odd problem that I hope someone can assist with.<br>
> >>>><br>
> >>>> My install is inserting a number of zeroes into the logfile between the<br>
> >>> date and the rest of the message.<br>
> >>>> Here are some details:<br>
> >>>><br>
> >>>> System is RHEL:<br>
> >>>><br>
> >>>> $ uname -a<br>
> >>>> Linux vxpip-eeisl001 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41<br>
> >>> EST 2013 x86_64 x86_64 x86_64 GNU/Linux<br>
> >>>> $<br>
> >>>><br>
> >>>><br>
> >>>> Syslog-ng was built on the box yesterday with these sources:<br>
> >>>> eventlog-0.2.12+20120504+1700<br>
> >>>> syslog-ng-3.5.3<br>
> >>>><br>
> >>>><br>
> >>>> $ /usr/local/sbin/syslog-ng --version<br>
> >>>> syslog-ng 3.5.3<br>
> >>>> Installer-Version: 3.5.3<br>
> >>>> Revision: ssh+git://algernon@git.balabit<br>
> >>> /var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#ccb05a22408ba4c837d998b2538854d994f845a5<br>
> >>>> Compile-Date: Mar 12 2014 11:37:32<br>
> >>>> Available-Modules:<br>
> >>> afmongodb,afstomp,syslogformat,affile,basicfuncs,csvparser,confgen,system-source,afamqp,linux-kmsg-format,afprog,afuser,afsocket,dbparser,cryptofuncs,afsocket-notls<br>
> >>>> Enable-Debug: off<br>
> >>>> Enable-GProf: off<br>
> >>>> Enable-Memtrace: off<br>
> >>>> Enable-IPv6: on<br>
> >>>> Enable-Spoof-Source: off<br>
> >>>> Enable-TCP-Wrapper: off<br>
> >>>> Enable-Linux-Caps: off<br>
> >>>> Enable-Pcre: off<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> This is the default config file that came with it:<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>> #############################################################################<br>
> >>>> # Default syslog-ng.conf file which collects all local logs into a<br>
> >>>> # single file called /var/log/messages.<br>
> >>>> #<br>
> >>>><br>
> >>>> @version: 3.5<br>
> >>>> @include "scl.conf"<br>
> >>>><br>
> >>>> source s_local {<br>
> >>>> system();<br>
> >>>> internal();<br>
> >>>> };<br>
> >>>><br>
> >>>> source s_network {<br>
> >>>> udp();<br>
> >>>> };<br>
> >>>><br>
> >>>> destination d_local {<br>
> >>>> file("/var/log/messages");<br>
> >>>> };<br>
> >>>><br>
> >>>> log {<br>
> >>>> source(s_local);<br>
> >>>><br>
> >>>> # uncomment this line to open port 514 to receive messages<br>
> >>>> #source(s_network);<br>
> >>>> destination(d_local);<br>
> >>>> };<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> Here is the file:<br>
> >>>><br>
> >>>><br>
> >>>> $ sudo file /var/log/messages<br>
> >>>> /var/log/messages: data<br>
> >>>> $ sudo cat /var/log/messages<br>
> >>>> Mar 13 10:43:00 syslog-ng[18451]: syslog-ng starting up; version='3.5.3'<br>
> >>>> Mar 13 10:43:01 CROND[18454]: (root) CMD (/util/avail/get_avail.sh<br>
> >>> 1>/util/avail/logs/get_avail.out 2>&1)<br>
> >>>> Mar 13 10:43:16 sudo: a0142566 : TTY=pts/2 ; PWD=/home/a0142566 ;<br>
> >>> USER=root ; COMMAND=/usr/bin/file /var/log/messages<br>
> >>>> And this is a capture from vi in hex mode on that file.<br>
> >>>><br>
> >>>><br>
> >>>> 0000000: 4d61 7220 3133 2031 303a 3433 3a30 3020 Mar 13 10:43:00<br>
> >>>> 0000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000100: 0000 0000 0000 0000 0000 0000 0000 0020 ...............<br>
> >>>> 0000110: 7379 736c 6f67 2d6e 675b 3138 3435 315d syslog-ng[18451]<br>
> >>>> 0000120: 3a20 7379 736c 6f67 2d6e 6720 7374 6172 : syslog-ng star<br>
> >>>> 0000130: 7469 6e67 2075 703b 2076 6572 7369 6f6e ting up; version<br>
> >>>> 0000140: 3d27 332e 352e 3327 0a4d 6172 2031 3320 ='3.5.3'.Mar 13<br>
> >>>> 0000150: 3130 3a34 333a 3031 2000 0000 0000 0000 10:43:01 .......<br>
> >>>> 0000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 00001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000200: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000210: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................<br>
> >>>> 0000250: 0000 0000 0000 0000 2043 524f 4e44 5b31 ........ CROND[1<br>
> >>>> 0000260: 3834 3534 5d3a 2028 726f 6f74 2920 434d 8454]: (root) CM<br>
> >>>> 0000270: 4420 282f 7574 696c 2f61 7661 696c 2f67 D (/util/avail/g<br>
> >>>> 0000280: 6574 5f61 7661 696c 2e73 6820 313e 2f75 et_avail.sh 1>/u<br>
> >>>> 0000290: 7469 6c2f 6176 6169 6c2f 6c6f 6773 2f67 til/avail/logs/g<br>
> >>>> 00002a0: 6574 5f61 7661 696c 2e6f 7574 2032 3e26 et_avail.out 2>&<br>
> >>>> 00002b0: 3129 0a4d 6172 2031 3320 3130 3a34 333a 1).<br>
> >>>><br>
> >>>><br>
> >>>> Any ideas?<br>
> >>>> Thanks in advance folks!!<br>
> >>>> Jim<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>> ______________________________________________________________________________<br>
> >>>> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> >>>> Documentation:<br>
> >>> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> >>>> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
> >>>><br>
> >>><br>
> >>> ______________________________________________________________________________<br>
> >>> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> >>> Documentation:<br>
> >>> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> >>> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
> >>><br>
> >>><br>
> >>><br>
> >>><br>
> >>> ______________________________________________________________________________<br>
> >>> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> >>> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> >>> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
> >>><br>
><br>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>