[syslog-ng] odd behavior writing "binary" logfiles - followup data
jrhendri at roadrunner.com
jrhendri at roadrunner.com
Mon Mar 17 16:20:32 CET 2014
and in the self-inflicted wounds category...
I did some additional digging on the box and discovered what certainly *appears* to be the cause:
[a0142566 at vxpip-eeisl001 ~]$ grep host /etc/nsswitch.conf
#hosts: db files nisplus nis dns
hosts: files dns
[a0142566 at vxpip-eeisl001 ~]$
[a0142566 at vxpip-eeisl001 ~]$ diff /etc/hosts /etc/hosts.03162014
2c2
< 10.177.74.55 vxpip-eeisl001
---
> 10.177.74.55 . vxpip-eeisl001
[a0142566 at vxpip-eeisl001 ~]$
I fixed this and then re-ran some tests using the attached syslog-ng.conf.DEBUG and the following invocation:
sudo strace /usr/local/sbin/syslog-ng -vde -f /usr/local/etc/syslog-ng.conf.DEBUG > /tmp/syslog-ng-DEBUG-strace 2>&1
and things look MUCH better.
[a0142566 at vxpip-eeisl001 etc]$ file /var/log/debug*
/var/log/debug.messages.vxpip-eeisl001.auth.info: ASCII text
/var/log/debug.messages.vxpip-eeisl001.authpriv.notice: ASCII text
/var/log/debug.messages.vxpip-eeisl001.cron.info: ASCII text
/var/log/debug.network_messages.vddp13e-0d6b677: ASCII text
/var/log/debug.template_messages: ASCII text
[a0142566 at vxpip-eeisl001 etc]$
Note: I also found another system that has the same config error:
[a0142566 at vxkip-eeisl001 ~]$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.181.73.32 . vxkip-eeisl001
and ran a quick test through a resolver perl script that does (basically) this:
my $iaddr = inet_aton($thingy);
if (my $name = gethostbyaddr($iaddr,AF_INET)) {
print "$thingy,\t$name\n";
}
[a0142566 at vxkip-eeisl001 ~]$ echo 10.181.73.32 | ~/bin/resolve.pl
10.181.73.32, .
Sorry again for bothering the list (I really should have caught that!!... And yes – I will be bringing this to the OS team’s attention :-)
Although it does raise a point that (maybe) a default value like "please_fix_my_hostname" could be used in case this happens elsewhere?
Just a thought.
Thank you again!!
Jim
---- Jim Hendrick <jrhendri at roadrunner.com> wrote:
> I will take some time today and re-run this with various debugging flags.
>
> (just realized I should have... )
>
> On 03/15/2014 09:37 PM, jrhendri at roadrunner.com wrote:
> > Odd - I do not see any gethost calls at all in the strace
> >
> > this is from "sudo strace /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf > syslog-strace 2>&1"
> >
> >
> > I include the config.status and config.log also to see if that may help.
> >
> > Jim
> >
> >
> > ---- Balazs Scheidler <bazsi77 at gmail.com> wrote:
> >> During startup syslog-ng queries the hostname using gethostname. Can you
> >> strace syslog-ng and look for this call to see what it returns?
> >>
> >> We are also resolving this using the dns which is also interesting.
> >>
> >> Thanks
> >> On Mar 13, 2014 7:08 PM, <jrhendri at roadrunner.com> wrote:
> >>
> >>> OK - after much poking around (including testing this on a different host
> >>> with this morning's latest source) here's what I found
> >>>
> >>> I believe that syslog-ng is (for some reason) unable to come up with a
> >>> correct hostname for the local system.
> >>> I would really like to know why this is happening
> >>> I would also really like to know if any of the source options I tried
> >>> should have worked (I was kind of desperate with some of the combinations...)
> >>>
> >>> That said - here's my summary:
> >>>
> >>> Examining the "binary" files, It was always writing 256 bytes of "0" where
> >>> ${HOST} should have been.
> >>>
> >>> It *never* happened with a network source (tried with Kiwi sysloggen using
> >>> RCF and non RCF messages)
> >>> It *always* happened with any local source including all the ones
> >>> commented out (I tested each separately)
> >>>
> >>> source s_local {
> >>> unix-dgram("/dev/log"); # standard Linux log source (this is the default
> >>> place for the syslog() function to send logs to)
> >>> #!# unix-stream("/dev/log"); # standard Linux log source (this is the
> >>> default place for the syslog() function to send logs to)
> >>> #!# system();
> >>> #!# internal();
> >>> };
> >>>
> >>> Once I identified this behavior, I tried a number of options /
> >>> combinations to work around this including (each separately)
> >>> source s_local {
> >>> system();
> >>> #!# system( keep_hostname(no) host_override("localhost") );
> >>> #!# system( keep_hostname(yes) host_override("localhost") );
> >>> #!# system( flags(no-parse) host_override("localhost") );
> >>> #!# system( flags(no-hostname) host_override("localhost") );
> >>> #!# system( flags(no-parse) );
> >>> #!# system( flags(no-hostname) );
> >>> #!# internal( );
> >>> };
> >>>
> >>> And nothing worked.
> >>>
> >>> Finally I kinda punted... I am now doing something really kludgy as a work
> >>> around:
> >>> source s_local {
> >>> system();
> >>> internal( );
> >>> };
> >>>
> >>> source s_network {
> >>> udp();
> >>> };
> >>>
> >>> destination d_local_template {
> >>>
> >>> file("/data/syslog-ng/$YEAR/$MONTH/$DAY/localhost/localhost.$FACILITY.$PRIORITY.$DATE"
> >>> template("${ISODATE} localhost ${PROGRAM} ${MESSAGE}\n") );
> >>> };
> >>>
> >>>
> >>> destination d_network {
> >>>
> >>> file("/data/syslog-ng/$YEAR/$MONTH/$DAY/$HOST_FROM/$HOST_FROM.$FACILITY.$PRIORITY.$DATE");
> >>> };
> >>>
> >>> log {
> >>> source(s_local);
> >>> destination(d_local_template);
> >>> };
> >>>
> >>> log {
> >>> source(s_network);
> >>> destination(d_network);
> >>> };
> >>>
> >>>
> >>> ---- jrhendri at roadrunner.com wrote:
> >>>> I have a really odd problem that I hope someone can assist with.
> >>>>
> >>>> My install is inserting a number of zeroes into the logfile between the
> >>> date and the rest of the message.
> >>>> Here are some details:
> >>>>
> >>>> System is RHEL:
> >>>>
> >>>> $ uname -a
> >>>> Linux vxpip-eeisl001 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41
> >>> EST 2013 x86_64 x86_64 x86_64 GNU/Linux
> >>>> $
> >>>>
> >>>>
> >>>> Syslog-ng was built on the box yesterday with these sources:
> >>>> eventlog-0.2.12+20120504+1700
> >>>> syslog-ng-3.5.3
> >>>>
> >>>>
> >>>> $ /usr/local/sbin/syslog-ng --version
> >>>> syslog-ng 3.5.3
> >>>> Installer-Version: 3.5.3
> >>>> Revision: ssh+git://algernon@git.balabit
> >>> /var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#ccb05a22408ba4c837d998b2538854d994f845a5
> >>>> Compile-Date: Mar 12 2014 11:37:32
> >>>> Available-Modules:
> >>> afmongodb,afstomp,syslogformat,affile,basicfuncs,csvparser,confgen,system-source,afamqp,linux-kmsg-format,afprog,afuser,afsocket,dbparser,cryptofuncs,afsocket-notls
> >>>> Enable-Debug: off
> >>>> Enable-GProf: off
> >>>> Enable-Memtrace: off
> >>>> Enable-IPv6: on
> >>>> Enable-Spoof-Source: off
> >>>> Enable-TCP-Wrapper: off
> >>>> Enable-Linux-Caps: off
> >>>> Enable-Pcre: off
> >>>>
> >>>>
> >>>>
> >>>> This is the default config file that came with it:
> >>>>
> >>>>
> >>>>
> >>> #############################################################################
> >>>> # Default syslog-ng.conf file which collects all local logs into a
> >>>> # single file called /var/log/messages.
> >>>> #
> >>>>
> >>>> @version: 3.5
> >>>> @include "scl.conf"
> >>>>
> >>>> source s_local {
> >>>> system();
> >>>> internal();
> >>>> };
> >>>>
> >>>> source s_network {
> >>>> udp();
> >>>> };
> >>>>
> >>>> destination d_local {
> >>>> file("/var/log/messages");
> >>>> };
> >>>>
> >>>> log {
> >>>> source(s_local);
> >>>>
> >>>> # uncomment this line to open port 514 to receive messages
> >>>> #source(s_network);
> >>>> destination(d_local);
> >>>> };
> >>>>
> >>>>
> >>>>
> >>>> Here is the file:
> >>>>
> >>>>
> >>>> $ sudo file /var/log/messages
> >>>> /var/log/messages: data
> >>>> $ sudo cat /var/log/messages
> >>>> Mar 13 10:43:00 syslog-ng[18451]: syslog-ng starting up; version='3.5.3'
> >>>> Mar 13 10:43:01 CROND[18454]: (root) CMD (/util/avail/get_avail.sh
> >>> 1>/util/avail/logs/get_avail.out 2>&1)
> >>>> Mar 13 10:43:16 sudo: a0142566 : TTY=pts/2 ; PWD=/home/a0142566 ;
> >>> USER=root ; COMMAND=/usr/bin/file /var/log/messages
> >>>> And this is a capture from vi in hex mode on that file.
> >>>>
> >>>>
> >>>> 0000000: 4d61 7220 3133 2031 303a 3433 3a30 3020 Mar 13 10:43:00
> >>>> 0000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000100: 0000 0000 0000 0000 0000 0000 0000 0020 ...............
> >>>> 0000110: 7379 736c 6f67 2d6e 675b 3138 3435 315d syslog-ng[18451]
> >>>> 0000120: 3a20 7379 736c 6f67 2d6e 6720 7374 6172 : syslog-ng star
> >>>> 0000130: 7469 6e67 2075 703b 2076 6572 7369 6f6e ting up; version
> >>>> 0000140: 3d27 332e 352e 3327 0a4d 6172 2031 3320 ='3.5.3'.Mar 13
> >>>> 0000150: 3130 3a34 333a 3031 2000 0000 0000 0000 10:43:01 .......
> >>>> 0000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 00001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000200: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000210: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> >>>> 0000250: 0000 0000 0000 0000 2043 524f 4e44 5b31 ........ CROND[1
> >>>> 0000260: 3834 3534 5d3a 2028 726f 6f74 2920 434d 8454]: (root) CM
> >>>> 0000270: 4420 282f 7574 696c 2f61 7661 696c 2f67 D (/util/avail/g
> >>>> 0000280: 6574 5f61 7661 696c 2e73 6820 313e 2f75 et_avail.sh 1>/u
> >>>> 0000290: 7469 6c2f 6176 6169 6c2f 6c6f 6773 2f67 til/avail/logs/g
> >>>> 00002a0: 6574 5f61 7661 696c 2e6f 7574 2032 3e26 et_avail.out 2>&
> >>>> 00002b0: 3129 0a4d 6172 2031 3320 3130 3a34 333a 1).
> >>>>
> >>>>
> >>>> Any ideas?
> >>>> Thanks in advance folks!!
> >>>> Jim
> >>>>
> >>>>
> >>>>
> >>> ______________________________________________________________________________
> >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>>> Documentation:
> >>> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>>
> >>>
> >>> ______________________________________________________________________________
> >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> Documentation:
> >>> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>
> >>>
> >>>
> >>>
> >>> ______________________________________________________________________________
> >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: syslog-ng.conf.DEBUG.gz
Type: application/x-gzip
Size: 445 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140317/0da9e89e/attachment-0002.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: syslog-ng-DEBUG-strace.gz
Type: application/x-gzip
Size: 10457 bytes
Desc: not available
Url : http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140317/0da9e89e/attachment-0003.bin
More information about the syslog-ng
mailing list