[syslog-ng] odd behavior writing "binary" logfiles - followup data
Balazs Scheidler
bazsi77 at gmail.com
Tue Mar 25 23:08:04 CET 2014
Great to hear you could finally nail it. As for the validation idea, I'll
give it a thought.
Thanks.
On Mar 17, 2014 4:20 PM, <jrhendri at roadrunner.com> wrote:
>
> and in the self-inflicted wounds category...
>
> I did some additional digging on the box and discovered what certainly
> *appears* to be the cause:
>
> [a0142566 at vxpip-eeisl001 ~]$ grep host /etc/nsswitch.conf
> #hosts: db files nisplus nis dns
> hosts: files dns
> [a0142566 at vxpip-eeisl001 ~]$
>
> [a0142566 at vxpip-eeisl001 ~]$ diff /etc/hosts /etc/hosts.03162014
> 2c2
> < 10.177.74.55 vxpip-eeisl001
> ---
> > 10.177.74.55 . vxpip-eeisl001
> [a0142566 at vxpip-eeisl001 ~]$
>
> I fixed this and then re-ran some tests using the attached
> syslog-ng.conf.DEBUG and the following invocation:
>
> sudo strace /usr/local/sbin/syslog-ng -vde -f
> /usr/local/etc/syslog-ng.conf.DEBUG > /tmp/syslog-ng-DEBUG-strace 2>&1
>
>
> and things look MUCH better.
>
>
> [a0142566 at vxpip-eeisl001 etc]$ file /var/log/debug*
> /var/log/debug.messages.vxpip-eeisl001.auth.info: ASCII text
> /var/log/debug.messages.vxpip-eeisl001.authpriv.notice: ASCII text
> /var/log/debug.messages.vxpip-eeisl001.cron.info: ASCII text
> /var/log/debug.network_messages.vddp13e-0d6b677: ASCII text
> /var/log/debug.template_messages: ASCII text
> [a0142566 at vxpip-eeisl001 etc]$
>
> Note: I also found another system that has the same config error:
>
> [a0142566 at vxkip-eeisl001 ~]$ cat /etc/hosts
> 127.0.0.1 localhost.localdomain localhost
> 10.181.73.32 . vxkip-eeisl001
>
> and ran a quick test through a resolver perl script that does (basically)
> this:
> my $iaddr = inet_aton($thingy);
> if (my $name = gethostbyaddr($iaddr,AF_INET)) {
> print "$thingy,\t$name\n";
> }
>
> [a0142566 at vxkip-eeisl001 ~]$ echo 10.181.73.32 | ~/bin/resolve.pl
> 10.181.73.32, .
>
> Sorry again for bothering the list (I really should have caught that!!...
> And yes - I will be bringing this to the OS team's attention :-)
>
> Although it does raise a point that (maybe) a default value like
> "please_fix_my_hostname" could be used in case this happens elsewhere?
>
> Just a thought.
>
> Thank you again!!
> Jim
>
>
>
>
> ---- Jim Hendrick <jrhendri at roadrunner.com> wrote:
> > I will take some time today and re-run this with various debugging flags.
> >
> > (just realized I should have... )
> >
> > On 03/15/2014 09:37 PM, jrhendri at roadrunner.com wrote:
> > > Odd - I do not see any gethost calls at all in the strace
> > >
> > > this is from "sudo strace /usr/local/sbin/syslog-ng -f
> /usr/local/etc/syslog-ng.conf > syslog-strace 2>&1"
> > >
> > >
> > > I include the config.status and config.log also to see if that may
> help.
> > >
> > > Jim
> > >
> > >
> > > ---- Balazs Scheidler <bazsi77 at gmail.com> wrote:
> > >> During startup syslog-ng queries the hostname using gethostname. Can
> you
> > >> strace syslog-ng and look for this call to see what it returns?
> > >>
> > >> We are also resolving this using the dns which is also interesting.
> > >>
> > >> Thanks
> > >> On Mar 13, 2014 7:08 PM, <jrhendri at roadrunner.com> wrote:
> > >>
> > >>> OK - after much poking around (including testing this on a different
> host
> > >>> with this morning's latest source) here's what I found
> > >>>
> > >>> I believe that syslog-ng is (for some reason) unable to come up with
> a
> > >>> correct hostname for the local system.
> > >>> I would really like to know why this is happening
> > >>> I would also really like to know if any of the source options I tried
> > >>> should have worked (I was kind of desperate with some of the
> combinations...)
> > >>>
> > >>> That said - here's my summary:
> > >>>
> > >>> Examining the "binary" files, It was always writing 256 bytes of "0"
> where
> > >>> ${HOST} should have been.
> > >>>
> > >>> It *never* happened with a network source (tried with Kiwi sysloggen
> using
> > >>> RCF and non RCF messages)
> > >>> It *always* happened with any local source including all the ones
> > >>> commented out (I tested each separately)
> > >>>
> > >>> source s_local {
> > >>> unix-dgram("/dev/log"); # standard Linux log source (this is the
> default
> > >>> place for the syslog() function to send logs to)
> > >>> #!# unix-stream("/dev/log"); # standard Linux log source (this is
> the
> > >>> default place for the syslog() function to send logs to)
> > >>> #!# system();
> > >>> #!# internal();
> > >>> };
> > >>>
> > >>> Once I identified this behavior, I tried a number of options /
> > >>> combinations to work around this including (each separately)
> > >>> source s_local {
> > >>> system();
> > >>> #!# system( keep_hostname(no) host_override("localhost") );
> > >>> #!# system( keep_hostname(yes) host_override("localhost") );
> > >>> #!# system( flags(no-parse) host_override("localhost") );
> > >>> #!# system( flags(no-hostname) host_override("localhost") );
> > >>> #!# system( flags(no-parse) );
> > >>> #!# system( flags(no-hostname) );
> > >>> #!# internal( );
> > >>> };
> > >>>
> > >>> And nothing worked.
> > >>>
> > >>> Finally I kinda punted... I am now doing something really kludgy as
> a work
> > >>> around:
> > >>> source s_local {
> > >>> system();
> > >>> internal( );
> > >>> };
> > >>>
> > >>> source s_network {
> > >>> udp();
> > >>> };
> > >>>
> > >>> destination d_local_template {
> > >>>
> > >>>
> file("/data/syslog-ng/$YEAR/$MONTH/$DAY/localhost/localhost.$FACILITY.$PRIORITY.$DATE"
> > >>> template("${ISODATE} localhost ${PROGRAM} ${MESSAGE}\n") );
> > >>> };
> > >>>
> > >>>
> > >>> destination d_network {
> > >>>
> > >>>
> file("/data/syslog-ng/$YEAR/$MONTH/$DAY/$HOST_FROM/$HOST_FROM.$FACILITY.$PRIORITY.$DATE");
> > >>> };
> > >>>
> > >>> log {
> > >>> source(s_local);
> > >>> destination(d_local_template);
> > >>> };
> > >>>
> > >>> log {
> > >>> source(s_network);
> > >>> destination(d_network);
> > >>> };
> > >>>
> > >>>
> > >>> ---- jrhendri at roadrunner.com wrote:
> > >>>> I have a really odd problem that I hope someone can assist with.
> > >>>>
> > >>>> My install is inserting a number of zeroes into the logfile between
> the
> > >>> date and the rest of the message.
> > >>>> Here are some details:
> > >>>>
> > >>>> System is RHEL:
> > >>>>
> > >>>> $ uname -a
> > >>>> Linux vxpip-eeisl001 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29
> 11:47:41
> > >>> EST 2013 x86_64 x86_64 x86_64 GNU/Linux
> > >>>> $
> > >>>>
> > >>>>
> > >>>> Syslog-ng was built on the box yesterday with these sources:
> > >>>> eventlog-0.2.12+20120504+1700
> > >>>> syslog-ng-3.5.3
> > >>>>
> > >>>>
> > >>>> $ /usr/local/sbin/syslog-ng --version
> > >>>> syslog-ng 3.5.3
> > >>>> Installer-Version: 3.5.3
> > >>>> Revision: ssh+git://algernon@git.balabit
> > >>>
> /var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#ccb05a22408ba4c837d998b2538854d994f845a5
> > >>>> Compile-Date: Mar 12 2014 11:37:32
> > >>>> Available-Modules:
> > >>>
> afmongodb,afstomp,syslogformat,affile,basicfuncs,csvparser,confgen,system-source,afamqp,linux-kmsg-format,afprog,afuser,afsocket,dbparser,cryptofuncs,afsocket-notls
> > >>>> Enable-Debug: off
> > >>>> Enable-GProf: off
> > >>>> Enable-Memtrace: off
> > >>>> Enable-IPv6: on
> > >>>> Enable-Spoof-Source: off
> > >>>> Enable-TCP-Wrapper: off
> > >>>> Enable-Linux-Caps: off
> > >>>> Enable-Pcre: off
> > >>>>
> > >>>>
> > >>>>
> > >>>> This is the default config file that came with it:
> > >>>>
> > >>>>
> > >>>>
> > >>>
> #############################################################################
> > >>>> # Default syslog-ng.conf file which collects all local logs into a
> > >>>> # single file called /var/log/messages.
> > >>>> #
> > >>>>
> > >>>> @version: 3.5
> > >>>> @include "scl.conf"
> > >>>>
> > >>>> source s_local {
> > >>>> system();
> > >>>> internal();
> > >>>> };
> > >>>>
> > >>>> source s_network {
> > >>>> udp();
> > >>>> };
> > >>>>
> > >>>> destination d_local {
> > >>>> file("/var/log/messages");
> > >>>> };
> > >>>>
> > >>>> log {
> > >>>> source(s_local);
> > >>>>
> > >>>> # uncomment this line to open port 514 to receive messages
> > >>>> #source(s_network);
> > >>>> destination(d_local);
> > >>>> };
> > >>>>
> > >>>>
> > >>>>
> > >>>> Here is the file:
> > >>>>
> > >>>>
> > >>>> $ sudo file /var/log/messages
> > >>>> /var/log/messages: data
> > >>>> $ sudo cat /var/log/messages
> > >>>> Mar 13 10:43:00 syslog-ng[18451]: syslog-ng starting up;
> version='3.5.3'
> > >>>> Mar 13 10:43:01 CROND[18454]: (root) CMD (/util/avail/get_avail.sh
> > >>> 1>/util/avail/logs/get_avail.out 2>&1)
> > >>>> Mar 13 10:43:16 sudo: a0142566 : TTY=pts/2 ; PWD=/home/a0142566 ;
> > >>> USER=root ; COMMAND=/usr/bin/file /var/log/messages
> > >>>> And this is a capture from vi in hex mode on that file.
> > >>>>
> > >>>>
> > >>>> 0000000: 4d61 7220 3133 2031 303a 3433 3a30 3020 Mar 13 10:43:00
> > >>>> 0000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000100: 0000 0000 0000 0000 0000 0000 0000 0020 ...............
> > >>>> 0000110: 7379 736c 6f67 2d6e 675b 3138 3435 315d syslog-ng[18451]
> > >>>> 0000120: 3a20 7379 736c 6f67 2d6e 6720 7374 6172 : syslog-ng star
> > >>>> 0000130: 7469 6e67 2075 703b 2076 6572 7369 6f6e ting up; version
> > >>>> 0000140: 3d27 332e 352e 3327 0a4d 6172 2031 3320 ='3.5.3'.Mar 13
> > >>>> 0000150: 3130 3a34 333a 3031 2000 0000 0000 0000 10:43:01 .......
> > >>>> 0000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00001c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 00001f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000200: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000210: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000220: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000230: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000240: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > >>>> 0000250: 0000 0000 0000 0000 2043 524f 4e44 5b31 ........ CROND[1
> > >>>> 0000260: 3834 3534 5d3a 2028 726f 6f74 2920 434d 8454]: (root) CM
> > >>>> 0000270: 4420 282f 7574 696c 2f61 7661 696c 2f67 D (/util/avail/g
> > >>>> 0000280: 6574 5f61 7661 696c 2e73 6820 313e 2f75 et_avail.sh 1>/u
> > >>>> 0000290: 7469 6c2f 6176 6169 6c2f 6c6f 6773 2f67 til/avail/logs/g
> > >>>> 00002a0: 6574 5f61 7661 696c 2e6f 7574 2032 3e26 et_avail.out 2>&
> > >>>> 00002b0: 3129 0a4d 6172 2031 3320 3130 3a34 333a 1).
> > >>>>
> > >>>>
> > >>>> Any ideas?
> > >>>> Thanks in advance folks!!
> > >>>> Jim
> > >>>>
> > >>>>
> > >>>>
> > >>>
> ______________________________________________________________________________
> > >>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > >>>> Documentation:
> > >>> http://www.balabit.com/support/documentation/?product=syslog-ng
> > >>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > >>>>
> > >>>
> > >>>
> ______________________________________________________________________________
> > >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > >>> Documentation:
> > >>> http://www.balabit.com/support/documentation/?product=syslog-ng
> > >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> ______________________________________________________________________________
> > >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > >>> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > >>>
> >
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20140325/787faeb7/attachment-0001.htm
More information about the syslog-ng
mailing list