[syslog-ng] v3.3 not flushing lines?

Clayton Dukes cdukes at gmail.com
Fri Jan 13 16:02:39 CET 2012


It's been 3 days, not hours :-)

Here's the user's config file:

#############################################################################
# Default syslog-ng.conf file which collects all local logs into a
# single file called /var/log/messages.
#

@version: 3.3
@include "scl.conf"

source s_local {
        system();
        internal();
};

source s_remote {
        udp(ip(0.0.0.0) port(514));
        tcp(ip(0.0.0.0) port(514));
};

destination d_separatedbyhosts {
        file("/var/log/syslog-ng/$HOST/messages" owner("root")
group("root") perm(0640) dir_perm(0750) create_dirs(yes)
        template("$R_YEAR-$R_MONTH-$R_DAY
$R_HOUR:$R_MIN:$S_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n")
        template_escape(yes)
        );
};

log {
        source(s_remote);

        # uncomment this line to open port 514 to receive messages
        #source(s_network);
        destination(d_separatedbyhosts);
};
# <lzconfig> BEGIN LogZilla v3.2 syslog-ng config file
# Please don't remove the lzconfig tags above and at the end of this block.
#
# NOTE:
# You may need to alter your receive buffer if you expect a large amount of
logs!
# Please read
http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers
# Example:  udp( so_rcvbuf(1048576));
#
#
###########################################################################################
# BEGIN LogZilla Config for syslog-ng
###########################################################################################
# Last updated on 2011-11-20
###########################################################################################
options {
      # long_hostnames(off);
      # doesn't actually help on Solaris, log(3) truncates at 1024 chars
      log_msg_size(8192);
      # buffer just a little for performance
      # sync(1); <- Deprecated - use flush_lines() instead
      flush_lines(1);
      # memory is cheap, buffer messages unable to write (like to loghost)
      log_fifo_size(16384);
      # Hosts we don't want syslog from
      #bad_hostname("^(ctld.|cmd|tmd|last)$");
      # The time to wait before a dead connection is reestablished (seconds)
      time_reopen(10);
      #Use DNS so that our good names are used, not hostnames
      use_dns(yes);
      dns_cache(yes);
      #Use the whole DNS name
      use_fqdn(yes);
      keep_hostname(yes);
      # chain_hostnames(no);
      #Read permission for everyone
      perm(0644);
      # The default action of syslog-ng 1.6.0 is to log a STATS line
      # to the file every 10 minutes.  That's pretty ugly after a while.
      # Change it to every 12 hours so you get a nice daily update of
      # # how many messages syslog-ng missed (0).
      # stats(43200);
};


# Note: LogZilla will ONLY process log entries in the format below.
# You can't run db_insert.pl on any log file without using this template.
# The reason is that messages vary in composition so the tab delimiters are
# needed to determine the tokens.
destination d_logzilla {
   program("/var/www/logzilla/scripts/db_insert.pl"
   template("$R_YEAR-$R_MONTH-$R_DAY
$R_HOUR:$R_MIN:$S_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n")
   template_escape(yes)
   );
};

destination df_logzilla {
   file("/var/log/logzilla/DEBUG.log"
   template("$R_YEAR-$R_MONTH-$R_DAY
$R_HOUR:$R_MIN:$R_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n")
   template_escape(yes)
   );
};

# Tell syslog-ng to log to our new destination
log {
   source(s_remote);
      destination(d_logzilla);
      # Uncomment below and restart syslog-ng for debugging
      destination(df_logzilla);
};
# END LogZilla Config for syslog-ng
###########################################################################################
# </lzconfig> END LogZilla v3.2 syslog-ng config file



______________________________________________________________

Clayton Dukes
______________________________________________________________


On Fri, Jan 13, 2012 at 9:54 AM, Patrick Hemmer <syslogng at feystorm.net>wrote:

> Sent: Fri Jan 13 2012 09:45:55 GMT-0500 (EST)
> From: Clayton Dukes <cdukes at gmail.com>
> To: Syslog-ng users' and developers' mailing list <
> syslog-ng at lists.balabit.hu>
> Subject: Re: [syslog-ng] v3.3 not flushing lines?
>
>> 3rd try...anyone?
>>
>> ______________________________**______________________________**__
>>
>> Clayton Dukes
>> ______________________________**______________________________**__
>>
>>
>> On Thu, Jan 12, 2012 at 3:48 PM, Clayton Dukes <cdukes at gmail.com <mailto:
>> cdukes at gmail.com>> wrote:
>>
>>    Can anyone help here? I'm out of ideas :-)
>>
>>    ______________________________**______________________________**__
>>
>>    Clayton Dukes
>>    ______________________________**______________________________**__
>>
>>
>>
>>    On Wed, Jan 11, 2012 at 8:05 PM, Clayton Dukes <cdukes at gmail.com
>>    <mailto:cdukes at gmail.com>> wrote:
>>
>>        Hey folks,
>>
>>        I have a user experiencing an issue where some of the incoming
>>        logs are not getting piped to my perl script until the second
>>        time we generate events.
>>
>>        I believe this is generally set using flush_lines(1), but it
>>        doesn’t seem to be honoring that setting in the config.
>>        I also tried adding flush_timeout(900), but that has no affect.
>>        In the output below, if I quit the telnet and start it again,
>>        the previously missing events are then received (but the
>>        current ones are missing) - basically, it doesn't appear to be
>>        flushing every single line.
>>        Possible other reason: has something changed between v2.x and
>>        3.x where the program() destination would possibly not send an
>>        EOF signal - i.e. is the pipe now kept open vs. an individual
>>        call to the program each time in v2.x?
>>
>>
>>        To verify that this is happening:
>>
>>
>>        Term 1:
>>        tail -f /tmp/logzilla_import.txt
>>        Term 2:
>>        /usr/local/sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -Fdv
>>        Term 3:
>>        telnet 192.168.254.1
>>
>>        Term 1 results:
>>        192.168.254.1 22 7 3732620769 .Jan 11 2012 19:29:02.284 EST:
>>        Telnet2: 1 1 251 1 3751981041 0 2012-01-11 19
>>        <tel:2012-01-11%2019>:29:02 2012-01-11 19
>>        <tel:2012-01-11%2019>:29:02
>>
>>        192.168.254.1 22 7 3732620769 .Jan 11 2012 19:29:02.284 EST:
>>        TCP2: Telnet sent WILL ECHO (1) 3751981041 0 2012-01-11 19
>>        <tel:2012-01-11%2019>:29:02 2012-01-11 19
>>        <tel:2012-01-11%2019>:29:02
>>
>>
>>        Term 2 results:
>>        Incoming log entry; line='<183>6987: .Jan 11 2012 19:29:02.284
>>        EST: TCP2: Telnet sent WILL ECHO (1)'
>>        Incoming log entry; line='<183>6988: .Jan 11 2012 19:29:02.284
>>        EST: Telnet2: 2 2 251 3'
>>        Incoming log entry; line='<183>6989: .Jan 11 2012 19:29:02.284
>>        EST: TCP2: Telnet sent WILL SUPPRESS-GA (3)'
>>        Incoming log entry; line='<183>6990: .Jan 11 2012 19:29:02.284
>>        EST: Telnet2: 80000 80000 253 24'
>>        Incoming log entry; line='<183>6991: .Jan 11 2012 19:29:02.284
>>        EST: TCP2: Telnet sent DO TTY-TYPE (24)'
>>        Incoming log entry; line='<183>6992: .Jan 11 2012 19:29:02.284
>>        EST: Telnet2: 10000000 10000000 253 31'
>>        Incoming log entry; line='<183>6993: .Jan 11 2012 19:29:02.284
>>        EST: TCP2: Telnet sent DO WINDOW-SIZE (31)'
>>        Incoming log entry; line='<183>6994: .Jan 11 2012 19:29:02.284
>>        EST: TCP2: Telnet received DO ENCRYPTION (38)'
>>        Incoming log entry; line='<183>6995: .Jan 11 2012 19:29:02.284
>>        EST: TCP2: Telnet sent WONT ENCRYPTION (38) (unimplemented)'
>>        Incoming log entry; line='<183>6996: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received WILL ENCRYPTION (38)'
>>        Incoming log entry; line='<183>6997: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet sent DONT ENCRYPTION (38) (unimplemented)'
>>        Incoming log entry; line='<183>6998: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received DO SUPPRESS-GA (3)'
>>        Incoming log entry; line='<183>6999: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received WILL TTY-TYPE (24)'
>>        Incoming log entry; line='<183>7000: .Jan 11 2012 19:29:02.292
>>        EST: Telnet2: Sent SB 24 1 '
>>        Incoming log entry; line='<183>7001: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received WILL WINDOW-SIZE (31)'
>>        Incoming log entry; line='<183>7002: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received WILL TTY-SPEED (32) (refused)'
>>        Incoming log entry; line='<183>7003: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet sent DONT TTY-SPEED (32)'
>>        Incoming log entry; line='<183>7004: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received WILL LOCAL-FLOW (33) (refused)'
>>        Incoming log entry; line='<183>7005: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet sent DONT LOCAL-FLOW (33)'
>>        Incoming log entry; line='<183>7006: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received WILL LINEMODE (34)'
>>        Incoming log entry; line='<183>7007: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet sent DONT LINEMODE (34) (unimplemented)'
>>        Incoming log entry; line='<183>7008: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received WILL NEW-ENVIRON (39)'
>>        Incoming log entry; line='<183>7009: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet sent DONT NEW-ENVIRON (39) (unimplemented)'
>>        Incoming log entry; line='<183>7010: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received DO STATUS (5)'
>>        Incoming log entry; line='<183>7011: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet sent WONT STATUS (5) (unimplemented)'
>>        Incoming log entry; line='<183>7012: .Jan 11 2012 19:29:02.292
>>        EST: TCP2: Telnet received DO ECHO (1)'
>>        Incoming log entry; line='<183>7013: .Jan 11 2012 19:29:02.292
>>        EST: Telnet2: recv SB NAWS 132 63'
>>        Incoming log entry; line='<183>7014: .Jan 11 2012 19:29:02.292
>>        EST: Telnet2: recv SB 24 0 LINUX'
>>        Incoming log entry; line='<183>7015: .Jan 11 2012 19:29:02.493
>>        EST: TCP2: Telnet received WILL ENVIRONMENT (36) (refused)'
>>        Incoming log entry; line='<183>7016: .Jan 11 2012 19:29:02.493
>>        EST: TCP2: Telnet sent DONT ENVIRONMENT (36)'
>>
>>
>>
>>
>>
>>
>>        ______________________________**______________________________**__
>>
>>        Clayton Dukes
>>        ______________________________**______________________________**__
>>
>>
>>
>>  This is a mailing list, repeating yourself isnt going to help. Not
> everyone checks it every few hours. Have patience.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20120113/7aa7e162/attachment-0001.htm 


More information about the syslog-ng mailing list