Re: [syslog-ng] v3.3 not flushing lines?

13 Jan 2012

      It's been 3 days, not hours :-)

Here's the user's config file:

#############################################################################
# Default syslog-ng.conf file which collects all local logs into a
# single file called /var/log/messages.
#

@version: 3.3
@include "scl.conf"

source s_local {
        system();
        internal();
};

source s_remote {
        udp(ip(0.0.0.0) port(514));
        tcp(ip(0.0.0.0) port(514));
};

destination d_separatedbyhosts {
        file("/var/log/syslog-ng/$HOST/messages" owner("root")
group("root") perm(0640) dir_perm(0750) create_dirs(yes)
        template("$R_YEAR-$R_MONTH-$R_DAY
$R_HOUR:$R_MIN:$S_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n")
        template_escape(yes)
        );
};

log {
        source(s_remote);

        # uncomment this line to open port 514 to receive messages
        #source(s_network);
        destination(d_separatedbyhosts);
};
# <lzconfig> BEGIN LogZilla v3.2 syslog-ng config file
# Please don't remove the lzconfig tags above and at the end of this block.
#
# NOTE:
# You may need to alter your receive buffer if you expect a large amount of
logs!
# Please read
http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers
# Example:  udp( so_rcvbuf(1048576));
#
#
###########################################################################################
# BEGIN LogZilla Config for syslog-ng
###########################################################################################
# Last updated on 2011-11-20
###########################################################################################
options {
      # long_hostnames(off);
      # doesn't actually help on Solaris, log(3) truncates at 1024 chars
      log_msg_size(8192);
      # buffer just a little for performance
      # sync(1); <- Deprecated - use flush_lines() instead
      flush_lines(1);
      # memory is cheap, buffer messages unable to write (like to loghost)
      log_fifo_size(16384);
      # Hosts we don't want syslog from
      #bad_hostname("^(ctld.|cmd|tmd|last)$");
      # The time to wait before a dead connection is reestablished (seconds)
      time_reopen(10);
      #Use DNS so that our good names are used, not hostnames
      use_dns(yes);
      dns_cache(yes);
      #Use the whole DNS name
      use_fqdn(yes);
      keep_hostname(yes);
      # chain_hostnames(no);
      #Read permission for everyone
      perm(0644);
      # The default action of syslog-ng 1.6.0 is to log a STATS line
      # to the file every 10 minutes.  That's pretty ugly after a while.
      # Change it to every 12 hours so you get a nice daily update of
      # # how many messages syslog-ng missed (0).
      # stats(43200);
};

# Note: LogZilla will ONLY process log entries in the format below.
# You can't run db_insert.pl on any log file without using this template.
# The reason is that messages vary in composition so the tab delimiters are
# needed to determine the tokens.
destination d_logzilla {
   program("/var/www/logzilla/scripts/db_insert.pl"
   template("$R_YEAR-$R_MONTH-$R_DAY
$R_HOUR:$R_MIN:$S_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n")
   template_escape(yes)
   );
};

destination df_logzilla {
   file("/var/log/logzilla/DEBUG.log"
   template("$R_YEAR-$R_MONTH-$R_DAY
$R_HOUR:$R_MIN:$R_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n")
   template_escape(yes)
   );
};

# Tell syslog-ng to log to our new destination
log {
   source(s_remote);
      destination(d_logzilla);
      # Uncomment below and restart syslog-ng for debugging
      destination(df_logzilla);
};
# END LogZilla Config for syslog-ng
###########################################################################################
# </lzconfig> END LogZilla v3.2 syslog-ng config file

______________________________________________________________

Clayton Dukes
______________________________________________________________

On Fri, Jan 13, 2012 at 9:54 AM, Patrick Hemmer <syslogng@feystorm.net>wrote:
...
Sent: Fri Jan 13 2012 09:45:55 GMT-0500 (EST)
From: Clayton Dukes <cdukes@gmail.com>
To: Syslog-ng users' and developers' mailing list <
syslog-ng@lists.balabit.hu>
Subject: Re: [syslog-ng] v3.3 not flushing lines?
...
3rd try...anyone?
______________________________**______________________________**__
Clayton Dukes
______________________________**______________________________**__
On Thu, Jan 12, 2012 at 3:48 PM, Clayton Dukes <cdukes@gmail.com <mailto:
cdukes@gmail.com>> wrote:
Can anyone help here? I'm out of ideas :-)
______________________________**______________________________**__
Clayton Dukes
   ______________________________**______________________________**__
On Wed, Jan 11, 2012 at 8:05 PM, Clayton Dukes <cdukes@gmail.com
   <mailto:cdukes@gmail.com>> wrote:
Hey folks,
I have a user experiencing an issue where some of the incoming
       logs are not getting piped to my perl script until the second
       time we generate events.
I believe this is generally set using flush_lines(1), but it
       doesn’t seem to be honoring that setting in the config.
       I also tried adding flush_timeout(900), but that has no affect.
       In the output below, if I quit the telnet and start it again,
       the previously missing events are then received (but the
       current ones are missing) - basically, it doesn't appear to be
       flushing every single line.
       Possible other reason: has something changed between v2.x and
       3.x where the program() destination would possibly not send an
       EOF signal - i.e. is the pipe now kept open vs. an individual
       call to the program each time in v2.x?
To verify that this is happening:
Term 1:
       tail -f /tmp/logzilla_import.txt
       Term 2:
       /usr/local/sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -Fdv
       Term 3:
       telnet 192.168.254.1
Term 1 results:
       192.168.254.1 22 7 3732620769 .Jan 11 2012 19:29:02.284 EST:
       Telnet2: 1 1 251 1 3751981041 0 2012-01-11 19
       <tel:2012-01-11%2019>:29:02 2012-01-11 19
       <tel:2012-01-11%2019>:29:02
192.168.254.1 22 7 3732620769 .Jan 11 2012 19:29:02.284 EST:
       TCP2: Telnet sent WILL ECHO (1) 3751981041 0 2012-01-11 19
       <tel:2012-01-11%2019>:29:02 2012-01-11 19
       <tel:2012-01-11%2019>:29:02
Term 2 results:
       Incoming log entry; line='<183>6987: .Jan 11 2012 19:29:02.284
       EST: TCP2: Telnet sent WILL ECHO (1)'
       Incoming log entry; line='<183>6988: .Jan 11 2012 19:29:02.284
       EST: Telnet2: 2 2 251 3'
       Incoming log entry; line='<183>6989: .Jan 11 2012 19:29:02.284
       EST: TCP2: Telnet sent WILL SUPPRESS-GA (3)'
       Incoming log entry; line='<183>6990: .Jan 11 2012 19:29:02.284
       EST: Telnet2: 80000 80000 253 24'
       Incoming log entry; line='<183>6991: .Jan 11 2012 19:29:02.284
       EST: TCP2: Telnet sent DO TTY-TYPE (24)'
       Incoming log entry; line='<183>6992: .Jan 11 2012 19:29:02.284
       EST: Telnet2: 10000000 10000000 253 31'
       Incoming log entry; line='<183>6993: .Jan 11 2012 19:29:02.284
       EST: TCP2: Telnet sent DO WINDOW-SIZE (31)'
       Incoming log entry; line='<183>6994: .Jan 11 2012 19:29:02.284
       EST: TCP2: Telnet received DO ENCRYPTION (38)'
       Incoming log entry; line='<183>6995: .Jan 11 2012 19:29:02.284
       EST: TCP2: Telnet sent WONT ENCRYPTION (38) (unimplemented)'
       Incoming log entry; line='<183>6996: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received WILL ENCRYPTION (38)'
       Incoming log entry; line='<183>6997: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet sent DONT ENCRYPTION (38) (unimplemented)'
       Incoming log entry; line='<183>6998: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received DO SUPPRESS-GA (3)'
       Incoming log entry; line='<183>6999: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received WILL TTY-TYPE (24)'
       Incoming log entry; line='<183>7000: .Jan 11 2012 19:29:02.292
       EST: Telnet2: Sent SB 24 1 '
       Incoming log entry; line='<183>7001: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received WILL WINDOW-SIZE (31)'
       Incoming log entry; line='<183>7002: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received WILL TTY-SPEED (32) (refused)'
       Incoming log entry; line='<183>7003: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet sent DONT TTY-SPEED (32)'
       Incoming log entry; line='<183>7004: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received WILL LOCAL-FLOW (33) (refused)'
       Incoming log entry; line='<183>7005: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet sent DONT LOCAL-FLOW (33)'
       Incoming log entry; line='<183>7006: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received WILL LINEMODE (34)'
       Incoming log entry; line='<183>7007: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet sent DONT LINEMODE (34) (unimplemented)'
       Incoming log entry; line='<183>7008: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received WILL NEW-ENVIRON (39)'
       Incoming log entry; line='<183>7009: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet sent DONT NEW-ENVIRON (39) (unimplemented)'
       Incoming log entry; line='<183>7010: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received DO STATUS (5)'
       Incoming log entry; line='<183>7011: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet sent WONT STATUS (5) (unimplemented)'
       Incoming log entry; line='<183>7012: .Jan 11 2012 19:29:02.292
       EST: TCP2: Telnet received DO ECHO (1)'
       Incoming log entry; line='<183>7013: .Jan 11 2012 19:29:02.292
       EST: Telnet2: recv SB NAWS 132 63'
       Incoming log entry; line='<183>7014: .Jan 11 2012 19:29:02.292
       EST: Telnet2: recv SB 24 0 LINUX'
       Incoming log entry; line='<183>7015: .Jan 11 2012 19:29:02.493
       EST: TCP2: Telnet received WILL ENVIRONMENT (36) (refused)'
       Incoming log entry; line='<183>7016: .Jan 11 2012 19:29:02.493
       EST: TCP2: Telnet sent DONT ENVIRONMENT (36)'
______________________________**______________________________**__
Clayton Dukes
       ______________________________**______________________________**__
This is a mailing list, repeating yourself isnt going to help. Not
everyone checks it every few hours. Have patience.