Systemd service times out for syslog-ng 3.16.1

16 May 2019

      I'm trying to switch from an init.d script to a system service and having problems getting the service to run.
The syslog-ng service gets timed out and never starts up.

My service looks like this:
[Unit]
Description="Syslog-ng an enhanced syslog"
Documentation=man:syslog-ng(8)
After=network.target
ConditionFileNotEmpty=/opt/syslog-ng/etc/syslog-ng.conf

[Service]
Type=notify
ExecStart=/opt/syslog-ng/sbin/syslog-ng --no-caps -p /var/run/syslogd.pid -rdvt
ExecReload=/bin/kill -HUP $MAINPID
StandardOutput=journal
StandardError=journal
Restart=on-failure
Environment=LD_LIBRARY_PATH=/usr/lib/jvm/java/jre/lib/amd64/server:$LD_LIBRARY_PATH

[Install]
WantedBy=multi-user.target

My config looks like this:
#############################################################################
# Default syslog-ng.conf.partial file which collects all local logs into a
# single file called /var/log/messages.
#

@version: 3.16
@module "mod-java"
@include "scl.conf"
@include "conf.d/gc-parsers.conf"

source s_internal {internal();};

source s_rtl_stream {
    unix-stream("/var/gamecircus/log/rtl-stream.sock" flags(no-parse) log_msg_size(1000000));
};

# Backup for when syslog-ng crashes and doesn't restart queues the messages
source s_rtl_backup {
    file("/var/gamecircus/log/rtl-backup.log" flags(no-parse) log_msg_size(1000000));
};

destination d_kafka_unstructured {
    kafka (
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/")
        kafka-bootstrap-servers(<redacted>)
        topic("syslog-ng-`DeployedEnvironment`")
    );
};

destination d_kafka_structured {
    kafka (
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/kafka/libs/")
        kafka-bootstrap-servers(<redacted>)
        topic("${topic}.${environment}")
        template("$(format-json --scope nv_pairs --exclude MESSAGE)\n")
    );
};

destination d_syslog_ng {
    file("/var/log/syslog-ng");
};

parser p_json { json-parser(); };

log {
    source(s_internal);
    destination(d_syslog_ng);
};

log {
    source(s_rtl_stream);
    parser(p_json);
    destination(d_kafka_structured);
};

log {
    source(s_rtl_backup);
    parser(p_json);
    destination(d_kafka_structured);
};

# Build script will add stuff below for the specific build.
###############################################################################

The logging that I turned on in the service above isbn't helping. Here's the last little bit

May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217091] Add path to classpath: /opt/kafka/libs/kafka-streams-0.11.0.0.jar;
May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217248] Add path to classpath: /opt/kafka/libs/rocksdbjni-5.0.1.jar;
May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.217309] Add path to classpath: /opt/kafka/libs/kafka-streams-examples-0.11.0.0.jar;
May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.361062] Add path to classpath: /opt/syslog-ng/lib/syslog-ng/java-modules/syslog-ng-core.jar;
May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365718] Running application hooks; hook='1'
May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.365930] Running application hooks; hook='3'
May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366137] syslog-ng starting up; version='3.16.1'
May 16 15:10:13 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:10:13.366480] Worker thread started; driver='d_kafka_structured#0'
May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service start operation timed out. Terminating.
May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273058] Running application hooks; hook='4'
May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.273074] syslog-ng shutting down; version='3.16.1'
May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.376882] Worker thread finished; driver='d_kafka_structured#0'
May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377030] Closing log transport fd; fd='92'
May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.377186] Java machine free;
May 16 15:11:43 ip-172-30-6-13.ec2.internal syslog-ng[23866]: [2019-05-16T15:11:43.378858] Running application hooks; hook='5'
May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Failed to start "Syslog-ng an enhanced syslog".
May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: Unit syslog-ng.service entered failed state.
May 16 15:11:43 ip-172-30-6-13.ec2.internal systemd[1]: syslog-ng.service failed.

Any suggestions on what me causing it to timeout? I'm new to Ssytemd but if I understand the notify type it would suggest that syslog-ng isn't notifying systemd that it's started. I'm not sure what would be holding it up.

Any help is much appreciated.

Ronald Fenner
Network Architect
Game Circus LLC.

rfenner@gamecircus.com

Ronald Fenner

Evan Rempel

Ronald Fenner

Ronald Fenner

Evan Rempel

Ronald Fenner

tags

participants (2)