[syslog-ng] Cross-compiling syslog-ng 3.11.1 for an embedded application.
Chris_Johnson at trendmicro.com
Chris_Johnson at trendmicro.com
Wed Oct 4 16:34:25 UTC 2017
I had similar issues when I first started using syslog-ng (I think we started with v2.8) in our embedded system. Specifically we were using SVN for source code control and the automated ‘build’ would do a fresh checkout every time. This screwed up the timestamps on the source files and syslog-ng was trying to rebuild files that really didn’t need it. My solution was to add a section to my Makefile that would ‘touch’ the relevant files to avoid this problem (also, the utilities syslog-ng uses were a newer version of the same utilities used elsewhere in our build, and each would break the other; so we couldn’t just let them be rebuild every time).
Definitions for vars used in the Makefile:
# define ROOTDIR if undefined (RELPATH is where we *know* this Makefile is located under ROOTDIR)
RELPATH := /ports/syslogd/syslog-ng
ROOTDIR ?= $(subst $(RELPATH),,$(CURDIR))
STAGEDIR = $(CURDIR)/stage
SOURCEDIR = syslog-ng-3.5.6
The relevant portion of the Makefile that resets the timestamps is:
# We never need to rebuild 'Makefile.in' or 'configure' in production
# commenting out rules for:
# $(SOURCEDIR)/Makefile.in
# $(SOURCEDIR)/configure.in
#
# commenting out pre-requisites for:
# $(SOURCEDIR)/Makefile
# config
#
# NOTE: syslog-ng 3.5.x needs to use autoconf v2.69
# automake v1.14
#
# WARNING: Currently (as of 11/11/13) the rest of the build needs version(s):
# autoconf v2.63
# automake v1.11
# and will fail when using the v2.69 (v1.14) versions!
#
# If 'configure' and/or any 'Makefile.in' file(s) need to be updated,
# you will have 'switch' between the two versions of autoconf/automake;
# using 2.69/1.14 for syslog-ng and 2.63/1.11 for the rest of the build.
#
# force 'foreign' strictness (this is how Makefiles from syslog-ng were made)
# see www.gnu.org/software/automake/manual for more info
#$(SOURCEDIR)/Makefile.in: $(SOURCEDIR)/Makefile.am
# cd $(SOURCEDIR); \
# automake --foreign
# 3.3.x uses 'older' nomenclature of 'configure.in' (instead of 'configure.ac')
# 3.5.x now uses 'configure.ac' nomenclature
# see www.gnu.org/software/autoconf/manual for more info
#$(SOURCEDIR)/configure: $(SOURCEDIR)/configure.in
# cd $(SOURCEDIR); \
# autoconf
# build a new 'Makefile' if 'Makefile.in' or 'configure' is newer
# or this Makefile is newer (assume a change to configure parameter list)
#$(SOURCEDIR)/Makefile: $(SOURCEDIR)/Makefile.in $(SOURCEDIR)/configure Makefile
#
# Force 'recipe' to NOT run in parallel (.NOTPARALLEL:).
# This allows the 'kludge' for timestamps to work correctly!
# Without this, the two sets of touches occur virtually simultaneously since
# each command in the 'recipe' runs in its own shell AND the shells run in
# parallel, the 'sleep' command isn't causing the second set of touches
# to generate a different timestamp from the first set of touches.
#
# This seems to be causing random build errors on the EC build box as each
# command is run in a different parallel order
#
.NOTPARALLEL:
$(SOURCEDIR)/Makefile:
make config
########################################################################
# kludge to fix 'svn checkout' timestamps messing up Makefile rules
# - touch files to restore proper relative timestamps
#
# cfg-grammar.y needs to be older that the other -grammar.y files
touch $(SOURCEDIR)/lib/cfg-grammar.y
sleep 1
find $(SOURCEDIR) -name .svn -prune -o -name cfg-grammar.y -prune -o -name "*-grammar.y" -exec touch {} \;
#
# touch 'configure' dependent files
for d in \
. \
lib/ivykis \
modules/afamqp/rabbitmq-c \
modules/afmongodb/libmongo-client \
;do for f in \
configure.ac \
aclocal.m4 \
Makefile.in \
config.h.in \
configure \
;do touch $(SOURCEDIR)/$$d/$$f ; \
done ; \
done
#
# force '-grammar.y' files to be older than '-grammar.[ch]' files
sleep 1
find $(SOURCEDIR) -name .svn -prune -o -name "*-grammar.[ch]" -exec touch {} \;
#
# force '.n' files to be younger than '.n.xml' files
for f in \
loggen.1 \
pdbtool.1 \
syslog-ng.8 \
syslog-ng.conf.5 \
syslog-ng-ctl.1 \
;do touch $(SOURCEDIR)/doc/man/$$f ; \
done
#
# touch any other files as needed
touch $(SOURCEDIR)/lib/cfg-lex.h
touch $(SOURCEDIR)/lib/cfg-lex.c
########################################################################
Also, related to this was the ‘clean’ feature that was removing too much:
clean distclean:
@# syslog-ng 3.5.x build system's clean is too clean and is deleting '-grammar' files
@# requiring them to be rebuild... BUT the utilities needed are either missing or the
@# wrong version to successfully be rebuild in the BUILD system(s)!
@#
@# So, manually clean!
@#
@#make -C $(SOURCEDIR) $@ || true
@rm -rf $(STAGEDIR)
@#
@# walk through a list of dirs deleting them
@for d in \
autom4te.cache \
.deps \
.libs \
;do find $(SOURCEDIR) -depth -name .svn -prune -o -name $$d -exec rm -rf {} \; ; \
done
@#
@# walk through a list of file patterns deleting them
@for fpat in \
config.h \
config.log \
config.status \
stamp-h1 \
libtool \
"*.pc" \
.dirstamp \
"*.l[ao]" \
"*.[ao]" \
;do find $(SOURCEDIR) -name .svn -prune -o -name $$fpat -exec rm -f {} \; ; \
done
@#
@# delete Makefile(s)
@for d in \
. \
lib/ivykis \
lib/ivykis/test \
lib/ivykis/src \
lib/ivykis/misc \
lib/ivykis/man3 \
lib/ivykis/contrib \
lib/ivykis/contrib/kojines \
lib/ivykis/contrib/iv_getaddrinfo \
modules/afamqp/rabbitmq-c \
;do rm -f $(SOURCEDIR)/$$d/Makefile;\
done
@#
@# delete specific files not already covered
@for f in \
scripts/update-patterndb \
tests/loggen/loggen \
lib/ivykis/src/include/iv.h \
modules/dbparser/pdbtool/pdbtool \
modules/afamqp/rabbitmq-c/examples/amqp_exchange_declare \
modules/afamqp/rabbitmq-c/examples/amqp_bind \
modules/afamqp/rabbitmq-c/examples/amqp_listenq \
modules/afamqp/rabbitmq-c/examples/amqp_unbind \
modules/afamqp/rabbitmq-c/examples/amqp_sendstring \
modules/afamqp/rabbitmq-c/examples/amqp_rpc_sendstring_client \
modules/afamqp/rabbitmq-c/examples/amqp_producer \
modules/afamqp/rabbitmq-c/examples/amqp_consumer \
modules/afamqp/rabbitmq-c/examples/amqp_listen \
syslog-ng/syslog-ng-ctl \
syslog-ng/syslog-ng \
;do rm -f $(SOURCEDIR)/$$f; \
done
Hope these snippets are useful starting points.
Chris
-------------------------------------
Christopher Johnson
chris_johnson at trendmicro.com<mailto:chris_johnson at trendmicro.com>
Trend Micro Inc. - Tippingpoint
-------------------------------------
From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Robert King
Sent: Wednesday, October 4, 2017 8:09 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] Cross-compiling syslog-ng 3.11.1 for an embedded application.
Not easily, and not without getting permission from the director of development.
Let’s start with an easy question. …/syslog-ng/syslog-ng is a script. Is this part of the *final* distribution to be installed on the target system? There is also …/syslog-ng/.libs/syslog-ng with is an ELF binary executable. Is this part of the *final* distribution to be installed on the target system? Why are there two executable files with the same name?
…/syslog-ng/.libs/syslog-ng is being natively built on a generic Linux Mint system, but not on my cross-development system. How do you suggest troubleshooting this? Going through the generated Makefile in the project root directory is a nightmare. There is no makefile in …/syslog-ng, so I can’t try to build just that.
Sorry that I was irritated earlier, but I’ve been fighting this for three weeks and my boss is getting angry.
From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu] On Behalf Of Budai, László
Sent: Wednesday, October 04, 2017 10:00 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Subject: Re: [syslog-ng] Cross-compiling syslog-ng 3.11.1 for an embedded application.
Hi,
could you share a build environment where we can easily reproduce the issue(eg.: a docker file, or, image )?
Laszlo Budai
On Wednesday, October 4, 2017, Robert King <robert.king at tellabs.com<mailto:robert.king at tellabs.com>> wrote:
From: syslog-ng [mailto:syslog-ng-bounces at lists.balabit.hu<javascript:_e(%7B%7D,'cvml','syslog-ng-bounces at lists.balabit.hu');>] On Behalf Of Scheidler, Balázs
Sent: Tuesday, October 03, 2017 11:32 PM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu<javascript:_e(%7B%7D,'cvml','syslog-ng at lists.balabit.hu');>>
Subject: Re: [syslog-ng] Cross-compiling syslog-ng 3.11.1 for an embedded application.
Hi,
Again, so without your help and a civilized exchange of emails this can't be troubleshot, as I never saw your environment, and can't check various details that would affect compilation.
What information would you like from me to progress?
We are using gmake to build the targets. We are calling the configure script with the following arguments:
PKG_CONFIG_PATH=$PCRERELEASE/lib/pkgconfig:$EVENTLOGRELEASE/lib/pkgconfig:$GLIBRELEASE/lib/pkgconfig \
SED=/bin/sed \
./configure \
--host=$HOST \
PKG_CONFIG_PATH=$PCRERELEASE/lib/pkgconfig:$EVENTLOGRELEASE/lib/pkgconfig:$GLIBRELEASE/lib/pkgconfig \
--build=$BUILD \
--prefix=$INSTALLDIR \
--bindir=$INSTALLDIR/bin \
--sbindir=$INSTALLDIR/sbin \
--includedir=$INSTALLDIR/include \
--libdir=$INSTALLDIR/lib \
--sharedstatedir=$INSTALLDIR/share \
--exec-prefix=$EXECDIR \
--libexecdir=$EXECDIR/lib \
--localstatedir=$EXECDIR/etc/syslog-ng/var \
--sysconfdir=$EXECDIR/etc/syslog-ng/etc \
--enable-shared=yes \
--enable-ssl \
--enable-static=yes \
--enable-dynamic-linking=yes \
--disable-json \
--enable-pcre=no \
--with-libmongo-client=no \
--with-librabbitmq-client=no \
CC=$MVL/bin/ppc_440-gcc \
CFLAGS="-g -O2" \
CPP=$MVL/bin/ppc_440-cpp \
CPPFLAGS="-I$EVENTLOGRELEASE/include -I$GLIBRELEASE/include -I$TARGET/usr/include -I$TARGET/usr/local/include" \
LDFLAGS="-L/vobs/tools/openssl/openssl/release/$OS/lib -L$EVENTLOGRELEASE/lib -L$GLIBRELEASE/lib -L$TARGET/lib -L$TARGET/usr/lib" \
OPENSSL_CFLAGS="${OPENSSL_CC_INC_PATH} ${OPENSSL_CC_FIPS_INC_PATH} -DBUILD_WITH_SSL" \
OPENSSL_LIBS="-lssl -lcrypto" \
LIBS="-lssl -lcrypto"
<table class="TM_EMAIL_NOTICE"><tr><td><pre>
TREND MICRO EMAIL NOTICE
The information contained in this email and any attachments is confidential
and may be subject to copyright or other intellectual property protection.
If you are not the intended recipient, you are not authorized to use or
disclose this information, and we request that you notify us by reply mail or
telephone and delete the original message from your mail system.
</pre></td></tr></table>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20171004/9608840b/attachment-0001.html>
More information about the syslog-ng
mailing list