[syslog-ng] Bazsi's blog: Project Lumberjack to improve Linux logging

[Balazs Scheidler]

Project Lumberjack to improve Linux logging

In a lively discussion at the RedHat offices two weeks ago in Brno, a
number of well respected individuals were discussing how logging in
general, and Linux logging in particular could be improved. As you may
have guessed I was invited because of syslog-ng, but representatives of
other logging related projects were also in nice numbers: Steve Gibbs
(auditd), Lennart Poettering (systemd, journald), Rainer Gerhards
(rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people
from the RedHat team.

We discussed a couple of pain points for logging, logging is usually an
afterthought during development, computer based processing, correllation
of application logs is nearly impossible. We roughly agreed that the key
to improve the situation is to involve the community at large, initiate
a momentum and try to get application developers on board and have them
create structured logs. We also agreed that this will not happen
overnight and we need to take a gradual approach.

To move into that direction, the benefits of good logging needs to be
communicated and delivered to both application developers and their
users.

We also talked about what kind of building blocks are needed to deliver
a solution fast, and concluded that we basically have everything
available, and even better they are open source. The key is to tie these
components together, document best practices and perhaps provide better
integration.

Thus project Lumberjack was born, hosted as a Fedora project at
https://fedorahosted.org/lumberjack/.

The building blocks that need some care are:


      * some applications already produce logs in structured format,
        those should be integrated (auditd for instance)
      * we need to define a mechanism to submit structured logs to local
        logging services  for further processing (ELAPI and some
        enhanced syslog)
      * we need to make sure that local logging services cope with
        structured data (already available for a long time now)
      * we need to define a mechanism to store messages in a structured
        form and a way query them
      * last, but not least we need to define a naming scheme for event
        data which CEE can bring to the table


Most of these is already possible by using a combination of tools and
proper configuration, however learning how to do this is not a trivial
undertaking for those who only want to develop or use applications.

Changing that is the primary aim of Project Lumberjack. If you are
interested in logging, make sure to check that out.

 

 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20120301/20431b43/attachment.htm