Project Lumberjack to improve Linux logging In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Gibbs (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team. We discussed a couple of pain points for logging, logging is usually an afterthought during development, computer based processing, correllation of application logs is nearly impossible. We roughly agreed that the key to improve the situation is to involve the community at large, initiate a momentum and try to get application developers on board and have them create structured logs. We also agreed that this will not happen overnight and we need to take a gradual approach. To move into that direction, the benefits of good logging needs to be communicated and delivered to both application developers and their users. We also talked about what kind of building blocks are needed to deliver a solution fast, and concluded that we basically have everything available, and even better they are open source. The key is to tie these components together, document best practices and perhaps provide better integration. Thus project Lumberjack was born, hosted as a Fedora project at https://fedorahosted.org/lumberjack/. The building blocks that need some care are: * some applications already produce logs in structured format, those should be integrated (auditd for instance) * we need to define a mechanism to submit structured logs to local logging services for further processing (ELAPI and some enhanced syslog) * we need to make sure that local logging services cope with structured data (already available for a long time now) * we need to define a mechanism to store messages in a structured form and a way query them * last, but not least we need to define a naming scheme for event data which CEE can bring to the table Most of these is already possible by using a combination of tools and proper configuration, however learning how to do this is not a trivial undertaking for those who only want to develop or use applications. Changing that is the primary aim of Project Lumberjack. If you are interested in logging, make sure to check that out.