[syslog-ng] Understanding Sources & Destinations

[Evan Rempel]

to address this test cycle we use the following setup:

1. Normal instance of syslog-ng that does the production processing. It also has a pipe source and tags the pipe source with a "TEST" tag. It does all of the normal processing on this source as any other production messages, however, any message with the TEST tag is written in json format to a file source with a "final" flag.

2. Run a second instance of syslog-ng that uses a dbparser to parse a line of the format that we write our syslog lines to file. This instance reads from a pipe source and reformats the syslog line to an "on the wire" format and writes it to the pipe destination matching the pipe source of the production instance

3. To test any messages, we just copy them from the raw syslog files and paste them into the test pipe in #2. Then we examine the json object in the TEST destination of #1.

We have a "test suite" of bad messages that we run as a regression test and compare the json output file to verify functionality.

Works well for us.

Evan.

On 04/10/2015 05:59 AM, Jim Hendrick wrote:
> This could be used for example to process historical log data as part of an investigation.  Say you have a search tool (like ELK or something that needs preprocessing to json, sql, etc).
> Being able to send those logs through a command line version that does the exact same processing as the real time one would be extremely valuable.
> Jim
>
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
> -------- Original message --------
> From: "Scheidler, Balázs" <balazs.scheidler at balabit.com>
> Date: 04/10/2015 12:39 AM (GMT-05:00)
> To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
> Subject: Re: [syslog-ng] Understanding Sources & Destinations
>
> Hi,
>
> The command line tool should work primarily between stdin/stdout just like for instance awk.
>
> We may make it possible to fire up tcp listeners but the primary use-case is batch processing already received data.
>
> Another use case is to debug configuration snippets in a faster way than edit/reload/test cycle that nromal syslog-ng offers.
>
> On Apr 10, 2015 12:55 AM, "Asadullah Hussain" <asadxflow at gmail.com <mailto:asadxflow at gmail.com>> wrote:
>
>     Hi Guys, I am working to understand the task of developing syslog-ng as a command line tool (GSoC2015) and I am trying to figure what "possible outputs" the command line tool will take. As per my understanding the input & outputs to syslog are defined as Sources & Destinations (which are places where applications output their logs from respectively).
>
>     I have read the documentation about sources & destinations [Table 6.2 ](http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#chapter-sources) and my understanding is that syslog has following 3 types of sources: files, streams on tcp/ip sockets and stdout from a program.
>
>     and 3 types of destinations: files or remote hosts (TCP/IP socket) and applications (mentioned in section 7).
>
>     So are these all possible types of inputs/outputs that syslog-ng can have? which have to be supported by the command line tool. Or am I missing some points?
>
>     -- 
>     Cheers,
>
>     Asadullah Hussain
>
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>     FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>


-- 
Evan Rempel                                      erempel at uvic.ca
Senior Systems Administrator                        250.721.7691
Data Centre Services, University Systems, University of Victoria

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20150410/06dcd672/attachment.htm