[syslog-ng] Best way to pipe "application" logs to central syslog-ng server.

[Delon Lee Di Lun]

Hi!

@György
My ultimate goal is to replicate the file/directory for the apache logs on
the remote server and create the same structure on my syslog server,
perhaps /var/log/$HOST/..... separated from the "OS" logs.

As i mention, this is the ultimate goal. With the example @Gergely
provided. It seems more possible now.

This is due to there are current mechanism of transferring application logs
over but its the least friendly approach. Hence the spark of this thread to
look into better delivery methods.

Since we are on this topic. I don't see the file module in syslog-ng having
any option that support writing these metadata into the respective fields
in the IETF syslog-ng variables. Means I have to use a rewrite module of
some sort to manipulate the metadata right?

I am getting your point and getting a rough picture of the "pipeline" in my
mind now.

@Gergely
I am unable to change the format of the apache logs.

Yes i understand it does not matter becox in the example. syslog-ng would
be using the classic BSD syslog.

Essentially, the entire csv parsed value would be in the MSG field in the
BSD syslog-ng. So its sort of "cut" in linux to split the filepath n the
log entry.

Let me give this two options a try. Using the IEFT syslog protocol vs
rewritting the MSG field .

Thanks!

Yours Sincerely,
Delon Lee


On Wed, 9 May 2018 at 21:18 Gergely Nagy <algernon at balabit.com> wrote:

> >>>>> "Delon" == Delon Lee Di Lun <lee.delon2005 at gmail.com> writes:
>
>     Delon> However, Would it be performance "greedy"?
>
> It will certainly be slower than if you changed Apache to log to a
> format that's easier to transport and work with on the server side. But
> the CSV parser is quite performant.
>
>     Delon> I read about the new BSD syslog protocol and IETF syslog
> protocol, doing
>     Delon> comparison on the cost-benifit analysis on "upgrading" to using
> the
>     Delon> protocol.
>
> In this case, it doesn't matter, because we don't use the protocol, at
> least, not in my example.
>
>     Delon> I saw that the new IETF syslog protocol cater for a "APP-NAME"
> variable.
>     Delon> Logically speaking, would I able to read in the logs, specify
> the
>     Delon> "APP-NAME", on the server site, filter out this "APP-NAME"?
>
> Yeah, that's a possibility too. But if all you want is store the logs
> as-is on the server side, parsing them fully is much more expensive than
> what I showed.
>
> You can use a rewrite rule to change the app name, and then you can
> filter on that on the server side. But if you use the filtering to route
> messages to files, you can just use a templated filename, which would be
> both faster, and the configuration would be a lot shorter too. The
> downside is that you need to trust the incoming logs to have the correct
> filename.
>
> --
> |8]
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180509/caee197f/attachment.html>