[syslog-ng] Multiple dates

Thu Jun 8 15:49:06 UTC 2017

Hi,

On Jun 8, 2017 14:41, "Sandbox" <sandboxheh at gmail.com> wrote:

Hi,

Thanks your answer.

On Jun 8, 2017 08:38, "Sandbox" <sandboxheh at gmail.com> wrote:

Hi,

I just started to test, learn etc  syslog-ng, my server configuration is
really basic:

Q: Can i filter (and mark them on client) the incoming logs, so i dont have
to open multiple ports for different logs?

| Sure, you can open one port and have it filtered using source ip
(netmask() filter), embedded hostname (host() filter) or even message
content.

I made some filters, eg:

filter testsrv_apache_access {
    match("apache-access-log")
};

Q: i tried to use the "program" filter, but for some reason would't work,
as you mentioned im using program-override("apache-access-log") on the
client and sat up the apache-access-log as filter match on the server side.
With this setting it complains about missing value setting.

The stored log:

Q: Why does it store the date 3 times in every logs?

| You seem to have received an rfc5424 formatted message, but it was not
parsed, maybe because you were using the wrong source driver (syslog() is
the one that should handle this format).
|
| Since it wasnt parsed, syslog-ng assumed the entire line is a $MSG, and
prepended its own syslog header. Also, apache itself contains date as well.
|
| The solution depends on your exact use case. If you want to transport
non-syslog data (like apache.log), you'll probably want to dedicate a port
to it (so it doesnt mix syslog), or you make sure you  | can identify it on
the server side.

| E.g.
|
| source { file("/var/log/apache/access.log" host-override("hostname")
program-override("apache-access-log") flags(no-parse))); };
|
| This would read the log file without parsing it, adds $HOST and $PROGRAM
fields, which would otherwise be missing.
|
| Then:
|
| * send it on to the server using whatever means (tcp and syslog both
works), on the wire, the syslog header will be prepended.

I sat up tcp driver and it stopped to send any log to the server. :)

|
| * On the server, identify that these are apache logs (based on the
$PROGRAM value), then write a file using a custom template, where you only
use $MSG:
|
|file("logfile" template("$MSG\n"));

I am still got this: <13>1 2017-06-08T14:53:54+02:00 testsrv_access
apache-access-log - - - 192.168.56.48 - - [08/Jun/2017:14:53:54 +0200] "GET
/index.php HTTP/1.1" 304 -

This means you are still receiving the client messaged using something like
tcp(flags(no-parse))

You have to match the destination on the client with the source on the
server. How do those look like?

| This would remove the syslog header in your output file.
| Hope this helps
| Bazsi

Thanks, Robert

2017-06-08 10:19 GMT+02:00 Scheidler, Balázs <balazs.scheidler at balabit.com>:

>
>
> On Jun 8, 2017 08:38, "Sandbox" <sandboxheh at gmail.com> wrote:
>
> Hi,
>
> I just started to test, learn etc  syslog-ng, my server configuration is
> really basic:
>
> Q: Can i filter (and mark them on client) the incoming logs, so i dont
> have to open multiple ports for different logs?
>
>
> Sure, you can open one port and have it filtered using source ip
> (netmask() filter), embedded hostname (host() filter) or even message
> content.
>
>
>
> The stored log:
>
> Q: Why does it store the date 3 times in every logs?
>
>
> You seem to have received an rfc5424 formatted message, but it was not
> parsed, maybe because you were using the wrong source driver (syslog() is
> the one that should handle this format).
>
> Since it wasnt parsed, syslog-ng assumed the entire line is a $MSG, and
> prepended its own syslog header. Also, apache itself contains date as well.
>
> The solution depends on your exact use case. If you want to transport
> non-syslog data (like apache.log), you'll probably want to dedicate a port
> to it (so it doesnt mix syslog), or you make sure you can identify it on
> the server side.
>
> E.g.
>
> source { file("/var/log/apache/access.log" host-override("hostname")
> program-override("apache-access-log") flags(no-parse))); };
>
> This would read the log file without parsing it, adds $HOST and $PROGRAM
> fields, which would otherwise be missing.
>
> Then:
>
> * send it on to the server using whatever means (tcp and syslog both
> works), on the wire, the syslog header will be prepended.
>
> * On the server, identify that these are apache logs (based on the
> $PROGRAM value), then write a file using a custom template, where you only
> use $MSG:
>
> file("logfile" template("$MSG\n"));
>
> This would remove the syslog header in your output file.
> Hope this helps
> Bazsi
>
>
>
>
>
> Jun  8 08:20:11 192.168.7.30 133 <13>1 2017-06-08T08:20:11+02:00 testweb01
> - - - [meta sequenceId="24"] :1 - - [08/Jun/2017:08:20:10 +0200] "GET /
> HTTP/1.1" 200 3004
>
>
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=
> syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170608/fbcc25ad/attachment-0001.html>