[syslog-ng] Multiple dates

Fri Jun 9 07:32:40 UTC 2017

Hi, dunno

My client config:

source s_apache_access {
    file("/var/log/apache2/access_testsrv_log" follow-freq(1)
flags("no-parse") host-override("testsrv_access") program-override("apache-
access-log"));
};

destination d_network_apache {.
    syslog("192.168.56.48"
    transport("tcp").
    ip-protocol(4)
    port("16602")
    persist-name("testsrv_apache"));
};

log { source(s_apache_access); destination(d_network_apache); };

And my server config for this:

source s_network_testsrv_apache {
    tcp(ip(192.168.56.48)
    port("16602"));
};

destination d_apache_access_testsrv {
    file("/var/log/syslog-ng/testsrv/apache2/$FULLHOST.access"
    owner("root")
    group("adm")
    perm(0640)
    create_dirs(yes)
    dir_owner("root")
    dir_group("adm")
    template ("${MSG}\n")
    persist-name("testsrv_access"));
....
    file("/var/log/archive/$R_YEAR/testsrv/apache/$R_MONTH/$
FULLHOST.access.$R_DAY"
    owner("root")
    group("adm")
    perm(0640)
    create_dirs(yes)
    dir_owner("root")
    dir_group("adm")
    template ("${MSG}\n")
    persist-name("testsrv_access_archive"));
};

filter testsrv_apache_access {
    match("apache-access-log")
};

log { source(s_network_testsrv_apache); filter(testsrv_apache_access);
destination(d_apache_access_testsrv); };

Thanks, Robert

2017-06-08 17:49 GMT+02:00 Scheidler, Balázs <balazs.scheidler at balabit.com>:

> Hi,
>
>
> On Jun 8, 2017 14:41, "Sandbox" <sandboxheh at gmail.com> wrote:
>
> Hi,
>
> Thanks your answer.
>
>
> On Jun 8, 2017 08:38, "Sandbox" <sandboxheh at gmail.com> wrote:
>
> Hi,
>
> I just started to test, learn etc  syslog-ng, my server configuration is
> really basic:
>
> Q: Can i filter (and mark them on client) the incoming logs, so i dont
> have to open multiple ports for different logs?
>
>
> | Sure, you can open one port and have it filtered using source ip
> (netmask() filter), embedded hostname (host() filter) or even message
> content.
>
> I made some filters, eg:
>
> filter testsrv_apache_access {
>     match("apache-access-log")
> };
>
> Q: i tried to use the "program" filter, but for some reason would't work,
> as you mentioned im using program-override("apache-access-log") on the
> client and sat up the apache-access-log as filter match on the server
> side. With this setting it complains about missing value setting.
>
>
>
> The stored log:
>
> Q: Why does it store the date 3 times in every logs?
>
>
> | You seem to have received an rfc5424 formatted message, but it was not
> parsed, maybe because you were using the wrong source driver (syslog() is
> the one that should handle this format).
> |
> | Since it wasnt parsed, syslog-ng assumed the entire line is a $MSG, and
> prepended its own syslog header. Also, apache itself contains date as well.
> |
> | The solution depends on your exact use case. If you want to transport
> non-syslog data (like apache.log), you'll probably want to dedicate a port
> to it (so it doesnt mix syslog), or you make sure you  | can identify it on
> the server side.
>
> | E.g.
> |
> | source { file("/var/log/apache/access.log" host-override("hostname")
> program-override("apache-access-log") flags(no-parse))); };
> |
> | This would read the log file without parsing it, adds $HOST and $PROGRAM
> fields, which would otherwise be missing.
> |
> | Then:
> |
> | * send it on to the server using whatever means (tcp and syslog both
> works), on the wire, the syslog header will be prepended.
>
> I sat up tcp driver and it stopped to send any log to the server. :)
>
> |
> | * On the server, identify that these are apache logs (based on the
> $PROGRAM value), then write a file using a custom template, where you only
> use $MSG:
> |
> |file("logfile" template("$MSG\n"));
>
> I am still got this: <13>1 2017-06-08T14:53:54+02:00 testsrv_access
> apache-access-log - - - 192.168.56.48 - - [08/Jun/2017:14:53:54 +0200] "GET
> /index.php HTTP/1.1" 304 -
>
>
>
> This means you are still receiving the client messaged using something
> like tcp(flags(no-parse))
>
> You have to match the destination on the client with the source on the
> server. How do those look like?
>
>
>
> | This would remove the syslog header in your output file.
> | Hope this helps
> | Bazsi
>
> Thanks, Robert
>
> 2017-06-08 10:19 GMT+02:00 Scheidler, Balázs <balazs.scheidler at balabit.com
> >:
>
>>
>>
>> On Jun 8, 2017 08:38, "Sandbox" <sandboxheh at gmail.com> wrote:
>>
>> Hi,
>>
>> I just started to test, learn etc  syslog-ng, my server configuration is
>> really basic:
>>
>> Q: Can i filter (and mark them on client) the incoming logs, so i dont
>> have to open multiple ports for different logs?
>>
>>
>> Sure, you can open one port and have it filtered using source ip
>> (netmask() filter), embedded hostname (host() filter) or even message
>> content.
>>
>>
>>
>> The stored log:
>>
>> Q: Why does it store the date 3 times in every logs?
>>
>>
>> You seem to have received an rfc5424 formatted message, but it was not
>> parsed, maybe because you were using the wrong source driver (syslog() is
>> the one that should handle this format).
>>
>> Since it wasnt parsed, syslog-ng assumed the entire line is a $MSG, and
>> prepended its own syslog header. Also, apache itself contains date as well.
>>
>> The solution depends on your exact use case. If you want to transport
>> non-syslog data (like apache.log), you'll probably want to dedicate a port
>> to it (so it doesnt mix syslog), or you make sure you can identify it on
>> the server side.
>>
>> E.g.
>>
>> source { file("/var/log/apache/access.log" host-override("hostname")
>> program-override("apache-access-log") flags(no-parse))); };
>>
>> This would read the log file without parsing it, adds $HOST and $PROGRAM
>> fields, which would otherwise be missing.
>>
>> Then:
>>
>> * send it on to the server using whatever means (tcp and syslog both
>> works), on the wire, the syslog header will be prepended.
>>
>> * On the server, identify that these are apache logs (based on the
>> $PROGRAM value), then write a file using a custom template, where you only
>> use $MSG:
>>
>> file("logfile" template("$MSG\n"));
>>
>> This would remove the syslog header in your output file.
>> Hope this helps
>> Bazsi
>>
>>
>>
>>
>>
>> Jun  8 08:20:11 192.168.7.30 133 <13>1 2017-06-08T08:20:11+02:00
>> testweb01 - - - [meta sequenceId="24"] :1 - - [08/Jun/2017:08:20:10 +0200]
>> "GET / HTTP/1.1" 200 3004
>>
>>
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170609/a6e9bc2f/attachment-0001.html>