Hi, 

Thanks your answer.

On Jun 8, 2017 08:38, "Sandbox" <sandboxheh@gmail.com> wrote:

Hi,

I just started to test, learn etc  syslog-ng, my server configuration is really basic:

Q: Can i filter (and mark them on client) the incoming logs, so i dont have to open multiple ports for different logs?

| Sure, you can open one port and have it filtered using source ip (netmask() filter), embedded hostname (host() filter) or even message content.

I made some filters, eg:

filter testsrv_apache_access {

    match("apache-access-log")

};

Q: i tried to use the "program" filter, but for some reason would't work, as you mentioned im using program-override("apache-access-log") on the client and sat up the apache-access-log as filter match on the server side. With this setting it complains about missing value setting.

The stored log:

Q: Why does it store the date 3 times in every logs?

| You seem to have received an rfc5424 formatted message, but it was not parsed, maybe because you were using the wrong source driver (syslog() is the one that should handle this format).

|

| Since it wasnt parsed, syslog-ng assumed the entire line is a $MSG, and prepended its own syslog header. Also, apache itself contains date as well.

|

| The solution depends on your exact use case. If you want to transport non-syslog data (like apache.log), you'll probably want to dedicate a port to it (so it doesnt mix syslog), or you make sure you  | can identify it on the server side.

| E.g.

|

| source { file("/var/log/apache/access.log" host-override("hostname") program-override("apache-access-log") flags(no-parse))); };

|

| This would read the log file without parsing it, adds $HOST and $PROGRAM fields, which would otherwise be missing.

|

| Then:

|

| * send it on to the server using whatever means (tcp and syslog both works), on the wire, the syslog header will be prepended.

I sat up tcp driver and it stopped to send any log to the server. :)

|

| * On the server, identify that these are apache logs (based on the $PROGRAM value), then write a file using a custom template, where you only use $MSG:

|

|file("logfile" template("$MSG\n"));

I am still got this: <13>1 2017-06-08T14:53:54+02:00 testsrv_access apache-access-log - - - 192.168.56.48 - - [08/Jun/2017:14:53:54 +0200] "GET /index.php HTTP/1.1" 304 -

| This would remove the syslog header in your output file.

| Hope this helps

| Bazsi

Thanks, Robert

2017-06-08 10:19 GMT+02:00 Scheidler, Balázs <balazs.scheidler@balabit.com>:

On Jun 8, 2017 08:38, "Sandbox" <sandboxheh@gmail.com> wrote:

Hi,

I just started to test, learn etc  syslog-ng, my server configuration is really basic:

Q: Can i filter (and mark them on client) the incoming logs, so i dont have to open multiple ports for different logs?

Sure, you can open one port and have it filtered using source ip (netmask() filter), embedded hostname (host() filter) or even message content.

The stored log:

Q: Why does it store the date 3 times in every logs?

You seem to have received an rfc5424 formatted message, but it was not parsed, maybe because you were using the wrong source driver (syslog() is the one that should handle this format).

Since it wasnt parsed, syslog-ng assumed the entire line is a $MSG, and prepended its own syslog header. Also, apache itself contains date as well.

The solution depends on your exact use case. If you want to transport non-syslog data (like apache.log), you'll probably want to dedicate a port to it (so it doesnt mix syslog), or you make sure you can identify it on the server side.

E.g.

source { file("/var/log/apache/access.log" host-override("hostname") program-override("apache-access-log") flags(no-parse))); };

This would read the log file without parsing it, adds $HOST and $PROGRAM fields, which would otherwise be missing.

Then:

* send it on to the server using whatever means (tcp and syslog both works), on the wire, the syslog header will be prepended.

* On the server, identify that these are apache logs (based on the $PROGRAM value), then write a file using a custom template, where you only use $MSG:

file("logfile" template("$MSG\n"));

This would remove the syslog header in your output file.

Hope this helps

Bazsi

Jun  8 08:20:11 192.168.7.30 133 <13>1 2017-06-08T08:20:11+02:00 testweb01 - - - [meta sequenceId="24"] :1 - - [08/Jun/2017:08:20:10 +0200] "GET / HTTP/1.1" 200 3004

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq