Best way to pipe "application" logs to central syslog-ng server.
Hi, I am working on a mini project that requires the receipt of "application" logs using rsyslog client to pipe it to a syslog-ng central server. An example of the "application" logs im referring to would be for instance Apache HTTPD logs, I want to separate the "application" logs, in this example, the apache logs, and the "OS" logs into different directories. I am thinking of certain crude way, using regex to filter the messages at my server end ,to do it which might not be as clean. Would like to ask the floor if anybody had experience with working in this environment. Would change my rsyslog client and my syslog-ng server to use RFC5424, IETF's compare to the legacy BSD syslog protocol help? Comparing the headers there is a "app-name" variable in the IETF's syslog protocol I can use? Yours Sincerely, Delon Lee
Hi!
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> An example of the "application" logs im referring to would be for instance Delon> Apache HTTPD logs, I want to separate the "application" logs, in this Delon> example, the apache logs, and the "OS" logs into different Delon> directories. There are a few ways I can think of that'd make this work. The simplest one is perhaps using a different port to send HTTPD logs than the one used for OS logs. Then you can set up different log paths, no explicit filtering required. The downside is that you'll have your syslog-ng listen on two ports, may need to adjust firewall rules, and so on. Another option would be to force the PROGRAM field of apache logs to "apache" or "httpd", and filter based on that on the syslog-ng side. No need to use regexps for this, and you don't need to open a separate port, either. The downside is that you need the PROGRAM field to be consistent: all apache logs should have it set to the same value, and no OS logs should reuse that value. A third option would be to add an SDATA field to the apache logs on the rsyslog side, and filter based on that on the syslog-ng side. I am not familiar with rsyslog all that much, and can't offer an example how to do that. But it shouldn't be too hard, I imagine. -- |8]
Hi, On Mon, May 07, 2018 at 03:26:21PM +0200, Gergely Nagy wrote:
A third option would be to add an SDATA field to the apache logs on the rsyslog side, and filter based on that on the syslog-ng side. I am not familiar with rsyslog all that much, and can't offer an example how to do that. But it shouldn't be too hard, I imagine.
One admittedly very hackish way to add SDATA to rsyslog is: $Template t_rfc5424,"<%pri%>1 %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [foo bar=\"baz\"] %msg:R,ERE,1,FIELD:^ (.*)--end%" *.* @remote_syslog:514;t_rfc5424 FWIW ;-)
Hi Gergely & Fabien, Thank you for your quick response. Gladly appreciate. In response to gergely, the 2nd option would require the changes to be made on the "apache side" of things right? If so, its unlikely possible in my use case. What I am looking for is ideally some method of tinkering around with the header portion of syslog protocol. Hopefully to leave the message portion of the logs untouch if possible. Hence the third method is what I am thinking initially but just asking around if anybody has explored a better option. In response to fabien, this was the "crude way" i was talking about. haha. Yours Sincerely, Delon Lee On Mon, 7 May 2018 at 21:35 Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 07, 2018 at 03:26:21PM +0200, Gergely Nagy wrote:
A third option would be to add an SDATA field to the apache logs on the rsyslog side, and filter based on that on the syslog-ng side. I am not familiar with rsyslog all that much, and can't offer an example how to do that. But it shouldn't be too hard, I imagine.
One admittedly very hackish way to add SDATA to rsyslog is:
$Template t_rfc5424,"<%pri%>1 %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% [foo bar=\"baz\"] %msg <https://maps.google.com/?q=%22%5D+%25msg&entry=gmail&source=g>:R,ERE,1,FIELD:^ (.*)--end%"
*.* @remote_syslog:514;t_rfc5424
FWIW ;-)
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> In response to gergely, the 2nd option would require the changes to be made Delon> on the "apache side" of things right? If so, its unlikely possible in my Delon> use case. No, you can do that with rsyslog and syslog-ng too. Both allow you to tinker with the syslog headers. Ideally, changing the Apache-generated log format to conform to a syslog RFC would be ideal, but I understand that's not something most are willing - or able/allowed - to make. So the next best option is to fiddle with the syslog fields on the syslog side of things. -- |8]
We write all of our apache logs to an application ErrorLog "|/path/to/our/script site.fqdn.name error" LogFormat "%h %l %u %t \"%r\" %>s %b" common CustomLog "|/path/to/our/script site.fqdn.name access" common This script just writes the log line to syslog via script specific syslog API with an application name of httpd and a line prefix of access: site.fqdn.name: or error: site.fqdn.name: This allows the receiving end (central syslog server) to strip off the header and recreate files specific to each virtual host access_site.fqdn.name_datestamp error_site.fqdn.name_datestamp And these destination files will have the EXACT content that apache would have logged to disk on the source server. This permits us to feed web analytic tools in real time and provide them the exact source logs that these tools support. Works for us. Evan. On 05/07/2018 08:58 AM, Gergely Nagy wrote:
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes: Delon> In response to gergely, the 2nd option would require the changes to be made Delon> on the "apache side" of things right? If so, its unlikely possible in my Delon> use case.
No, you can do that with rsyslog and syslog-ng too. Both allow you to tinker with the syslog headers.
Ideally, changing the Apache-generated log format to conform to a syslog RFC would be ideal, but I understand that's not something most are willing - or able/allowed - to make. So the next best option is to fiddle with the syslog fields on the syslog side of things.
Hi, Thank you for your suggestion however this is not possible in my setup. Yours Sincerely, Delon Lee On Tue, 8 May 2018 at 03:52 Evan Rempel <erempel@uvic.ca> wrote:
We write all of our apache logs to an application
ErrorLog "|/path/to/our/script site.fqdn.name error"
LogFormat "%h %l %u %t \"%r\" %>s %b" common CustomLog "|/path/to/our/script site.fqdn.name access" common
This script just writes the log line to syslog via script specific syslog API with an application name of httpd and a line prefix of
access: site.fqdn.name: or error: site.fqdn.name:
This allows the receiving end (central syslog server) to strip off the header and recreate files specific to each virtual host
access_site.fqdn.name_datestamp error_site.fqdn.name_datestamp
And these destination files will have the EXACT content that apache would have logged to disk on the source server.
This permits us to feed web analytic tools in real time and provide them the exact source logs that these tools support.
Works for us.
Evan.
On 05/07/2018 08:58 AM, Gergely Nagy wrote:
> "Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes: Delon> In response to gergely, the 2nd option would require the changes to be made Delon> on the "apache side" of things right? If so, its unlikely possible in my Delon> use case.
No, you can do that with rsyslog and syslog-ng too. Both allow you to tinker with the syslog headers.
Ideally, changing the Apache-generated log format to conform to a syslog RFC would be ideal, but I understand that's not something most are willing - or able/allowed - to make. So the next best option is to fiddle with the syslog fields on the syslog side of things.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Possible to wipe up a sample config? Might clear things up? My apache is writing logs into /var/log/apache/www.contoso.com. This is for example one of my apache virtual host. Thanks in advance. Yours Sincerely, Delon Lee On Mon, 7 May 2018 at 23:57 Gergely Nagy <algernon@balabit.com> wrote:
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> In response to gergely, the 2nd option would require the changes to be made Delon> on the "apache side" of things right? If so, its unlikely possible in my Delon> use case.
No, you can do that with rsyslog and syslog-ng too. Both allow you to tinker with the syslog headers.
Ideally, changing the Apache-generated log format to conform to a syslog RFC would be ideal, but I understand that's not something most are willing - or able/allowed - to make. So the next best option is to fiddle with the syslog fields on the syslog side of things.
-- |8]
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> Possible to wipe up a sample config? Delon> Might clear things up? This is a slightly different solution, but should work nevertheless: Client: @version: 3.15 source s_apache_logs { wildcard-file( base-dir("/var/log/apache") filename-pattern("www.*") flags(no-parse) ); }; destination d_central { network("1.2.3.4" template("$(basename ${FILE_NAME}),${MSG}\n")); }; log { source(s_apache_logs); destination(d_central); }; Server: @version: 3.15 source s_net { network("1.2.3.4" flags(no-parse)); }; parser p_apache { csv-parser( columns("apache.FILE_NAME", "apache.MESSAGE"); flags(greedy); ); }; destination d_central_apache { file("/var/log/apache/${apache.FILE_NAME}" template("${apache.MESSAGE}\n")); }; log { source(s_net); parser(p_apache); destination(d_central_apache); }; ------------ * -------------- The idea here is that on the client, we read the apache logs as-is, and forward them with the filename prepended. On the server side, we split the message into filename and message, and use the first part to determine which file to save the message to. Then we write the rest of the line to that file. This way you'll end up with the same contents on both sides, in files that have the same name (but perhaps different path, that part is up to you). Hope this helps. -- |8]
Hi, Thank you spending the time with the sample config. It looks like what I want! However, Would it be performance "greedy"? I read about the new BSD syslog protocol and IETF syslog protocol, doing comparison on the cost-benifit analysis on "upgrading" to using the protocol. I saw that the new IETF syslog protocol cater for a "APP-NAME" variable. Logically speaking, would I able to read in the logs, specify the "APP-NAME", on the server site, filter out this "APP-NAME"? I have not had the time to really sit down and experiment and look though all the modules options and switches. I want to gather more information before i sit down and start the "trial and error" process. Yours Sincerely, Delon Lee On Wed, 9 May 2018 at 18:48 Gergely Nagy <algernon@balabit.com> wrote:
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> Possible to wipe up a sample config? Delon> Might clear things up?
This is a slightly different solution, but should work nevertheless:
Client:
@version: 3.15
source s_apache_logs { wildcard-file( base-dir("/var/log/apache") filename-pattern("www.*") flags(no-parse) ); };
destination d_central { network("1.2.3.4" template("$(basename ${FILE_NAME}),${MSG}\n")); };
log { source(s_apache_logs); destination(d_central); };
Server:
@version: 3.15
source s_net { network("1.2.3.4" flags(no-parse)); };
parser p_apache { csv-parser( columns("apache.FILE_NAME", "apache.MESSAGE"); flags(greedy); ); };
destination d_central_apache { file("/var/log/apache/${apache.FILE_NAME}" template("${apache.MESSAGE}\n")); };
log { source(s_net); parser(p_apache); destination(d_central_apache); };
------------ * --------------
The idea here is that on the client, we read the apache logs as-is, and forward them with the filename prepended. On the server side, we split the message into filename and message, and use the first part to determine which file to save the message to. Then we write the rest of the line to that file.
This way you'll end up with the same contents on both sides, in files that have the same name (but perhaps different path, that part is up to you).
Hope this helps.
-- |8]
Hi, "Delon Lee Di Lun" <lee.delon2005@gmail.com> írta 2018-05-09 12:11-kor:
I read about the new BSD syslog protocol and IETF syslog protocol, doing comparison on the cost-benifit analysis on "upgrading" to using the protocol.
+1 about the ietf protocol! ;-)
I saw that the new IETF syslog protocol cater for a "APP-NAME" variable. Logically speaking, would I able to read in the logs, specify the "APP-NAME", on the server site, filter out this "APP-NAME"?
I don't really see your point. What's your goal? Process in those log files from apache, then drop them on server side, or what? If you want to channel those log events to a different destination using a filter, then the answer is yes. It could work: you can set anything you want as APP-NAME, and you can filter on it on the server side. Btw.: Using the ietf protocoll syslog-ng also gives you a couple of useful metadata about the read logmessage: It will add a field with the filename and file position where the logevent comes from. Cheers, Gyu
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> However, Would it be performance "greedy"? It will certainly be slower than if you changed Apache to log to a format that's easier to transport and work with on the server side. But the CSV parser is quite performant. Delon> I read about the new BSD syslog protocol and IETF syslog protocol, doing Delon> comparison on the cost-benifit analysis on "upgrading" to using the Delon> protocol. In this case, it doesn't matter, because we don't use the protocol, at least, not in my example. Delon> I saw that the new IETF syslog protocol cater for a "APP-NAME" variable. Delon> Logically speaking, would I able to read in the logs, specify the Delon> "APP-NAME", on the server site, filter out this "APP-NAME"? Yeah, that's a possibility too. But if all you want is store the logs as-is on the server side, parsing them fully is much more expensive than what I showed. You can use a rewrite rule to change the app name, and then you can filter on that on the server side. But if you use the filtering to route messages to files, you can just use a templated filename, which would be both faster, and the configuration would be a lot shorter too. The downside is that you need to trust the incoming logs to have the correct filename. -- |8]
Hi! @György My ultimate goal is to replicate the file/directory for the apache logs on the remote server and create the same structure on my syslog server, perhaps /var/log/$HOST/..... separated from the "OS" logs. As i mention, this is the ultimate goal. With the example @Gergely provided. It seems more possible now. This is due to there are current mechanism of transferring application logs over but its the least friendly approach. Hence the spark of this thread to look into better delivery methods. Since we are on this topic. I don't see the file module in syslog-ng having any option that support writing these metadata into the respective fields in the IETF syslog-ng variables. Means I have to use a rewrite module of some sort to manipulate the metadata right? I am getting your point and getting a rough picture of the "pipeline" in my mind now. @Gergely I am unable to change the format of the apache logs. Yes i understand it does not matter becox in the example. syslog-ng would be using the classic BSD syslog. Essentially, the entire csv parsed value would be in the MSG field in the BSD syslog-ng. So its sort of "cut" in linux to split the filepath n the log entry. Let me give this two options a try. Using the IEFT syslog protocol vs rewritting the MSG field . Thanks! Yours Sincerely, Delon Lee On Wed, 9 May 2018 at 21:18 Gergely Nagy <algernon@balabit.com> wrote:
"Delon" == Delon Lee Di Lun <lee.delon2005@gmail.com> writes:
Delon> However, Would it be performance "greedy"?
It will certainly be slower than if you changed Apache to log to a format that's easier to transport and work with on the server side. But the CSV parser is quite performant.
Delon> I read about the new BSD syslog protocol and IETF syslog protocol, doing Delon> comparison on the cost-benifit analysis on "upgrading" to using the Delon> protocol.
In this case, it doesn't matter, because we don't use the protocol, at least, not in my example.
Delon> I saw that the new IETF syslog protocol cater for a "APP-NAME" variable. Delon> Logically speaking, would I able to read in the logs, specify the Delon> "APP-NAME", on the server site, filter out this "APP-NAME"?
Yeah, that's a possibility too. But if all you want is store the logs as-is on the server side, parsing them fully is much more expensive than what I showed.
You can use a rewrite rule to change the app name, and then you can filter on that on the server side. But if you use the filtering to route messages to files, you can just use a templated filename, which would be both faster, and the configuration would be a lot shorter too. The downside is that you need to trust the incoming logs to have the correct filename.
-- |8]
participants (5)
-
Delon Lee Di Lun
-
Evan Rempel
-
Fabien Wernli
-
Gergely Nagy
-
PÁSZTOR György