[syslog-ng] Syslog-ng and OSQUERY
Czanik, Péter
peter.czanik at balabit.com
Thu Apr 20 10:36:33 UTC 2017
Hi,
Just a heads up: I also got it working after a bit of debugging. The
problem was, that in the above configuration sample there are spaces in the
template. After removing those, it worked. Here is my config:
[root at localhost conf.d]# cat oq.conf
rewrite r_csv_message {
set("$MESSAGE", value("CSVMESSAGE") );
subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
};
template t_csv {
template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${FACILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n");
template_escape(no);
};
destination d_osquery_copy {
file("/var/log/csv_osquery" template(t_csv));
};
destination d_osquery {
pipe("/var/osquery/syslog_pipe" template(t_csv));
};
log {
source(s_sys);
rewrite(r_csv_message);
destination(d_osquery);
destination(d_osquery_copy);
};
[root at localhost conf.d]#
I figured it out by installing rsyslog and looking at the differences in
the output.
I plan to summarize my experiences in a blog in a week or two.
Bye,
Peter Czanik (CzP) <peter.czanik at balabit.com>
Balabit / syslog-ng upstream
https://www.balabit.com/blog/author/peterczanik/
https://twitter.com/PCzanik
On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
> Hi
> Evan
> Your suggestion works flawlessly. The syslog table in OSQUERY
> gets filled up with logs. The missing part is the rewrite rule
> r_csv_message. Many many thanks to you.
>
> Regards
> Dwijadas Dey
>
>
>
> On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel at uvic.ca> wrote:
>>
>>> The fact that your error "Received more fields than expected" went away
>>> implies that the number of fields is correct.
>>> Without any errors or any data in the table your trouble shooting
>>> options are limited.
>>>
>>> I would make another file based destination for syslog-ng
>>>
>>> destination d_osquery_copy {
>>> file("/var/osquery/syslog" template(t_csv));
>>> };
>>>
>>> And add this destination to your log statement.
>>>
>>> log {
>>> source(s_osquery);
>>> destination(d_osquery);
>>> destination(d_osquery_copy);
>>> };
>>>
>>>
>>> Then you will have a copy of the data that is being sent to osquery and
>>> you should be able to get help from the osquery community.
>>>
>>>
>>> One other thing to note is that I did not provide you with the correct
>>> CSV of the MESSAGE portion. If the $MESSAGE contains double quotes
>>> then this will not be a correctly formatted CSV field.
>>>
>>> you can make a rewrite rule for the message
>>>
>>> rewrite r_csv_message {
>>> set("$MESSAGE", value("CSVMESSAGE") );
>>> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
>>> };
>>>
>>> then you need to invoke this rewrite rule in your log statement.
>>>
>>> log {
>>> source(s_osquery);
>>> rewrite(r_csv_message);
>>> destination(d_osquery);
>>> destination(d_osquery_copy);
>>> };
>>>
>>> And finally your template needs to use the CSVMESSAGE rather than the
>>> MESSAGE
>>>
>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n");
>>> template_escape(no); };
>>>
>>>
>>> I hope that helps too.
>>>
>>> Evan.
>>>
>>>
>>> On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
>>>
>>> Hi
>>> Evan
>>> Thanks you for a quick reply. After changing the template as
>>> suggested by you, the error goes away but the syslog table in OSQUERY does
>>> not get filled up. May be the OSQUERY expects 7 entry for the syslog table
>>> while the template has six fields.
>>>
>>> > osquery> .schema syslog
>>> > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>>> > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>
>>> No verbose error as well.
>>>
>>> Regards
>>>
>>> On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>>
>>>> The documentation from OSQuery is for rsyslog and shows that a csv set
>>>> of values is needed.
>>>>
>>>> string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys
>>>> logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::
>>>> csv%,%msg:::csv%\n"
>>>>
>>>> In syslog-ng this format becomes
>>>>
>>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n");
>>>> template_escape(no); };
>>>>
>>>> Give that a try and see how things go.
>>>>
>>>>
>>>>
>>>> On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
>>>>
>>>> Hi
>>>> Peter
>>>> I am trying to send syslogs to a named pipe and on the
>>>> other end OSQUERY will consume the syslogs from the named pipe. Once
>>>> OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes
>>>> along with OSQUERY. I have been able to send the syslogs to named pipe (
>>>> verified with cat command ) but on the other hand OSQUERY did consume the
>>>> logs but could not send these logs to the table due to format error.
>>>>
>>>> The schema of syslog table in OSQUERY
>>>> ------------------------------------------------------------
>>>> osquery> .schema syslog
>>>> CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>>>> `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>>
>>>> Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)
>>>> ------------------------------------------------------------
>>>> ----------------------
>>>> source s_osquery {
>>>> system();
>>>> };
>>>>
>>>> template t_csv {
>>>> template("'${HOUR}${MIN}${SEC}
>>>> ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA
>>>> CILITY}',\t'${MSG}'\n");
>>>> # template("$timestamp\t${ISODAT
>>>> E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
>>>> template_escape(no);
>>>> };
>>>>
>>>> destination d_osquery {
>>>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>>>> };
>>>>
>>>> log {
>>>> source(s_osquery);
>>>> destination(d_osquery);
>>>> };
>>>>
>>>> I am trying to match the above template to rsyslog format for OSQUERY
>>>>
>>>> https://osquery.readthedocs.io/en/stable/deployment/syslog/#
>>>> rsyslog-versions-7_1
>>>>
>>>> If i cat the pipe, i can see the syslogs.
>>>>
>>>> # cat /var/osquery/syslog_pipe
>>>>
>>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu',
>>>> '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876
>>>> [preauth]'
>>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu',
>>>> '55', 'notice', 'authpriv', 'PAM 2 more authentication
>>>> failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51
>>>> user=root'
>>>>
>>>>
>>>> The above logs contains exactly 7 fields as required by OSQUERY syslog
>>>> table as described above.
>>>>
>>>>
>>>> The error that i am getting at the moment -
>>>> ------------------------------------------------------------
>>>> E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than
>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>> message=Added file event listener to: /root/.ssh/**
>>>> E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than
>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>> message=Added file event listener to: /home/*/.ssh/**
>>>> E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than
>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>> message=Added file event listener to: /tmp/**
>>>> I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run
>>>> loop terminated for reason: Too many errors in syslog parsing.
>>>>
>>>> I think the issue is with the template definition which needs to match
>>>> with the template with rsyslog as described in the above link.
>>>>
>>>> I will appreciate if someone can point out the issues in template and
>>>> how it should be in syslog-ng.
>>>>
>>>>
>>>> Regards
>>>>
>>>>
>>>>
>>>> On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <
>>>> peter.czanik at balabit.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> What do you try to achieve? Sending syslog messages to OSquery or
>>>>> collecting OSquery logs by syslog-ng?
>>>>>
>>>>> /me now has a test environment installed
>>>>>
>>>>> Bye,
>>>>>
>>>>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>>>>> Balabit / syslog-ng upstream
>>>>> https://www.balabit.com/blog/author/peterczanik/
>>>>> https://twitter.com/PCzanik
>>>>>
>>>>> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>> Robert
>>>>>> You are right, i am trying the same with a named pipe
>>>>>> so that OSQUERY consume syslogs as pointed by Evan. There are plenty of
>>>>>> documents showing the same with rsyslog but not with syslog-ng.
>>>>>>
>>>>>> This is what my syslog configuration for osquery:-
>>>>>>
>>>>>> /etc/syslog-ng/conf.d/osquery.conf
>>>>>>
>>>>>> source s_osquery {
>>>>>> # system();
>>>>>> pipe("/var/osquery/syslog_pipe");
>>>>>> # unix-stream("/dev/log");
>>>>>> };
>>>>>> #filter osqueryd {
>>>>>> # program("^osqueryd.*");
>>>>>> #};
>>>>>> destination d_osquery {
>>>>>> file("/var/log/osquery/osqueryd.results.log"
>>>>>> template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
>>>>>> };
>>>>>> log {
>>>>>> source(s_osquery);
>>>>>> # filter(osqueryd);
>>>>>> destination(d_osquery);
>>>>>> };
>>>>>>
>>>>>> But this does not produce any logs for OSQUERY. I have checked , the
>>>>>> name piped has been created.
>>>>>>
>>>>>> # ls -l /var/osquery/syslog_pipe
>>>>>> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
>>>>>>
>>>>>> But when i try to check what logs are passing through the pipe using
>>>>>> following command, no message shows up.
>>>>>> # cat /var/osquery/syslog_pipe
>>>>>>
>>>>>> I have correct options set in OSQUERY configuration file in
>>>>>> /etc/osquery/osquery.conf.
>>>>>>
>>>>>> ..................
>>>>>> ..................
>>>>>> "logger_plugin": "syslog",
>>>>>> "enable_syslog": "true",
>>>>>> "syslog_pipe_path": "/var/osquery/syslog_pipe",
>>>>>> ..................
>>>>>> ..................
>>>>>> I think Evan can point me the right configuration for syslog-ng (
>>>>>> version 3.5.6 in ubuntu 16 )
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <
>>>>>> robert.fekete at balabit.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It seems that by default, osquery logs JSON messages into a file. (
>>>>>>> https://osquery.readthedocs.io/en/latest/deployment/logging/ )
>>>>>>> You can use this file in a syslog-ng source, and parse the JSON
>>>>>>> messages with the json parser (note that you need a recent syslog-ng OSE
>>>>>>> for this), see https://www.balabit.com/docume
>>>>>>> nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin
>>>>>>> /html/json-parser.html .
>>>>>>>
>>>>>>>
>>>>>>> The above Osquery page mentions that it can send log messages
>>>>>>> directly to syslog (instead of a file), but I haven't found how you can
>>>>>>> actually configure it.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Robert
>>>>>>>
>>>>>>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>> List users
>>>>>>>> Is it possible to send OSQUERY logs to syslog-ng
>>>>>>>> 3.5 In the OSQUERY docs
>>>>>>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/>
>>>>>>>> rsyslog is configured to write logs to syslog. Does the same method applies
>>>>>>>> to syslog-ng 3.5 ?
>>>>>>>>
>>>>>>>> Thanks and regards
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>> --
>>> Evan Rempel erempel at uvic.ca
>>> Senior Systems Administrator 250.721.7691 <(250)%20721-7691>
>>> Data Centre Services, University Systems, University of Victoria
>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170420/9d2ba440/attachment-0001.html>
More information about the syslog-ng
mailing list