[syslog-ng] Syslog-ng and OSQUERY
Czanik, Péter
peter.czanik at balabit.com
Fri Apr 28 11:37:28 UTC 2017
Hi,
You can read my blog about syslog-ng and osquery at
https://www.balabit.com/blog/endpoint-visibility-and-monitoring-using-osquery-and-syslog-ng/
Bye,
Peter Czanik (CzP) <peter.czanik at balabit.com>
Balabit / syslog-ng upstream
https://www.balabit.com/blog/author/peterczanik/
https://twitter.com/PCzanik
On Thu, Apr 20, 2017 at 3:58 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
> Hi Evan
> Does not matter since your previous suggestion worked in my
> case ( many interpretation ). Now i can test with your new rewrite rules.
>
> Regards
>
>
> On Thu, Apr 20, 2017 at 6:56 PM, Evan Rempel <erempel at uvic.ca> wrote:
>
>> I should have read the RFC on CSV prior to drafting the format.
>>
>> https://tools.ietf.org/html/rfc4180
>>
>> Turns out Peter and I both got it wrong. The rewrite should be
>>
>> rewrite r_csv_message {
>> set("$MESSAGE", value("CSVMESSAGE") );
>> subst("\"","\"\"", value("CSVMESSAGE"), flags(global) );
>> };
>>
>> Of course, there are many "interpretations" of CSV so your consumer may
>> not conform to the official standard.
>>
>>
>> On 04/20/2017 03:36 AM, Czanik, Péter wrote:
>>
>> Hi,
>>
>> Just a heads up: I also got it working after a bit of debugging. The
>> problem was, that in the above configuration sample there are spaces in the
>> template. After removing those, it worked. Here is my config:
>>
>> [root at localhost conf.d]# cat oq.conf
>> rewrite r_csv_message {
>> set("$MESSAGE", value("CSVMESSAGE") );
>> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
>> };
>>
>> template t_csv {
>> template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${FA
>> CILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n");
>> template_escape(no);
>> };
>>
>> destination d_osquery_copy {
>> file("/var/log/csv_osquery" template(t_csv));
>> };
>>
>> destination d_osquery {
>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>> };
>>
>> log {
>> source(s_sys);
>> rewrite(r_csv_message);
>> destination(d_osquery);
>> destination(d_osquery_copy);
>> };
>> [root at localhost conf.d]#
>>
>> I figured it out by installing rsyslog and looking at the differences in
>> the output.
>>
>> I plan to summarize my experiences in a blog in a week or two.
>>
>> Bye,
>>
>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>> Balabit / syslog-ng upstream
>> https://www.balabit.com/blog/author/peterczanik/
>> https://twitter.com/PCzanik
>>
>> On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
>>
>>> Hi
>>> Evan
>>> Your suggestion works flawlessly. The syslog table in
>>> OSQUERY gets filled up with logs. The missing part is the rewrite rule
>>> r_csv_message. Many many thanks to you.
>>>
>>> Regards
>>> Dwijadas Dey
>>>
>>>
>>>
>>> On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel < <erempel at uvic.ca>
>>>> erempel at uvic.ca> wrote:
>>>>
>>>>> The fact that your error "Received more fields than expected" went
>>>>> away implies that the number of fields is correct.
>>>>> Without any errors or any data in the table your trouble shooting
>>>>> options are limited.
>>>>>
>>>>> I would make another file based destination for syslog-ng
>>>>>
>>>>> destination d_osquery_copy {
>>>>> file("/var/osquery/syslog" template(t_csv));
>>>>> };
>>>>>
>>>>> And add this destination to your log statement.
>>>>>
>>>>> log {
>>>>> source(s_osquery);
>>>>> destination(d_osquery);
>>>>> destination(d_osquery_copy);
>>>>> };
>>>>>
>>>>>
>>>>> Then you will have a copy of the data that is being sent to osquery
>>>>> and you should be able to get help from the osquery community.
>>>>>
>>>>>
>>>>> One other thing to note is that I did not provide you with the correct
>>>>> CSV of the MESSAGE portion. If the $MESSAGE contains double quotes
>>>>> then this will not be a correctly formatted CSV field.
>>>>>
>>>>> you can make a rewrite rule for the message
>>>>>
>>>>> rewrite r_csv_message {
>>>>> set("$MESSAGE", value("CSVMESSAGE") );
>>>>> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
>>>>> };
>>>>>
>>>>> then you need to invoke this rewrite rule in your log statement.
>>>>>
>>>>> log {
>>>>> source(s_osquery);
>>>>> rewrite(r_csv_message);
>>>>> destination(d_osquery);
>>>>> destination(d_osquery_copy);
>>>>> };
>>>>>
>>>>> And finally your template needs to use the CSVMESSAGE rather than the
>>>>> MESSAGE
>>>>>
>>>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n");
>>>>> template_escape(no); };
>>>>>
>>>>>
>>>>> I hope that helps too.
>>>>>
>>>>> Evan.
>>>>>
>>>>>
>>>>> On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
>>>>>
>>>>> Hi
>>>>> Evan
>>>>> Thanks you for a quick reply. After changing the template
>>>>> as suggested by you, the error goes away but the syslog table in OSQUERY
>>>>> does not get filled up. May be the OSQUERY expects 7 entry for the syslog
>>>>> table while the template has six fields.
>>>>>
>>>>> > osquery> .schema syslog
>>>>> > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>>>>> > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>>>
>>>>> No verbose error as well.
>>>>>
>>>>> Regards
>>>>>
>>>>> On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel < <erempel at uvic.ca>
>>>>> erempel at uvic.ca> wrote:
>>>>>
>>>>>> The documentation from OSQuery is for rsyslog and shows that a csv
>>>>>> set of values is needed.
>>>>>>
>>>>>> string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys
>>>>>> logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::
>>>>>> csv%,%msg:::csv%\n"
>>>>>>
>>>>>> In syslog-ng this format becomes
>>>>>>
>>>>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>>>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n");
>>>>>> template_escape(no); };
>>>>>>
>>>>>> Give that a try and see how things go.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
>>>>>>
>>>>>> Hi
>>>>>> Peter
>>>>>> I am trying to send syslogs to a named pipe and on the
>>>>>> other end OSQUERY will consume the syslogs from the named pipe. Once
>>>>>> OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes
>>>>>> along with OSQUERY. I have been able to send the syslogs to named pipe (
>>>>>> verified with cat command ) but on the other hand OSQUERY did consume the
>>>>>> logs but could not send these logs to the table due to format error.
>>>>>>
>>>>>> The schema of syslog table in OSQUERY
>>>>>> ------------------------------------------------------------
>>>>>> osquery> .schema syslog
>>>>>> CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host`
>>>>>> TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>>>>
>>>>>> Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)
>>>>>> ------------------------------------------------------------
>>>>>> ----------------------
>>>>>> source s_osquery {
>>>>>> system();
>>>>>> };
>>>>>>
>>>>>> template t_csv {
>>>>>> template("'${HOUR}${MIN}${SEC}
>>>>>> ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA
>>>>>> CILITY}',\t'${MSG}'\n");
>>>>>> # template("$timestamp\t${ISODAT
>>>>>> E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
>>>>>> template_escape(no);
>>>>>> };
>>>>>>
>>>>>> destination d_osquery {
>>>>>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>>>>>> };
>>>>>>
>>>>>> log {
>>>>>> source(s_osquery);
>>>>>> destination(d_osquery);
>>>>>> };
>>>>>>
>>>>>> I am trying to match the above template to rsyslog format for OSQUERY
>>>>>>
>>>>>>
>>>>>> <https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1>
>>>>>> https://osquery.readthedocs.io/en/stable/deployment/syslog/#
>>>>>> rsyslog-versions-7_1
>>>>>>
>>>>>> If i cat the pipe, i can see the syslogs.
>>>>>>
>>>>>> # cat /var/osquery/syslog_pipe
>>>>>>
>>>>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu',
>>>>>> '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876
>>>>>> [preauth]'
>>>>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu',
>>>>>> '55', 'notice', 'authpriv', 'PAM 2 more authentication
>>>>>> failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51
>>>>>> user=root'
>>>>>>
>>>>>>
>>>>>> The above logs contains exactly 7 fields as required by OSQUERY
>>>>>> syslog table as described above.
>>>>>>
>>>>>>
>>>>>> The error that i am getting at the moment -
>>>>>> ------------------------------------------------------------
>>>>>> E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than
>>>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>>> message=Added file event listener to: /root/.ssh/**
>>>>>> E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than
>>>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>>> message=Added file event listener to: /home/*/.ssh/**
>>>>>> E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than
>>>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>>> message=Added file event listener to: /tmp/**
>>>>>> I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog
>>>>>> run loop terminated for reason: Too many errors in syslog parsing.
>>>>>>
>>>>>> I think the issue is with the template definition which needs to
>>>>>> match with the template with rsyslog as described in the above link.
>>>>>>
>>>>>> I will appreciate if someone can point out the issues in template and
>>>>>> how it should be in syslog-ng.
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <
>>>>>> <peter.czanik at balabit.com>peter.czanik at balabit.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> What do you try to achieve? Sending syslog messages to OSquery or
>>>>>>> collecting OSquery logs by syslog-ng?
>>>>>>>
>>>>>>> /me now has a test environment installed
>>>>>>>
>>>>>>> Bye,
>>>>>>>
>>>>>>> Peter Czanik (CzP) < <peter.czanik at balabit.com>
>>>>>>> peter.czanik at balabit.com>
>>>>>>> Balabit / syslog-ng upstream
>>>>>>> <https://www.balabit.com/blog/author/peterczanik/>
>>>>>>> https://www.balabit.com/blog/author/peterczanik/
>>>>>>> <https://twitter.com/PCzanik>https://twitter.com/PCzanik
>>>>>>>
>>>>>>> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey < <dwijad at gmail.com>
>>>>>>> dwijad at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>> Robert
>>>>>>>> You are right, i am trying the same with a named pipe
>>>>>>>> so that OSQUERY consume syslogs as pointed by Evan. There are plenty of
>>>>>>>> documents showing the same with rsyslog but not with syslog-ng.
>>>>>>>>
>>>>>>>> This is what my syslog configuration for osquery:-
>>>>>>>>
>>>>>>>> /etc/syslog-ng/conf.d/osquery.conf
>>>>>>>>
>>>>>>>> source s_osquery {
>>>>>>>> # system();
>>>>>>>> pipe("/var/osquery/syslog_pipe");
>>>>>>>> # unix-stream("/dev/log");
>>>>>>>> };
>>>>>>>> #filter osqueryd {
>>>>>>>> # program("^osqueryd.*");
>>>>>>>> #};
>>>>>>>> destination d_osquery {
>>>>>>>> file("/var/log/osquery/osqueryd.results.log"
>>>>>>>> template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
>>>>>>>> };
>>>>>>>> log {
>>>>>>>> source(s_osquery);
>>>>>>>> # filter(osqueryd);
>>>>>>>> destination(d_osquery);
>>>>>>>> };
>>>>>>>>
>>>>>>>> But this does not produce any logs for OSQUERY. I have checked ,
>>>>>>>> the name piped has been created.
>>>>>>>>
>>>>>>>> # ls -l /var/osquery/syslog_pipe
>>>>>>>> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
>>>>>>>>
>>>>>>>> But when i try to check what logs are passing through the pipe
>>>>>>>> using following command, no message shows up.
>>>>>>>> # cat /var/osquery/syslog_pipe
>>>>>>>>
>>>>>>>> I have correct options set in OSQUERY configuration file in
>>>>>>>> /etc/osquery/osquery.conf.
>>>>>>>>
>>>>>>>> ..................
>>>>>>>> ..................
>>>>>>>> "logger_plugin": "syslog",
>>>>>>>> "enable_syslog": "true",
>>>>>>>> "syslog_pipe_path": "/var/osquery/syslog_pipe",
>>>>>>>> ..................
>>>>>>>> ..................
>>>>>>>> I think Evan can point me the right configuration for syslog-ng (
>>>>>>>> version 3.5.6 in ubuntu 16 )
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <
>>>>>>>> <robert.fekete at balabit.com>robert.fekete at balabit.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> It seems that by default, osquery logs JSON messages into a file.
>>>>>>>>> ( <https://osquery.readthedocs.io/en/latest/deployment/logging/>
>>>>>>>>> https://osquery.readthedocs.io/en/latest/deployment/logging/ )
>>>>>>>>> You can use this file in a syslog-ng source, and parse the JSON
>>>>>>>>> messages with the json parser (note that you need a recent syslog-ng OSE
>>>>>>>>> for this), see
>>>>>>>>> <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html>
>>>>>>>>> https://www.balabit.com/documents/syslog-ng-ose-latest-guide
>>>>>>>>> s/en/syslog-ng-ose-guide-admin/html/json-parser.html .
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The above Osquery page mentions that it can send log messages
>>>>>>>>> directly to syslog (instead of a file), but I haven't found how you can
>>>>>>>>> actually configure it.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Robert
>>>>>>>>>
>>>>>>>>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey < <dwijad at gmail.com>
>>>>>>>>> dwijad at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>> List users
>>>>>>>>>> Is it possible to send OSQUERY logs to
>>>>>>>>>> syslog-ng 3.5 In the OSQUERY docs
>>>>>>>>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/>
>>>>>>>>>> rsyslog is configured to write logs to syslog. Does the same method applies
>>>>>>>>>> to syslog-ng 3.5 ?
>>>>>>>>>>
>>>>>>>>>> Thanks and regards
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170428/cbe7b5e2/attachment-0001.html>
More information about the syslog-ng
mailing list