[syslog-ng] Syslog-ng and OSQUERY
Dwijadas Dey
dwijad at gmail.com
Fri Apr 28 16:22:28 UTC 2017
Hi
Peter
Awesome blog post on syslog-ng and osquery....as always
Regards
On Fri, Apr 28, 2017 at 5:07 PM, Czanik, Péter <peter.czanik at balabit.com>
wrote:
> Hi,
> You can read my blog about syslog-ng and osquery at
> https://www.balabit.com/blog/endpoint-visibility-and-
> monitoring-using-osquery-and-syslog-ng/
> Bye,
>
> Peter Czanik (CzP) <peter.czanik at balabit.com>
> Balabit / syslog-ng upstream
> https://www.balabit.com/blog/author/peterczanik/
> https://twitter.com/PCzanik
>
> On Thu, Apr 20, 2017 at 3:58 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
>
>> Hi Evan
>> Does not matter since your previous suggestion worked in my
>> case ( many interpretation ). Now i can test with your new rewrite rules.
>>
>> Regards
>>
>>
>> On Thu, Apr 20, 2017 at 6:56 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>
>>> I should have read the RFC on CSV prior to drafting the format.
>>>
>>> https://tools.ietf.org/html/rfc4180
>>>
>>> Turns out Peter and I both got it wrong. The rewrite should be
>>>
>>> rewrite r_csv_message {
>>> set("$MESSAGE", value("CSVMESSAGE") );
>>> subst("\"","\"\"", value("CSVMESSAGE"), flags(global) );
>>> };
>>>
>>> Of course, there are many "interpretations" of CSV so your consumer may
>>> not conform to the official standard.
>>>
>>>
>>> On 04/20/2017 03:36 AM, Czanik, Péter wrote:
>>>
>>> Hi,
>>>
>>> Just a heads up: I also got it working after a bit of debugging. The
>>> problem was, that in the above configuration sample there are spaces in the
>>> template. After removing those, it worked. Here is my config:
>>>
>>> [root at localhost conf.d]# cat oq.conf
>>> rewrite r_csv_message {
>>> set("$MESSAGE", value("CSVMESSAGE") );
>>> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
>>> };
>>>
>>> template t_csv {
>>> template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${FA
>>> CILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n");
>>> template_escape(no);
>>> };
>>>
>>> destination d_osquery_copy {
>>> file("/var/log/csv_osquery" template(t_csv));
>>> };
>>>
>>> destination d_osquery {
>>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>>> };
>>>
>>> log {
>>> source(s_sys);
>>> rewrite(r_csv_message);
>>> destination(d_osquery);
>>> destination(d_osquery_copy);
>>> };
>>> [root at localhost conf.d]#
>>>
>>> I figured it out by installing rsyslog and looking at the differences in
>>> the output.
>>>
>>> I plan to summarize my experiences in a blog in a week or two.
>>>
>>> Bye,
>>>
>>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>>> Balabit / syslog-ng upstream
>>> https://www.balabit.com/blog/author/peterczanik/
>>> https://twitter.com/PCzanik
>>>
>>> On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
>>>
>>>> Hi
>>>> Evan
>>>> Your suggestion works flawlessly. The syslog table in
>>>> OSQUERY gets filled up with logs. The missing part is the rewrite rule
>>>> r_csv_message. Many many thanks to you.
>>>>
>>>> Regards
>>>> Dwijadas Dey
>>>>
>>>>
>>>>
>>>> On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel < <erempel at uvic.ca>
>>>>> erempel at uvic.ca> wrote:
>>>>>
>>>>>> The fact that your error "Received more fields than expected" went
>>>>>> away implies that the number of fields is correct.
>>>>>> Without any errors or any data in the table your trouble shooting
>>>>>> options are limited.
>>>>>>
>>>>>> I would make another file based destination for syslog-ng
>>>>>>
>>>>>> destination d_osquery_copy {
>>>>>> file("/var/osquery/syslog" template(t_csv));
>>>>>> };
>>>>>>
>>>>>> And add this destination to your log statement.
>>>>>>
>>>>>> log {
>>>>>> source(s_osquery);
>>>>>> destination(d_osquery);
>>>>>> destination(d_osquery_copy);
>>>>>> };
>>>>>>
>>>>>>
>>>>>> Then you will have a copy of the data that is being sent to osquery
>>>>>> and you should be able to get help from the osquery community.
>>>>>>
>>>>>>
>>>>>> One other thing to note is that I did not provide you with the
>>>>>> correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes
>>>>>> then this will not be a correctly formatted CSV field.
>>>>>>
>>>>>> you can make a rewrite rule for the message
>>>>>>
>>>>>> rewrite r_csv_message {
>>>>>> set("$MESSAGE", value("CSVMESSAGE") );
>>>>>> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
>>>>>> };
>>>>>>
>>>>>> then you need to invoke this rewrite rule in your log statement.
>>>>>>
>>>>>> log {
>>>>>> source(s_osquery);
>>>>>> rewrite(r_csv_message);
>>>>>> destination(d_osquery);
>>>>>> destination(d_osquery_copy);
>>>>>> };
>>>>>>
>>>>>> And finally your template needs to use the CSVMESSAGE rather than the
>>>>>> MESSAGE
>>>>>>
>>>>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>>>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n");
>>>>>> template_escape(no); };
>>>>>>
>>>>>>
>>>>>> I hope that helps too.
>>>>>>
>>>>>> Evan.
>>>>>>
>>>>>>
>>>>>> On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
>>>>>>
>>>>>> Hi
>>>>>> Evan
>>>>>> Thanks you for a quick reply. After changing the template
>>>>>> as suggested by you, the error goes away but the syslog table in OSQUERY
>>>>>> does not get filled up. May be the OSQUERY expects 7 entry for the syslog
>>>>>> table while the template has six fields.
>>>>>>
>>>>>> > osquery> .schema syslog
>>>>>> > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>>>>>> > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>>>>
>>>>>> No verbose error as well.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel < <erempel at uvic.ca>
>>>>>> erempel at uvic.ca> wrote:
>>>>>>
>>>>>>> The documentation from OSQuery is for rsyslog and shows that a csv
>>>>>>> set of values is needed.
>>>>>>>
>>>>>>> string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys
>>>>>>> logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::
>>>>>>> csv%,%msg:::csv%\n"
>>>>>>>
>>>>>>> In syslog-ng this format becomes
>>>>>>>
>>>>>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>>>>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n");
>>>>>>> template_escape(no); };
>>>>>>>
>>>>>>> Give that a try and see how things go.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>> Peter
>>>>>>> I am trying to send syslogs to a named pipe and on the
>>>>>>> other end OSQUERY will consume the syslogs from the named pipe. Once
>>>>>>> OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes
>>>>>>> along with OSQUERY. I have been able to send the syslogs to named pipe (
>>>>>>> verified with cat command ) but on the other hand OSQUERY did consume the
>>>>>>> logs but could not send these logs to the table due to format error.
>>>>>>>
>>>>>>> The schema of syslog table in OSQUERY
>>>>>>> ------------------------------------------------------------
>>>>>>> osquery> .schema syslog
>>>>>>> CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host`
>>>>>>> TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>>>>>
>>>>>>> Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)
>>>>>>> ------------------------------------------------------------
>>>>>>> ----------------------
>>>>>>> source s_osquery {
>>>>>>> system();
>>>>>>> };
>>>>>>>
>>>>>>> template t_csv {
>>>>>>> template("'${HOUR}${MIN}${SEC}
>>>>>>> ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA
>>>>>>> CILITY}',\t'${MSG}'\n");
>>>>>>> # template("$timestamp\t${ISODAT
>>>>>>> E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
>>>>>>> template_escape(no);
>>>>>>> };
>>>>>>>
>>>>>>> destination d_osquery {
>>>>>>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>>>>>>> };
>>>>>>>
>>>>>>> log {
>>>>>>> source(s_osquery);
>>>>>>> destination(d_osquery);
>>>>>>> };
>>>>>>>
>>>>>>> I am trying to match the above template to rsyslog format for OSQUERY
>>>>>>>
>>>>>>>
>>>>>>> <https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1>
>>>>>>> https://osquery.readthedocs.io/en/stable/deployment/syslog/#
>>>>>>> rsyslog-versions-7_1
>>>>>>>
>>>>>>> If i cat the pipe, i can see the syslogs.
>>>>>>>
>>>>>>> # cat /var/osquery/syslog_pipe
>>>>>>>
>>>>>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu',
>>>>>>> '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876
>>>>>>> [preauth]'
>>>>>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu',
>>>>>>> '55', 'notice', 'authpriv', 'PAM 2 more authentication
>>>>>>> failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51
>>>>>>> user=root'
>>>>>>>
>>>>>>>
>>>>>>> The above logs contains exactly 7 fields as required by OSQUERY
>>>>>>> syslog table as described above.
>>>>>>>
>>>>>>>
>>>>>>> The error that i am getting at the moment -
>>>>>>> ------------------------------------------------------------
>>>>>>> E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields
>>>>>>> than expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>>>> message=Added file event listener to: /root/.ssh/**
>>>>>>> E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields
>>>>>>> than expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>>>> message=Added file event listener to: /home/*/.ssh/**
>>>>>>> E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields
>>>>>>> than expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>>>> message=Added file event listener to: /tmp/**
>>>>>>> I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog
>>>>>>> run loop terminated for reason: Too many errors in syslog parsing.
>>>>>>>
>>>>>>> I think the issue is with the template definition which needs to
>>>>>>> match with the template with rsyslog as described in the above link.
>>>>>>>
>>>>>>> I will appreciate if someone can point out the issues in template
>>>>>>> and how it should be in syslog-ng.
>>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <
>>>>>>> <peter.czanik at balabit.com>peter.czanik at balabit.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> What do you try to achieve? Sending syslog messages to OSquery or
>>>>>>>> collecting OSquery logs by syslog-ng?
>>>>>>>>
>>>>>>>> /me now has a test environment installed
>>>>>>>>
>>>>>>>> Bye,
>>>>>>>>
>>>>>>>> Peter Czanik (CzP) < <peter.czanik at balabit.com>
>>>>>>>> peter.czanik at balabit.com>
>>>>>>>> Balabit / syslog-ng upstream
>>>>>>>> <https://www.balabit.com/blog/author/peterczanik/>
>>>>>>>> https://www.balabit.com/blog/author/peterczanik/
>>>>>>>> <https://twitter.com/PCzanik>https://twitter.com/PCzanik
>>>>>>>>
>>>>>>>> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey < <dwijad at gmail.com>
>>>>>>>> dwijad at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>> Robert
>>>>>>>>> You are right, i am trying the same with a named
>>>>>>>>> pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty
>>>>>>>>> of documents showing the same with rsyslog but not with syslog-ng.
>>>>>>>>>
>>>>>>>>> This is what my syslog configuration for osquery:-
>>>>>>>>>
>>>>>>>>> /etc/syslog-ng/conf.d/osquery.conf
>>>>>>>>>
>>>>>>>>> source s_osquery {
>>>>>>>>> # system();
>>>>>>>>> pipe("/var/osquery/syslog_pipe");
>>>>>>>>> # unix-stream("/dev/log");
>>>>>>>>> };
>>>>>>>>> #filter osqueryd {
>>>>>>>>> # program("^osqueryd.*");
>>>>>>>>> #};
>>>>>>>>> destination d_osquery {
>>>>>>>>> file("/var/log/osquery/osqueryd.results.log"
>>>>>>>>> template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
>>>>>>>>> };
>>>>>>>>> log {
>>>>>>>>> source(s_osquery);
>>>>>>>>> # filter(osqueryd);
>>>>>>>>> destination(d_osquery);
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> But this does not produce any logs for OSQUERY. I have checked ,
>>>>>>>>> the name piped has been created.
>>>>>>>>>
>>>>>>>>> # ls -l /var/osquery/syslog_pipe
>>>>>>>>> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
>>>>>>>>>
>>>>>>>>> But when i try to check what logs are passing through the pipe
>>>>>>>>> using following command, no message shows up.
>>>>>>>>> # cat /var/osquery/syslog_pipe
>>>>>>>>>
>>>>>>>>> I have correct options set in OSQUERY configuration file in
>>>>>>>>> /etc/osquery/osquery.conf.
>>>>>>>>>
>>>>>>>>> ..................
>>>>>>>>> ..................
>>>>>>>>> "logger_plugin": "syslog",
>>>>>>>>> "enable_syslog": "true",
>>>>>>>>> "syslog_pipe_path": "/var/osquery/syslog_pipe",
>>>>>>>>> ..................
>>>>>>>>> ..................
>>>>>>>>> I think Evan can point me the right configuration for syslog-ng (
>>>>>>>>> version 3.5.6 in ubuntu 16 )
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <
>>>>>>>>> <robert.fekete at balabit.com>robert.fekete at balabit.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> It seems that by default, osquery logs JSON messages into a
>>>>>>>>>> file. (
>>>>>>>>>> <https://osquery.readthedocs.io/en/latest/deployment/logging/>
>>>>>>>>>> https://osquery.readthedocs.io/en/latest/deployment/logging/ )
>>>>>>>>>> You can use this file in a syslog-ng source, and parse the JSON
>>>>>>>>>> messages with the json parser (note that you need a recent syslog-ng OSE
>>>>>>>>>> for this), see
>>>>>>>>>> <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html>
>>>>>>>>>> https://www.balabit.com/documents/syslog-ng-ose-latest-guide
>>>>>>>>>> s/en/syslog-ng-ose-guide-admin/html/json-parser.html .
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The above Osquery page mentions that it can send log messages
>>>>>>>>>> directly to syslog (instead of a file), but I haven't found how you can
>>>>>>>>>> actually configure it.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Robert
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <
>>>>>>>>>> <dwijad at gmail.com>dwijad at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>> List users
>>>>>>>>>>> Is it possible to send OSQUERY logs to
>>>>>>>>>>> syslog-ng 3.5 In the OSQUERY docs
>>>>>>>>>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/>
>>>>>>>>>>> rsyslog is configured to write logs to syslog. Does the same method applies
>>>>>>>>>>> to syslog-ng 3.5 ?
>>>>>>>>>>>
>>>>>>>>>>> Thanks and regards
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170428/83d45a56/attachment-0001.html>
More information about the syslog-ng
mailing list