[syslog-ng] Syslog-ng and OSQUERY
Evan Rempel
erempel at uvic.ca
Thu Apr 20 13:26:47 UTC 2017
I should have read the RFC on CSV prior to drafting the format.
https://tools.ietf.org/html/rfc4180
Turns out Peter and I both got it wrong. The rewrite should be
rewrite r_csv_message {
set("$MESSAGE", value("CSVMESSAGE") );
subst("\"","\"\"", value("CSVMESSAGE"), flags(global) );
};
Of course, there are many "interpretations" of CSV so your consumer may
not conform to the official standard.
On 04/20/2017 03:36 AM, Czanik, Péter wrote:
> Hi,
>
> Just a heads up: I also got it working after a bit of debugging. The
> problem was, that in the above configuration sample there are spaces
> in the template. After removing those, it worked. Here is my config:
>
> [root at localhost conf.d]# cat oq.conf
> rewrite r_csv_message {
> set("$MESSAGE", value("CSVMESSAGE") );
> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
> };
>
> template t_csv {
> template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${FACILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n");
> template_escape(no);
> };
>
> destination d_osquery_copy {
> file("/var/log/csv_osquery" template(t_csv));
> };
>
> destination d_osquery {
> pipe("/var/osquery/syslog_pipe" template(t_csv));
> };
>
> log {
> source(s_sys);
> rewrite(r_csv_message);
> destination(d_osquery);
> destination(d_osquery_copy);
> };
> [root at localhost conf.d]#
>
> I figured it out by installing rsyslog and looking at the differences
> in the output.
>
> I plan to summarize my experiences in a blog in a week or two.
>
> Bye,
>
> Peter Czanik (CzP) <peter.czanik at balabit.com
> <mailto:peter.czanik at balabit.com>>
> Balabit / syslog-ng upstream
> https://www.balabit.com/blog/author/peterczanik/
> https://twitter.com/PCzanik
>
> On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad at gmail.com
> <mailto:dwijad at gmail.com>> wrote:
>
> Hi
> Evan
> Your suggestion works flawlessly. The syslog table in
> OSQUERY gets filled up with logs. The missing part is the rewrite
> rule r_csv_message. Many many thanks to you.
>
> Regards
> Dwijadas Dey
>
>
>
> On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel at uvic.ca
> <mailto:erempel at uvic.ca>> wrote:
>
> The fact that your error "Received more fields than
> expected" went away implies that the number of fields is
> correct.
> Without any errors or any data in the table your trouble
> shooting options are limited.
>
> I would make another file based destination for syslog-ng
>
> destination d_osquery_copy {
> file("/var/osquery/syslog" template(t_csv));
> };
>
> And add this destination to your log statement.
>
> log {
> source(s_osquery);
> destination(d_osquery);
> destination(d_osquery_copy);
> };
>
>
> Then you will have a copy of the data that is being sent
> to osquery and you should be able to get help from the
> osquery community.
>
>
> One other thing to note is that I did not provide you with
> the correct CSV of the MESSAGE portion. If the $MESSAGE
> contains double quotes
> then this will not be a correctly formatted CSV field.
>
> you can make a rewrite rule for the message
>
> rewrite r_csv_message {
> set("$MESSAGE", value("CSVMESSAGE") );
> subst("\"","\\\"", value("CSVMESSAGE"),
> flags(global) );
> };
>
> then you need to invoke this rewrite rule in your log
> statement.
>
> log {
> source(s_osquery);
> rewrite(r_csv_message);
> destination(d_osquery);
> destination(d_osquery_copy);
> };
>
> And finally your template needs to use the CSVMESSAGE
> rather than the MESSAGE
>
> template t_csv { template("\"${ISODATE}\",
> \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\",
> \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
>
>
> I hope that helps too.
>
> Evan.
>
>
> On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
>> Hi
>> Evan
>> Thanks you for a quick reply. After changing
>> the template as suggested by you, the error goes away but
>> the syslog table in OSQUERY does not get filled up. May
>> be the OSQUERY expects 7 entry for the syslog table while
>> the template has six fields.
>>
>> > osquery> .schema syslog
>> > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>> > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>> No verbose error as well.
>>
>> Regards
>>
>> On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel
>> <erempel at uvic.ca <mailto:erempel at uvic.ca>> wrote:
>>
>> The documentation from OSQuery is for rsyslog and
>> shows that a csv set of values is needed.
>>
>> string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n"
>>
>> In syslog-ng this format becomes
>>
>> template t_csv { template("\"${ISODATE}\",
>> \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\",
>> \"${PROGRAM}\", \"${MESSAGE}\"\n");
>> template_escape(no); };
>>
>> Give that a try and see how things go.
>>
>>
>>
>> On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
>>> Hi
>>> Peter
>>> I am trying to send syslogs to a named
>>> pipe and on the other end OSQUERY will consume the
>>> syslogs from the named pipe. Once OSQUERY consumes
>>> syslogs, it will sends the logs to RocksDB that
>>> comes along with OSQUERY. I have been able to send
>>> the syslogs to named pipe ( verified with cat
>>> command ) but on the other hand OSQUERY did consume
>>> the logs but could not send these logs to the table
>>> due to format error.
>>>
>>> The schema of syslog table in OSQUERY
>>> ------------------------------------------------------------
>>> osquery> .schema syslog
>>> CREATE TABLE syslog_events(`time` BIGINT, `datetime`
>>> TEXT, `host` TEXT, `severity` INTEGER, `facility`
>>> TEXT, `tag` TEXT, `message` TEXT);
>>>
>>> Conf file in syslog-ng
>>> (/etc/syslog-ng/conf.d/osquery.conf)
>>> ----------------------------------------------------------------------------------
>>> source s_osquery {
>>> system();
>>> };
>>>
>>> template t_csv {
>>>
>>> template("'${HOUR}${MIN}${SEC}',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FACILITY}',\t'${MSG}'\n");
>>> #
>>> template("$timestamp\t${ISODATE}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
>>> template_escape(no);
>>> };
>>>
>>> destination d_osquery {
>>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>>> };
>>>
>>> log {
>>> source(s_osquery);
>>> destination(d_osquery);
>>> };
>>>
>>> I am trying to match the above template to rsyslog
>>> format for OSQUERY
>>>
>>> https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1
>>> <https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1>
>>>
>>> If i cat the pipe, i can see the syslogs.
>>>
>>> # cat /var/osquery/syslog_pipe
>>>
>>> '155349', '2017-04-18T15:53:49+00:00',
>>> 'ubuntu', '26', 'info', 'auth', 'Disconnected from
>>> 61.177.172.51 port 20876 [preauth]'
>>> '155349', '2017-04-18T15:53:49+00:00',
>>> 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2
>>> more authentication failures; logname= uid=0 euid=0
>>> tty=ssh ruser= rhost=61.177.172.51 user=root'
>>>
>>>
>>> The above logs contains exactly 7 fields as required
>>> by OSQUERY syslog table as described above.
>>>
>>>
>>> The error that i am getting at the moment -
>>> ------------------------------------------------------------
>>> E0418 15:50:39.131995 4229 syslog.cpp:173] Received
>>> more fields than expected in line: ''154852',
>>> '2017-04-18T15:48:52+00:00', 'ubuntu', '9b',
>>> 'err', 'local3', 'severity=2 location=syslog.cpp:173
>>> message=Received more fields than expected in line:
>>> ''154852', '2017-04-18T15:48:52+00:00',
>>> 'ubuntu', '9d', 'notice',
>>> 'local3', 'severity=0
>>> location=file_events.cpp:68 message=Added file event
>>> listener to: /root/.ssh/**
>>> E0418 15:50:39.132355 4229 syslog.cpp:173] Received
>>> more fields than expected in line: ''154852',
>>> '2017-04-18T15:48:52+00:00', 'ubuntu', '9b',
>>> 'err', 'local3', 'severity=2 location=syslog.cpp:173
>>> message=Received more fields than expected in line:
>>> ''154852', '2017-04-18T15:48:52+00:00',
>>> 'ubuntu', '9d', 'notice',
>>> 'local3', 'severity=0
>>> location=file_events.cpp:68 message=Added file event
>>> listener to: /home/*/.ssh/**
>>> E0418 15:50:39.132758 4229 syslog.cpp:173] Received
>>> more fields than expected in line: ''154852',
>>> '2017-04-18T15:48:52+00:00', 'ubuntu', '9b',
>>> 'err', 'local3', 'severity=2 location=syslog.cpp:173
>>> message=Received more fields than expected in line:
>>> ''154852', '2017-04-18T15:48:52+00:00',
>>> 'ubuntu', '9d', 'notice',
>>> 'local3', 'severity=0
>>> location=file_events.cpp:68 message=Added file event
>>> listener to: /tmp/**
>>> I0418 15:50:39.133230 4229 events.cpp:767] Event
>>> publisher syslog run loop terminated for reason: Too
>>> many errors in syslog parsing.
>>>
>>> I think the issue is with the template definition
>>> which needs to match with the template with rsyslog
>>> as described in the above link.
>>>
>>> I will appreciate if someone can point out the
>>> issues in template and how it should be in syslog-ng.
>>>
>>>
>>> Regards
>>>
>>>
>>>
>>> On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter
>>> <peter.czanik at balabit.com
>>> <mailto:peter.czanik at balabit.com>> wrote:
>>>
>>> Hi,
>>>
>>> What do you try to achieve? Sending syslog
>>> messages to OSquery or collecting OSquery logs
>>> by syslog-ng?
>>>
>>> /me now has a test environment installed
>>>
>>> Bye,
>>>
>>> Peter Czanik (CzP) <peter.czanik at balabit.com
>>> <mailto:peter.czanik at balabit.com>>
>>> Balabit / syslog-ng upstream
>>> https://www.balabit.com/blog/author/peterczanik/
>>> <https://www.balabit.com/blog/author/peterczanik/>
>>> https://twitter.com/PCzanik
>>>
>>> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey
>>> <dwijad at gmail.com <mailto:dwijad at gmail.com>> wrote:
>>>
>>> Hi
>>> Robert
>>> You are right, i am trying the same with a
>>> named pipe so that OSQUERY consume syslogs
>>> as pointed by Evan. There are plenty of
>>> documents showing the same with rsyslog but
>>> not with syslog-ng.
>>>
>>> This is what my syslog configuration for
>>> osquery:-
>>>
>>> /etc/syslog-ng/conf.d/osquery.conf
>>>
>>> source s_osquery {
>>> # system();
>>> pipe("/var/osquery/syslog_pipe");
>>> # unix-stream("/dev/log");
>>> };
>>> #filter osqueryd {
>>> # program("^osqueryd.*");
>>> #};
>>> destination d_osquery {
>>> file("/var/log/osquery/osqueryd.results.log"
>>> template("$(format-json --scope
>>> selected_macros --scope nv_pairs)\n"));
>>> };
>>> log {
>>> source(s_osquery);
>>> # filter(osqueryd);
>>> destination(d_osquery);
>>> };
>>>
>>> But this does not produce any logs for
>>> OSQUERY. I have checked , the name piped has
>>> been created.
>>>
>>> # ls -l /var/osquery/syslog_pipe
>>> pr--rw---- 1 root adm 0 Apr 14 15:41
>>> /var/osquery/syslog_pipe
>>>
>>> But when i try to check what logs are
>>> passing through the pipe using following
>>> command, no message shows up.
>>> # cat /var/osquery/syslog_pipe
>>>
>>> I have correct options set in OSQUERY
>>> configuration file in /etc/osquery/osquery.conf.
>>>
>>> ..................
>>> ..................
>>> "logger_plugin": "syslog",
>>> "enable_syslog": "true",
>>> "syslog_pipe_path": "/var/osquery/syslog_pipe",
>>> ..................
>>> ..................
>>> I think Evan can point me the right
>>> configuration for syslog-ng ( version 3.5.6
>>> in ubuntu 16 )
>>>
>>> Regards
>>>
>>>
>>>
>>>
>>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete,
>>> Róbert <robert.fekete at balabit.com
>>> <mailto:robert.fekete at balabit.com>> wrote:
>>>
>>> Hi,
>>>
>>> It seems that by default, osquery logs
>>> JSON messages into a file. (
>>> https://osquery.readthedocs.io/en/latest/deployment/logging/
>>> <https://osquery.readthedocs.io/en/latest/deployment/logging/>
>>> )
>>> You can use this file in a syslog-ng
>>> source, and parse the JSON messages with
>>> the json parser (note that you need a
>>> recent syslog-ng OSE for this), see
>>> https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html
>>> <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html>
>>> .
>>>
>>>
>>> The above Osquery page mentions that it
>>> can send log messages directly to syslog
>>> (instead of a file), but I haven't found
>>> how you can actually configure it.
>>>
>>> Regards,
>>>
>>> Robert
>>>
>>> On Fri, Apr 14, 2017 at 9:46 PM,
>>> Dwijadas Dey <dwijad at gmail.com
>>> <mailto:dwijad at gmail.com>> wrote:
>>>
>>> Hi
>>> List users
>>> Is it possible to
>>> send OSQUERY logs to syslog-ng 3.5
>>> In the OSQUERY docs
>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/>
>>> rsyslog is configured to write logs
>>> to syslog. Does the same method
>>> applies to syslog-ng 3.5 ?
>>>
>>> Thanks and regards
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170420/65cf61d3/attachment-0001.html>
More information about the syslog-ng
mailing list