[syslog-ng] Syslog-ng and OSQUERY
Dwijadas Dey
dwijad at gmail.com
Thu Apr 20 13:15:00 UTC 2017
Hi
Peter
Congratulation ! Yes, the template took most of the time from
me as well. Just waiting to check your experiences in the blog.
Regards
Dwijadas Dey
On Thu, Apr 20, 2017 at 4:06 PM, Czanik, Péter <peter.czanik at balabit.com>
wrote:
> Hi,
>
> Just a heads up: I also got it working after a bit of debugging. The
> problem was, that in the above configuration sample there are spaces in the
> template. After removing those, it worked. Here is my config:
>
> [root at localhost conf.d]# cat oq.conf
> rewrite r_csv_message {
> set("$MESSAGE", value("CSVMESSAGE") );
> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
> };
>
> template t_csv {
> template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${
> FACILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n");
> template_escape(no);
> };
>
> destination d_osquery_copy {
> file("/var/log/csv_osquery" template(t_csv));
> };
>
> destination d_osquery {
> pipe("/var/osquery/syslog_pipe" template(t_csv));
> };
>
> log {
> source(s_sys);
> rewrite(r_csv_message);
> destination(d_osquery);
> destination(d_osquery_copy);
> };
> [root at localhost conf.d]#
>
> I figured it out by installing rsyslog and looking at the differences in
> the output.
>
> I plan to summarize my experiences in a blog in a week or two.
>
> Bye,
>
> Peter Czanik (CzP) <peter.czanik at balabit.com>
> Balabit / syslog-ng upstream
> https://www.balabit.com/blog/author/peterczanik/
> https://twitter.com/PCzanik
>
> On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
>
>> Hi
>> Evan
>> Your suggestion works flawlessly. The syslog table in OSQUERY
>> gets filled up with logs. The missing part is the rewrite rule
>> r_csv_message. Many many thanks to you.
>>
>> Regards
>> Dwijadas Dey
>>
>>
>>
>> On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel at uvic.ca> wrote:
>>>
>>>> The fact that your error "Received more fields than expected" went away
>>>> implies that the number of fields is correct.
>>>> Without any errors or any data in the table your trouble shooting
>>>> options are limited.
>>>>
>>>> I would make another file based destination for syslog-ng
>>>>
>>>> destination d_osquery_copy {
>>>> file("/var/osquery/syslog" template(t_csv));
>>>> };
>>>>
>>>> And add this destination to your log statement.
>>>>
>>>> log {
>>>> source(s_osquery);
>>>> destination(d_osquery);
>>>> destination(d_osquery_copy);
>>>> };
>>>>
>>>>
>>>> Then you will have a copy of the data that is being sent to osquery and
>>>> you should be able to get help from the osquery community.
>>>>
>>>>
>>>> One other thing to note is that I did not provide you with the correct
>>>> CSV of the MESSAGE portion. If the $MESSAGE contains double quotes
>>>> then this will not be a correctly formatted CSV field.
>>>>
>>>> you can make a rewrite rule for the message
>>>>
>>>> rewrite r_csv_message {
>>>> set("$MESSAGE", value("CSVMESSAGE") );
>>>> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
>>>> };
>>>>
>>>> then you need to invoke this rewrite rule in your log statement.
>>>>
>>>> log {
>>>> source(s_osquery);
>>>> rewrite(r_csv_message);
>>>> destination(d_osquery);
>>>> destination(d_osquery_copy);
>>>> };
>>>>
>>>> And finally your template needs to use the CSVMESSAGE rather than the
>>>> MESSAGE
>>>>
>>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n");
>>>> template_escape(no); };
>>>>
>>>>
>>>> I hope that helps too.
>>>>
>>>> Evan.
>>>>
>>>>
>>>> On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
>>>>
>>>> Hi
>>>> Evan
>>>> Thanks you for a quick reply. After changing the template as
>>>> suggested by you, the error goes away but the syslog table in OSQUERY does
>>>> not get filled up. May be the OSQUERY expects 7 entry for the syslog table
>>>> while the template has six fields.
>>>>
>>>> > osquery> .schema syslog
>>>> > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>>>> > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>>
>>>> No verbose error as well.
>>>>
>>>> Regards
>>>>
>>>> On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>>>
>>>>> The documentation from OSQuery is for rsyslog and shows that a csv set
>>>>> of values is needed.
>>>>>
>>>>> string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys
>>>>> logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::
>>>>> csv%,%msg:::csv%\n"
>>>>>
>>>>> In syslog-ng this format becomes
>>>>>
>>>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n");
>>>>> template_escape(no); };
>>>>>
>>>>> Give that a try and see how things go.
>>>>>
>>>>>
>>>>>
>>>>> On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
>>>>>
>>>>> Hi
>>>>> Peter
>>>>> I am trying to send syslogs to a named pipe and on the
>>>>> other end OSQUERY will consume the syslogs from the named pipe. Once
>>>>> OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes
>>>>> along with OSQUERY. I have been able to send the syslogs to named pipe (
>>>>> verified with cat command ) but on the other hand OSQUERY did consume the
>>>>> logs but could not send these logs to the table due to format error.
>>>>>
>>>>> The schema of syslog table in OSQUERY
>>>>> ------------------------------------------------------------
>>>>> osquery> .schema syslog
>>>>> CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host`
>>>>> TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>>>
>>>>> Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)
>>>>> ------------------------------------------------------------
>>>>> ----------------------
>>>>> source s_osquery {
>>>>> system();
>>>>> };
>>>>>
>>>>> template t_csv {
>>>>> template("'${HOUR}${MIN}${SEC}
>>>>> ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA
>>>>> CILITY}',\t'${MSG}'\n");
>>>>> # template("$timestamp\t${ISODAT
>>>>> E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
>>>>> template_escape(no);
>>>>> };
>>>>>
>>>>> destination d_osquery {
>>>>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>>>>> };
>>>>>
>>>>> log {
>>>>> source(s_osquery);
>>>>> destination(d_osquery);
>>>>> };
>>>>>
>>>>> I am trying to match the above template to rsyslog format for OSQUERY
>>>>>
>>>>> https://osquery.readthedocs.io/en/stable/deployment/syslog/#
>>>>> rsyslog-versions-7_1
>>>>>
>>>>> If i cat the pipe, i can see the syslogs.
>>>>>
>>>>> # cat /var/osquery/syslog_pipe
>>>>>
>>>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu',
>>>>> '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876
>>>>> [preauth]'
>>>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu',
>>>>> '55', 'notice', 'authpriv', 'PAM 2 more authentication
>>>>> failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51
>>>>> user=root'
>>>>>
>>>>>
>>>>> The above logs contains exactly 7 fields as required by OSQUERY syslog
>>>>> table as described above.
>>>>>
>>>>>
>>>>> The error that i am getting at the moment -
>>>>> ------------------------------------------------------------
>>>>> E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than
>>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>> message=Added file event listener to: /root/.ssh/**
>>>>> E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than
>>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>> message=Added file event listener to: /home/*/.ssh/**
>>>>> E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than
>>>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>>>> message=Added file event listener to: /tmp/**
>>>>> I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run
>>>>> loop terminated for reason: Too many errors in syslog parsing.
>>>>>
>>>>> I think the issue is with the template definition which needs to match
>>>>> with the template with rsyslog as described in the above link.
>>>>>
>>>>> I will appreciate if someone can point out the issues in template and
>>>>> how it should be in syslog-ng.
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <
>>>>> peter.czanik at balabit.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What do you try to achieve? Sending syslog messages to OSquery or
>>>>>> collecting OSquery logs by syslog-ng?
>>>>>>
>>>>>> /me now has a test environment installed
>>>>>>
>>>>>> Bye,
>>>>>>
>>>>>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>>>>>> Balabit / syslog-ng upstream
>>>>>> https://www.balabit.com/blog/author/peterczanik/
>>>>>> https://twitter.com/PCzanik
>>>>>>
>>>>>> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>> Robert
>>>>>>> You are right, i am trying the same with a named pipe
>>>>>>> so that OSQUERY consume syslogs as pointed by Evan. There are plenty of
>>>>>>> documents showing the same with rsyslog but not with syslog-ng.
>>>>>>>
>>>>>>> This is what my syslog configuration for osquery:-
>>>>>>>
>>>>>>> /etc/syslog-ng/conf.d/osquery.conf
>>>>>>>
>>>>>>> source s_osquery {
>>>>>>> # system();
>>>>>>> pipe("/var/osquery/syslog_pipe");
>>>>>>> # unix-stream("/dev/log");
>>>>>>> };
>>>>>>> #filter osqueryd {
>>>>>>> # program("^osqueryd.*");
>>>>>>> #};
>>>>>>> destination d_osquery {
>>>>>>> file("/var/log/osquery/osqueryd.results.log"
>>>>>>> template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
>>>>>>> };
>>>>>>> log {
>>>>>>> source(s_osquery);
>>>>>>> # filter(osqueryd);
>>>>>>> destination(d_osquery);
>>>>>>> };
>>>>>>>
>>>>>>> But this does not produce any logs for OSQUERY. I have checked , the
>>>>>>> name piped has been created.
>>>>>>>
>>>>>>> # ls -l /var/osquery/syslog_pipe
>>>>>>> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
>>>>>>>
>>>>>>> But when i try to check what logs are passing through the pipe using
>>>>>>> following command, no message shows up.
>>>>>>> # cat /var/osquery/syslog_pipe
>>>>>>>
>>>>>>> I have correct options set in OSQUERY configuration file in
>>>>>>> /etc/osquery/osquery.conf.
>>>>>>>
>>>>>>> ..................
>>>>>>> ..................
>>>>>>> "logger_plugin": "syslog",
>>>>>>> "enable_syslog": "true",
>>>>>>> "syslog_pipe_path": "/var/osquery/syslog_pipe",
>>>>>>> ..................
>>>>>>> ..................
>>>>>>> I think Evan can point me the right configuration for syslog-ng (
>>>>>>> version 3.5.6 in ubuntu 16 )
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <
>>>>>>> robert.fekete at balabit.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> It seems that by default, osquery logs JSON messages into a file.
>>>>>>>> ( https://osquery.readthedocs.io/en/latest/deployment/logging/ )
>>>>>>>> You can use this file in a syslog-ng source, and parse the JSON
>>>>>>>> messages with the json parser (note that you need a recent syslog-ng OSE
>>>>>>>> for this), see https://www.balabit.com/docume
>>>>>>>> nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin
>>>>>>>> /html/json-parser.html .
>>>>>>>>
>>>>>>>>
>>>>>>>> The above Osquery page mentions that it can send log messages
>>>>>>>> directly to syslog (instead of a file), but I haven't found how you can
>>>>>>>> actually configure it.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Robert
>>>>>>>>
>>>>>>>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>> List users
>>>>>>>>> Is it possible to send OSQUERY logs to
>>>>>>>>> syslog-ng 3.5 In the OSQUERY docs
>>>>>>>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/>
>>>>>>>>> rsyslog is configured to write logs to syslog. Does the same method applies
>>>>>>>>> to syslog-ng 3.5 ?
>>>>>>>>>
>>>>>>>>> Thanks and regards
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>> ____________________________________________________________
>>>>> __________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support
>>>>> /documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>> --
>>>> Evan Rempel erempel at uvic.ca
>>>> Senior Systems Administrator 250.721.7691 <(250)%20721-7691>
>>>> Data Centre Services, University Systems, University of Victoria
>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170420/46c557af/attachment-0001.html>
More information about the syslog-ng
mailing list