[syslog-ng] Syslog-ng and OSQUERY
Dwijadas Dey
dwijad at gmail.com
Tue Apr 18 18:58:25 UTC 2017
Hi
Evan
Your suggestion works flawlessly. The syslog table in OSQUERY
gets filled up with logs. The missing part is the rewrite rule
r_csv_message. Many many thanks to you.
Regards
Dwijadas Dey
On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel at uvic.ca> wrote:
>
>> The fact that your error "Received more fields than expected" went away
>> implies that the number of fields is correct.
>> Without any errors or any data in the table your trouble shooting options
>> are limited.
>>
>> I would make another file based destination for syslog-ng
>>
>> destination d_osquery_copy {
>> file("/var/osquery/syslog" template(t_csv));
>> };
>>
>> And add this destination to your log statement.
>>
>> log {
>> source(s_osquery);
>> destination(d_osquery);
>> destination(d_osquery_copy);
>> };
>>
>>
>> Then you will have a copy of the data that is being sent to osquery and
>> you should be able to get help from the osquery community.
>>
>>
>> One other thing to note is that I did not provide you with the correct
>> CSV of the MESSAGE portion. If the $MESSAGE contains double quotes
>> then this will not be a correctly formatted CSV field.
>>
>> you can make a rewrite rule for the message
>>
>> rewrite r_csv_message {
>> set("$MESSAGE", value("CSVMESSAGE") );
>> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
>> };
>>
>> then you need to invoke this rewrite rule in your log statement.
>>
>> log {
>> source(s_osquery);
>> rewrite(r_csv_message);
>> destination(d_osquery);
>> destination(d_osquery_copy);
>> };
>>
>> And finally your template needs to use the CSVMESSAGE rather than the
>> MESSAGE
>>
>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n");
>> template_escape(no); };
>>
>>
>> I hope that helps too.
>>
>> Evan.
>>
>>
>> On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
>>
>> Hi
>> Evan
>> Thanks you for a quick reply. After changing the template as
>> suggested by you, the error goes away but the syslog table in OSQUERY does
>> not get filled up. May be the OSQUERY expects 7 entry for the syslog table
>> while the template has six fields.
>>
>> > osquery> .schema syslog
>> > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>> > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>
>> No verbose error as well.
>>
>> Regards
>>
>> On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>
>>> The documentation from OSQuery is for rsyslog and shows that a csv set
>>> of values is needed.
>>>
>>> string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys
>>> logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::
>>> csv%,%msg:::csv%\n"
>>>
>>> In syslog-ng this format becomes
>>>
>>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n");
>>> template_escape(no); };
>>>
>>> Give that a try and see how things go.
>>>
>>>
>>>
>>> On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
>>>
>>> Hi
>>> Peter
>>> I am trying to send syslogs to a named pipe and on the other
>>> end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY
>>> consumes syslogs, it will sends the logs to RocksDB that comes along with
>>> OSQUERY. I have been able to send the syslogs to named pipe ( verified with
>>> cat command ) but on the other hand OSQUERY did consume the logs but could
>>> not send these logs to the table due to format error.
>>>
>>> The schema of syslog table in OSQUERY
>>> ------------------------------------------------------------
>>> osquery> .schema syslog
>>> CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>>> `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>>
>>> Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)
>>> ------------------------------------------------------------
>>> ----------------------
>>> source s_osquery {
>>> system();
>>> };
>>>
>>> template t_csv {
>>> template("'${HOUR}${MIN}${SEC}
>>> ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA
>>> CILITY}',\t'${MSG}'\n");
>>> # template("$timestamp\t${ISODAT
>>> E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
>>> template_escape(no);
>>> };
>>>
>>> destination d_osquery {
>>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>>> };
>>>
>>> log {
>>> source(s_osquery);
>>> destination(d_osquery);
>>> };
>>>
>>> I am trying to match the above template to rsyslog format for OSQUERY
>>>
>>> https://osquery.readthedocs.io/en/stable/deployment/syslog/#
>>> rsyslog-versions-7_1
>>>
>>> If i cat the pipe, i can see the syslogs.
>>>
>>> # cat /var/osquery/syslog_pipe
>>>
>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26',
>>> 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]'
>>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55',
>>> 'notice', 'authpriv', 'PAM 2 more authentication failures;
>>> logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
>>>
>>>
>>> The above logs contains exactly 7 fields as required by OSQUERY syslog
>>> table as described above.
>>>
>>>
>>> The error that i am getting at the moment -
>>> ------------------------------------------------------------
>>> E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than
>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>> message=Added file event listener to: /root/.ssh/**
>>> E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than
>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>> message=Added file event listener to: /home/*/.ssh/**
>>> E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than
>>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>>> location=syslog.cpp:173 message=Received more fields than expected in line:
>>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>>> message=Added file event listener to: /tmp/**
>>> I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run
>>> loop terminated for reason: Too many errors in syslog parsing.
>>>
>>> I think the issue is with the template definition which needs to match
>>> with the template with rsyslog as described in the above link.
>>>
>>> I will appreciate if someone can point out the issues in template and
>>> how it should be in syslog-ng.
>>>
>>>
>>> Regards
>>>
>>>
>>>
>>> On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik at balabit.com
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> What do you try to achieve? Sending syslog messages to OSquery or
>>>> collecting OSquery logs by syslog-ng?
>>>>
>>>> /me now has a test environment installed
>>>>
>>>> Bye,
>>>>
>>>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>>>> Balabit / syslog-ng upstream
>>>> https://www.balabit.com/blog/author/peterczanik/
>>>> https://twitter.com/PCzanik
>>>>
>>>> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
>>>>
>>>>> Hi
>>>>> Robert
>>>>> You are right, i am trying the same with a named pipe so
>>>>> that OSQUERY consume syslogs as pointed by Evan. There are plenty of
>>>>> documents showing the same with rsyslog but not with syslog-ng.
>>>>>
>>>>> This is what my syslog configuration for osquery:-
>>>>>
>>>>> /etc/syslog-ng/conf.d/osquery.conf
>>>>>
>>>>> source s_osquery {
>>>>> # system();
>>>>> pipe("/var/osquery/syslog_pipe");
>>>>> # unix-stream("/dev/log");
>>>>> };
>>>>> #filter osqueryd {
>>>>> # program("^osqueryd.*");
>>>>> #};
>>>>> destination d_osquery {
>>>>> file("/var/log/osquery/osqueryd.results.log"
>>>>> template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
>>>>> };
>>>>> log {
>>>>> source(s_osquery);
>>>>> # filter(osqueryd);
>>>>> destination(d_osquery);
>>>>> };
>>>>>
>>>>> But this does not produce any logs for OSQUERY. I have checked , the
>>>>> name piped has been created.
>>>>>
>>>>> # ls -l /var/osquery/syslog_pipe
>>>>> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
>>>>>
>>>>> But when i try to check what logs are passing through the pipe using
>>>>> following command, no message shows up.
>>>>> # cat /var/osquery/syslog_pipe
>>>>>
>>>>> I have correct options set in OSQUERY configuration file in
>>>>> /etc/osquery/osquery.conf.
>>>>>
>>>>> ..................
>>>>> ..................
>>>>> "logger_plugin": "syslog",
>>>>> "enable_syslog": "true",
>>>>> "syslog_pipe_path": "/var/osquery/syslog_pipe",
>>>>> ..................
>>>>> ..................
>>>>> I think Evan can point me the right configuration for syslog-ng (
>>>>> version 3.5.6 in ubuntu 16 )
>>>>>
>>>>> Regards
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <
>>>>> robert.fekete at balabit.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> It seems that by default, osquery logs JSON messages into a file. (
>>>>>> https://osquery.readthedocs.io/en/latest/deployment/logging/ )
>>>>>> You can use this file in a syslog-ng source, and parse the JSON
>>>>>> messages with the json parser (note that you need a recent syslog-ng OSE
>>>>>> for this), see https://www.balabit.com/docume
>>>>>> nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin
>>>>>> /html/json-parser.html .
>>>>>>
>>>>>>
>>>>>> The above Osquery page mentions that it can send log messages
>>>>>> directly to syslog (instead of a file), but I haven't found how you can
>>>>>> actually configure it.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Robert
>>>>>>
>>>>>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>> List users
>>>>>>> Is it possible to send OSQUERY logs to syslog-ng
>>>>>>> 3.5 In the OSQUERY docs
>>>>>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/>
>>>>>>> rsyslog is configured to write logs to syslog. Does the same method applies
>>>>>>> to syslog-ng 3.5 ?
>>>>>>>
>>>>>>> Thanks and regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>> --
>> Evan Rempel erempel at uvic.ca
>> Senior Systems Administrator 250.721.7691
>> Data Centre Services, University Systems, University of Victoria
>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170419/f2abba89/attachment-0001.html>
More information about the syslog-ng
mailing list