[syslog-ng] Syslog-ng and OSQUERY
Dwijadas Dey
dwijad at gmail.com
Tue Apr 18 18:53:31 UTC 2017
Hi
Evan
Your suggestion works flawlessly. The syslog table in OSQUERY
gets filled up with logs 😀 The missing part is the rewrite rule
r_csv_message. Many many thanks to you.
Regards
Dwijadas Dey
On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel at uvic.ca> wrote:
> The fact that your error "Received more fields than expected" went away
> implies that the number of fields is correct.
> Without any errors or any data in the table your trouble shooting options
> are limited.
>
> I would make another file based destination for syslog-ng
>
> destination d_osquery_copy {
> file("/var/osquery/syslog" template(t_csv));
> };
>
> And add this destination to your log statement.
>
> log {
> source(s_osquery);
> destination(d_osquery);
> destination(d_osquery_copy);
> };
>
>
> Then you will have a copy of the data that is being sent to osquery and
> you should be able to get help from the osquery community.
>
>
> One other thing to note is that I did not provide you with the correct CSV
> of the MESSAGE portion. If the $MESSAGE contains double quotes
> then this will not be a correctly formatted CSV field.
>
> you can make a rewrite rule for the message
>
> rewrite r_csv_message {
> set("$MESSAGE", value("CSVMESSAGE") );
> subst("\"","\\\"", value("CSVMESSAGE"), flags(global) );
> };
>
> then you need to invoke this rewrite rule in your log statement.
>
> log {
> source(s_osquery);
> rewrite(r_csv_message);
> destination(d_osquery);
> destination(d_osquery_copy);
> };
>
> And finally your template needs to use the CSVMESSAGE rather than the
> MESSAGE
>
> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n");
> template_escape(no); };
>
>
> I hope that helps too.
>
> Evan.
>
>
> On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
>
> Hi
> Evan
> Thanks you for a quick reply. After changing the template as
> suggested by you, the error goes away but the syslog table in OSQUERY does
> not get filled up. May be the OSQUERY expects 7 entry for the syslog table
> while the template has six fields.
>
> > osquery> .schema syslog
> > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
> > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>
> No verbose error as well.
>
> Regards
>
> On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel at uvic.ca> wrote:
>
>> The documentation from OSQuery is for rsyslog and shows that a csv set of
>> values is needed.
>>
>> string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%
>> syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag
>> :::csv%,%msg:::csv%\n"
>>
>> In syslog-ng this format becomes
>>
>> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
>> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n");
>> template_escape(no); };
>>
>> Give that a try and see how things go.
>>
>>
>>
>> On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
>>
>> Hi
>> Peter
>> I am trying to send syslogs to a named pipe and on the other
>> end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY
>> consumes syslogs, it will sends the logs to RocksDB that comes along with
>> OSQUERY. I have been able to send the syslogs to named pipe ( verified with
>> cat command ) but on the other hand OSQUERY did consume the logs but could
>> not send these logs to the table due to format error.
>>
>> The schema of syslog table in OSQUERY
>> ------------------------------------------------------------
>> osquery> .schema syslog
>> CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
>> `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>>
>> Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)
>> ------------------------------------------------------------
>> ----------------------
>> source s_osquery {
>> system();
>> };
>>
>> template t_csv {
>> template("'${HOUR}${MIN}${SEC}
>> ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA
>> CILITY}',\t'${MSG}'\n");
>> # template("$timestamp\t${ISODAT
>> E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
>> template_escape(no);
>> };
>>
>> destination d_osquery {
>> pipe("/var/osquery/syslog_pipe" template(t_csv));
>> };
>>
>> log {
>> source(s_osquery);
>> destination(d_osquery);
>> };
>>
>> I am trying to match the above template to rsyslog format for OSQUERY
>>
>> https://osquery.readthedocs.io/en/stable/deployment/syslog/#
>> rsyslog-versions-7_1
>>
>> If i cat the pipe, i can see the syslogs.
>>
>> # cat /var/osquery/syslog_pipe
>>
>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26',
>> 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]'
>> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55',
>> 'notice', 'authpriv', 'PAM 2 more authentication failures;
>> logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
>>
>>
>> The above logs contains exactly 7 fields as required by OSQUERY syslog
>> table as described above.
>>
>>
>> The error that i am getting at the moment -
>> ------------------------------------------------------------
>> E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than
>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>> location=syslog.cpp:173 message=Received more fields than expected in line:
>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>> message=Added file event listener to: /root/.ssh/**
>> E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than
>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>> location=syslog.cpp:173 message=Received more fields than expected in line:
>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>> message=Added file event listener to: /home/*/.ssh/**
>> E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than
>> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
>> 'ubuntu', '9b', 'err', 'local3', 'severity=2
>> location=syslog.cpp:173 message=Received more fields than expected in line:
>> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
>> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
>> message=Added file event listener to: /tmp/**
>> I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run
>> loop terminated for reason: Too many errors in syslog parsing.
>>
>> I think the issue is with the template definition which needs to match
>> with the template with rsyslog as described in the above link.
>>
>> I will appreciate if someone can point out the issues in template and how
>> it should be in syslog-ng.
>>
>>
>> Regards
>>
>>
>>
>> On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik at balabit.com>
>> wrote:
>>
>>> Hi,
>>>
>>> What do you try to achieve? Sending syslog messages to OSquery or
>>> collecting OSquery logs by syslog-ng?
>>>
>>> /me now has a test environment installed
>>>
>>> Bye,
>>>
>>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>>> Balabit / syslog-ng upstream
>>> https://www.balabit.com/blog/author/peterczanik/
>>> https://twitter.com/PCzanik
>>>
>>> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
>>>
>>>> Hi
>>>> Robert
>>>> You are right, i am trying the same with a named pipe so
>>>> that OSQUERY consume syslogs as pointed by Evan. There are plenty of
>>>> documents showing the same with rsyslog but not with syslog-ng.
>>>>
>>>> This is what my syslog configuration for osquery:-
>>>>
>>>> /etc/syslog-ng/conf.d/osquery.conf
>>>>
>>>> source s_osquery {
>>>> # system();
>>>> pipe("/var/osquery/syslog_pipe");
>>>> # unix-stream("/dev/log");
>>>> };
>>>> #filter osqueryd {
>>>> # program("^osqueryd.*");
>>>> #};
>>>> destination d_osquery {
>>>> file("/var/log/osquery/osqueryd.results.log"
>>>> template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
>>>> };
>>>> log {
>>>> source(s_osquery);
>>>> # filter(osqueryd);
>>>> destination(d_osquery);
>>>> };
>>>>
>>>> But this does not produce any logs for OSQUERY. I have checked , the
>>>> name piped has been created.
>>>>
>>>> # ls -l /var/osquery/syslog_pipe
>>>> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
>>>>
>>>> But when i try to check what logs are passing through the pipe using
>>>> following command, no message shows up.
>>>> # cat /var/osquery/syslog_pipe
>>>>
>>>> I have correct options set in OSQUERY configuration file in
>>>> /etc/osquery/osquery.conf.
>>>>
>>>> ..................
>>>> ..................
>>>> "logger_plugin": "syslog",
>>>> "enable_syslog": "true",
>>>> "syslog_pipe_path": "/var/osquery/syslog_pipe",
>>>> ..................
>>>> ..................
>>>> I think Evan can point me the right configuration for syslog-ng (
>>>> version 3.5.6 in ubuntu 16 )
>>>>
>>>> Regards
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <
>>>> robert.fekete at balabit.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> It seems that by default, osquery logs JSON messages into a file. (
>>>>> https://osquery.readthedocs.io/en/latest/deployment/logging/ )
>>>>> You can use this file in a syslog-ng source, and parse the JSON
>>>>> messages with the json parser (note that you need a recent syslog-ng OSE
>>>>> for this), see https://www.balabit.com/docume
>>>>> nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin
>>>>> /html/json-parser.html .
>>>>>
>>>>>
>>>>> The above Osquery page mentions that it can send log messages directly
>>>>> to syslog (instead of a file), but I haven't found how you can actually
>>>>> configure it.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Robert
>>>>>
>>>>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>> List users
>>>>>> Is it possible to send OSQUERY logs to syslog-ng
>>>>>> 3.5 In the OSQUERY docs
>>>>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/>
>>>>>> rsyslog is configured to write logs to syslog. Does the same method applies
>>>>>> to syslog-ng 3.5 ?
>>>>>>
>>>>>> Thanks and regards
>>>>>>
>>>>>>
>>>>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> --
> Evan Rempel erempel at uvic.ca
> Senior Systems Administrator 250.721.7691
> Data Centre Services, University Systems, University of Victoria
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170418/d0a7a528/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: osqueryi1.png
Type: image/png
Size: 71048 bytes
Desc: not available
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170418/d0a7a528/attachment.png>
More information about the syslog-ng
mailing list