[syslog-ng] Syslog-ng and OSQUERY
Dwijadas Dey
dwijad at gmail.com
Tue Apr 18 17:22:17 UTC 2017
Hi
Evan
Thanks you for a quick reply. After changing the template as
suggested by you, the error goes away but the syslog table in OSQUERY does
not get filled up. May be the OSQUERY expects 7 entry for the syslog table
while the template has six fields.
> osquery> .schema syslog
> CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
> `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel at uvic.ca> wrote:
> The documentation from OSQuery is for rsyslog and shows that a csv set of
> values is needed.
>
> string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,
> %syslogseverity:::csv%,%syslogfacility-text:::csv%,%
> syslogtag:::csv%,%msg:::csv%\n"
>
> In syslog-ng this format becomes
>
> template t_csv { template("\"${ISODATE}\", \"${HOST}\",
> \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n");
> template_escape(no); };
>
> Give that a try and see how things go.
>
>
>
> On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
>
> Hi
> Peter
> I am trying to send syslogs to a named pipe and on the other
> end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY
> consumes syslogs, it will sends the logs to RocksDB that comes along with
> OSQUERY. I have been able to send the syslogs to named pipe ( verified with
> cat command ) but on the other hand OSQUERY did consume the logs but could
> not send these logs to the table due to format error.
>
> The schema of syslog table in OSQUERY
> ------------------------------------------------------------
> osquery> .schema syslog
> CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT,
> `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
>
> Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)
> ------------------------------------------------------------
> ----------------------
> source s_osquery {
> system();
> };
>
> template t_csv {
> template("'${HOUR}${MIN}${SEC}
> ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${
> FACILITY}',\t'${MSG}'\n");
> # template("$timestamp\t${ISODATE}\t{$HOST}\t$
> syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
> template_escape(no);
> };
>
> destination d_osquery {
> pipe("/var/osquery/syslog_pipe" template(t_csv));
> };
>
> log {
> source(s_osquery);
> destination(d_osquery);
> };
>
> I am trying to match the above template to rsyslog format for OSQUERY
>
> https://osquery.readthedocs.io/en/stable/deployment/
> syslog/#rsyslog-versions-7_1
>
> If i cat the pipe, i can see the syslogs.
>
> # cat /var/osquery/syslog_pipe
>
> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26',
> 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]'
> '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55',
> 'notice', 'authpriv', 'PAM 2 more authentication failures;
> logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
>
>
> The above logs contains exactly 7 fields as required by OSQUERY syslog
> table as described above.
>
>
> The error that i am getting at the moment -
> ------------------------------------------------------------
> E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than
> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
> 'ubuntu', '9b', 'err', 'local3', 'severity=2
> location=syslog.cpp:173 message=Received more fields than expected in line:
> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
> message=Added file event listener to: /root/.ssh/**
> E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than
> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
> 'ubuntu', '9b', 'err', 'local3', 'severity=2
> location=syslog.cpp:173 message=Received more fields than expected in line:
> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
> message=Added file event listener to: /home/*/.ssh/**
> E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than
> expected in line: ''154852', '2017-04-18T15:48:52+00:00',
> 'ubuntu', '9b', 'err', 'local3', 'severity=2
> location=syslog.cpp:173 message=Received more fields than expected in line:
> ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
> 'notice', 'local3', 'severity=0 location=file_events.cpp:68
> message=Added file event listener to: /tmp/**
> I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run
> loop terminated for reason: Too many errors in syslog parsing.
>
> I think the issue is with the template definition which needs to match
> with the template with rsyslog as described in the above link.
>
> I will appreciate if someone can point out the issues in template and how
> it should be in syslog-ng.
>
>
> Regards
>
>
>
> On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik at balabit.com>
> wrote:
>
>> Hi,
>>
>> What do you try to achieve? Sending syslog messages to OSquery or
>> collecting OSquery logs by syslog-ng?
>>
>> /me now has a test environment installed
>>
>> Bye,
>>
>> Peter Czanik (CzP) <peter.czanik at balabit.com>
>> Balabit / syslog-ng upstream
>> https://www.balabit.com/blog/author/peterczanik/
>> https://twitter.com/PCzanik
>>
>> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
>>
>>> Hi
>>> Robert
>>> You are right, i am trying the same with a named pipe so
>>> that OSQUERY consume syslogs as pointed by Evan. There are plenty of
>>> documents showing the same with rsyslog but not with syslog-ng.
>>>
>>> This is what my syslog configuration for osquery:-
>>>
>>> /etc/syslog-ng/conf.d/osquery.conf
>>>
>>> source s_osquery {
>>> # system();
>>> pipe("/var/osquery/syslog_pipe");
>>> # unix-stream("/dev/log");
>>> };
>>> #filter osqueryd {
>>> # program("^osqueryd.*");
>>> #};
>>> destination d_osquery {
>>> file("/var/log/osquery/osqueryd.results.log"
>>> template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
>>> };
>>> log {
>>> source(s_osquery);
>>> # filter(osqueryd);
>>> destination(d_osquery);
>>> };
>>>
>>> But this does not produce any logs for OSQUERY. I have checked , the
>>> name piped has been created.
>>>
>>> # ls -l /var/osquery/syslog_pipe
>>> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
>>>
>>> But when i try to check what logs are passing through the pipe using
>>> following command, no message shows up.
>>> # cat /var/osquery/syslog_pipe
>>>
>>> I have correct options set in OSQUERY configuration file in
>>> /etc/osquery/osquery.conf.
>>>
>>> ..................
>>> ..................
>>> "logger_plugin": "syslog",
>>> "enable_syslog": "true",
>>> "syslog_pipe_path": "/var/osquery/syslog_pipe",
>>> ..................
>>> ..................
>>> I think Evan can point me the right configuration for syslog-ng (
>>> version 3.5.6 in ubuntu 16 )
>>>
>>> Regards
>>>
>>>
>>>
>>>
>>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <
>>> robert.fekete at balabit.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> It seems that by default, osquery logs JSON messages into a file. (
>>>> https://osquery.readthedocs.io/en/latest/deployment/logging/ )
>>>> You can use this file in a syslog-ng source, and parse the JSON
>>>> messages with the json parser (note that you need a recent syslog-ng OSE
>>>> for this), see https://www.balabit.com/docume
>>>> nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin
>>>> /html/json-parser.html .
>>>>
>>>>
>>>> The above Osquery page mentions that it can send log messages directly
>>>> to syslog (instead of a file), but I haven't found how you can actually
>>>> configure it.
>>>>
>>>> Regards,
>>>>
>>>> Robert
>>>>
>>>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad at gmail.com> wrote:
>>>>
>>>>> Hi
>>>>> List users
>>>>> Is it possible to send OSQUERY logs to syslog-ng
>>>>> 3.5 In the OSQUERY docs
>>>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/>
>>>>> rsyslog is configured to write logs to syslog. Does the same method applies
>>>>> to syslog-ng 3.5 ?
>>>>>
>>>>> Thanks and regards
>>>>>
>>>>>
>>>>>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170418/d86a65db/attachment-0001.html>
More information about the syslog-ng
mailing list