I am trying to send syslogs to a
named pipe and on the other end OSQUERY will
consume the syslogs from the named pipe. Once
OSQUERY consumes syslogs, it will sends the logs
to RocksDB that comes along with OSQUERY. I have
been able to send the syslogs to named pipe (
verified with cat command ) but on the other
hand OSQUERY did consume the logs but could not
send these logs to the table due to format
error.
The schema of syslog table in OSQUERY
------------------------------------------------------------
osquery> .schema syslog
CREATE TABLE syslog_events(`time` BIGINT,
`datetime` TEXT, `host` TEXT, `severity` INTEGER,
`facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng
(/etc/syslog-ng/conf.d/osquery.conf)
----------------------------------------------------------------------------------
source s_osquery {
system();
};
template t_csv {
template("'${HOUR}${MIN}${SEC}',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FACILITY}',\t'${MSG}'\n");
#
template("$timestamp\t${ISODATE}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
template_escape(no);
};
destination d_osquery {
pipe("/var/osquery/syslog_pipe"
template(t_csv));
};
log {
source(s_osquery);
destination(d_osquery);
};
I am trying to match the above template to rsyslog
format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00',
'ubuntu', '26', 'info', 'auth', 'Disconnected
from 61.177.172.51 port 20876 [preauth]'
'155349', '2017-04-18T15:53:49+00:00',
'ubuntu', '55', 'notice',
'authpriv', 'PAM 2 more authentication failures;
logname= uid=0 euid=0 tty=ssh ruser=
rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by
OSQUERY syslog table as described above.
The error that i am getting at the moment -
------------------------------------------------------------
E0418 15:50:39.131995 4229 syslog.cpp:173] Received more
fields than expected in line: ''154852',
'2017-04-18T15:48:52+00:00', 'ubuntu', '9b',
'err', 'local3', 'severity=2
location=syslog.cpp:173 message=Received more fields than
expected in line: ''154852',
'2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
'notice', 'local3', 'severity=0
location=file_events.cpp:68 message=Added file event
listener to: /root/.ssh/**
E0418 15:50:39.132355 4229 syslog.cpp:173] Received more
fields than expected in line: ''154852',
'2017-04-18T15:48:52+00:00', 'ubuntu', '9b',
'err', 'local3', 'severity=2
location=syslog.cpp:173 message=Received more fields than
expected in line: ''154852',
'2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
'notice', 'local3', 'severity=0
location=file_events.cpp:68 message=Added file event
listener to: /home/*/.ssh/**
E0418 15:50:39.132758 4229 syslog.cpp:173] Received more
fields than expected in line: ''154852',
'2017-04-18T15:48:52+00:00', 'ubuntu', '9b',
'err', 'local3', 'severity=2
location=syslog.cpp:173 message=Received more fields than
expected in line: ''154852',
'2017-04-18T15:48:52+00:00', 'ubuntu', '9d',
'notice', 'local3', 'severity=0
location=file_events.cpp:68 message=Added file event
listener to: /tmp/**
I0418 15:50:39.133230 4229 events.cpp:767] Event
publisher syslog run loop terminated for reason: Too many
errors in syslog parsing.
I think the issue is with the template definition which
needs to match with the template with rsyslog as described
in the above link.