I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY
------------------------------------------------------------
osquery> .schema syslog
CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)
----------------------------------------------------------------------------------
source s_osquery {
system();
};
template t_csv {
template("'${HOUR}${MIN}${SEC}',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FACILITY}',\t'${MSG}'\n");
# template("$timestamp\t${ISODATE}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");
template_escape(no);
};
destination d_osquery {
pipe("/var/osquery/syslog_pipe" template(t_csv));
};
log {
source(s_osquery);
destination(d_osquery);
};
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]'
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment -
------------------------------------------------------------
E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/**
E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/**
E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/**
I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.