<div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Hi<br></div> Peter<br></div> I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.<br><br></div>The schema of syslog table in OSQUERY<br>------------------------------------------------------------<br>osquery> .schema syslog<br>CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);<br><br>Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf)<br>----------------------------------------------------------------------------------<br>source s_osquery {<br> system(); <br>};<br><br>template t_csv {<br> template("'${HOUR}${MIN}${SEC}',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FACILITY}',\t'${MSG}'\n");<br> # template("$timestamp\t${ISODATE}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n");<br> template_escape(no);<br>};<br><br>destination d_osquery {<br> pipe("/var/osquery/syslog_pipe" template(t_csv));<br>};<br><br>log {<br> source(s_osquery);<br> destination(d_osquery);<br>};<br><br></div>I am trying to match the above template to rsyslog format for OSQUERY<br><br><a href="https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1">https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1</a><br><br></div>If i cat the pipe, i can see the syslogs.<br><br># cat /var/osquery/syslog_pipe<br><br>'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]'<br>'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'<br><br><br></div>The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.<br><br><br></div>The error that i am getting at the moment -<br>------------------------------------------------------------<br>E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/**<br>E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/**<br>E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/**<br>I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.<br><br></div>I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.<br><br></div>I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.<br><br><br></div>Regards<br><div><div><div><div><br><div><div><div><br></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <span dir="ltr"><<a href="mailto:peter.czanik@balabit.com" target="_blank">peter.czanik@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi,<br></div><br>What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?<br><br></div>/me now has a test environment installed<br><br></div>Bye,<br></div><div class="gmail_extra"><br clear="all"><div><div class="m_1327877505004078358gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Peter Czanik (CzP) <<a href="mailto:peter.czanik@balabit.com" target="_blank">peter.czanik@balabit.com</a>><br>Balabit / syslog-ng upstream<br><a href="https://www.balabit.com/blog/author/peterczanik/" target="_blank">https://www.balabit.com/blog/<wbr>author/peterczanik/</a><br><a href="https://twitter.com/PCzanik" target="_blank">https://twitter.com/PCzanik</a></div></div></div></div><div><div class="h5">
<br><div class="gmail_quote">On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <span dir="ltr"><<a href="mailto:dwijad@gmail.com" target="_blank">dwijad@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Hi<br></div> Robert<br></div> You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.<br><br></div>This is what my syslog configuration for osquery:-<br><br>/etc/syslog-ng/conf.d/osquery.<wbr>conf<br><br>source s_osquery {<br> # system();<br> pipe("/var/osquery/syslog_pipe<wbr>");<br> # unix-stream("/dev/log");<br>};<br>#filter osqueryd {<br> # program("^osqueryd.*");<br>#};<br>destination d_osquery {<br> file("/var/log/osquery/osquery<wbr>d.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n"));<br>};<br>log {<br> source(s_osquery);<br> # filter(osqueryd);<br> destination(d_osquery);<br>};<br><br></div>But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.<br><br># ls -l /var/osquery/syslog_pipe<br>pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe<br><br>But when i try to check what logs are passing through the pipe using following command, no message shows up.<br># cat /var/osquery/syslog_pipe<br><br></div><div>I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.<br><br>..................<br>..................<br> "logger_plugin": "syslog",<br>"enable_syslog": "true",<br>"syslog_pipe_path": "/var/osquery/syslog_pipe",<br>..................<br>..................<br></div>I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )<br><br></div>Regards<div><div class="m_1327877505004078358h5"><br><div><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <span dir="ltr"><<a href="mailto:robert.fekete@balabit.com" target="_blank">robert.fekete@balabit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi, <br><br></div>It seems that by default, osquery logs JSON messages into a file. ( <a href="https://osquery.readthedocs.io/en/latest/deployment/logging/" target="_blank">https://osquery.readthedocs.io<wbr>/en/latest/deployment/logging/</a> )<br></div>You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see <a href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html" target="_blank">https://www.balabit.com/docume<wbr>nts/syslog-ng-ose-latest-guide<wbr>s/en/syslog-ng-ose-guide-admin<wbr>/html/json-parser.html</a> .<br><br><br></div>The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.<br><br></div><div>Regards, <br><br></div><div>Robert<br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_1327877505004078358m_-3763707499530419432h5">On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <span dir="ltr"><<a href="mailto:dwijad@gmail.com" target="_blank">dwijad@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_1327877505004078358m_-3763707499530419432h5"><div dir="ltr"><div><div><div>Hi<br></div> List users<br></div> Is it possible to send OSQUERY logs to syslog-ng 3.5 In the <a href="https://osquery.readthedocs.io/en/latest/deployment/syslog/" target="_blank">OSQUERY docs</a> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?<br><br></div>Thanks and regards<br></div>
<br></div></div>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div></div></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>