Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ? Thanks and regards
The documentation you point to is for OSQuery to consume syslog messages, not for OSQuery to send its logs to syslog. The same *method* applies for syslog-ng to send the system logs to a named pipe for OSQuery to consume, however, the configuration details are completely different. Evan. On 04/14/2017 12:46 PM, Dwijadas Dey wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
Hi, It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... . The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it. Regards, Robert On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng. This is what my syslog configuration for osquery:- /etc/syslog-ng/conf.d/osquery.conf source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); }; But this does not produce any logs for OSQUERY. I have checked , the name piped has been created. # ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf. .................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 ) Regards On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <robert.fekete@balabit.com> wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/documents/syslog-ng-ose- latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng? /me now has a test environment installed Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <robert.fekete@balabit.com
wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/documents/syslog-ng-ose-latest- guides/en/syslog-ng-ose-guide-admin/html/json-parser.html .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error. The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT); Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ---------------------------------------------------------------------------------- source s_osquery { system(); }; template t_csv { template("'${HOUR}${MIN}${SEC}',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FACILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODATE}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); }; destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); }; log { source(s_osquery); destination(d_osquery); }; I am trying to match the above template to rsyslog format for OSQUERY https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions... If i cat the pipe, i can see the syslogs. # cat /var/osquery/syslog_pipe '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root' The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above. The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing. I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link. I will appreciate if someone can point out the issues in template and how it should be in syslog-ng. Regards On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < robert.fekete@balabit.com> wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/documents/syslog-ng-ose-latest-guide s/en/syslog-ng-ose-guide-admin/html/json-parser.html .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed. string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n" In syslog-ng this format becomes template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); }; Give that a try and see how things go. On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ---------------------------------------------------------------------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC}',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FACILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODATE}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions...
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ <https://www.balabit.com/blog/author/peterczanik/> https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com <mailto:dwijad@gmail.com>> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <robert.fekete@balabit.com <mailto:robert.fekete@balabit.com>> wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ <https://osquery.readthedocs.io/en/latest/deployment/logging/> ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html> .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com <mailto:dwijad@gmail.com>> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well. Regards On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel@uvic.ca> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%, %syslogseverity:::csv%,%syslogfacility-text:::csv%,% syslogtag:::csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ------------------------------------------------------------ ---------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC} ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${ FACILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODATE}\t{$HOST}\t$ syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/ syslog/#rsyslog-versions-7_1
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < robert.fekete@balabit.com> wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/docume nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin /html/json-parser.html .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited. I would make another file based destination for syslog-ng destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); }; And add this destination to your log statement. log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); }; Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community. One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field. you can make a rewrite rule for the message rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); }; then you need to invoke this rewrite rule in your log statement. log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); }; And finally your template needs to use the CSVMESSAGE rather than the MESSAGE template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); }; I hope that helps too. Evan. On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT); No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ---------------------------------------------------------------------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC}',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FACILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODATE}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions... <https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1>
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ <https://www.balabit.com/blog/author/peterczanik/> https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com <mailto:dwijad@gmail.com>> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <robert.fekete@balabit.com <mailto:robert.fekete@balabit.com>> wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ <https://osquery.readthedocs.io/en/latest/deployment/logging/> ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html> .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com <mailto:dwijad@gmail.com>> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
Hi Evan Your suggestion works flawlessly. The syslog table in OSQUERY gets filled up with logs 😀 The missing part is the rewrite rule r_csv_message. Many many thanks to you. Regards Dwijadas Dey On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel@uvic.ca> wrote:
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited.
I would make another file based destination for syslog-ng
destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); };
And add this destination to your log statement.
log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); };
Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community.
One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field.
you can make a rewrite rule for the message
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
then you need to invoke this rewrite rule in your log statement.
log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); };
And finally your template needs to use the CSVMESSAGE rather than the MESSAGE
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
I hope that helps too.
Evan.
On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel@uvic.ca> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,% syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag :::csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ------------------------------------------------------------ ---------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC} ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA CILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODAT E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/# rsyslog-versions-7_1
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < robert.fekete@balabit.com> wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/docume nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin /html/json-parser.html .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Evan Your suggestion works flawlessly. The syslog table in OSQUERY gets filled up with logs. The missing part is the rewrite rule r_csv_message. Many many thanks to you. Regards Dwijadas Dey On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel@uvic.ca> wrote:
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited.
I would make another file based destination for syslog-ng
destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); };
And add this destination to your log statement.
log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); };
Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community.
One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field.
you can make a rewrite rule for the message
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
then you need to invoke this rewrite rule in your log statement.
log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); };
And finally your template needs to use the CSVMESSAGE rather than the MESSAGE
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
I hope that helps too.
Evan.
On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel@uvic.ca> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag::: csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ------------------------------------------------------------ ---------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC} ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA CILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODAT E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/# rsyslog-versions-7_1
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik@balabit.com
wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < robert.fekete@balabit.com> wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/docume nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin /html/json-parser.html .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
> Hi > List users > Is it possible to send OSQUERY logs to syslog-ng > 3.5 In the OSQUERY docs > <https://osquery.readthedocs.io/en/latest/deployment/syslog/> > rsyslog is configured to write logs to syslog. Does the same method applies > to syslog-ng 3.5 ? > > Thanks and regards > > >
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, Just a heads up: I also got it working after a bit of debugging. The problem was, that in the above configuration sample there are spaces in the template. After removing those, it worked. Here is my config: [root@localhost conf.d]# cat oq.conf rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); }; template t_csv { template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${FACILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n"); template_escape(no); }; destination d_osquery_copy { file("/var/log/csv_osquery" template(t_csv)); }; destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); }; log { source(s_sys); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); }; [root@localhost conf.d]# I figured it out by installing rsyslog and looking at the differences in the output. I plan to summarize my experiences in a blog in a week or two. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Evan Your suggestion works flawlessly. The syslog table in OSQUERY gets filled up with logs. The missing part is the rewrite rule r_csv_message. Many many thanks to you.
Regards Dwijadas Dey
On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel@uvic.ca> wrote:
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited.
I would make another file based destination for syslog-ng
destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); };
And add this destination to your log statement.
log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); };
Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community.
One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field.
you can make a rewrite rule for the message
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
then you need to invoke this rewrite rule in your log statement.
log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); };
And finally your template needs to use the CSVMESSAGE rather than the MESSAGE
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
I hope that helps too.
Evan.
On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel@uvic.ca> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag::: csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ------------------------------------------------------------ ---------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC} ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA CILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODAT E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/# rsyslog-versions-7_1
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter < peter.czanik@balabit.com> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < robert.fekete@balabit.com> wrote:
> Hi, > > It seems that by default, osquery logs JSON messages into a file. ( > https://osquery.readthedocs.io/en/latest/deployment/logging/ ) > You can use this file in a syslog-ng source, and parse the JSON > messages with the json parser (note that you need a recent syslog-ng OSE > for this), see https://www.balabit.com/docume > nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin > /html/json-parser.html . > > > The above Osquery page mentions that it can send log messages > directly to syslog (instead of a file), but I haven't found how you can > actually configure it. > > Regards, > > Robert > > On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> > wrote: > >> Hi >> List users >> Is it possible to send OSQUERY logs to syslog-ng >> 3.5 In the OSQUERY docs >> <https://osquery.readthedocs.io/en/latest/deployment/syslog/> >> rsyslog is configured to write logs to syslog. Does the same method applies >> to syslog-ng 3.5 ? >> >> Thanks and regards >> >> >>
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 <(250)%20721-7691> Data Centre Services, University Systems, University of Victoria
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Peter Congratulation ! Yes, the template took most of the time from me as well. Just waiting to check your experiences in the blog. Regards Dwijadas Dey On Thu, Apr 20, 2017 at 4:06 PM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
Just a heads up: I also got it working after a bit of debugging. The problem was, that in the above configuration sample there are spaces in the template. After removing those, it worked. Here is my config:
[root@localhost conf.d]# cat oq.conf rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
template t_csv { template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${ FACILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n"); template_escape(no); };
destination d_osquery_copy { file("/var/log/csv_osquery" template(t_csv)); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_sys); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); }; [root@localhost conf.d]#
I figured it out by installing rsyslog and looking at the differences in the output.
I plan to summarize my experiences in a blog in a week or two.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Evan Your suggestion works flawlessly. The syslog table in OSQUERY gets filled up with logs. The missing part is the rewrite rule r_csv_message. Many many thanks to you.
Regards Dwijadas Dey
On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel@uvic.ca> wrote:
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited.
I would make another file based destination for syslog-ng
destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); };
And add this destination to your log statement.
log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); };
Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community.
One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field.
you can make a rewrite rule for the message
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
then you need to invoke this rewrite rule in your log statement.
log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); };
And finally your template needs to use the CSVMESSAGE rather than the MESSAGE
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
I hope that helps too.
Evan.
On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel@uvic.ca> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag::: csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ------------------------------------------------------------ ---------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC} ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA CILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODAT E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/# rsyslog-versions-7_1
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter < peter.czanik@balabit.com> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
> Hi > Robert > You are right, i am trying the same with a named pipe > so that OSQUERY consume syslogs as pointed by Evan. There are plenty of > documents showing the same with rsyslog but not with syslog-ng. > > This is what my syslog configuration for osquery:- > > /etc/syslog-ng/conf.d/osquery.conf > > source s_osquery { > # system(); > pipe("/var/osquery/syslog_pipe"); > # unix-stream("/dev/log"); > }; > #filter osqueryd { > # program("^osqueryd.*"); > #}; > destination d_osquery { > file("/var/log/osquery/osqueryd.results.log" > template("$(format-json --scope selected_macros --scope nv_pairs)\n")); > }; > log { > source(s_osquery); > # filter(osqueryd); > destination(d_osquery); > }; > > But this does not produce any logs for OSQUERY. I have checked , the > name piped has been created. > > # ls -l /var/osquery/syslog_pipe > pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe > > But when i try to check what logs are passing through the pipe using > following command, no message shows up. > # cat /var/osquery/syslog_pipe > > I have correct options set in OSQUERY configuration file in > /etc/osquery/osquery.conf. > > .................. > .................. > "logger_plugin": "syslog", > "enable_syslog": "true", > "syslog_pipe_path": "/var/osquery/syslog_pipe", > .................. > .................. > I think Evan can point me the right configuration for syslog-ng ( > version 3.5.6 in ubuntu 16 ) > > Regards > > > > > On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < > robert.fekete@balabit.com> wrote: > >> Hi, >> >> It seems that by default, osquery logs JSON messages into a file. >> ( https://osquery.readthedocs.io/en/latest/deployment/logging/ ) >> You can use this file in a syslog-ng source, and parse the JSON >> messages with the json parser (note that you need a recent syslog-ng OSE >> for this), see https://www.balabit.com/docume >> nts/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin >> /html/json-parser.html . >> >> >> The above Osquery page mentions that it can send log messages >> directly to syslog (instead of a file), but I haven't found how you can >> actually configure it. >> >> Regards, >> >> Robert >> >> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com> >> wrote: >> >>> Hi >>> List users >>> Is it possible to send OSQUERY logs to >>> syslog-ng 3.5 In the OSQUERY docs >>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/> >>> rsyslog is configured to write logs to syslog. Does the same method applies >>> to syslog-ng 3.5 ? >>> >>> Thanks and regards >>> >>> >>>
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 <(250)%20721-7691> Data Centre Services, University Systems, University of Victoria
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I should have read the RFC on CSV prior to drafting the format. https://tools.ietf.org/html/rfc4180 Turns out Peter and I both got it wrong. The rewrite should be rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\"\"", value("CSVMESSAGE"), flags(global) ); }; Of course, there are many "interpretations" of CSV so your consumer may not conform to the official standard. On 04/20/2017 03:36 AM, Czanik, Péter wrote:
Hi,
Just a heads up: I also got it working after a bit of debugging. The problem was, that in the above configuration sample there are spaces in the template. After removing those, it worked. Here is my config:
[root@localhost conf.d]# cat oq.conf rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
template t_csv { template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${FACILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n"); template_escape(no); };
destination d_osquery_copy { file("/var/log/csv_osquery" template(t_csv)); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_sys); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); }; [root@localhost conf.d]#
I figured it out by installing rsyslog and looking at the differences in the output.
I plan to summarize my experiences in a blog in a week or two.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad@gmail.com <mailto:dwijad@gmail.com>> wrote:
Hi Evan Your suggestion works flawlessly. The syslog table in OSQUERY gets filled up with logs. The missing part is the rewrite rule r_csv_message. Many many thanks to you.
Regards Dwijadas Dey
On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited.
I would make another file based destination for syslog-ng
destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); };
And add this destination to your log statement.
log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); };
Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community.
One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field.
you can make a rewrite rule for the message
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
then you need to invoke this rewrite rule in your log statement.
log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); };
And finally your template needs to use the CSVMESSAGE rather than the MESSAGE
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
I hope that helps too.
Evan.
On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
> osquery> .schema syslog > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT); No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ---------------------------------------------------------------------------------- source s_osquery { system(); };
template t_csv {
template("'${HOUR}${MIN}${SEC}',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FACILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODATE}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions... <https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1>
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ <https://www.balabit.com/blog/author/peterczanik/> https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey <dwijad@gmail.com <mailto:dwijad@gmail.com>> wrote:
Hi Robert You are right, i am trying the same with a named pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty of documents showing the same with rsyslog but not with syslog-ng.
This is what my syslog configuration for osquery:-
/etc/syslog-ng/conf.d/osquery.conf
source s_osquery { # system(); pipe("/var/osquery/syslog_pipe"); # unix-stream("/dev/log"); }; #filter osqueryd { # program("^osqueryd.*"); #}; destination d_osquery { file("/var/log/osquery/osqueryd.results.log" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; log { source(s_osquery); # filter(osqueryd); destination(d_osquery); };
But this does not produce any logs for OSQUERY. I have checked , the name piped has been created.
# ls -l /var/osquery/syslog_pipe pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe
But when i try to check what logs are passing through the pipe using following command, no message shows up. # cat /var/osquery/syslog_pipe
I have correct options set in OSQUERY configuration file in /etc/osquery/osquery.conf.
.................. .................. "logger_plugin": "syslog", "enable_syslog": "true", "syslog_pipe_path": "/var/osquery/syslog_pipe", .................. .................. I think Evan can point me the right configuration for syslog-ng ( version 3.5.6 in ubuntu 16 )
Regards
On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert <robert.fekete@balabit.com <mailto:robert.fekete@balabit.com>> wrote:
Hi,
It seems that by default, osquery logs JSON messages into a file. ( https://osquery.readthedocs.io/en/latest/deployment/logging/ <https://osquery.readthedocs.io/en/latest/deployment/logging/> ) You can use this file in a syslog-ng source, and parse the JSON messages with the json parser (note that you need a recent syslog-ng OSE for this), see https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html> .
The above Osquery page mentions that it can send log messages directly to syslog (instead of a file), but I haven't found how you can actually configure it.
Regards,
Robert
On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey <dwijad@gmail.com <mailto:dwijad@gmail.com>> wrote:
Hi List users Is it possible to send OSQUERY logs to syslog-ng 3.5 In the OSQUERY docs <https://osquery.readthedocs.io/en/latest/deployment/syslog/> rsyslog is configured to write logs to syslog. Does the same method applies to syslog-ng 3.5 ?
Thanks and regards
Hi Evan Does not matter since your previous suggestion worked in my case ( many interpretation ). Now i can test with your new rewrite rules. Regards On Thu, Apr 20, 2017 at 6:56 PM, Evan Rempel <erempel@uvic.ca> wrote:
I should have read the RFC on CSV prior to drafting the format.
https://tools.ietf.org/html/rfc4180
Turns out Peter and I both got it wrong. The rewrite should be
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\"\"", value("CSVMESSAGE"), flags(global) ); };
Of course, there are many "interpretations" of CSV so your consumer may not conform to the official standard.
On 04/20/2017 03:36 AM, Czanik, Péter wrote:
Hi,
Just a heads up: I also got it working after a bit of debugging. The problem was, that in the above configuration sample there are spaces in the template. After removing those, it worked. Here is my config:
[root@localhost conf.d]# cat oq.conf rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
template t_csv { template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${ FACILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n"); template_escape(no); };
destination d_osquery_copy { file("/var/log/csv_osquery" template(t_csv)); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_sys); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); }; [root@localhost conf.d]#
I figured it out by installing rsyslog and looking at the differences in the output.
I plan to summarize my experiences in a blog in a week or two.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Evan Your suggestion works flawlessly. The syslog table in OSQUERY gets filled up with logs. The missing part is the rewrite rule r_csv_message. Many many thanks to you.
Regards Dwijadas Dey
On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel < <erempel@uvic.ca>
erempel@uvic.ca> wrote:
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited.
I would make another file based destination for syslog-ng
destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); };
And add this destination to your log statement.
log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); };
Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community.
One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field.
you can make a rewrite rule for the message
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
then you need to invoke this rewrite rule in your log statement.
log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); };
And finally your template needs to use the CSVMESSAGE rather than the MESSAGE
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
I hope that helps too.
Evan.
On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel < <erempel@uvic.ca> erempel@uvic.ca> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag::: csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ------------------------------------------------------------ ---------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC} ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA CILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODAT E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
<https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1> https://osquery.readthedocs.io/en/stable/deployment/syslog/# rsyslog-versions-7_1
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter < <peter.czanik@balabit.com>peter.czanik@balabit.com> wrote:
Hi,
What do you try to achieve? Sending syslog messages to OSquery or collecting OSquery logs by syslog-ng?
/me now has a test environment installed
Bye,
Peter Czanik (CzP) < <peter.czanik@balabit.com> peter.czanik@balabit.com> Balabit / syslog-ng upstream <https://www.balabit.com/blog/author/peterczanik/> https://www.balabit.com/blog/author/peterczanik/ <https://twitter.com/PCzanik>https://twitter.com/PCzanik
On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey < <dwijad@gmail.com> dwijad@gmail.com> wrote:
> Hi > Robert > You are right, i am trying the same with a named pipe > so that OSQUERY consume syslogs as pointed by Evan. There are plenty of > documents showing the same with rsyslog but not with syslog-ng. > > This is what my syslog configuration for osquery:- > > /etc/syslog-ng/conf.d/osquery.conf > > source s_osquery { > # system(); > pipe("/var/osquery/syslog_pipe"); > # unix-stream("/dev/log"); > }; > #filter osqueryd { > # program("^osqueryd.*"); > #}; > destination d_osquery { > file("/var/log/osquery/osqueryd.results.log" > template("$(format-json --scope selected_macros --scope nv_pairs)\n")); > }; > log { > source(s_osquery); > # filter(osqueryd); > destination(d_osquery); > }; > > But this does not produce any logs for OSQUERY. I have checked , the > name piped has been created. > > # ls -l /var/osquery/syslog_pipe > pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe > > But when i try to check what logs are passing through the pipe using > following command, no message shows up. > # cat /var/osquery/syslog_pipe > > I have correct options set in OSQUERY configuration file in > /etc/osquery/osquery.conf. > > .................. > .................. > "logger_plugin": "syslog", > "enable_syslog": "true", > "syslog_pipe_path": "/var/osquery/syslog_pipe", > .................. > .................. > I think Evan can point me the right configuration for syslog-ng ( > version 3.5.6 in ubuntu 16 ) > > Regards > > > > > On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < > <robert.fekete@balabit.com>robert.fekete@balabit.com> wrote: > >> Hi, >> >> It seems that by default, osquery logs JSON messages into a file. >> ( <https://osquery.readthedocs.io/en/latest/deployment/logging/> >> https://osquery.readthedocs.io/en/latest/deployment/logging/ ) >> You can use this file in a syslog-ng source, and parse the JSON >> messages with the json parser (note that you need a recent syslog-ng OSE >> for this), see >> <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html> >> https://www.balabit.com/documents/syslog-ng-ose-latest-guide >> s/en/syslog-ng-ose-guide-admin/html/json-parser.html . >> >> >> The above Osquery page mentions that it can send log messages >> directly to syslog (instead of a file), but I haven't found how you can >> actually configure it. >> >> Regards, >> >> Robert >> >> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey < <dwijad@gmail.com> >> dwijad@gmail.com> wrote: >> >>> Hi >>> List users >>> Is it possible to send OSQUERY logs to >>> syslog-ng 3.5 In the OSQUERY docs >>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/> >>> rsyslog is configured to write logs to syslog. Does the same method applies >>> to syslog-ng 3.5 ? >>> >>> Thanks and regards >>> >>> >>>
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, You can read my blog about syslog-ng and osquery at https://www.balabit.com/blog/endpoint-visibility-and-monitoring-using-osquer... Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Thu, Apr 20, 2017 at 3:58 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Evan Does not matter since your previous suggestion worked in my case ( many interpretation ). Now i can test with your new rewrite rules.
Regards
On Thu, Apr 20, 2017 at 6:56 PM, Evan Rempel <erempel@uvic.ca> wrote:
I should have read the RFC on CSV prior to drafting the format.
https://tools.ietf.org/html/rfc4180
Turns out Peter and I both got it wrong. The rewrite should be
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\"\"", value("CSVMESSAGE"), flags(global) ); };
Of course, there are many "interpretations" of CSV so your consumer may not conform to the official standard.
On 04/20/2017 03:36 AM, Czanik, Péter wrote:
Hi,
Just a heads up: I also got it working after a bit of debugging. The problem was, that in the above configuration sample there are spaces in the template. After removing those, it worked. Here is my config:
[root@localhost conf.d]# cat oq.conf rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
template t_csv { template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${FA CILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n"); template_escape(no); };
destination d_osquery_copy { file("/var/log/csv_osquery" template(t_csv)); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_sys); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); }; [root@localhost conf.d]#
I figured it out by installing rsyslog and looking at the differences in the output.
I plan to summarize my experiences in a blog in a week or two.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Evan Your suggestion works flawlessly. The syslog table in OSQUERY gets filled up with logs. The missing part is the rewrite rule r_csv_message. Many many thanks to you.
Regards Dwijadas Dey
On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel < <erempel@uvic.ca>
erempel@uvic.ca> wrote:
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited.
I would make another file based destination for syslog-ng
destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); };
And add this destination to your log statement.
log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); };
Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community.
One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field.
you can make a rewrite rule for the message
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
then you need to invoke this rewrite rule in your log statement.
log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); };
And finally your template needs to use the CSVMESSAGE rather than the MESSAGE
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
I hope that helps too.
Evan.
On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel < <erempel@uvic.ca> erempel@uvic.ca> wrote:
The documentation from OSQuery is for rsyslog and shows that a csv set of values is needed.
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag::: csv%,%msg:::csv%\n"
In syslog-ng this format becomes
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); template_escape(no); };
Give that a try and see how things go.
On 04/18/2017 08:57 AM, Dwijadas Dey wrote:
Hi Peter I am trying to send syslogs to a named pipe and on the other end OSQUERY will consume the syslogs from the named pipe. Once OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes along with OSQUERY. I have been able to send the syslogs to named pipe ( verified with cat command ) but on the other hand OSQUERY did consume the logs but could not send these logs to the table due to format error.
The schema of syslog table in OSQUERY ------------------------------------------------------------ osquery> .schema syslog CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) ------------------------------------------------------------ ---------------------- source s_osquery { system(); };
template t_csv { template("'${HOUR}${MIN}${SEC} ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA CILITY}',\t'${MSG}'\n"); # template("$timestamp\t${ISODAT E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); template_escape(no); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_osquery); destination(d_osquery); };
I am trying to match the above template to rsyslog format for OSQUERY
<https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1> https://osquery.readthedocs.io/en/stable/deployment/syslog/# rsyslog-versions-7_1
If i cat the pipe, i can see the syslogs.
# cat /var/osquery/syslog_pipe
'155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 [preauth]' '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', '55', 'notice', 'authpriv', 'PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 user=root'
The above logs contains exactly 7 fields as required by OSQUERY syslog table as described above.
The error that i am getting at the moment - ------------------------------------------------------------ E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /root/.ssh/** E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /home/*/.ssh/** E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9b', 'err', 'local3', 'severity=2 location=syslog.cpp:173 message=Received more fields than expected in line: ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', 'notice', 'local3', 'severity=0 location=file_events.cpp:68 message=Added file event listener to: /tmp/** I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog run loop terminated for reason: Too many errors in syslog parsing.
I think the issue is with the template definition which needs to match with the template with rsyslog as described in the above link.
I will appreciate if someone can point out the issues in template and how it should be in syslog-ng.
Regards
On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter < <peter.czanik@balabit.com>peter.czanik@balabit.com> wrote:
> Hi, > > What do you try to achieve? Sending syslog messages to OSquery or > collecting OSquery logs by syslog-ng? > > /me now has a test environment installed > > Bye, > > Peter Czanik (CzP) < <peter.czanik@balabit.com> > peter.czanik@balabit.com> > Balabit / syslog-ng upstream > <https://www.balabit.com/blog/author/peterczanik/> > https://www.balabit.com/blog/author/peterczanik/ > <https://twitter.com/PCzanik>https://twitter.com/PCzanik > > On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey < <dwijad@gmail.com> > dwijad@gmail.com> wrote: > >> Hi >> Robert >> You are right, i am trying the same with a named pipe >> so that OSQUERY consume syslogs as pointed by Evan. There are plenty of >> documents showing the same with rsyslog but not with syslog-ng. >> >> This is what my syslog configuration for osquery:- >> >> /etc/syslog-ng/conf.d/osquery.conf >> >> source s_osquery { >> # system(); >> pipe("/var/osquery/syslog_pipe"); >> # unix-stream("/dev/log"); >> }; >> #filter osqueryd { >> # program("^osqueryd.*"); >> #}; >> destination d_osquery { >> file("/var/log/osquery/osqueryd.results.log" >> template("$(format-json --scope selected_macros --scope nv_pairs)\n")); >> }; >> log { >> source(s_osquery); >> # filter(osqueryd); >> destination(d_osquery); >> }; >> >> But this does not produce any logs for OSQUERY. I have checked , >> the name piped has been created. >> >> # ls -l /var/osquery/syslog_pipe >> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe >> >> But when i try to check what logs are passing through the pipe >> using following command, no message shows up. >> # cat /var/osquery/syslog_pipe >> >> I have correct options set in OSQUERY configuration file in >> /etc/osquery/osquery.conf. >> >> .................. >> .................. >> "logger_plugin": "syslog", >> "enable_syslog": "true", >> "syslog_pipe_path": "/var/osquery/syslog_pipe", >> .................. >> .................. >> I think Evan can point me the right configuration for syslog-ng ( >> version 3.5.6 in ubuntu 16 ) >> >> Regards >> >> >> >> >> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < >> <robert.fekete@balabit.com>robert.fekete@balabit.com> wrote: >> >>> Hi, >>> >>> It seems that by default, osquery logs JSON messages into a file. >>> ( <https://osquery.readthedocs.io/en/latest/deployment/logging/> >>> https://osquery.readthedocs.io/en/latest/deployment/logging/ ) >>> You can use this file in a syslog-ng source, and parse the JSON >>> messages with the json parser (note that you need a recent syslog-ng OSE >>> for this), see >>> <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html> >>> https://www.balabit.com/documents/syslog-ng-ose-latest-guide >>> s/en/syslog-ng-ose-guide-admin/html/json-parser.html . >>> >>> >>> The above Osquery page mentions that it can send log messages >>> directly to syslog (instead of a file), but I haven't found how you can >>> actually configure it. >>> >>> Regards, >>> >>> Robert >>> >>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey < <dwijad@gmail.com> >>> dwijad@gmail.com> wrote: >>> >>>> Hi >>>> List users >>>> Is it possible to send OSQUERY logs to >>>> syslog-ng 3.5 In the OSQUERY docs >>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/> >>>> rsyslog is configured to write logs to syslog. Does the same method applies >>>> to syslog-ng 3.5 ? >>>> >>>> Thanks and regards >>>> >>>> >>>>
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Peter Awesome blog post on syslog-ng and osquery....as always Regards On Fri, Apr 28, 2017 at 5:07 PM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi, You can read my blog about syslog-ng and osquery at https://www.balabit.com/blog/endpoint-visibility-and- monitoring-using-osquery-and-syslog-ng/ Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Thu, Apr 20, 2017 at 3:58 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Evan Does not matter since your previous suggestion worked in my case ( many interpretation ). Now i can test with your new rewrite rules.
Regards
On Thu, Apr 20, 2017 at 6:56 PM, Evan Rempel <erempel@uvic.ca> wrote:
I should have read the RFC on CSV prior to drafting the format.
https://tools.ietf.org/html/rfc4180
Turns out Peter and I both got it wrong. The rewrite should be
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\"\"", value("CSVMESSAGE"), flags(global) ); };
Of course, there are many "interpretations" of CSV so your consumer may not conform to the official standard.
On 04/20/2017 03:36 AM, Czanik, Péter wrote:
Hi,
Just a heads up: I also got it working after a bit of debugging. The problem was, that in the above configuration sample there are spaces in the template. After removing those, it worked. Here is my config:
[root@localhost conf.d]# cat oq.conf rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
template t_csv { template("\"${ISODATE}\",\"${HOST}\",\"${LEVEL_NUM}\",\"${FA CILITY}\",\"${PROGRAM}\",\"${CSVMESSAGE}\"\n"); template_escape(no); };
destination d_osquery_copy { file("/var/log/csv_osquery" template(t_csv)); };
destination d_osquery { pipe("/var/osquery/syslog_pipe" template(t_csv)); };
log { source(s_sys); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); }; [root@localhost conf.d]#
I figured it out by installing rsyslog and looking at the differences in the output.
I plan to summarize my experiences in a blog in a week or two.
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik
On Tue, Apr 18, 2017 at 8:58 PM, Dwijadas Dey <dwijad@gmail.com> wrote:
Hi Evan Your suggestion works flawlessly. The syslog table in OSQUERY gets filled up with logs. The missing part is the rewrite rule r_csv_message. Many many thanks to you.
Regards Dwijadas Dey
On Wed, Apr 19, 2017 at 12:06 AM, Evan Rempel < <erempel@uvic.ca>
erempel@uvic.ca> wrote:
The fact that your error "Received more fields than expected" went away implies that the number of fields is correct. Without any errors or any data in the table your trouble shooting options are limited.
I would make another file based destination for syslog-ng
destination d_osquery_copy { file("/var/osquery/syslog" template(t_csv)); };
And add this destination to your log statement.
log { source(s_osquery); destination(d_osquery); destination(d_osquery_copy); };
Then you will have a copy of the data that is being sent to osquery and you should be able to get help from the osquery community.
One other thing to note is that I did not provide you with the correct CSV of the MESSAGE portion. If the $MESSAGE contains double quotes then this will not be a correctly formatted CSV field.
you can make a rewrite rule for the message
rewrite r_csv_message { set("$MESSAGE", value("CSVMESSAGE") ); subst("\"","\\\"", value("CSVMESSAGE"), flags(global) ); };
then you need to invoke this rewrite rule in your log statement.
log { source(s_osquery); rewrite(r_csv_message); destination(d_osquery); destination(d_osquery_copy); };
And finally your template needs to use the CSVMESSAGE rather than the MESSAGE
template t_csv { template("\"${ISODATE}\", \"${HOST}\", \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${CSVMESSAGE}\"\n"); template_escape(no); };
I hope that helps too.
Evan.
On 04/18/2017 10:22 AM, Dwijadas Dey wrote:
Hi Evan Thanks you for a quick reply. After changing the template as suggested by you, the error goes away but the syslog table in OSQUERY does not get filled up. May be the OSQUERY expects 7 entry for the syslog table while the template has six fields.
> osquery> .schema syslog > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` TEXT, > `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT);
No verbose error as well.
Regards
On Tue, Apr 18, 2017 at 9:45 PM, Evan Rempel < <erempel@uvic.ca> erempel@uvic.ca> wrote:
> The documentation from OSQuery is for rsyslog and shows that a csv > set of values is needed. > > string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sys > logseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag::: > csv%,%msg:::csv%\n" > > In syslog-ng this format becomes > > template t_csv { template("\"${ISODATE}\", \"${HOST}\", > \"${LEVEL_NUM}\", \"${FACILITY}\", \"${PROGRAM}\", \"${MESSAGE}\"\n"); > template_escape(no); }; > > Give that a try and see how things go. > > > > On 04/18/2017 08:57 AM, Dwijadas Dey wrote: > > Hi > Peter > I am trying to send syslogs to a named pipe and on the > other end OSQUERY will consume the syslogs from the named pipe. Once > OSQUERY consumes syslogs, it will sends the logs to RocksDB that comes > along with OSQUERY. I have been able to send the syslogs to named pipe ( > verified with cat command ) but on the other hand OSQUERY did consume the > logs but could not send these logs to the table due to format error. > > The schema of syslog table in OSQUERY > ------------------------------------------------------------ > osquery> .schema syslog > CREATE TABLE syslog_events(`time` BIGINT, `datetime` TEXT, `host` > TEXT, `severity` INTEGER, `facility` TEXT, `tag` TEXT, `message` TEXT); > > Conf file in syslog-ng (/etc/syslog-ng/conf.d/osquery.conf) > ------------------------------------------------------------ > ---------------------- > source s_osquery { > system(); > }; > > template t_csv { > template("'${HOUR}${MIN}${SEC} > ',\t'${ISODATE}',\t'${HOST}',\t'${TAG}',\t'${LEVEL}',\t'${FA > CILITY}',\t'${MSG}'\n"); > # template("$timestamp\t${ISODAT > E}\t{$HOST}\t$syslogseverity\t$syslogfacility\t$syslogtag\t$msg\n"); > template_escape(no); > }; > > destination d_osquery { > pipe("/var/osquery/syslog_pipe" template(t_csv)); > }; > > log { > source(s_osquery); > destination(d_osquery); > }; > > I am trying to match the above template to rsyslog format for OSQUERY > > > <https://osquery.readthedocs.io/en/stable/deployment/syslog/#rsyslog-versions-7_1> > https://osquery.readthedocs.io/en/stable/deployment/syslog/# > rsyslog-versions-7_1 > > If i cat the pipe, i can see the syslogs. > > # cat /var/osquery/syslog_pipe > > '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', > '26', 'info', 'auth', 'Disconnected from 61.177.172.51 port 20876 > [preauth]' > '155349', '2017-04-18T15:53:49+00:00', 'ubuntu', > '55', 'notice', 'authpriv', 'PAM 2 more authentication > failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.177.172.51 > user=root' > > > The above logs contains exactly 7 fields as required by OSQUERY > syslog table as described above. > > > The error that i am getting at the moment - > ------------------------------------------------------------ > E0418 15:50:39.131995 4229 syslog.cpp:173] Received more fields > than expected in line: ''154852', '2017-04-18T15:48:52+00:00', > 'ubuntu', '9b', 'err', 'local3', 'severity=2 > location=syslog.cpp:173 message=Received more fields than expected in line: > ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', > 'notice', 'local3', 'severity=0 location=file_events.cpp:68 > message=Added file event listener to: /root/.ssh/** > E0418 15:50:39.132355 4229 syslog.cpp:173] Received more fields > than expected in line: ''154852', '2017-04-18T15:48:52+00:00', > 'ubuntu', '9b', 'err', 'local3', 'severity=2 > location=syslog.cpp:173 message=Received more fields than expected in line: > ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', > 'notice', 'local3', 'severity=0 location=file_events.cpp:68 > message=Added file event listener to: /home/*/.ssh/** > E0418 15:50:39.132758 4229 syslog.cpp:173] Received more fields > than expected in line: ''154852', '2017-04-18T15:48:52+00:00', > 'ubuntu', '9b', 'err', 'local3', 'severity=2 > location=syslog.cpp:173 message=Received more fields than expected in line: > ''154852', '2017-04-18T15:48:52+00:00', 'ubuntu', '9d', > 'notice', 'local3', 'severity=0 location=file_events.cpp:68 > message=Added file event listener to: /tmp/** > I0418 15:50:39.133230 4229 events.cpp:767] Event publisher syslog > run loop terminated for reason: Too many errors in syslog parsing. > > I think the issue is with the template definition which needs to > match with the template with rsyslog as described in the above link. > > I will appreciate if someone can point out the issues in template > and how it should be in syslog-ng. > > > Regards > > > > On Tue, Apr 18, 2017 at 7:12 PM, Czanik, Péter < > <peter.czanik@balabit.com>peter.czanik@balabit.com> wrote: > >> Hi, >> >> What do you try to achieve? Sending syslog messages to OSquery or >> collecting OSquery logs by syslog-ng? >> >> /me now has a test environment installed >> >> Bye, >> >> Peter Czanik (CzP) < <peter.czanik@balabit.com> >> peter.czanik@balabit.com> >> Balabit / syslog-ng upstream >> <https://www.balabit.com/blog/author/peterczanik/> >> https://www.balabit.com/blog/author/peterczanik/ >> <https://twitter.com/PCzanik>https://twitter.com/PCzanik >> >> On Mon, Apr 17, 2017 at 4:32 PM, Dwijadas Dey < <dwijad@gmail.com> >> dwijad@gmail.com> wrote: >> >>> Hi >>> Robert >>> You are right, i am trying the same with a named >>> pipe so that OSQUERY consume syslogs as pointed by Evan. There are plenty >>> of documents showing the same with rsyslog but not with syslog-ng. >>> >>> This is what my syslog configuration for osquery:- >>> >>> /etc/syslog-ng/conf.d/osquery.conf >>> >>> source s_osquery { >>> # system(); >>> pipe("/var/osquery/syslog_pipe"); >>> # unix-stream("/dev/log"); >>> }; >>> #filter osqueryd { >>> # program("^osqueryd.*"); >>> #}; >>> destination d_osquery { >>> file("/var/log/osquery/osqueryd.results.log" >>> template("$(format-json --scope selected_macros --scope nv_pairs)\n")); >>> }; >>> log { >>> source(s_osquery); >>> # filter(osqueryd); >>> destination(d_osquery); >>> }; >>> >>> But this does not produce any logs for OSQUERY. I have checked , >>> the name piped has been created. >>> >>> # ls -l /var/osquery/syslog_pipe >>> pr--rw---- 1 root adm 0 Apr 14 15:41 /var/osquery/syslog_pipe >>> >>> But when i try to check what logs are passing through the pipe >>> using following command, no message shows up. >>> # cat /var/osquery/syslog_pipe >>> >>> I have correct options set in OSQUERY configuration file in >>> /etc/osquery/osquery.conf. >>> >>> .................. >>> .................. >>> "logger_plugin": "syslog", >>> "enable_syslog": "true", >>> "syslog_pipe_path": "/var/osquery/syslog_pipe", >>> .................. >>> .................. >>> I think Evan can point me the right configuration for syslog-ng ( >>> version 3.5.6 in ubuntu 16 ) >>> >>> Regards >>> >>> >>> >>> >>> On Mon, Apr 17, 2017 at 6:24 PM, Fekete, Róbert < >>> <robert.fekete@balabit.com>robert.fekete@balabit.com> wrote: >>> >>>> Hi, >>>> >>>> It seems that by default, osquery logs JSON messages into a >>>> file. ( >>>> <https://osquery.readthedocs.io/en/latest/deployment/logging/> >>>> https://osquery.readthedocs.io/en/latest/deployment/logging/ ) >>>> You can use this file in a syslog-ng source, and parse the JSON >>>> messages with the json parser (note that you need a recent syslog-ng OSE >>>> for this), see >>>> <https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/json-parser.html> >>>> https://www.balabit.com/documents/syslog-ng-ose-latest-guide >>>> s/en/syslog-ng-ose-guide-admin/html/json-parser.html . >>>> >>>> >>>> The above Osquery page mentions that it can send log messages >>>> directly to syslog (instead of a file), but I haven't found how you can >>>> actually configure it. >>>> >>>> Regards, >>>> >>>> Robert >>>> >>>> On Fri, Apr 14, 2017 at 9:46 PM, Dwijadas Dey < >>>> <dwijad@gmail.com>dwijad@gmail.com> wrote: >>>> >>>>> Hi >>>>> List users >>>>> Is it possible to send OSQUERY logs to >>>>> syslog-ng 3.5 In the OSQUERY docs >>>>> <https://osquery.readthedocs.io/en/latest/deployment/syslog/> >>>>> rsyslog is configured to write logs to syslog. Does the same method applies >>>>> to syslog-ng 3.5 ? >>>>> >>>>> Thanks and regards >>>>> >>>>> >>>>> >
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Czanik, Péter
-
Dwijadas Dey
-
Evan Rempel
-
Fekete, Róbert