Trying to get json out of 3.7beta2
Hi Thanks to various folks I managed to build 3.7b2 with json. I am now trying to convert my ELSA config to produce a parsed output in json that I can feed into Elastic Search. Having failed to make my full config work (I had to try ;) so I tried a basic one based on https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g... [ Aside: minor syntax error on this page — lnside log {} you can not name parser elements ] here is my conf: @version: 3.7 source s_json { network(port(1514) flags(no-parse)); }; destination d_json { file("/data/russell/test.json” template("$(format-json --scope dot-nv-pairs)\n")); }; log { source(s_json); parser { json-parser (prefix(".json.")); }; destination(d_json); }; I get no output and ‘stats’ shows: [rful011@secmgrprd01 ~]$ sudo /usr/local/syslog-ng/sbin/syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number src.none;;;a;processed;0 src.none;;;a;stamp;0 source;s_json;;a;processed;19375 global;payload_reallocs;;a;processed;25710 global;msg_clones;;a;processed;0 destination;d_json;;a;processed;0 center;;queued;a;processed;0 global;sdata_updates;;a;processed;0 center;;received;a;processed;19375 global;internal_queue_length;;a;processed;19378 Which is the same as I get with my full config with lots of patterns. As usual am missing something basic!
Seems that jsonparser drops messages for some reason. I don't have the code handy but perhaps you could check the error cases. It should report failures though. On Jul 6, 2015 6:12 AM, "Russell Fulton" <r.fulton@auckland.ac.nz> wrote:
Hi
Thanks to various folks I managed to build 3.7b2 with json.
I am now trying to convert my ELSA config to produce a parsed output in json that I can feed into Elastic Search.
Having failed to make my full config work (I had to try ;) so I tried a basic one based on
https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g...
[ Aside: minor syntax error on this page — lnside log {} you can not name parser elements ]
here is my conf:
@version: 3.7
source s_json { network(port(1514) flags(no-parse)); };
destination d_json { file("/data/russell/test.json” template("$(format-json --scope dot-nv-pairs)\n")); };
log { source(s_json); parser { json-parser (prefix(".json.")); }; destination(d_json); };
I get no output and ‘stats’ shows:
[rful011@secmgrprd01 ~]$ sudo /usr/local/syslog-ng/sbin/syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number src.none;;;a;processed;0 src.none;;;a;stamp;0 source;s_json;;a;processed;19375 global;payload_reallocs;;a;processed;25710 global;msg_clones;;a;processed;0 destination;d_json;;a;processed;0 center;;queued;a;processed;0 global;sdata_updates;;a;processed;0 center;;received;a;processed;19375 global;internal_queue_length;;a;processed;19378
Which is the same as I get with my full config with lots of patterns.
As usual am missing something basic!
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 7 Jul 2015, at 08:57, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
Seems that jsonparser drops messages for some reason. I don't have the code handy but perhaps you could check the error cases.
sorry, not sure what you mean by 'check the error cases’. There is nothing I can find in /var/log BTW This is a corporate managed RHEL box that runs rsyslog for ‘normal’ logging. I am running syslog-ng listening on non standard ports as I have done on other systems.
It should report failures though.
Let me know if there is anything I can do to help diagnose this. I am keen to get this going. Is there any docs apart from the section in the user manual? If anyone can point me to a non trivial example (including using pattern parsing) that would be really helpful. Russell
Hi, I meant the c source code for json-parser(), which is modules/json/json-parser.c, more specifically the json_parser_process() function: ``` static gboolean json_parser_process(LogParser *s, LogMessage **pmsg, const LogPathOptions *path_options, const gchar *input, gsize input_len) { JSONParser *self = (JSONParser *) s; struct json_object *jso; struct json_tokener *tok; if (self->marker) { if (strncmp(input, self->marker, self->marker_len) != 0) return FALSE; input += self->marker_len; while (isspace(*input)) input++; } tok = json_tokener_new(); jso = json_tokener_parse_ex(tok, input, input_len); if (tok->err != json_tokener_success || !jso) { msg_error("Unparsable JSON stream encountered", evt_tag_str ("input", input), tok->err != json_tokener_success ? evt_tag_str ("error", json_tokener_error_desc(tok->err)) : NULL, NULL); json_tokener_free (tok); return FALSE; } json_tokener_free(tok); log_msg_make_writable(pmsg, path_options); if (!json_parser_extract(self, jso, *pmsg)) { msg_error("Error extracting JSON members into LogMessage as the top-level JSON object is not an object", evt_tag_str ("input", input), NULL); json_object_put(jso); return FALSE; } json_object_put(jso); return TRUE; } ``` As it seems, there are three cases where json-parser() drops messages: 1) a marker() option is specified and the message doesn't start with that value 2) json parse error, in which case an error is printed 3) if the top-level json object in the input is not an Object (but a list or a single value), but again an error message is printed in this case too I didn't see the marker option in your configuration, so either of the error cases should be logged as an internal message, at the error level. Can you check that? Cheers, Bazsi On Tue, Jul 7, 2015 at 8:48 PM, Russell Fulton <r.fulton@auckland.ac.nz> wrote:
On 7 Jul 2015, at 08:57, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
Seems that jsonparser drops messages for some reason. I don't have the code handy but perhaps you could check the error cases.
sorry, not sure what you mean by 'check the error cases’. There is nothing I can find in /var/log
BTW This is a corporate managed RHEL box that runs rsyslog for ‘normal’ logging. I am running syslog-ng listening on non standard ports as I have done on other systems.
It should report failures though.
Let me know if there is anything I can do to help diagnose this. I am keen to get this going.
Is there any docs apart from the section in the user manual? If anyone can point me to a non trivial example (including using pattern parsing) that would be really helpful.
Russell
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
As it seems, there are three cases where json-parser() drops messages: 1) a marker() option is specified and the message doesn't start with that value 2) json parse error, in which case an error is printed 3) if the top-level json object in the input is not an Object (but a list or a single value), but again an error message is printed in this case too
I didn't see the marker option in your configuration, so either of the error cases should be logged as an internal message, at the error level.
Where should the errors get logged to? I have been looking for anything in /var/log/. Do I need to add an log{} entry for the errors? BTW This is a corporate managed RHEL box that runs rsyslog for ‘normal’ logging. I am running syslog-ng listening on non standard ports as I have done on other systems. [rful011@secmgrprd01 ~]$ sudo grep syslog /var/log/messages Jul 6 12:04:15 secmgrprd01 syslog-ng[24493]: syslog-ng starting up; version='3.7.0beta2' Jul 6 12:05:08 secmgrprd01 sudo[24497]: rful011 : TTY=pts/0 ; PWD=/usr/local ; USER=root ; COMMAND=/sbin/service syslog-ng stop Jul 8 08:43:50 secmgrprd01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1523" x-info="http://www.rsyslog.com"] exiting on signal 15. Jul 8 08:46:04 secmgrprd01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="12946" x-info="http://www.rsyslog.com"] start Jul 8 08:52:27 secmgrprd01 supervise/syslog-ng[12924]: Daemon was killed, not restarting; exitcode=‘9’ What is odd is the I did not get start messages for the most recent restart and I am sure that it has been restarted many more times than twice in the last couple of days. Russell
OK I now know what is going on. Firstly in my minimalist config I did not have a log destination for internal() so I was not getting the log messages. Secondly I had misunderstood what that there are two parts to the json support: Input and output. All I wanted was output but I specified input as well so it barfed. This became clear as soon as I had the error messages :) After a few hours fiddling I now have a nice json file with my parsed syslog records! Elastic Search here we come Thanks for your help and patience! :) Russell
Hi, On 07/08/2015 06:32 AM, Russell Fulton wrote:
After a few hours fiddling I now have a nice json file with my parsed syslog records! Elastic Search here we come
At the beginning of the thread you mentioned, that you are converting a syslog-ng configuration from ELSA. Could you share your configuration here? Of course only after removing any sensitive information from it. I'm at a conference right now ( https://2015.rmll.info/ ) and some people were asking just for this. Also: the pattern database coming with ELSA uses s1, s1, i1, i2, etc. for naming value pairs and the real names are in the MySQL database. Do you have a workaround for this? Bye, -- Peter Czanik (CzP) <peter.czanik@balabit.com> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
Hi, My initial thought is that your pattern matching is simply failing. I had an issue where an oddly formatted log source was not populating the program part of the syslog message correctly, causing syslog-ng to not match what I thought it should. You might try using some of the built-in macros (without json initially) to see what is getting parsed into which macros currently. You can play with this a bit to figure out what is happening. One way is to define templates in a test destination, and then you can see what is being parsed. Here is an example where I was unsure of what was being parsed into the date destination d_local_template { file("/var/log/template_messages" template("date ${ISODATE} host ${HOST} program ${PROGRAM} message ${MESSAGE}\n") ); }; Please look at the admin guide for other macros you can play with, but this should give you some basic tools to work with. Also - Have you looked at patterndb ? Here are a couple snippets that are working well for me: parser p_proxy { db-parser(file("/usr/local/etc/patterndb.d/proxy.xml")); }; destination d_redis { redis ( host("localhost") command("LPUSH", "logstash", "$(format-json type=proxyproxy_time=${PROXY.TIME} proxy_time_taken=${PROXY.TIME_TAKEN} proxy_c_ip=${PROXY.C_IP} proxy_sc_status=${PROXY.SC_STATUS} proxy_s_action=${PROXY.S_ACTION} proxy_sc_bytes=int64(${PROXY.SC_BYTES}) proxy_cs_bytes=int64(${PROXY.CS_BYTES}) proxy_cs_method=${PROXY.CS_METHOD} proxy_cs_uri_scheme=${PROXY.CS_URI_SCHEME} proxy_cs_host=${PROXY.CS_HOST} proxy_cs_uri_port=${PROXY.CS_URI_PORT} proxy_cs_uri_path=${PROXY.CS_URI_PATH} proxy_cs_uri_equery=${PROXY.CS_URI_EQUERY} proxy_cs_username=${PROXY.CS_USERNAME} proxy_cs_auth_group=${PROXY.CS_AUTH__GROUP} proxy_s_supplier_name=${PROXY.S_SUPPLIER_NAME} proxy_content_type=${PROXY.CONTENT_TYPE} proxy_referrer=${PROXY.REFERRER} proxy_user_agent=${PROXY.USER_AGENT} proxy_filter_result=${PROXY.FILTER_RESULT} proxy_cs_categories=${PROXY.CS_CATEGORIES} proxy_x_virus_id=${PROXY.X_VIRUS_ID} proxy_s_ip=${PROXY.S_IP} proxy_any=${PROXY.ANYREST})\n") ); }; log { source(s_network); parser(p_proxy); destination(d_redis); }; And from redis I (currently) use logstash to pull events and feed elasticsearch (acting as a very fast buffer between syslog-ng and elasticsearch). Hope this helps. Jim ---- Russell Fulton <r.fulton@auckland.ac.nz> wrote:
Hi
Thanks to various folks I managed to build 3.7b2 with json. I am now trying to convert my ELSA config to produce a parsed output in json that I can feed into Elastic Search. Having failed to make my full config work (I had to try ;) so I tried a basic one based on https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-g... [ Aside: minor syntax error on this page — lnside log {} you can not name parser elements ] here is my conf: @version: 3.7 source s_json { network(port(1514) flags(no-parse)); }; destination d_json { file("/data/russell/test.json” template("$(format-json --scope dot-nv-pairs)\n")); }; log { source(s_json); parser { json-parser (prefix(".json.")); }; destination(d_json); }; I get no output and ‘stats’ shows: [rful011@secmgrprd01 ~]$ sudo /usr/local/syslog-ng/sbin/syslog-ng-ctl stats SourceName;SourceId;SourceInstance;State;Type;Number src.none;;;a;processed;0 src.none;;;a;stamp;0 source;s_json;;a;processed;19375 global;payload_reallocs;;a;processed;25710 global;msg_clones;;a;processed;0 destination;d_json;;a;processed;0 center;;queued;a;processed;0 global;sdata_updates;;a;processed;0 center;;received;a;processed;19375 global;internal_queue_length;;a;processed;19378 Which is the same as I get with my full config with lots of patterns. As usual am missing something basic! ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (5)
-
Balazs Scheidler
-
jrhendri＠roadrunner.com
-
Peter Czanik
-
Russell Fulton
-
Scheidler, Balázs