<div dir="ltr"><div>Found this Gem! <a href="https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#json-parser-options">https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#json-parser-options</a></div><div><table border="0" class="gmail-admon" summary="Example: Convert logstash eventlog format v0 to v1" style="border-collapse:collapse;padding:5px 5px 5px 0px;width:589.906px;max-width:95%;display:block;margin-left:0em;font-family:"droid sans",verdana,helvetica,sans-serif;border:inherit;background-image:inherit;background-position:inherit;background-size:inherit;background-repeat:inherit;background-origin:inherit;background-clip:inherit;color:inherit"><tbody style="width:560px"><tr style="width:560px"><th class="gmail-admonheader" align="left" style="border-collapse:collapse;padding:0.5em 0px 0.5em 1.5em;font-size:0.8em;text-align:left;line-height:1.5em;width:540.812px;border:0px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;color:rgb(29,89,135)">Example 12.8. Convert logstash eventlog format v0 to v1</th></tr><tr style="width:560px"><td class="gmail-admontext" align="left" valign="top" style="font-size:0.8em;border-collapse:collapse;padding:0px 1em 0.5em 1.5em;display:block;max-width:95%;line-height:1.5em;width:532px;border:0px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;color:rgb(0,0,0)"><p style="word-wrap:break-word;list-style:square;margin-top:0.5em;margin-bottom:0em">The following parser converts messages in the logstash eventlog v0 format to the v1 format</p><p style="word-wrap:break-word;list-style:square;margin-top:0.5em;margin-bottom:0em"></p><div class="gmail-simplesect" style="color:rgb(0,0,0);font-family:"droid sans",verdana,helvetica,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div class="gmail-titlepage" style="max-width:99%;margin:3em auto 0em;width:640px;color:black;line-height:1.5"><div><div></div></div></div></div><p></p><div class="gmail-simplesect" style="color:rgb(0,0,0);font-family:"droid sans",verdana,helvetica,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div class="example" style="margin-top:2em;margin-bottom:2em;background:inherit;color:inherit;border:inherit"><table border="0" class="gmail-admon" summary="Example: Convert logstash eventlog format v0 to v1" style="border-collapse:collapse;padding:5px 5px 5px 0px;width:589.906px;max-width:95%;display:block;margin-left:0em;border:inherit;background:inherit;color:inherit"><tbody style="width:560px"><tr style="width:560px"><td class="gmail-admontext" align="left" valign="top" style="font-size:0.8em;border-collapse:collapse;padding:0px 1em 0.5em 1.5em;display:block;max-width:95%;line-height:1.5em;width:532px;border:0px;background:rgb(255,255,255);color:rgb(0,0,0)"><pre class="gmail-prettyprint gmail-synopsis gmail-prettyprinted" style="overflow:auto;white-space:pre-wrap;word-wrap:normal;font-family:courier,fixed;font-size:0.8em;background-color:rgb(224,224,224);color:rgb(0,0,0);padding:2px;border:1px solid rgb(136,136,136)"><span class="gmail-pln" style="color:rgb(0,0,0)">parser p_jsoneventv0 </span><span class="gmail-pun" style="color:rgb(102,102,0)">{</span><span class="gmail-pln" style="color:rgb(0,0,0)">
channel </span><span class="gmail-pun" style="color:rgb(102,102,0)">{</span><span class="gmail-pln" style="color:rgb(0,0,0)">
parser </span><span class="gmail-pun" style="color:rgb(102,102,0)">{</span><span class="gmail-pln" style="color:rgb(0,0,0)"> json</span><span class="gmail-pun" style="color:rgb(102,102,0)">-</span><span class="gmail-pln" style="color:rgb(0,0,0)">parser</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-pln" style="color:rgb(0,0,0)">extract</span><span class="gmail-pun" style="color:rgb(102,102,0)">-</span><span class="gmail-pln" style="color:rgb(0,0,0)">prefix</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-str" style="color:rgb(0,136,0)">"@fields"</span><span class="gmail-pun" style="color:rgb(102,102,0)">));</span><span class="gmail-pln" style="color:rgb(0,0,0)"> </span><span class="gmail-pun" style="color:rgb(102,102,0)">};</span><span class="gmail-pln" style="color:rgb(0,0,0)">
parser </span><span class="gmail-pun" style="color:rgb(102,102,0)">{</span><span class="gmail-pln" style="color:rgb(0,0,0)"> json</span><span class="gmail-pun" style="color:rgb(102,102,0)">-</span><span class="gmail-pln" style="color:rgb(0,0,0)">parser</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-pln" style="color:rgb(0,0,0)">prefix</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-str" style="color:rgb(0,136,0)">".json."</span><span class="gmail-pun" style="color:rgb(102,102,0)">));</span><span class="gmail-pln" style="color:rgb(0,0,0)"> </span><span class="gmail-pun" style="color:rgb(102,102,0)">};</span><span class="gmail-pln" style="color:rgb(0,0,0)">
rewrite </span><span class="gmail-pun" style="color:rgb(102,102,0)">{</span><span class="gmail-pln" style="color:rgb(0,0,0)">
</span><span class="gmail-kwd" style="color:rgb(0,0,136)">set</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-str" style="color:rgb(0,136,0)">"1"</span><span class="gmail-pln" style="color:rgb(0,0,0)"> value</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-str" style="color:rgb(0,136,0)">"@version"</span><span class="gmail-pun" style="color:rgb(102,102,0)">));</span><span class="gmail-pln" style="color:rgb(0,0,0)">
</span><span class="gmail-kwd" style="color:rgb(0,0,136)">set</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-str" style="color:rgb(0,136,0)">"${.json.@timestamp}"</span><span class="gmail-pln" style="color:rgb(0,0,0)"> value</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-str" style="color:rgb(0,136,0)">"@timestamp"</span><span class="gmail-pun" style="color:rgb(102,102,0)">));</span><span class="gmail-pln" style="color:rgb(0,0,0)">
</span><span class="gmail-kwd" style="color:rgb(0,0,136)">set</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-str" style="color:rgb(0,136,0)">"${.json.@message}"</span><span class="gmail-pln" style="color:rgb(0,0,0)"> value</span><span class="gmail-pun" style="color:rgb(102,102,0)">(</span><span class="gmail-str" style="color:rgb(0,136,0)">"message"</span><span class="gmail-pun" style="color:rgb(102,102,0)">));</span><span class="gmail-pln" style="color:rgb(0,0,0)">
</span><span class="gmail-pun" style="color:rgb(102,102,0)">};</span><span class="gmail-pln" style="color:rgb(0,0,0)">
</span><span class="gmail-pun" style="color:rgb(102,102,0)">};</span><span class="gmail-pln" style="color:rgb(0,0,0)">
</span><span class="gmail-pun" style="color:rgb(102,102,0)">};</span></pre></td></tr></tbody></table></div></div></td></tr></tbody></table></div><div>Added parser and updated log statement. </div><div><div>source s_BEATS<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>{network(port(5140) log-msg-size(65536) flags(no-parse));};</div><div>destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));};<br></div><div>log { source(s_BEATS); parser(p_jsoneventv0); destination (d_jfile); };</div></div><div><br></div><div><br></div><div><br></div><div>[2017-05-11T21:17:19.433352] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":6000},"monitor":"http@<a href="http://192.168.1.16:9200">http://192.168.1.16:9200</a>","type":"http","http_rtt":{"us":2000},"url":"<a href="http://192.168.1.16:9200">http://192.168.1.16:9200</a>","tags":["beats_input_raw_event"],"duration":{"us":8000},"@timestamp":"2017-05-12T01:17:23.258Z","rtt":{"us":8000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}'</div><div>[2017-05-11T21:17:19.433471] <b>Error extracting JSON members into LogMessage as the top-level JSON object is not an objec</b>t; input='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":6000},"monitor":"http@<a href="http://192.168.1.16:9200">http://192.168.1.16:9200</a>","type":"http","http_rtt":{"us":2000},"url":"<a href="http://192.168.1.16:9200">http://192.168.1.16:9200</a>","tags":["beats_input_raw_event"],"duration":{"us":8000},"@timestamp":"2017-05-12T01:17:23.258Z","rtt":{"us":8000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}'</div><div>[2017-05-11T21:17:19.433495] Message parsing complete; result='0', rule='p_jsoneventv0', location='/etc/syslog-ng/syslog-ng.conf:18:14'</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 11, 2017 at 8:51 PM, Scot <span dir="ltr"><<a href="mailto:scotrn@gmail.com" target="_blank">scotrn@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm almost there I think! <div>An idea why is outputting message ={} nothing ? What is rule='#anon-parser0<div><br><div><b>relative lines from syslog-ng.conf</b></div><div>source s_BEATS<span class="m_8147996228759702976gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>{network(port(<b>5140</b>) log-msg-size(65536) flags(no-parse));};</div><div>destination d_jfile { file("/opt/syslog-ng/logs/$<wbr>HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));};<br></div><div>log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };</div><div><br></div><div><div><b><i>/etc/logstash/conf.d/logstash.<wbr>conf</i></b></div><div>input {</div><div> beats {</div><div> port => 5044</div><div> }</div><div>}</div><div><br></div><div>output {</div><div> tcp {</div><div> host => "192.168.1.16"</div><div> port => "<b>5140</b>"</div><div> mode => "client"</div><div> }<br></div></div><div><br><div><br></div><div><b><u>Running syslog-ng in foreground. </u></b></div><div><br></div><div><div>[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(<a href="http://192.168.1.16">192.168.1.16</a>:<wbr>60660)', local='AF_INET(<a href="http://0.0.0.0:5140" target="_blank">0.0.0.0:5140</a>)'</div><div>[2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"<wbr>192.168.1.16","tcp_connect_<wbr>rtt":{"us":2000},"monitor":"<wbr>http@<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:9200</a>"<wbr>,"type":"http","http_rtt":{"<wbr>us":2000},"url":"<a href="http://192.168.1.16:9200" target="_blank">http://192.<wbr>168.1.16:9200</a>","tags":["beats_<wbr>input_raw_event"],"duration":{<wbr>"us":5000},"@timestamp":"2017-<wbr>05-12T00:30:32.020Z","rtt":{"<wbr>us":5000},"port":9200,"<wbr>response":{"status":200},"<wbr>beat":{"hostname":"TYLER-<wbr>LAPTOP","name":"TYLER-LAPTOP",<wbr>"version":"5.4.0"},"@version":<wbr>"1","host":"TYLER-LAPTOP","up"<wbr>:true}{"scheme":"http","ip":"<wbr>192.168.1.16","tcp_connect_<wbr>rtt":{"us":2000},"monitor":"<wbr>http@<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:9200</a>"<wbr>,"type":"http","http_rtt":{"<wbr>us":5000},"url":"<a href="http://192.168.1.16:9200" target="_blank">http://192.<wbr>168.1.16:9200</a>","tags":["beats_<wbr>input_raw_event"],"duration":{<wbr>"us":7000},"rtt":{"us":7000},"<wbr>@timestamp":"2017-05-12T00:30:<wbr>42.020Z","port":9200,"<wbr>response":{"status":200},"<wbr>beat":{"hostname":"TYLER-<wbr>LAPTOP","name":"TYLER-LAPTOP",<wbr>"version":"5.4.0"},"@version":<wbr>"1","host":"TYLER-LAPTOP","up"<wbr>:true}{"scheme":"http","ip":"<wbr>192.168.1.16","tcp_connect_<wbr>rtt":{"us":2000},"monitor":"<wbr>http@<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:9200</a>"<wbr>,"type":"'</div><div><br></div><div>[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/<wbr>syslog-ng.conf:60:33'</div><div>[2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(<a href="http://192.168.1.16">192.168.1.16</a>:<wbr>60660)', local='AF_INET(<a href="http://0.0.0.0:5140" target="_blank">0.0.0.0:5140</a>)'</div><div>[2017-05-11T20:38:49.900384] Outgoing message; message='{}</div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_8147996228759702976m_4754309162587128553moz-cite-prefix">Since you already have the no-parse
flag on the source, everything goes into $MESSAGE<br>
<br>
Make a file destination with the template of "$MESSAGE\n" and it
should contain the entire payload.<span class="m_8147996228759702976HOEnZb"><font color="#888888"><br>
<br>
Evan.</font></span><div><div class="m_8147996228759702976h5"><br>
<br>
On 05/10/2017 12:57 PM, Scot wrote:<br>
</div></div></div><div><div class="m_8147996228759702976h5">
<blockquote type="cite">
<div dir="ltr">
<div>Thanks Evan, </div>
<div><br>
</div>
<div>Bumped it up to 32768 </div>
<div><br>
</div>
Error extracting JSON members into LogMessage as the top-level
JSON object is not an object; input='":"A <br>
<div>I think there may be something else I need to do with the
payload. </div>
<div><br>
</div>
<div>How would I dump everything to a file to look at it ? </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 10, 2017 at 2:10 PM, Evan
Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_8147996228759702976m_4754309162587128553m_2935620687240762513moz-cite-prefix">looks
like you might be running into the maximum message size.<br>
Try setting the syslog-ng configuration item<br>
<br>
log_msg_size(64K);
<div>
<div class="m_8147996228759702976m_4754309162587128553h5"><br>
<br>
<br>
On 05/10/2017 10:50 AM, Scot wrote:<br>
</div>
</div>
</div>
<div>
<div class="m_8147996228759702976m_4754309162587128553h5">
<blockquote type="cite">
<div dir="ltr">Using a RAW TCP seems to be loosing
some of the beats header data and messages are
getting concatenated.
<div>
<div>Trying different options but I'm fumbling. <br>
</div>
<div><br>
</div>
<div>
<div> syslog-ng[4596]: Unparsable JSON stream
encountered; input='=net"},"message":"Synch<wbr>ronization
of a replica of an Active Directory naming
context has begun.\n\nDestination
DRA:\tCN=NTDS Settings,CN=...blaaa"</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>source s_BEATS {network(port(5140)
flags(no-parse));}</div>
<div>
<div>parser p_json {</div>
<div> json-parser (prefix(".json."));</div>
<div>};</div>
</div>
<div>log { source(s_BEATS); parser(p_json);
destination (d_file); };<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Anyone have a howto or blog for using
syslog-ng with json inputs ? </div>
<div>I'm looking at the syslog-ng-ose-latest-guide<wbr>s
but it's hard to put all the input output and
parser requirements together. </div>
<div><br>
</div>
<div>Trying to get here </div>
<div>winlogbeat->syslog-ng->ES </div>
<div>winlogbeat->syslog-ng->SPLUNKF<wbr>orwader</div>
<div>winlogbeat->syslog-ng->/opt/sy<wbr>slog-ng/logs/$FROM_HOST.json <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">or </div>
<div class="gmail_quote">
<div>winlogbeat->logstash->syslog-n<wbr>g->ES
</div>
<div>...</div>
<div><br>
</div>
</div>
<div class="gmail_quote">On Tue, May 9, 2017
at 3:27 AM, Fabien Wernli <span dir="ltr"><<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<span><br>
On Mon, May 08, 2017 at 11:30:14PM
+0000, Scot wrote:<br>
> I'm trying to find a solution that
will let me mirror my beats data like<br>
> syslog-ng lets me do with syslog
traffic.<br>
<br>
</span>As far as I know those tools simply
send the data over TCP in JSON format.<br>
If you just need to do routing using
syslog-ng, you can simply use network<br>
source with flags(no-parse). If you need
to process the data using<br>
syslog-ng, you'll also need the
json-parser().<br>
<br>
Cheers<br>
<br>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</div></div></div>
<br></div></div><span class="">______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></span></blockquote></div><br></div>
</blockquote></div><br></div>