<div dir="ltr">I think it's working with kv-parser!! <div>#</div><div>source s_BEATS          {network(port(5140) log-msg-size(65536) flags(no-parse));};</div><div>#</div><div># Gives Outgoing message; message='{}<br><div><div># destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; </div><div>#</div><div>destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"); };</div></div></div><div>#<br>#</div><div>log { source(s_BEATS);  parser {kv-parser();}; destination (d_jfile); };<br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 11, 2017 at 9:44 PM, Scot <span dir="ltr"><<a href="mailto:scotrn@gmail.com" target="_blank">scotrn@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Comparing the output from running syslog in --debug -F  I can see the payload of the json before and after parsing.  They look the same.  I think it's the json parsing on the receiving side. I added an option to logstash to force one line per event.  codec => "json_lines"<div><div><div><br></div><div>This is the output from logstash writing to local file. <br>{"scheme":"http","ip":"192.<wbr>168.1.16","tcp_connect_rtt":{"<wbr>us":5000},"monitor":"http@<a href="http://192.168.1.16:9200" target="_blank">http<wbr>://192.168.1.16:9200</a>","type":"<wbr>http","http_rtt":{"us":8000},"<wbr>url":"<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:9200</a><wbr>","tags":["beats_input_raw_<wbr>event"],"duration":{"us":<wbr>14000},"@timestamp":"2017-05-<wbr>12T01:32:13.258Z","rtt":{"us":<wbr>14000},"port":9200,"response":<wbr>{"status":200},"beat":{"<wbr>hostname":"TYLER-LAPTOP","<wbr>name":"TYLER-LAPTOP","version"<wbr>:"5.4.0"},"@version":"1","<wbr>host":"TYLER-LAPTOP","up":<wbr>true}<br></div><div><br></div><div><br></div><div>This is the stdout from syslog-ng.  Looks like everything is there its just in there parsing.  </div><div><b>Incoming log entry; line='</b></div><div>{"scheme":"http","ip":"192.<wbr>168.1.16","tcp_connect_rtt":{"<wbr>us":5000},"monitor":"http@<a href="http://192.168.1.16:9200" target="_blank">http<wbr>://192.168.1.16:9200</a>","type":"<wbr>http","http_rtt":{"us":4000},"<wbr>url":"<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:9200</a><wbr>","tags":["beats_input_raw_<wbr>event"],"duration":{"us":9000}<wbr>,"rtt":{"us":9000},"@<wbr>timestamp":"2017-05-12T01:31:<wbr>39.258Z","port":9200,"<wbr>response":{"status":200},"<wbr>beat":{"hostname":"TYLER-<wbr>LAPTOP","name":"TYLER-LAPTOP",<wbr>"version":"5.4.0"},"@version":<wbr>"1","host":"TYLER-LAPTOP","up"<wbr>:true}'<br></div><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, May 11, 2017 at 9:28 PM, Evan Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <div class="m_-7503688148693866562m_7271332207509791338moz-cite-prefix">What ever is feeding this source
      appears to be truncating the message to 1024 characters. I assume
      it is some kind of syslog stream because 1024 is the syslog limit
      for entire message size.<br>
      <br>
      I think you need to adjust your configuration of the originating
      software.<span class="m_-7503688148693866562HOEnZb"><font color="#888888"><br>
      <br>
      Evan.</font></span><div><div class="m_-7503688148693866562h5"><br>
      <br>
      <br>
      On 05/11/2017 05:51 PM, Scot wrote:<br>
    </div></div></div><div><div class="m_-7503688148693866562h5">
    <blockquote type="cite">
      
      <div dir="ltr">I'm almost there I think! 
        <div>An idea why is outputting message ={} nothing ?  What is
          rule='#anon-parser0
          <div><br>
            <div><b>relative lines from syslog-ng.conf</b></div>
            <div>source s_BEATS<span class="m_-7503688148693866562m_7271332207509791338gmail-Apple-tab-span" style="white-space:pre-wrap">              </span>{network(port(<b>5140</b>)
              log-msg-size(65536) flags(no-parse));};</div>
            <div>destination d_jfile {
              file("/opt/syslog-ng/logs/$HOS<wbr>T_FROM-$R_HOUR.json"
              template("$(format-json --scope dot-nv-pairs)\n"));};<br>
            </div>
            <div>log { source(s_BEATS);  parser {json-parser();};
              destination (d_jfile); };</div>
            <div><br>
            </div>
            <div>
              <div><b><i>/etc/logstash/conf.d/logstash.<wbr>conf</i></b></div>
              <div>input {</div>
              <div>  beats {</div>
              <div>    port => 5044</div>
              <div>  }</div>
              <div>}</div>
              <div><br>
              </div>
              <div>output {</div>
              <div>  tcp {</div>
              <div>    host => "192.168.1.16"</div>
              <div>    port => "<b>5140</b>"</div>
              <div>    mode => "client"</div>
              <div>  }<br>
              </div>
            </div>
            <div><br>
              <div><br>
              </div>
              <div><b><u>Running syslog-ng in foreground.  </u></b></div>
              <div><br>
              </div>
              <div>
                <div>[2017-05-11T20:30:53.529215] Syslog connection
                  accepted; fd='88',
                  client='AF_INET(<a href="http://192.168.1.16" target="_blank">192.168.1.16</a>:6<wbr>0660)', local='AF_INET(<a href="http://0.0.0.0:5140" target="_blank">0.0.0.0:5140</a>)'</div>
                <div>[2017-05-11T20:38:49.899997] Incoming log entry;
line='{"scheme":"http","ip":"1<wbr>92.168.1.16","tcp_connect_rtt"<wbr>:{"us":2000},"monitor":"http@<a href="http://192.168.1.16:9200" target="_blank">h<wbr>ttp://192.168.1.16:9200</a>","<wbr>type":"http","http_rtt":{"us":<wbr>2000},"url":"<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.<wbr>16:9200</a>","tags":["beats_input_<wbr>raw_event"],"duration":{"us":<wbr>5000},"@timestamp":"2017-05-<wbr>12T00:30:32.020Z","rtt":{"us":<wbr>5000},"port":9200,"response":{<wbr>"status":200},"beat":{"<wbr>hostname":"TYLER-LAPTOP","<wbr>name":"TYLER-LAPTOP","version"<wbr>:"5.4.0"},"@version":"1","<wbr>host":"TYLER-LAPTOP","up":<wbr>true}{"scheme":"http","ip":"19<wbr>2.168.1.16","tcp_connect_rtt":<wbr>{"us":2000},"monitor":"http@<a href="http://192.168.1.16:9200" target="_blank">ht<wbr>tp://192.168.1.16:9200</a>","type"<wbr>:"http","http_rtt":{"us":5000}<wbr>,"url":"<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:<wbr>9200</a>","tags":["beats_input_<wbr>raw_event"],"duration":{"us":<wbr>7000},"rtt":{"us":7000},"@<wbr>timestamp":"2017-05-12T00:30:4<wbr>2.020Z","port":9200,"response"<wbr>:{"status":200},"beat":{"<wbr>hostname":"TYLER-LAPTOP","<wbr>name":"TYLER-LAPTOP","version"<wbr>:"5.4.0"},"@version":"1","<wbr>host":"TYLER-LAPTOP","up":<wbr>true}{"scheme":"http","ip":"19<wbr>2.168.1.16","tcp_connect_rtt":<wbr>{"us":2000},"monitor":"http@<a href="http://192.168.1.16:9200" target="_blank">ht<wbr>tp://192.168.1.16:9200</a>","type"<wbr>:"'</div>
                <div><br>
                </div>
                <div>[2017-05-11T20:38:49.900179] Message parsing
                  complete; result='1', rule='#anon-parser0',
                  location='/etc/syslog-ng/syslo<wbr>g-ng.conf:60:33'</div>
                <div>[2017-05-11T20:38:49.900324] Syslog connection
                  closed; fd='88', client='AF_INET(<a href="http://192.168.1.16" target="_blank">192.168.1.16</a>:6<wbr>0660)',
                  local='AF_INET(<a href="http://0.0.0.0:5140" target="_blank">0.0.0.0:5140</a>)'</div>
                <div>[2017-05-11T20:38:49.900384] Outgoing message;
                  message='{}</div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Wed, May 10, 2017 at 4:01 PM, Evan
          Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <div class="m_-7503688148693866562m_7271332207509791338m_4754309162587128553moz-cite-prefix">Since
                you already have the no-parse flag on the source,
                everything goes into $MESSAGE<br>
                <br>
                Make a file destination with the template of
                "$MESSAGE\n" and it should contain the entire payload.<span class="m_-7503688148693866562m_7271332207509791338HOEnZb"><font color="#888888"><br>
                    <br>
                    Evan.</font></span>
                <div>
                  <div class="m_-7503688148693866562m_7271332207509791338h5"><br>
                    <br>
                    On 05/10/2017 12:57 PM, Scot wrote:<br>
                  </div>
                </div>
              </div>
              <div>
                <div class="m_-7503688148693866562m_7271332207509791338h5">
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div>Thanks Evan, </div>
                      <div><br>
                      </div>
                      <div>Bumped it up to 32768  </div>
                      <div><br>
                      </div>
                       Error extracting JSON members into LogMessage as
                      the top-level JSON object is not an object;
                      input='":"A  <br>
                      <div>I think there may be something else I need to
                        do with the payload. </div>
                      <div><br>
                      </div>
                      <div>How would I dump everything to a file to look
                        at it ? </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                    </div>
                    <div class="gmail_extra"><br>
                      <div class="gmail_quote">On Wed, May 10, 2017 at
                        2:10 PM, Evan Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF">
                            <div class="m_-7503688148693866562m_7271332207509791338m_4754309162587128553m_2935620687240762513moz-cite-prefix">looks
                              like you might be running into the maximum
                              message size.<br>
                              Try setting the syslog-ng configuration
                              item<br>
                              <br>
                              log_msg_size(64K);
                              <div>
                                <div class="m_-7503688148693866562m_7271332207509791338m_4754309162587128553h5"><br>
                                  <br>
                                  <br>
                                  On 05/10/2017 10:50 AM, Scot wrote:<br>
                                </div>
                              </div>
                            </div>
                            <div>
                              <div class="m_-7503688148693866562m_7271332207509791338m_4754309162587128553h5">
                                <blockquote type="cite">
                                  <div dir="ltr">Using a RAW TCP seems
                                    to be loosing some of the beats
                                    header data and messages are getting
                                    concatenated. 
                                    <div>
                                      <div>Trying different options but
                                        I'm fumbling. <br>
                                      </div>
                                      <div><br>
                                      </div>
                                      <div>
                                        <div>  syslog-ng[4596]:
                                          Unparsable JSON stream
                                          encountered;
                                          input='=net"},"message":"Synch<wbr>ronization
                                          of a replica of an Active
                                          Directory naming context has
                                          begun.\n\nDestination
                                          DRA:\tCN=NTDS
                                          Settings,CN=...blaaa"</div>
                                      </div>
                                      <div><br>
                                      </div>
                                      <div><br>
                                      </div>
                                      <div>source s_BEATS        
                                         {network(port(5140)
                                        flags(no-parse));}</div>
                                      <div>
                                        <div>parser p_json {</div>
                                        <div>    json-parser
                                          (prefix(".json."));</div>
                                        <div>};</div>
                                      </div>
                                      <div>log { source(s_BEATS);
                                         parser(p_json); destination
                                        (d_file); };<br>
                                      </div>
                                      <div><br>
                                      </div>
                                      <div><br>
                                      </div>
                                      <div>Anyone have a howto or blog
                                        for using syslog-ng with json
                                        inputs ?  </div>
                                      <div>I'm looking at
                                        the syslog-ng-ose-latest-guide<wbr>s
                                        but it's hard to put all the
                                        input output and parser
                                        requirements together.  </div>
                                      <div><br>
                                      </div>
                                      <div>Trying to get here </div>
                                      <div>winlogbeat->syslog-ng->ES
                                          </div>
                                      <div>winlogbeat->syslog-ng->SPLUNKF<wbr>orwader</div>
                                      <div>winlogbeat->syslog-ng->/opt/sy<wbr>slog-ng/logs/$FROM_HOST.json <br>
                                      </div>
                                      <div class="gmail_extra"><br>
                                        <div class="gmail_quote">or </div>
                                        <div class="gmail_quote">
                                          <div>winlogbeat->logstash->syslog-n<wbr>g->ES
                                              </div>
                                          <div>...</div>
                                          <div><br>
                                          </div>
                                        </div>
                                        <div class="gmail_quote">On Tue,
                                          May 9, 2017 at 3:27 AM, Fabien
                                          Wernli <span dir="ltr"><<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>></span>
                                          wrote:<br>
                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
                                            <span><br>
                                              On Mon, May 08, 2017 at
                                              11:30:14PM +0000, Scot
                                              wrote:<br>
                                              > I'm trying to find a
                                              solution that will let me
                                              mirror my beats data like<br>
                                              > syslog-ng lets me do
                                              with syslog traffic.<br>
                                              <br>
                                            </span>As far as I know
                                            those tools simply send the
                                            data over TCP in JSON
                                            format.<br>
                                            If you just need to do
                                            routing using syslog-ng, you
                                            can simply use network<br>
                                            source with flags(no-parse).
                                            If you need to process the
                                            data using<br>
                                            syslog-ng, you'll also need
                                            the json-parser().<br>
                                            <br>
                                            Cheers<br>
                                            <br>
                                          </blockquote>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </blockquote>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                    </div>
                  </blockquote>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
      </div>
    </blockquote>
    <br>
  </div></div></div>

<br></div></div><span class="">______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></span></blockquote></div><br></div>
</blockquote></div><br></div>