<div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">YouTube specified --scope dot-nv-pairs tó format-json, which only includes name-value pairs that start with a  dot.</div><div dir="auto"><br></div><div dir="auto"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017. máj. 12. de. 2:51 ezt írta ("Scot" <<a href="mailto:scotrn@gmail.com">scotrn@gmail.com</a>>):<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm almost there I think! <div>An idea why is outputting message ={} nothing ?  What is rule='#anon-parser0<div><br><div><b>relative lines from syslog-ng.conf</b></div><div>source s_BEATS<span class="m_-3838061240380507976gmail-Apple-tab-span" style="white-space:pre-wrap">            </span>{network(port(<b>5140</b>) log-msg-size(65536) flags(no-parse));};</div><div>destination d_jfile { file("/opt/syslog-ng/logs/$<wbr>HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));};<br></div><div>log { source(s_BEATS);  parser {json-parser();}; destination (d_jfile); };</div><div><br></div><div><div><b><i>/etc/logstash/conf.d/logstash.<wbr>conf</i></b></div><div>input {</div><div>  beats {</div><div>    port => 5044</div><div>  }</div><div>}</div><div><br></div><div>output {</div><div>  tcp {</div><div>    host => "192.168.1.16"</div><div>    port => "<b>5140</b>"</div><div>    mode => "client"</div><div>  }<br></div></div><div><br><div><br></div><div><b><u>Running syslog-ng in foreground.  </u></b></div><div><br></div><div><div>[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(<a href="http://192.168.1.16">192.168.1.16</a>:<wbr>60660)', local='AF_INET(<a href="http://0.0.0.0:5140" target="_blank">0.0.0.0:5140</a>)'</div><div>[2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"<wbr>192.168.1.16","tcp_connect_<wbr>rtt":{"us":2000},"monitor":"<wbr>http@<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:9200</a>"<wbr>,"type":"http","http_rtt":{"<wbr>us":2000},"url":"<a href="http://192.168.1.16:9200" target="_blank">http://192.<wbr>168.1.16:9200</a>","tags":["beats_<wbr>input_raw_event"],"duration":{<wbr>"us":5000},"@timestamp":"2017-<wbr>05-12T00:30:32.020Z","rtt":{"<wbr>us":5000},"port":9200,"<wbr>response":{"status":200},"<wbr>beat":{"hostname":"TYLER-<wbr>LAPTOP","name":"TYLER-LAPTOP",<wbr>"version":"5.4.0"},"@version":<wbr>"1","host":"TYLER-LAPTOP","up"<wbr>:true}{"scheme":"http","ip":"<wbr>192.168.1.16","tcp_connect_<wbr>rtt":{"us":2000},"monitor":"<wbr>http@<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:9200</a>"<wbr>,"type":"http","http_rtt":{"<wbr>us":5000},"url":"<a href="http://192.168.1.16:9200" target="_blank">http://192.<wbr>168.1.16:9200</a>","tags":["beats_<wbr>input_raw_event"],"duration":{<wbr>"us":7000},"rtt":{"us":7000},"<wbr>@timestamp":"2017-05-12T00:30:<wbr>42.020Z","port":9200,"<wbr>response":{"status":200},"<wbr>beat":{"hostname":"TYLER-<wbr>LAPTOP","name":"TYLER-LAPTOP",<wbr>"version":"5.4.0"},"@version":<wbr>"1","host":"TYLER-LAPTOP","up"<wbr>:true}{"scheme":"http","ip":"<wbr>192.168.1.16","tcp_connect_<wbr>rtt":{"us":2000},"monitor":"<wbr>http@<a href="http://192.168.1.16:9200" target="_blank">http://192.168.1.16:9200</a>"<wbr>,"type":"'</div><div><br></div><div>[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/<wbr>syslog-ng.conf:60:33'</div><div>[2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(<a href="http://192.168.1.16">192.168.1.16</a>:<wbr>60660)', local='AF_INET(<a href="http://0.0.0.0:5140" target="_blank">0.0.0.0:5140</a>)'</div><div>[2017-05-11T20:38:49.900384] Outgoing message; message='{}</div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <div class="m_-3838061240380507976m_4754309162587128553moz-cite-prefix">Since you already have the no-parse
      flag on the source, everything goes into $MESSAGE<br>
      <br>
      Make a file destination with the template of "$MESSAGE\n" and it
      should contain the entire payload.<span class="m_-3838061240380507976HOEnZb"><font color="#888888"><br>
      <br>
      Evan.</font></span><div><div class="m_-3838061240380507976h5"><br>
      <br>
      On 05/10/2017 12:57 PM, Scot wrote:<br>
    </div></div></div><div><div class="m_-3838061240380507976h5">
    <blockquote type="cite">
      
      <div dir="ltr">
        <div>Thanks Evan, </div>
        <div><br>
        </div>
        <div>Bumped it up to 32768  </div>
        <div><br>
        </div>
         Error extracting JSON members into LogMessage as the top-level
        JSON object is not an object; input='":"A  <br>
        <div>I think there may be something else I need to do with the
          payload. </div>
        <div><br>
        </div>
        <div>How would I dump everything to a file to look at it ? </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Wed, May 10, 2017 at 2:10 PM, Evan
          Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <div class="m_-3838061240380507976m_4754309162587128553m_2935620687240762513moz-cite-prefix">looks
                like you might be running into the maximum message size.<br>
                Try setting the syslog-ng configuration item<br>
                <br>
                log_msg_size(64K);
                <div>
                  <div class="m_-3838061240380507976m_4754309162587128553h5"><br>
                    <br>
                    <br>
                    On 05/10/2017 10:50 AM, Scot wrote:<br>
                  </div>
                </div>
              </div>
              <div>
                <div class="m_-3838061240380507976m_4754309162587128553h5">
                  <blockquote type="cite">
                    <div dir="ltr">Using a RAW TCP seems to be loosing
                      some of the beats header data and messages are
                      getting concatenated. 
                      <div>
                        <div>Trying different options but I'm fumbling. <br>
                        </div>
                        <div><br>
                        </div>
                        <div>
                          <div>  syslog-ng[4596]: Unparsable JSON stream
                            encountered; input='=net"},"message":"Synch<wbr>ronization
                            of a replica of an Active Directory naming
                            context has begun.\n\nDestination
                            DRA:\tCN=NTDS Settings,CN=...blaaa"</div>
                        </div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div>source s_BEATS          {network(port(5140)
                          flags(no-parse));}</div>
                        <div>
                          <div>parser p_json {</div>
                          <div>    json-parser (prefix(".json."));</div>
                          <div>};</div>
                        </div>
                        <div>log { source(s_BEATS);  parser(p_json);
                          destination (d_file); };<br>
                        </div>
                        <div><br>
                        </div>
                        <div><br>
                        </div>
                        <div>Anyone have a howto or blog for using
                          syslog-ng with json inputs ?  </div>
                        <div>I'm looking at the syslog-ng-ose-latest-guide<wbr>s
                          but it's hard to put all the input output and
                          parser requirements together.  </div>
                        <div><br>
                        </div>
                        <div>Trying to get here </div>
                        <div>winlogbeat->syslog-ng->ES   </div>
                        <div>winlogbeat->syslog-ng->SPLUNKF<wbr>orwader</div>
                        <div>winlogbeat->syslog-ng->/opt/sy<wbr>slog-ng/logs/$FROM_HOST.json <br>
                        </div>
                        <div class="gmail_extra"><br>
                          <div class="gmail_quote">or </div>
                          <div class="gmail_quote">
                            <div>winlogbeat->logstash->syslog-n<wbr>g->ES
                                </div>
                            <div>...</div>
                            <div><br>
                            </div>
                          </div>
                          <div class="gmail_quote">On Tue, May 9, 2017
                            at 3:27 AM, Fabien Wernli <span dir="ltr"><<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>></span>
                            wrote:<br>
                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
                              <span><br>
                                On Mon, May 08, 2017 at 11:30:14PM
                                +0000, Scot wrote:<br>
                                > I'm trying to find a solution that
                                will let me mirror my beats data like<br>
                                > syslog-ng lets me do with syslog
                                traffic.<br>
                                <br>
                              </span>As far as I know those tools simply
                              send the data over TCP in JSON format.<br>
                              If you just need to do routing using
                              syslog-ng, you can simply use network<br>
                              source with flags(no-parse). If you need
                              to process the data using<br>
                              syslog-ng, you'll also need the
                              json-parser().<br>
                              <br>
                              Cheers<br>
                              <br>
                            </blockquote>
                          </div>
                        </div>
                      </div>
                    </div>
                  </blockquote>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
      </div>
    </blockquote>
    <br>
  </div></div></div>

<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=<wbr>syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/<wbr>mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/<wbr>support/documentation/?<wbr>product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/<wbr>syslog-ng-faq</a><br>
<br>
<br></blockquote></div></div>