<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">What ever is feeding this source
appears to be truncating the message to 1024 characters. I assume
it is some kind of syslog stream because 1024 is the syslog limit
for entire message size.<br>
<br>
I think you need to adjust your configuration of the originating
software.<br>
<br>
Evan.<br>
<br>
<br>
On 05/11/2017 05:51 PM, Scot wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAOxbc8Fm_ykB5_tAdOen3io-15VbEjRR=YkQSj20tq49b4O-mw@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">I'm almost there I think!
<div>An idea why is outputting message ={} nothing ? What is
rule='#anon-parser0
<div><br>
<div><b>relative lines from syslog-ng.conf</b></div>
<div>source s_BEATS<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>{network(port(<b>5140</b>)
log-msg-size(65536) flags(no-parse));};</div>
<div>destination d_jfile {
file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"
template("$(format-json --scope dot-nv-pairs)\n"));};<br>
</div>
<div>log { source(s_BEATS); parser {json-parser();};
destination (d_jfile); };</div>
<div><br>
</div>
<div>
<div><b><i>/etc/logstash/conf.d/logstash.conf</i></b></div>
<div>input {</div>
<div> beats {</div>
<div> port => 5044</div>
<div> }</div>
<div>}</div>
<div><br>
</div>
<div>output {</div>
<div> tcp {</div>
<div> host => "192.168.1.16"</div>
<div> port => "<b>5140</b>"</div>
<div> mode => "client"</div>
<div> }<br>
</div>
</div>
<div><br>
<div><br>
</div>
<div><b><u>Running syslog-ng in foreground. </u></b></div>
<div><br>
</div>
<div>
<div>[2017-05-11T20:30:53.529215] Syslog connection
accepted; fd='88',
client='AF_INET(192.168.1.16:60660)', local='AF_INET(<a
href="http://0.0.0.0:5140" moz-do-not-send="true">0.0.0.0:5140</a>)'</div>
<div>[2017-05-11T20:38:49.899997] Incoming log entry;
line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@<a
href="http://192.168.1.16:9200"
moz-do-not-send="true">http://192.168.1.16:9200</a>","type":"http","http_rtt":{"us":2000},"url":"<a
href="http://192.168.1.16:9200"
moz-do-not-send="true">http://192.168.1.16:9200</a>","tags":["beats_input_raw_event"],"duration":{"us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@<a
href="http://192.168.1.16:9200"
moz-do-not-send="true">http://192.168.1.16:9200</a>","type":"http","http_rtt":{"us":5000},"url":"<a
href="http://192.168.1.16:9200"
moz-do-not-send="true">http://192.168.1.16:9200</a>","tags":["beats_input_raw_event"],"duration":{"us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@<a
href="http://192.168.1.16:9200"
moz-do-not-send="true">http://192.168.1.16:9200</a>","type":"'</div>
<div><br>
</div>
<div>[2017-05-11T20:38:49.900179] Message parsing
complete; result='1', rule='#anon-parser0',
location='/etc/syslog-ng/syslog-ng.conf:60:33'</div>
<div>[2017-05-11T20:38:49.900324] Syslog connection
closed; fd='88', client='AF_INET(192.168.1.16:60660)',
local='AF_INET(<a href="http://0.0.0.0:5140"
moz-do-not-send="true">0.0.0.0:5140</a>)'</div>
<div>[2017-05-11T20:38:49.900384] Outgoing message;
message='{}</div>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 10, 2017 at 4:01 PM, Evan
Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca"
target="_blank" moz-do-not-send="true">erempel@uvic.ca</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_4754309162587128553moz-cite-prefix">Since
you already have the no-parse flag on the source,
everything goes into $MESSAGE<br>
<br>
Make a file destination with the template of
"$MESSAGE\n" and it should contain the entire payload.<span
class="HOEnZb"><font color="#888888"><br>
<br>
Evan.</font></span>
<div>
<div class="h5"><br>
<br>
On 05/10/2017 12:57 PM, Scot wrote:<br>
</div>
</div>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div>Thanks Evan, </div>
<div><br>
</div>
<div>Bumped it up to 32768 </div>
<div><br>
</div>
Error extracting JSON members into LogMessage as
the top-level JSON object is not an object;
input='":"A <br>
<div>I think there may be something else I need to
do with the payload. </div>
<div><br>
</div>
<div>How would I dump everything to a file to look
at it ? </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 10, 2017 at
2:10 PM, Evan Rempel <span dir="ltr"><<a
href="mailto:erempel@uvic.ca"
target="_blank" moz-do-not-send="true">erempel@uvic.ca</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div
class="m_4754309162587128553m_2935620687240762513moz-cite-prefix">looks
like you might be running into the maximum
message size.<br>
Try setting the syslog-ng configuration
item<br>
<br>
log_msg_size(64K);
<div>
<div class="m_4754309162587128553h5"><br>
<br>
<br>
On 05/10/2017 10:50 AM, Scot wrote:<br>
</div>
</div>
</div>
<div>
<div class="m_4754309162587128553h5">
<blockquote type="cite">
<div dir="ltr">Using a RAW TCP seems
to be loosing some of the beats
header data and messages are getting
concatenated.
<div>
<div>Trying different options but
I'm fumbling. <br>
</div>
<div><br>
</div>
<div>
<div> syslog-ng[4596]:
Unparsable JSON stream
encountered;
input='=net"},"message":"Synch<wbr>ronization
of a replica of an Active
Directory naming context has
begun.\n\nDestination
DRA:\tCN=NTDS
Settings,CN=...blaaa"</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>source s_BEATS
{network(port(5140)
flags(no-parse));}</div>
<div>
<div>parser p_json {</div>
<div> json-parser
(prefix(".json."));</div>
<div>};</div>
</div>
<div>log { source(s_BEATS);
parser(p_json); destination
(d_file); };<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Anyone have a howto or blog
for using syslog-ng with json
inputs ? </div>
<div>I'm looking at
the syslog-ng-ose-latest-guide<wbr>s
but it's hard to put all the
input output and parser
requirements together. </div>
<div><br>
</div>
<div>Trying to get here </div>
<div>winlogbeat->syslog-ng->ES
</div>
<div>winlogbeat->syslog-ng->SPLUNKF<wbr>orwader</div>
<div>winlogbeat->syslog-ng->/opt/sy<wbr>slog-ng/logs/$FROM_HOST.json <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">or </div>
<div class="gmail_quote">
<div>winlogbeat->logstash->syslog-n<wbr>g->ES
</div>
<div>...</div>
<div><br>
</div>
</div>
<div class="gmail_quote">On Tue,
May 9, 2017 at 3:27 AM, Fabien
Wernli <span dir="ltr"><<a
href="mailto:wernli@in2p3.fr" target="_blank" moz-do-not-send="true">wernli@in2p3.fr</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Hi,<br>
<span><br>
On Mon, May 08, 2017 at
11:30:14PM +0000, Scot
wrote:<br>
> I'm trying to find a
solution that will let me
mirror my beats data like<br>
> syslog-ng lets me do
with syslog traffic.<br>
<br>
</span>As far as I know
those tools simply send the
data over TCP in JSON
format.<br>
If you just need to do
routing using syslog-ng, you
can simply use network<br>
source with flags(no-parse).
If you need to process the
data using<br>
syslog-ng, you'll also need
the json-parser().<br>
<br>
Cheers<br>
<br>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</body>
</html>