[syslog-ng] Syslog-ng input for beats ?
Scot
scotrn at gmail.com
Fri May 12 04:50:16 UTC 2017
Youtube ?
Got the \n off the json log by just removing the \n from "$MESSAGE\n"
Now that I have the json TCP stream from logstash into syslog-ng I want to
spit it back out to multiple destinations.
Sending this beats data back into elastic search is causing the message
block to go into ES non parsed.
How can I debug that ES output ?
source s_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));};
destination d_es_beats {
elasticsearch2(
disk-buffer(
reliable(no) # If set to no, the normal disk-buffer will
be used. This provides a faster, option
dir("/opt/syslog-ng/buffer")
disk-buf-size(10485760)
mem-buf-length(100000) # number of messages stored in
overflow queue
) # END DiskBuffer
client-mode("http")
index("winlogbeat-${YEAR}.${MONTH}.${DAY}")
type("winlogbeat") # Description: The type of the index.
For example, type("test")
#template("$(format-json --scope rfc3164 --scope nv-pairs
--exclude R_DATE --key ISODATE)\n")
#template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p
@message=$MESSAGE)")
template("${MESSAGE}")
cluster-url("http://192.168.1.16:9200/")
concurrent-requests("5") # Number of concurrrent batches
flush_limit("5000") # The number of messages in a single
batch
skip-cluster-health-check("yes")
cluster("hal")
client_lib_dir("/usr/share/elasticsearch/lib")
);
};
log { source(s_BEATS); parser {kv-parser();}; destination (d_es_beats); };
On Fri, May 12, 2017 at 12:09 AM, Balazs Scheidler <bazsi77 at gmail.com>
wrote:
> Hi,
>
> YouTube specified --scope dot-nv-pairs tó format-json, which only includes
> name-value pairs that start with a dot.
>
>
>
> 2017. máj. 12. de. 2:51 ezt írta ("Scot" <scotrn at gmail.com>):
>
>> I'm almost there I think!
>> An idea why is outputting message ={} nothing ? What is
>> rule='#anon-parser0
>>
>> *relative lines from syslog-ng.conf*
>> source s_BEATS {network(port(*5140*) log-msg-size(65536)
>> flags(no-parse));};
>> destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"
>> template("$(format-json --scope dot-nv-pairs)\n"));};
>> log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
>>
>> */etc/logstash/conf.d/logstash.conf*
>> input {
>> beats {
>> port => 5044
>> }
>> }
>>
>> output {
>> tcp {
>> host => "192.168.1.16"
>> port => "*5140*"
>> mode => "client"
>> }
>>
>>
>> *Running syslog-ng in foreground. *
>>
>> [2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88',
>> client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)'
>> [2017-05-11T20:38:49.899997] Incoming log entry;
>> line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt"
>> :{"us":2000},"monitor":"http at http://192.168.1.16:9200","
>> type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200
>> ","tags":["beats_input_raw_event"],"duration":{"us":
>> 5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us":
>> 5000},"port":9200,"response":{"status":200},"beat":{"
>> hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version"
>> :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":
>> true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":
>> {"us":2000},"monitor":"http at http://192.168.1.16:9200","type"
>> :"http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200
>> ","tags":["beats_input_raw_event"],"duration":{"us":
>> 7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:4
>> 2.020Z","port":9200,"response":{"status":200},"beat":{"
>> hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version"
>> :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":
>> true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":
>> {"us":2000},"monitor":"http at http://192.168.1.16:9200","type":"'
>>
>> [2017-05-11T20:38:49.900179] Message parsing complete; result='1',
>> rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33'
>> [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88',
>> client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)'
>> [2017-05-11T20:38:49.900384] Outgoing message; message='{}
>>
>> On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>
>>> Since you already have the no-parse flag on the source, everything goes
>>> into $MESSAGE
>>>
>>> Make a file destination with the template of "$MESSAGE\n" and it should
>>> contain the entire payload.
>>>
>>> Evan.
>>>
>>>
>>> On 05/10/2017 12:57 PM, Scot wrote:
>>>
>>> Thanks Evan,
>>>
>>> Bumped it up to 32768
>>>
>>> Error extracting JSON members into LogMessage as the top-level JSON
>>> object is not an object; input='":"A
>>> I think there may be something else I need to do with the payload.
>>>
>>> How would I dump everything to a file to look at it ?
>>>
>>>
>>>
>>>
>>> On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>>
>>>> looks like you might be running into the maximum message size.
>>>> Try setting the syslog-ng configuration item
>>>>
>>>> log_msg_size(64K);
>>>>
>>>>
>>>>
>>>> On 05/10/2017 10:50 AM, Scot wrote:
>>>>
>>>> Using a RAW TCP seems to be loosing some of the beats header data and
>>>> messages are getting concatenated.
>>>> Trying different options but I'm fumbling.
>>>>
>>>> syslog-ng[4596]: Unparsable JSON stream encountered;
>>>> input='=net"},"message":"Synchronization of a replica of an Active
>>>> Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS
>>>> Settings,CN=...blaaa"
>>>>
>>>>
>>>> source s_BEATS {network(port(5140) flags(no-parse));}
>>>> parser p_json {
>>>> json-parser (prefix(".json."));
>>>> };
>>>> log { source(s_BEATS); parser(p_json); destination (d_file); };
>>>>
>>>>
>>>> Anyone have a howto or blog for using syslog-ng with json inputs ?
>>>> I'm looking at the syslog-ng-ose-latest-guides but it's hard to put
>>>> all the input output and parser requirements together.
>>>>
>>>> Trying to get here
>>>> winlogbeat->syslog-ng->ES
>>>> winlogbeat->syslog-ng->SPLUNKForwader
>>>> winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
>>>>
>>>> or
>>>> winlogbeat->logstash->syslog-ng->ES
>>>> ...
>>>>
>>>> On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
>>>>> > I'm trying to find a solution that will let me mirror my beats data
>>>>> like
>>>>> > syslog-ng lets me do with syslog traffic.
>>>>>
>>>>> As far as I know those tools simply send the data over TCP in JSON
>>>>> format.
>>>>> If you just need to do routing using syslog-ng, you can simply use
>>>>> network
>>>>> source with flags(no-parse). If you need to process the data using
>>>>> syslog-ng, you'll also need the json-parser().
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170512/75eda01d/attachment-0001.html>
More information about the syslog-ng
mailing list