[syslog-ng] Verifying local logs and remote logs are being sent to destination graylog

Fri Nov 23 16:54:44 UTC 2018

Dear Rodney,
 I loaded your config into a freshly built syslog-ng OSE master. (And also
tried to set up a graylog environment, more on that in the second point.)

1)
I replaced the graylog destination address with a localhost one.
I started a netcat instace with the following command: *nc -kl 127.0.0.1
12201*
I started to send logs into syslog with lgger: *logger "hello world"*

I have successfuly received the "hello world" message in RFC3164 format on
the 12201 port wit netcat. So your config seems to be working.

2)
I have set up a graylog environment following the steps on this page:
https://hub.docker.com/r/graylog/graylog/ using docker-compose.
The started up graylog instance throws a tons of java exceptions and error
messages. But I can reach it's WebUI interface.
Inside the interface (despite my attempts to find a solution online) I can
not query any data, since there is an issue with the internal API
communication:

Could not load field information
>
> Loading field information failed with status: cannot GET
>> http://127.0.0.1:9000/api/system/fields (500)
>
>

NOTE: At this point I did not take the effort to test out the GELF protocol
(port 12201) so just used the old RFC3164 and the "Syslog TCP input" with
the described settings.

I setted up a new "input" on the administration interface. Details:
 - type: syslog TCP
 - name: syslog
 - bind address: 0.0.0.0 (left it on default)
 - port: 514 (left it on default)

With a "netstat" command I confirmed, it is listening on port 514 and I
also changed the syslog-ng configuration to send the messages to this port.

Despite there is no way to query data, I have found a way to confirm that
graylog server actually received the messages:
Under the http://127.0.0.1:9000/system/inputs menu item, on the right side
of the inputs, there is a button "More actions". In the drop down list
there is a menu item: "Show metrics" (I can not send you a direct link to
this menu item because it contains the UUID of the selected input.)

On this page there are multiple keys to different metrics.
- Under the: org.graylog2.inputs.syslog.tcp.SyslogTCPInput.<hexadecimal
ID>.read_bytes_total
  I can see the increasing number of read data.

- Under the: org.graylog2.inputs.codecs.SyslogCodec.<hexadecimal
ID>.processedMessages
 I can see the processed messages, which were identical with the number of
logger commands I executed.

Agaiin: logger -> syslog-ng -> inputs -> graylog
I got a clear flow of messages.

3)
To pull out syslog-ng from the picture I did one more step.
Stopped the syslog-ng service, and copied a well formatted RFC3164 message
from the description of the protocol itself:
https://tools.ietf.org/html/rfc3164#section-5.4

I sent this message with netcat into the "Syslog TCP input", which also
lead to increasing the metrics on the described page.

My final conclusion in the topic:

- I think your configuration is good. (There might be some other programs
on your system which may conflict with the syslog-ng process, but I think
nobody can tell it remotely.)

- I recommend you to check again the port numbers and the desired
protocols. Syslog-ng by default uses the RFC3164 format on the network
destination, but you set the port number to 12201. (Which indicates that
you are trying to use the GELF format.)
 I can not tell if it is wrong or not, because you can set any kind of
inputs onto any desired port inside the graylog administration settings.

- Trying to narrow the problem scope, I recommend you the following "men in
the middle" approach:
1) test syslog-ng with a localhost address, with the netcat application, as
seen in my first paragraph. (This way you can also verify your syslog-ng
configuration.)
2) try the same test with the original destination address. BUT stop the
graylog service, and use netcat again to receive the logs. (This way you
can verify your network setup.)
3) try to input data into graylog from localhost (on your graylog server)
with netcat. I think this step is the most important. (This way you can
verify if graylog is accepting data at all.)

If everything works flawlessly I think you are ready to put the whole thing
together.
At this moment I think the community (Thank You, everybody, for your
time!!!) can not do anymore to solve this problem remotely.

Best regards,
László Szemere

On Mon, Nov 19, 2018 at 10:39 PM Rodney Bizzell <hardworker30 at gmail.com>
wrote:

> So I made the changes to syslog-ng config for graylog destination
> configuration and started the debug mode again hopped on another terminal
> and ran logger command with a test message but I still see nothing getting
> into my graylog server.  I am uploading the debug file again to see if
> there is something I am missing. On my graylog side. I have inputs setup
> Syslog I have Gelf input setup as well
>
> On Wed, Nov 14, 2018 at 1:31 PM PÁSZTOR György <
> pasztor at linux.gyakg.u-szeged.hu> wrote:
>
>> Hi,
>>
>> I took a look at your debug bundle.
>>
>> As far as I see the pcap file, tcp communication happens, but it's not
>> graylog's "protocol", it's the legacy rfc3164.
>> Unless graylog expecting legacy protocol, that's not correct.
>> Then I spotted this:
>>
>> "Rodney Bizzell" <hardworker30 at gmail.com> írta 2018-11-12 14:59-kor:
>> > destination d_graylog {
>> > tcp("hostname of Graylog server"
>> > port (12201)
>> > );
>> > };
>>
>> This configures a legacy tcp destination. It won't format the messages for
>> gelf.
>> At least in case of graylog, the best choice for protocol would be gelf.
>> The correct configuration for gelf destination would be:
>> destination d_graylog {
>>   graylog2( host("hostname of Graylog server") );
>> };
>>
>> Or if you want something more detailed or specific stuff, then you can use
>> this:
>> destination d_graylog {
>>   network(
>>     "hostname of Graylog server"
>>     port(12201)
>>     transport(tcp)
>>     template("$(format-json --scope all-nv-pairs)\n")
>>   );
>> };
>>
>> Of course, if gelf destination expects zero terminated strings, than you
>> can replace the \n to \0 at the end of the template string.
>>
>> Regards,
>> Gyu
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181123/2354e979/attachment.html>