[syslog-ng] syslog-ng forwarding and processing issue
Scheidler, Balázs
balazs.scheidler at balabit.com
Fri Sep 30 15:47:29 CEST 2016
Hi,
Yup, either rfc5424 based format or simple plain old syslog, depending on
what the SIEM can accept.
syslog(transport(tcp)) is good as it supports both rfc5424 and traditional
and has framing, thus can support multiline logs.
--
Bazsi
On Fri, Sep 30, 2016 at 11:48 AM, Denis Dolinský <denis.dolinsky at gmail.com>
wrote:
> Hi Balazs,
>
> I totally agree with you. New log management concept I am going to produce
> shortly, I just wanted this od config to do the job till I will not migrate
> it to new solution.
>
> So, in new solution, what kind of transport should I consider - TCP ? From
> source to log processor and then to SIEM ?
> I don't want to use DNS, I want to keep hostnames, encryption is not
> required - maybe I would consider it in log processor to SIEM traffic.
>
> Thanks.
>
> Denis
>
> 2016-09-29 17:38 GMT+02:00 Scheidler, Balázs <balazs.scheidler at balabit.com
> >:
>
>> Source spoofing is the remnant of a world where devices didn't have tcp
>> based logging, and were broken enough not to recognise the source hostname
>> out of a syslog message.
>>
>> The world us supposed to have changed and I would drop spoof source any
>> time if it weren't for backward compatibility.
>>
>> Using spoof source and hence UDP is not a good way to deploy a serious
>> log management system in 2016.
>>
>> On Sep 29, 2016 09:09, "Denis Dolinský" <denis.dolinsky at gmail.com> wrote:
>>
>>> hi Sandor,
>>>
>>> these are my global options:
>>>
>>> #
>>> # Global options.
>>> #
>>> options { chain_hostnames(yes); keep_hostname(yes); keep_timestamp(yes);
>>> flush_lines(0); perm(0640); stats_freq(3600); };
>>>
>>> so spoof source is not necessary here ?
>>>
>>> Thanks.
>>>
>>> Denis
>>>
>>> 2016-09-29 9:49 GMT+02:00 Sandor Geller <sandor.geller at ericsson.com>:
>>>
>>>> Hi,
>>>>
>>>> Source spoofing fakes the source IP address of the outgoing packets, as
>>>> this can't work with connection-oriented protocols it is usable only
>>>> with UDP datagrams.
>>>>
>>>> Actually source spoofing isn't needed in most cases and won't even work
>>>> when spoofing protection is enabled in firewalls / routers and not all
>>>> hosts are on the same subnet.
>>>>
>>>> Take a look at keep-hostname() and chain-hostnames()
>>>>
>>>> Regards,
>>>>
>>>> Sandor
>>>>
>>>> On 09/29/2016 09:29 AM, Denis Dolinský wrote:
>>>> > Hi,
>>>> >
>>>> > yes, I need spoof_source to be enabled for source identification ...
>>>> >
>>>> > Denis
>>>> >
>>>> > 2016-09-28 16:44 GMT+02:00 Szalai, Attila
>>>> > <Attila.Szalai at morganstanley.com <mailto:Attila.Szalai at morganst
>>>> anley.com>>:
>>>> >
>>>> > Just a quick note.____
>>>> >
>>>> > The warning message about the binding issue caused by the
>>>> > spoof_source option. Is that option necessary?____
>>>> >
>>>> > __ __
>>>> >
>>>> > *From:*syslog-ng-bounces at lists.balabit.hu
>>>> > <mailto:syslog-ng-bounces at lists.balabit.hu>
>>>> > [mailto:syslog-ng-bounces at lists.balabit.hu
>>>> > <mailto:syslog-ng-bounces at lists.balabit.hu>] *On Behalf Of *Denis
>>>> > Dolinský
>>>> > *Sent:* Wednesday, September 28, 2016 3:47 PM
>>>> >
>>>> >
>>>> > *To:* Syslog-ng users' and developers' mailing list
>>>> > *Subject:* Re: [syslog-ng] syslog-ng forwarding and processing
>>>> issue____
>>>> >
>>>> > __ __
>>>> >
>>>> > hi guys,____
>>>> >
>>>> > __ __
>>>> >
>>>> > this is stats:____
>>>> >
>>>> > __ __
>>>> >
>>>> > destination;d_net_udp514;;a;processed;13
>>>> > source;s_net_udp514;;a;processed;3
>>>> > dst.syslog;d_net_udp514#0;udp,192.168.3.1:514;a;dropped;0
>>>> > dst.syslog;d_net_udp514#0;udp,192.168.3.1:514;a;processed;10
>>>> > dst.syslog;d_net_udp514#0;udp,192.168.3.1:514;a;stored;0____
>>>> >
>>>> > __ __
>>>> >
>>>> > from debug:____
>>>> >
>>>> > __ __
>>>> >
>>>> > Incoming log entry; source='s_net_udp514#0',
>>>> > line='<78> remote_server /usr/sbin/cron[24934]:
>>>> > Can't bind hostname for the IP address, therefore using IP address
>>>> > as hostname; IP address='192.168.2.1'____
>>>> >
>>>> > __ __
>>>> >
>>>> > Do you see anything what I do not do ?____
>>>> >
>>>> > __ __
>>>> >
>>>> > Thanks.____
>>>> >
>>>> > __ __
>>>> >
>>>> > Denis____
>>>> >
>>>> > __ __
>>>> >
>>>> > 2016-09-28 14:02 GMT+02:00 Szalai, Attila
>>>> > <Attila.Szalai at morganstanley.com
>>>> > <mailto:Attila.Szalai at morganstanley.com>>:____
>>>> >
>>>> > Hi,____
>>>> >
>>>> > ____
>>>> >
>>>> > In case of udp, the syslog source should handle receiving logs
>>>> with
>>>> > old and the new version too. (But that is more an exception than
>>>> the
>>>> > rule, so matching the receiver and the sender is a good idea
>>>> > generaly.)____
>>>> >
>>>> > ____
>>>> >
>>>> > Before anything else I would check if the logs arrive to the
>>>> > anonymizer host or not. The statistics can help on this. Also, if
>>>> > there are parsing issue, the syslog-ng would tell this through its
>>>> > log.____
>>>> >
>>>> > ____
>>>> >
>>>> > After that starting the syslog-ng with enabled debug logs can also
>>>> > help on discovering what happening with the received log.____
>>>> >
>>>> > ____
>>>> >
>>>> > *From:*syslog-ng-bounces at lists.balabit.hu
>>>> > <mailto:syslog-ng-bounces at lists.balabit.hu>
>>>> > [mailto:syslog-ng-bounces at lists.balabit.hu
>>>> > <mailto:syslog-ng-bounces at lists.balabit.hu>] *On Behalf Of
>>>> *Fekete,
>>>> > Róbert
>>>> > *Sent:* Wednesday, September 28, 2016 1:47 PM
>>>> > *To:* Syslog-ng users' and developers' mailing list
>>>> > *Subject:* Re: [syslog-ng] syslog-ng forwarding and processing
>>>> issue____
>>>> >
>>>> > ____
>>>> >
>>>> > Hi, ____
>>>> >
>>>> > ____
>>>> >
>>>> > The destination on your remote server and the source on the
>>>> > pseudomizer host do not match: the first one uses the udp() driver
>>>> > (RFC3164 protocol), while the second uses the syslog() driver
>>>> > (RFC5424) protocol. ____
>>>> >
>>>> > ____
>>>> >
>>>> > Change the destination driver to syslog() on the remote server.
>>>> (For
>>>> > more possibilities, see
>>>> > https://www.balabit.com/documents/syslog-ng-pe-latest-guide
>>>> s/en/syslog-ng-pe-guide-admin/html/concepts-things-to-consider.html
>>>> > <https://www.balabit.com/documents/syslog-ng-pe-latest-guid
>>>> es/en/syslog-ng-pe-guide-admin/html/concepts-things-to-consider.html>
>>>> )____
>>>> >
>>>> > ____
>>>> >
>>>> > HTH____
>>>> >
>>>> > ____
>>>> >
>>>> > Robert____
>>>> >
>>>> > ____
>>>> >
>>>> > On Wed, Sep 28, 2016 at 1:17 PM, Denis Dolinský
>>>> > <denis.dolinsky at gmail.com <mailto:denis.dolinsky at gmail.com>>
>>>> wrote:____
>>>> >
>>>> > Hi guys,____
>>>> >
>>>> > I have following setup in place:____
>>>> >
>>>> > remote server - 192.168.1.10____
>>>> >
>>>> > pseudomizer - syslog-ng PE in client mode - 192.168.2.10____
>>>> >
>>>> > SIEM - 192.168.3.10____
>>>> >
>>>> > So I am sending syslog logs from remote server to pseudomizer:____
>>>> >
>>>> > source src { internal()};____
>>>> >
>>>> > destination dst { udp ("192.168.2.10) port (514);};____
>>>> >
>>>> > log { source(src); destination (dst);____
>>>> >
>>>> > this is very old config from syslog v4____
>>>> >
>>>> > Then on pseudomizer - syslog-ng LTS 6.0.1 PE, I am collecting the
>>>> > logs, processing them - removing private data, putting pseudonyms
>>>> > instead and forwarding them to SIEM.
>>>> >
>>>> > source s_net_udp514 {
>>>> > syslog(
>>>> > ip(192.168.2.10)
>>>> > ip-protocol(4)
>>>> > transport("udp")
>>>> > so_rcvbuf(2097152)
>>>> > );
>>>> > };
>>>> >
>>>> > source src {
>>>> > internal();
>>>> > unix-dgram("/dev/log");
>>>> > system ();
>>>> > };
>>>> >
>>>> > destination d_net_udp514 {
>>>> > syslog (
>>>> > "192.168.3.10"
>>>> > port(514)
>>>> > transport(udp)
>>>> > spoof_source(yes)
>>>> > mark_mode(periodical));};
>>>> > rewrite r_rewrite {
>>>> > subst("admin", "pseudonym000001", value("MESSAGE"),
>>>> flags("global"));
>>>> >
>>>> > log {
>>>> > source(s_net_udp514); source (src);
>>>> > rewrite(r_rewrite); # do the pseudomizing
>>>> > destination(d_net_udp514);
>>>> > };____
>>>> >
>>>> > On SIEM device, I can see only pseudomizer internal logs (src),
>>>> not
>>>> > processed logs from remote server.____
>>>> >
>>>> > Any advice ?____
>>>> >
>>>> > Many thanks.____
>>>> >
>>>> > Denis____
>>>> >
>>>> >
>>>> > ___________________________________________________________
>>>> ___________________
>>>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> > <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
>>>> > Documentation:
>>>> > http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> > <http://www.balabit.com/support/documentation/?product=syslog-ng>
>>>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>> > <http://www.balabit.com/wiki/syslog-ng-faq>____
>>>> >
>>>> > ____
>>>> >
>>>> > __ __
>>>> >
>>>> > -----------------------------------------------------------
>>>> -------------
>>>> >
>>>> >
>>>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and
>>>> the
>>>> > opinions or views contained herein are not intended to be, and do
>>>> > not constitute, advice within the meaning of Section 975 of the
>>>> > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
>>>> > have received this communication in error, please destroy all
>>>> > electronic and paper copies and notify the sender immediately.
>>>> > Mistransmission is not intended to waive confidentiality or
>>>> > privilege. Morgan Stanley reserves the right, to the extent
>>>> > permitted under applicable law, to monitor electronic
>>>> > communications. This message is subject to terms available at the
>>>> > following link: http://www.morganstanley.com/disclaimers
>>>> > <http://www.morganstanley.com/disclaimers> If you cannot access
>>>> > these links, please notify us by reply message and we will send
>>>> the
>>>> > contents to you. By communicating with Morgan Stanley you consent
>>>> to
>>>> > the foregoing and to the voice recording of conversations with
>>>> > personnel of Morgan Stanley.____
>>>> >
>>>> >
>>>> > ___________________________________________________________
>>>> ___________________
>>>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> > <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
>>>> > Documentation:
>>>> > http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> > <http://www.balabit.com/support/documentation/?product=syslog-ng>
>>>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>> > <http://www.balabit.com/wiki/syslog-ng-faq>
>>>> >
>>>> > ____
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -- ____
>>>> >
>>>> > Ing. Denis Dolinský
>>>> > denis.dolinsky at gmail.com <mailto:denis.dolinsky at gmail.com>
>>>> > private cell: _+421 907 530711 <tel:%2B421%20907%20530711>_____
>>>> >
>>>> >
>>>> >
>>>> > -----------------------------------------------------------
>>>> -------------
>>>> >
>>>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and
>>>> the
>>>> > opinions or views contained herein are not intended to be, and do
>>>> > not constitute, advice within the meaning of Section 975 of the
>>>> > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
>>>> > have received this communication in error, please destroy all
>>>> > electronic and paper copies and notify the sender immediately.
>>>> > Mistransmission is not intended to waive confidentiality or
>>>> > privilege. Morgan Stanley reserves the right, to the extent
>>>> > permitted under applicable law, to monitor electronic
>>>> > communications. This message is subject to terms available at the
>>>> > following link: http://www.morganstanley.com/disclaimers
>>>> > <http://www.morganstanley.com/disclaimers> If you cannot access
>>>> > these links, please notify us by reply message and we will send
>>>> the
>>>> > contents to you. By communicating with Morgan Stanley you consent
>>>> to
>>>> > the foregoing and to the voice recording of conversations with
>>>> > personnel of Morgan Stanley.
>>>> >
>>>> >
>>>> > ___________________________________________________________
>>>> ___________________
>>>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> > <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
>>>> > Documentation:
>>>> > http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> > <http://www.balabit.com/support/documentation/?product=syslog-ng>
>>>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>> > <http://www.balabit.com/wiki/syslog-ng-faq>
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Ing. Denis Dolinský
>>>> > denis.dolinsky at gmail.com <mailto:denis.dolinsky at gmail.com>
>>>> > private cell: _+421 907 530711_
>>>> >
>>>> >
>>>> > ____________________________________________________________
>>>> __________________
>>>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> > Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>> >
>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>
>>>
>>> --
>>> Ing. Denis Dolinský
>>> denis.dolinsky at gmail.com
>>> private cell: *+421 907 530711 <%2B421%20907%20530711>*
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
> Ing. Denis Dolinský
> denis.dolinsky at gmail.com
> private cell: *+421 907 530711 <%2B421%20907%20530711>*
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160930/956124a2/attachment-0001.htm
More information about the syslog-ng
mailing list