[syslog-ng] Better performance between udp, unix-stream or pipe ?

[Matthew Hall]

As a connectionless protocol UDP cannot always detect existing listeners on all platforms. However if you think about it it's clear why this wouldn't work. How would it know which messages should be parse or noparse? It can't read your mind! ;)

Matthew.

Sent from my mobile device

"Yann I." <yann.frm at gmail.com> wrote:

>>
>> Does this work?
>>
>> source s_udp_not_parsed { udp(port(514) flags(no-parse)); };
>> source s_udp { udp(port(514)); };
>>
>> (...)
>>
>>
>I tried that solution few days ago and it didn't work. Two processes
>"syslog-ng" listened on port UDP/514 : the messages are not handled
>correctly. They would be handled by "s_udp_not_parsed" or "s_udp".
>
>nb : it's strange that two processes can listen on the same port (for the
>protocol UDP) isn't it ? This is the same behaviour with "netcat". I tried
>with netcat (example : nc -l -u 1234) and I have two processes which listen
>the port UDP/1234. Messages send by "nc" on the port 1234 are received by
>the last "netcat" which has been started.
>
>
>For passing yourself back the reparsed message I would recommend
>> unix-dgram. AF_UNIX is usually better than pipes for me at least.
>>
>
>Thank you ! I'll use AF_UNIX.
>
>Regards,
>
>Yann I.
>
>
>2010/11/16 Matthew Hall <mhall at mhcomputing.net>
>
>> For passing yourself back the reparsed message I would recommend
>> unix-dgram. AF_UNIX is usually better than pipes for me at least.
>>
>> Matthew.
>>
>> On Tue, Nov 16, 2010 at 12:14:53PM -0600, Martin Holste wrote:
>> > Does this work?
>> >
>> > source s_udp_not_parsed { udp(port(514) flags(no-parse)); };
>> > source s_udp { udp(port(514)); };
>> >
>> > log {
>> >   source(s_udp);
>> >   parser(db-parser());
>> >   destination(d_parsed);
>> > };
>> > log {
>> >   source(s_udp_not_parsed);
>> >   destination(d_not_parsed);
>> > };
>> >
>> > Otherwise, try reassembling a no-parse like message with a different
>> > output template.
>> >
>> > On Tue, Nov 16, 2010 at 11:13 AM, Yann I. <yann.frm at gmail.com> wrote:
>> > > Well I'm not sure because of the flag I used for the UDP source which
>> is set
>> > > to "no-parse".
>> > >
>> > > Here is my problem. From the UDP source, I may receive logs which are
>> not
>> > > "syslog compliance". So I'm using the flag 'no-parse' then I rewrite
>> the
>> > > message. After that rewrite, I forward the new message to the same
>> syslog-ng
>> > > server.
>> > > Then... I can apply filter, parser, etc on that new message which is
>> now
>> > > "syslog compliance" :-)
>> > >
>> > > So, I think I can't use log statement. I need to use that mecanism...
>> > > There might be another solution but this one seems to be a good
>> solution.
>> > >
>> > >
>> > > 2010/11/16 Martin Holste <mcholste at gmail.com>
>> > >>
>> > >> Ok, then this should be accomplished with a standard log statement
>> > >> like you've already begun to write.  What do your destinations look
>> > >> like?
>> > >>
>> > >> On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm at gmail.com> wrote:
>> > >> > In fact, this is the same process... There is only one process.
>> > >> >
>> > >> >
>> > >> > 2010/11/16 Martin Holste <mcholste at gmail.com>
>> > >> >>
>> > >> >> Why do you need separate syslog-ng processes running?
>> > >> >>
>> > >> >> On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm at gmail.com>
>> wrote:
>> > >> >> > Hi !
>> > >> >> >
>> > >> >> > I have a question about the use of udp, unix-stream or pipe. I
>> would
>> > >> >> > like to
>> > >> >> > forward a syslog message to the same syslog server like this :
>> > >> >> >
>> > >> >> > |  log {
>> > >> >> > |     source (s_r_udp);    (<-- listen on UDP/514)
>> > >> >> > |
>> > >> >> > |     filter (....);
>> > >> >> > |     filter (....);
>> > >> >> > |     parser (...);
>> > >> >> > |
>> > >> >> > |     destination (d_local_syslog);  (<-- send the message to a
>> local
>> > >> >> > syslog
>> > >> >> > by using unix-stream, udp or pipe mecanism)
>> > >> >> > |  };
>> > >> >> >
>> > >> >> > (...)
>> > >> >> >
>> > >> >> > |  log {
>> > >> >> > |     source (s_local_syslog);    (<--- here I receive the
>> messages
>> > >> >> > sent
>> > >> >> > by
>> > >> >> > the "d_syslog_loop")
>> > >> >> > |
>> > >> >> > |     filter (...);
>> > >> >> > |     filter (...);
>> > >> >> > |     parser (...);
>> > >> >> > |
>> > >> >> > |     destination (d_remote_syslog);
>> > >> >> >
>> > >> >> > I'm looking for the better way to send syslog message to the same
>> > >> >> > syslog
>> > >> >> > server : which mecanism provides the better performances : pipe,
>> udp
>> > >> >> > (by
>> > >> >> > using network) or unix-stream ?
>> > >> >> > Maybe the "pipe" is the better solution ?...
>> > >> >> >
>> > >> >> > I'm using the syslog-ng OSE 3.1.2 on CentOS.
>> > >> >> >
>> > >> >> > Regards,
>> > >> >> >
>> > >> >> > Yann I.
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> ______________________________________________________________________________
>> > >> >> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > >> >> > Documentation:
>> > >> >> > http://www.balabit.com/support/documentation/?product=syslog-ng
>> > >> >> > FAQ: http://www.campin.net/syslog-ng/faq.html
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >>
>> > >> >>
>> > >> >>
>> ______________________________________________________________________________
>> > >> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > >> >> Documentation:
>> > >> >> http://www.balabit.com/support/documentation/?product=syslog-ng
>> > >> >> FAQ: http://www.campin.net/syslog-ng/faq.html
>> > >> >>
>> > >> >
>> > >> >
>> > >>
>> > >>
>> ______________________________________________________________________________
>> > >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > >> Documentation:
>> > >> http://www.balabit.com/support/documentation/?product=syslog-ng
>> > >> FAQ: http://www.campin.net/syslog-ng/faq.html
>> > >>
>> > >
>> > >
>> >
>> ______________________________________________________________________________
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> > FAQ: http://www.campin.net/syslog-ng/faq.html
>> >
>>