Better performance between udp, unix-stream or pipe ?
Hi ! I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this : | log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | }; (...) | log { | source (s_local_syslog); (<--- here I receive the messages sent by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog); I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?... I'm using the syslog-ng OSE 3.1.2 on CentOS. Regards, Yann I.
Why do you need separate syslog-ng processes running? On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com> wrote:
Hi !
I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this :
| log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | };
(...)
| log { | source (s_local_syslog); (<--- here I receive the messages sent by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog);
I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?...
I'm using the syslog-ng OSE 3.1.2 on CentOS.
Regards,
Yann I.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
In fact, this is the same process... There is only one process. 2010/11/16 Martin Holste <mcholste@gmail.com>
Why do you need separate syslog-ng processes running?
On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com> wrote:
Hi !
I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this :
| log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | };
(...)
| log { | source (s_local_syslog); (<--- here I receive the messages sent by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog);
I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?...
I'm using the syslog-ng OSE 3.1.2 on CentOS.
Regards,
Yann I.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Ok, then this should be accomplished with a standard log statement like you've already begun to write. What do your destinations look like? On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm@gmail.com> wrote:
In fact, this is the same process... There is only one process.
2010/11/16 Martin Holste <mcholste@gmail.com>
Why do you need separate syslog-ng processes running?
On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com> wrote:
Hi !
I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this :
| log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | };
(...)
| log { | source (s_local_syslog); (<--- here I receive the messages sent by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog);
I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?...
I'm using the syslog-ng OSE 3.1.2 on CentOS.
Regards,
Yann I.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Well I'm not sure because of the flag I used for the UDP source which is set to "no-parse". Here is my problem. From the UDP source, I may receive logs which are not "syslog compliance". So I'm using the flag 'no-parse' then I rewrite the message. After that rewrite, I forward the new message to the same syslog-ng server. Then... I can apply filter, parser, etc on that new message which is now "syslog compliance" :-) So, I think I can't use log statement. I need to use that mecanism... There might be another solution but this one seems to be a good solution. 2010/11/16 Martin Holste <mcholste@gmail.com>
Ok, then this should be accomplished with a standard log statement like you've already begun to write. What do your destinations look like?
On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm@gmail.com> wrote:
In fact, this is the same process... There is only one process.
2010/11/16 Martin Holste <mcholste@gmail.com>
Why do you need separate syslog-ng processes running?
On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com> wrote:
Hi !
I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this :
| log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | };
(...)
| log { | source (s_local_syslog); (<--- here I receive the messages
sent
by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog);
I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?...
I'm using the syslog-ng OSE 3.1.2 on CentOS.
Regards,
Yann I.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Does this work? source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; source s_udp { udp(port(514)); }; log { source(s_udp); parser(db-parser()); destination(d_parsed); }; log { source(s_udp_not_parsed); destination(d_not_parsed); }; Otherwise, try reassembling a no-parse like message with a different output template. On Tue, Nov 16, 2010 at 11:13 AM, Yann I. <yann.frm@gmail.com> wrote:
Well I'm not sure because of the flag I used for the UDP source which is set to "no-parse".
Here is my problem. From the UDP source, I may receive logs which are not "syslog compliance". So I'm using the flag 'no-parse' then I rewrite the message. After that rewrite, I forward the new message to the same syslog-ng server. Then... I can apply filter, parser, etc on that new message which is now "syslog compliance" :-)
So, I think I can't use log statement. I need to use that mecanism... There might be another solution but this one seems to be a good solution.
2010/11/16 Martin Holste <mcholste@gmail.com>
Ok, then this should be accomplished with a standard log statement like you've already begun to write. What do your destinations look like?
On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm@gmail.com> wrote:
In fact, this is the same process... There is only one process.
2010/11/16 Martin Holste <mcholste@gmail.com>
Why do you need separate syslog-ng processes running?
On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com> wrote:
Hi !
I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this :
| log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | };
(...)
| log { | source (s_local_syslog); (<--- here I receive the messages sent by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog);
I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?...
I'm using the syslog-ng OSE 3.1.2 on CentOS.
Regards,
Yann I.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
For passing yourself back the reparsed message I would recommend unix-dgram. AF_UNIX is usually better than pipes for me at least. Matthew. On Tue, Nov 16, 2010 at 12:14:53PM -0600, Martin Holste wrote:
Does this work?
source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; source s_udp { udp(port(514)); };
log { source(s_udp); parser(db-parser()); destination(d_parsed); }; log { source(s_udp_not_parsed); destination(d_not_parsed); };
Otherwise, try reassembling a no-parse like message with a different output template.
On Tue, Nov 16, 2010 at 11:13 AM, Yann I. <yann.frm@gmail.com> wrote:
Well I'm not sure because of the flag I used for the UDP source which is set to "no-parse".
Here is my problem. From the UDP source, I may receive logs which are not "syslog compliance". So I'm using the flag 'no-parse' then I rewrite the message. After that rewrite, I forward the new message to the same syslog-ng server. Then... I can apply filter, parser, etc on that new message which is now "syslog compliance" :-)
So, I think I can't use log statement. I need to use that mecanism... There might be another solution but this one seems to be a good solution.
2010/11/16 Martin Holste <mcholste@gmail.com>
Ok, then this should be accomplished with a standard log statement like you've already begun to write. What do your destinations look like?
On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm@gmail.com> wrote:
In fact, this is the same process... There is only one process.
2010/11/16 Martin Holste <mcholste@gmail.com>
Why do you need separate syslog-ng processes running?
On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com> wrote:
Hi !
I have a question about the use of udp, unix-stream or pipe. I would like to forward a syslog message to the same syslog server like this :
| log { | source (s_r_udp); (<-- listen on UDP/514) | | filter (....); | filter (....); | parser (...); | | destination (d_local_syslog); (<-- send the message to a local syslog by using unix-stream, udp or pipe mecanism) | };
(...)
| log { | source (s_local_syslog); (<--- here I receive the messages sent by the "d_syslog_loop") | | filter (...); | filter (...); | parser (...); | | destination (d_remote_syslog);
I'm looking for the better way to send syslog message to the same syslog server : which mecanism provides the better performances : pipe, udp (by using network) or unix-stream ? Maybe the "pipe" is the better solution ?...
I'm using the syslog-ng OSE 3.1.2 on CentOS.
Regards,
Yann I.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Does this work?
source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; source s_udp { udp(port(514)); };
(...)
I tried that solution few days ago and it didn't work. Two processes "syslog-ng" listened on port UDP/514 : the messages are not handled correctly. They would be handled by "s_udp_not_parsed" or "s_udp". nb : it's strange that two processes can listen on the same port (for the protocol UDP) isn't it ? This is the same behaviour with "netcat". I tried with netcat (example : nc -l -u 1234) and I have two processes which listen the port UDP/1234. Messages send by "nc" on the port 1234 are received by the last "netcat" which has been started. For passing yourself back the reparsed message I would recommend
unix-dgram. AF_UNIX is usually better than pipes for me at least.
Thank you ! I'll use AF_UNIX. Regards, Yann I. 2010/11/16 Matthew Hall <mhall@mhcomputing.net>
For passing yourself back the reparsed message I would recommend unix-dgram. AF_UNIX is usually better than pipes for me at least.
Matthew.
On Tue, Nov 16, 2010 at 12:14:53PM -0600, Martin Holste wrote:
Does this work?
source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; source s_udp { udp(port(514)); };
log { source(s_udp); parser(db-parser()); destination(d_parsed); }; log { source(s_udp_not_parsed); destination(d_not_parsed); };
Otherwise, try reassembling a no-parse like message with a different output template.
On Tue, Nov 16, 2010 at 11:13 AM, Yann I. <yann.frm@gmail.com> wrote:
Well I'm not sure because of the flag I used for the UDP source which is set to "no-parse".
Here is my problem. From the UDP source, I may receive logs which are not "syslog compliance". So I'm using the flag 'no-parse' then I rewrite the message. After that rewrite, I forward the new message to the same syslog-ng server. Then... I can apply filter, parser, etc on that new message which is now "syslog compliance" :-)
So, I think I can't use log statement. I need to use that mecanism... There might be another solution but this one seems to be a good solution.
2010/11/16 Martin Holste <mcholste@gmail.com>
Ok, then this should be accomplished with a standard log statement like you've already begun to write. What do your destinations look like?
On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm@gmail.com> wrote:
In fact, this is the same process... There is only one process.
2010/11/16 Martin Holste <mcholste@gmail.com>
Why do you need separate syslog-ng processes running?
On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com>
wrote:
> Hi ! > > I have a question about the use of udp, unix-stream or pipe. I would > like to > forward a syslog message to the same syslog server like this : > > | log { > | source (s_r_udp); (<-- listen on UDP/514) > | > | filter (....); > | filter (....); > | parser (...); > | > | destination (d_local_syslog); (<-- send the message to a local > syslog > by using unix-stream, udp or pipe mecanism) > | }; > > (...) > > | log { > | source (s_local_syslog); (<--- here I receive the messages > sent > by > the "d_syslog_loop") > | > | filter (...); > | filter (...); > | parser (...); > | > | destination (d_remote_syslog); > > I'm looking for the better way to send syslog message to the same > syslog > server : which mecanism provides the better performances : pipe, udp > (by > using network) or unix-stream ? > Maybe the "pipe" is the better solution ?... > > I'm using the syslog-ng OSE 3.1.2 on CentOS. > > Regards, > > Yann I. > > > >
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html > > >
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
participants (3)
-
Martin Holste
-
Matthew Hall
-
Yann I.