Re: [syslog-ng] Better performance between udp, unix-stream or pipe ?
As a connectionless protocol UDP cannot always detect existing listeners on all platforms. However if you think about it it's clear why this wouldn't work. How would it know which messages should be parse or noparse? It can't read your mind! ;) Matthew. Sent from my mobile device "Yann I." <yann.frm@gmail.com> wrote:
Does this work?
source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; source s_udp { udp(port(514)); };
(...)
I tried that solution few days ago and it didn't work. Two processes "syslog-ng" listened on port UDP/514 : the messages are not handled correctly. They would be handled by "s_udp_not_parsed" or "s_udp".
nb : it's strange that two processes can listen on the same port (for the protocol UDP) isn't it ? This is the same behaviour with "netcat". I tried with netcat (example : nc -l -u 1234) and I have two processes which listen the port UDP/1234. Messages send by "nc" on the port 1234 are received by the last "netcat" which has been started.
For passing yourself back the reparsed message I would recommend
unix-dgram. AF_UNIX is usually better than pipes for me at least.
Thank you ! I'll use AF_UNIX.
Regards,
Yann I.
2010/11/16 Matthew Hall <mhall@mhcomputing.net>
For passing yourself back the reparsed message I would recommend unix-dgram. AF_UNIX is usually better than pipes for me at least.
Matthew.
On Tue, Nov 16, 2010 at 12:14:53PM -0600, Martin Holste wrote:
Does this work?
source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; source s_udp { udp(port(514)); };
log { source(s_udp); parser(db-parser()); destination(d_parsed); }; log { source(s_udp_not_parsed); destination(d_not_parsed); };
Otherwise, try reassembling a no-parse like message with a different output template.
On Tue, Nov 16, 2010 at 11:13 AM, Yann I. <yann.frm@gmail.com> wrote:
Well I'm not sure because of the flag I used for the UDP source which is set to "no-parse".
Here is my problem. From the UDP source, I may receive logs which are not "syslog compliance". So I'm using the flag 'no-parse' then I rewrite the message. After that rewrite, I forward the new message to the same syslog-ng server. Then... I can apply filter, parser, etc on that new message which is now "syslog compliance" :-)
So, I think I can't use log statement. I need to use that mecanism... There might be another solution but this one seems to be a good solution.
2010/11/16 Martin Holste <mcholste@gmail.com>
Ok, then this should be accomplished with a standard log statement like you've already begun to write. What do your destinations look like?
On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm@gmail.com> wrote:
In fact, this is the same process... There is only one process.
2010/11/16 Martin Holste <mcholste@gmail.com> > > Why do you need separate syslog-ng processes running? > > On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com>
wrote:
> > Hi ! > > > > I have a question about the use of udp, unix-stream or pipe. I would > > like to > > forward a syslog message to the same syslog server like this : > > > > | log { > > | source (s_r_udp); (<-- listen on UDP/514) > > | > > | filter (....); > > | filter (....); > > | parser (...); > > | > > | destination (d_local_syslog); (<-- send the message to a local > > syslog > > by using unix-stream, udp or pipe mecanism) > > | }; > > > > (...) > > > > | log { > > | source (s_local_syslog); (<--- here I receive the messages > > sent > > by > > the "d_syslog_loop") > > | > > | filter (...); > > | filter (...); > > | parser (...); > > | > > | destination (d_remote_syslog); > > > > I'm looking for the better way to send syslog message to the same > > syslog > > server : which mecanism provides the better performances : pipe, udp > > (by > > using network) or unix-stream ? > > Maybe the "pipe" is the better solution ?... > > > > I'm using the syslog-ng OSE 3.1.2 on CentOS. > > > > Regards, > > > > Yann I. > > > > > > > >
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: > > http://www.balabit.com/support/documentation/?product=syslog-ng > > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > > > > >
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.campin.net/syslog-ng/faq.html >
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
You're right Matthew and this is why I won't use that solution (2 UDP sources). It was just a test. :-) Instead, I'll use only one UDP source with the flag "no-parse" and a patterndb. Regards, Yann 2010/11/17 Matthew Hall <mhall@mhcomputing.net>
As a connectionless protocol UDP cannot always detect existing listeners on all platforms. However if you think about it it's clear why this wouldn't work. How would it know which messages should be parse or noparse? It can't read your mind! ;)
Matthew.
Sent from my mobile device
"Yann I." <yann.frm@gmail.com> wrote:
Does this work?
source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; source s_udp { udp(port(514)); };
(...)
I tried that solution few days ago and it didn't work. Two processes "syslog-ng" listened on port UDP/514 : the messages are not handled correctly. They would be handled by "s_udp_not_parsed" or "s_udp".
nb : it's strange that two processes can listen on the same port (for the protocol UDP) isn't it ? This is the same behaviour with "netcat". I tried with netcat (example : nc -l -u 1234) and I have two processes which listen the port UDP/1234. Messages send by "nc" on the port 1234 are received by the last "netcat" which has been started.
For passing yourself back the reparsed message I would recommend
unix-dgram. AF_UNIX is usually better than pipes for me at least.
Thank you ! I'll use AF_UNIX.
Regards,
Yann I.
2010/11/16 Matthew Hall <mhall@mhcomputing.net>
For passing yourself back the reparsed message I would recommend unix-dgram. AF_UNIX is usually better than pipes for me at least.
Matthew.
On Tue, Nov 16, 2010 at 12:14:53PM -0600, Martin Holste wrote:
Does this work?
source s_udp_not_parsed { udp(port(514) flags(no-parse)); }; source s_udp { udp(port(514)); };
log { source(s_udp); parser(db-parser()); destination(d_parsed); }; log { source(s_udp_not_parsed); destination(d_not_parsed); };
Otherwise, try reassembling a no-parse like message with a different output template.
On Tue, Nov 16, 2010 at 11:13 AM, Yann I. <yann.frm@gmail.com> wrote:
Well I'm not sure because of the flag I used for the UDP source which is set to "no-parse".
Here is my problem. From the UDP source, I may receive logs which are not "syslog compliance". So I'm using the flag 'no-parse' then I rewrite the message. After that rewrite, I forward the new message to the same syslog-ng server. Then... I can apply filter, parser, etc on that new message which is now "syslog compliance" :-)
So, I think I can't use log statement. I need to use that mecanism... There might be another solution but this one seems to be a good solution.
2010/11/16 Martin Holste <mcholste@gmail.com>
Ok, then this should be accomplished with a standard log statement like you've already begun to write. What do your destinations look like?
On Tue, Nov 16, 2010 at 10:58 AM, Yann I. <yann.frm@gmail.com>
wrote:
> In fact, this is the same process... There is only one process. > > > 2010/11/16 Martin Holste <mcholste@gmail.com> >> >> Why do you need separate syslog-ng processes running? >> >> On Tue, Nov 16, 2010 at 10:49 AM, Yann I. <yann.frm@gmail.com> wrote: >> > Hi ! >> > >> > I have a question about the use of udp, unix-stream or pipe. I would >> > like to >> > forward a syslog message to the same syslog server like this : >> > >> > | log { >> > | source (s_r_udp); (<-- listen on UDP/514) >> > | >> > | filter (....); >> > | filter (....); >> > | parser (...); >> > | >> > | destination (d_local_syslog); (<-- send the message to a local >> > syslog >> > by using unix-stream, udp or pipe mecanism) >> > | }; >> > >> > (...) >> > >> > | log { >> > | source (s_local_syslog); (<--- here I receive the messages >> > sent >> > by >> > the "d_syslog_loop") >> > | >> > | filter (...); >> > | filter (...); >> > | parser (...); >> > | >> > | destination (d_remote_syslog); >> > >> > I'm looking for the better way to send syslog message to the same >> > syslog >> > server : which mecanism provides the better performances : pipe, udp >> > (by >> > using network) or unix-stream ? >> > Maybe the "pipe" is the better solution ?... >> > >> > I'm using the syslog-ng OSE 3.1.2 on CentOS. >> > >> > Regards, >> > >> > Yann I. >> > >> >
On Thursday, November 18, 2010 07:55:00 Yann I. wrote:
You're right Matthew and this is why I won't use that solution (2 UDP sources). It was just a test. :-) Instead, I'll use only one UDP source with the flag "no-parse" and a patterndb.
You can use two UDP sources if you need it of course but they just can't be on the same port. ;-)
Regards, Yann
Matthew.
Yes of course... I could use several IP addresses too. ;-) In fact the devices which send messages can't be configured to send these messages to another port than 514. Yann 2010/11/18 Matthew Hall <mhall@mhcomputing.net>
On Thursday, November 18, 2010 07:55:00 Yann I. wrote:
You're right Matthew and this is why I won't use that solution (2 UDP sources). It was just a test. :-) Instead, I'll use only one UDP source with the flag "no-parse" and a patterndb.
You can use two UDP sources if you need it of course but they just can't be on the same port. ;-)
Regards, Yann
Matthew.
participants (2)
-
Matthew Hall
-
Yann I.