Hi all I am planning to do some capacity and load testing our syslog-ng servers over TCP. Towards that end, I will be writing a script to send packets to a syslog-ng host. Has anyone done anything similar? I was just thinking of writing a simple script that will open multiple sockets to the server and send a message (based upon our current calculated average message size). Any other suggestions are appreciated. Thanks,
On Thu, Sep 28, 2006 at 09:36:10AM -0700, Vaibhav Goel wrote:
I am planning to do some capacity and load testing our syslog-ng servers over TCP. Towards that end, I will be writing a script to send packets to a syslog-ng host. Has anyone done anything similar? I was just thinking of writing a simple script that will open multiple sockets to the server and send a message (based upon our current calculated average message size). Any other suggestions are appreciated.
The graphs here: http://www.campin.net/syslog-ng/faq.html#perf Are from a perl script that simply looks in syslog-ng logs for a particular "starting test" string that includes how many test messages are going to be sent, then looks for that many messages all with a sequence number appended to real-world syslog messages (a collection of about 50 messages I grabbed), then reports any numbers missing from the sequence when it gets near the total number of messages (doesn't depend on the actual last log making it in case it's lost), and reports on the time period between when the first and last message were logged to the file. Pretty simple. I did the log scraper in perl to generate those graphs via gd::graph or something like that, but you could just spit out CSV and graph it with something like (*gasp*) Excel. I can't share those scripts since they were written while working for a large, evil corporation, but the logic is quite simple. I had it measure CPU and also network stats (network stats were a separate graph not pictured on the site) but that's certainly not mandatory. Just make sure that if you measure CPU that you also see how much CPU the testing/analysis script uses - might be more than you'd think. -- Nate There are two ways to write error-free programs. Only the third one works. -Anon.
I am thinking of a way to use regular expressions more efficiently by adding a fixed string match along with the regular expression. filter f_xntp_filter_no_regexp { # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); program("xntpd") and match("time error .* is way too large .* set clock manually"); }; which will evaluate the regular expression for all messages. The syslog-ng FAQ recommends the filter filter f_xntp_filter_no_regexp { # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); program("xntpd") and match("time error") and match("is way too large") and match("set clock manually"); }; but that would match the message "xntpd[1234]: length is way too large of a time error to make set clock manually succeed." which isn't what I want to match. I would like to do filter f_xntp_filter_no_regexp { # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); program("xntpd") and match("time error") and match("is way too large") and match("set clock manually") and match("time error .* is way too large .* set clock manually"); }; So, the question is if the message does not match any of the fixed strings, does the regular expression get evaluate, or does the first non-matching fixed string cause the filter evaluation to abort? Anyone have any insights?
I was doing this on AIX 5.2 If I have multiple destination defined to the same destination (because I want different formats) I get the Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' This destination host had 9 coresponding "destination" and "log" configurations. Evan.
On Thu, 2006-10-05 at 13:43 -0700, Evan Rempel wrote:
I was doing this on AIX 5.2
If I have multiple destination defined to the same destination (because I want different formats) I get the Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))'
This destination host had 9 coresponding "destination" and "log" configurations.
Hmm.. each destination has a generated name, which is supposed to be unique, and that does not include the template string. Is it a problem if processed/dropped counters are shared between these writers? -- Bazsi
Balazs Scheidler wrote:
On Thu, 2006-10-05 at 13:43 -0700, Evan Rempel wrote:
I was doing this on AIX 5.2
If I have multiple destination defined to the same destination (because I want different formats) I get the Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))' Duplicate stats counter; counter='udp(AF_INET(xxx))'
This destination host had 9 coresponding "destination" and "log" configurations.
Hmm.. each destination has a generated name, which is supposed to be unique, and that does not include the template string. Is it a problem if processed/dropped counters are shared between these writers?
For statistics, I don't have any problem that the same counter is used. Currently, I get a single element dropped statistic for the protocal/IP pair, which in my opinion is goo since it is a single communications channel, and is sufficient for trouble shooting and auditing. Example: Log statistics; dropped='tcp(AF_INET(142.104.5.206:514))=0', dropped='udp(AF_INET(142.104.5.76:514))=0', processed='center(queued)=393', processed='center(received)=131', processed='destination(syslogServer1.local2.debug)=0', processed='destination(syslogServer1.local2.notice)=0', processed='destination(syslogServer1.local2.crit)=0', processed='destination(syslogServer1.local2.err)=0', processed='destination(syslogServer2)=131', processed='destination(syslogServer1.local2.alert)=0', processed='destination(syslogServer1)=131', processed='destination(syslog)=131', processed='destination(syslogServer1.local2.info)=0', processed='destination(syslogServer1.local2.emerg)=0', processed='destination(syslogServer1.local2.warning)=0', processed='source(local)=131' If everything is working as designed, and correctly, I would NOT expect to get the messages about Duplicate stats counters which imply that there is some kind of problem. Perhaps those messages could be removed. Evan.
Running Syslog-ng 2.0rc3 compiled on AIX 5.2 with xlc TCP connection to syslong-ng 1.6.9 on Linux. AIX config has options { sync(0); log_fifo_size(10000); use_fqdn(yes); keep_hostname(no); chain_hostnames(no); time_reap(60); time_reopen(5); }; When I hup the receiving syslong-ng, the TCP connection is dropped. The AIX syslog-ng 2.0rc3 does NOT reconnect after 5 seconds. sending a SIGHUP to the sylog-ng process causes it to reconnect, but the queued messages do not get sent. I am not sure if syslog-ng was unaware of the failed connection, resulting in it not queueing the messages, or if syslong-ng dropped its queued messages when it recieved a SIGHUP. local files continue to get the messages and remote udp continue to get the messages, but they don't get delivered to the tcp destination. Evan.
OK, where do I go from here. The reconnect feature is mandatory to get TCP/IP syslog functioning in a useful manner. Evan. Evan Rempel wrote:
Running Syslog-ng 2.0rc3 compiled on AIX 5.2 with xlc TCP connection to syslong-ng 1.6.9 on Linux.
AIX config has
options { sync(0); log_fifo_size(10000); use_fqdn(yes); keep_hostname(no); chain_hostnames(no); time_reap(60); time_reopen(5); };
When I hup the receiving syslong-ng, the TCP connection is dropped. The AIX syslog-ng 2.0rc3 does NOT reconnect after 5 seconds. sending a SIGHUP to the sylog-ng process causes it to reconnect, but the queued messages do not get sent.
I am not sure if syslog-ng was unaware of the failed connection, resulting in it not queueing the messages, or if syslong-ng dropped its queued messages when it recieved a SIGHUP.
local files continue to get the messages and remote udp continue to get the messages, but they don't get delivered to the tcp destination.
Evan. _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
On Tue, 2006-10-17 at 09:29 -0700, Evan Rempel wrote:
OK, where do I go from here. The reconnect feature is mandatory to get TCP/IP syslog functioning in a useful manner.
I'm looking into it as soon as possible, whenever that is. -- Bazsi
This problem is resolved. I think it was an issue with glib 2.12.3 The change log for glib 2.12.4 contains Overview of Changes from GLib 2.12.3 to GLib 2.12.4 =================================================== * Fix build problems related to Posix timers. * Bugs fixed 321974 nanosleep on AIX / g_timer API using high resolution timers which I think is the real answer. I built this and used syslog-ng 2.0.0 and this problem is gone. Evan. Evan Rempel wrote:
Running Syslog-ng 2.0rc3 compiled on AIX 5.2 with xlc TCP connection to syslong-ng 1.6.9 on Linux.
AIX config has
options { sync(0); log_fifo_size(10000); use_fqdn(yes); keep_hostname(no); chain_hostnames(no); time_reap(60); time_reopen(5); };
When I hup the receiving syslong-ng, the TCP connection is dropped. The AIX syslog-ng 2.0rc3 does NOT reconnect after 5 seconds. sending a SIGHUP to the sylog-ng process causes it to reconnect, but the queued messages do not get sent.
I am not sure if syslog-ng was unaware of the failed connection, resulting in it not queueing the messages, or if syslong-ng dropped its queued messages when it recieved a SIGHUP.
local files continue to get the messages and remote udp continue to get the messages, but they don't get delivered to the tcp destination.
Evan. _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
When logging from an AIX server, the format of the message can be <$PRI>$DATE Message forwarded from $HOST: $MESSAGE and syslog-ng handles this quite nicely, however, if an AIX machine is configured to use the "-s" option (short version) to the AIX syslogd subsystem, the message may be of the format <$PRI>$DATE From $HOST: $MESSAGE It would be nice if syslog-ng handled this as well. I realize that I am asking for syslog-ng to "fix" another vendors problem, but in IBM's defense, starting in AIX 5.2 there is a "-n" option to syslogd that prevents it from prepending anything to a message, resulting in <$PRI>$DATE $MESSAGE unfortunately, there is no host at all. -- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
On Tue, Oct 10, 2006 at 12:30:36PM -0700, Evan Rempel wrote:
When logging from an AIX server, the format of the message can be
<$PRI>$DATE Message forwarded from $HOST: $MESSAGE
and syslog-ng handles this quite nicely, however, if an AIX machine is configured to use the "-s" option (short version) to the AIX syslogd subsystem, the message may be of the format
<$PRI>$DATE From $HOST: $MESSAGE
It would be nice if syslog-ng handled this as well.
I realize that I am asking for syslog-ng to "fix" another vendors problem, but in IBM's defense, starting in AIX 5.2 there is a "-n" option to syslogd that prevents it from prepending anything to a message, resulting in <$PRI>$DATE $MESSAGE
unfortunately, there is no host at all.
This is identical to how Solaris sends syslog messages. See: http://www.campin.net/syslog-ng/syslog.html#problems syslog-ng generally deals well with it, unless you get a program name with a space in it. The config directive bad_hostnames() was added to deal with them. I can explain in more detail if needed. This thread is the one that prompted Bazsi to add the feature: https://lists.balabit.hu/pipermail/syslog-ng/2003-January/004345.html -- Nate I've seen things you people wouldn't believe. Attack ships on fire off the shoulder of Orion. I watched C-beams glitter in the dark near the Tannhauser gate. All those moments will be lost in time, like tears in rain. Time to die. -- Roy Batty, Blade Runner
Nate Campi wrote:
On Tue, Oct 10, 2006 at 12:30:36PM -0700, Evan Rempel wrote:
When logging from an AIX server, the format of the message can be
<$PRI>$DATE Message forwarded from $HOST: $MESSAGE
and syslog-ng handles this quite nicely, however, if an AIX machine is configured to use the "-s" option (short version) to the AIX syslogd subsystem, the message may be of the format
<$PRI>$DATE From $HOST: $MESSAGE
It would be nice if syslog-ng handled this as well.
I realize that I am asking for syslog-ng to "fix" another vendors problem, but in IBM's defense, starting in AIX 5.2 there is a "-n" option to syslogd that prevents it from prepending anything to a message, resulting in <$PRI>$DATE $MESSAGE
unfortunately, there is no host at all.
This is identical to how Solaris sends syslog messages. See:
http://www.campin.net/syslog-ng/syslog.html#problems
syslog-ng generally deals well with it, unless you get a program name with a space in it. The config directive bad_hostnames() was added to deal with them.
I can explain in more detail if needed. This thread is the one that prompted Bazsi to add the feature:
https://lists.balabit.hu/pipermail/syslog-ng/2003-January/004345.html
Yes, except that you missed the part about a message of the format <$PRI>$DATE From $HOST: $MESSAGE that does have a host in it, but it is not the first word, and it also has a : in it. Looking at the source code, this specific format is not handled the same as the format <$PRI>$DATE Message forwarded from $HOST: $MESSAGE which is handled explicitly. It is this shortened relay format that I would like to have added to syslong-ng. Evan. -- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
On Wed, Oct 11, 2006 at 09:00:25AM -0700, Evan Rempel wrote:
Yes, except that you missed the part about a message of the format <$PRI>$DATE From $HOST: $MESSAGE that does have a host in it, but it is not the first word, and it also has a : in it. Looking at the source code, this specific format is not handled the same as the format <$PRI>$DATE Message forwarded from $HOST: $MESSAGE which is handled explicitly.
It is this shortened relay format that I would like to have added to syslong-ng.
Didn't miss it, just letting you know that syslog-ng will deal with the other one. -- Nate Your mantra for today is: Don't let data from the network near a shell. Bad things happen. -- Randall Schwartz
Nate Campi wrote:
On Wed, Oct 11, 2006 at 09:00:25AM -0700, Evan Rempel wrote:
Yes, except that you missed the part about a message of the format <$PRI>$DATE From $HOST: $MESSAGE that does have a host in it, but it is not the first word, and it also has a : in it. Looking at the source code, this specific format is not handled the same as the format <$PRI>$DATE Message forwarded from $HOST: $MESSAGE which is handled explicitly.
It is this shortened relay format that I would like to have added to syslong-ng.
Didn't miss it, just letting you know that syslog-ng will deal with the other one.
OK. Thanks. Evan.
On Thu, 2006-09-28 at 13:02 -0700, Evan Rempel wrote:
I am thinking of a way to use regular expressions more efficiently by adding a fixed string match along with the regular expression.
filter f_xntp_filter_no_regexp { # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); program("xntpd") and match("time error .* is way too large .* set clock manually"); };
which will evaluate the regular expression for all messages.
The syslog-ng FAQ recommends the filter
filter f_xntp_filter_no_regexp { # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); program("xntpd") and match("time error") and match("is way too large") and match("set clock manually");
};
but that would match the message
"xntpd[1234]: length is way too large of a time error to make set clock manually succeed."
which isn't what I want to match.
I would like to do
filter f_xntp_filter_no_regexp { # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); program("xntpd") and match("time error") and match("is way too large") and match("set clock manually") and match("time error .* is way too large .* set clock manually"); };
So, the question is if the message does not match any of the fixed strings, does the regular expression get evaluate, or does the first non-matching fixed string cause the filter evaluation to abort?
syslog-ng bails out if the first operand to an AND operation fails, e.g. it short circuits evaluation just like C (and all sane languages) do. I'm not sure though that doing 3 iterations through the string using the fixed patterns justify for a single regexp evaluation. I'd think that one fixed pattern, at the beginning of the message should filter out the majority of the messages. (and use '^' in the fixed pattern, it should also help) -- Bazsi
participants (4)
-
Balazs Scheidler
-
Evan Rempel
-
Nate Campi
-
Vaibhav Goel