Auto routing to specific relay based on hostname
Guys, I am a newbie to syslog-ng solution. I am trying to create a common config file to be distributed across client machines in San Jose, Dallas and Dublin colos. Every colo has 1 dedicated relay which forwards log data to a central server (SJ|DU|DA)_ client --> (SJ|DU|DA)_ relay --> SYSLOG-NG server the clients located in these colos have sj,du & da in their hostnames eg da1xxx058, du1xxx059, sj1xxx060. I want to setup the config in a way that based on hostname the config decides what relay to connect to. Is this possible ? -- Regards, Sagar Naravane
sagar naravane <sagar.naravane@gmail.com> writes:
I am a newbie to syslog-ng solution. I am trying to create a common config file to be distributed across client machines in San Jose, Dallas and Dublin colos. Every colo has 1 dedicated relay which forwards log data to a central server
(SJ|DU|DA)_ client --> (SJ|DU|DA)_ relay --> SYSLOG-NG server
the clients located in these colos have sj,du & da in their hostnames eg da1xxx058, du1xxx059, sj1xxx060. I want to setup the config in a way that based on hostname the config decides what relay to connect to.
Is this possible ?
Yep, it is. One way to do it is to use filters: you set up a filter that matches each colo, and set up your destinations so that each message that matches a given colo filter, goes to the appropriate relay. Something along these lines: filter f_colo_sj { host("^sj"); }; filter f_colo_du { host("^du"); }; filter f_colo_da { host("^da"); }; destination d_relay_sj { ... }; ... log { source(s_local); filter(f_colo_sj); destination(d_relay_sh); flags(final); }; log { source(s_local); filter(f_colo_du); destination(d_relay_du); flags(final); }; ...and so on and so forth. The disadvantage here is that you need a filter and a destination for each relay, but with only 3 relays, that's bearable. -- |8]
Shouldn't it be filter f_colo_sj { host("^sj1*"); }; filter f_colo_du { host("^du1*"); }; filter f_colo_da { host("^da1*"); }; Sagar On Wed, Aug 22, 2012 at 2:55 PM, Gergely Nagy <algernon@balabit.hu> wrote:
sagar naravane <sagar.naravane@gmail.com> writes:
I am a newbie to syslog-ng solution. I am trying to create a common config file to be distributed across client machines in San Jose, Dallas and Dublin colos. Every colo has 1 dedicated relay which forwards log data to a central server
(SJ|DU|DA)_ client --> (SJ|DU|DA)_ relay --> SYSLOG-NG server
the clients located in these colos have sj,du & da in their hostnames eg da1xxx058, du1xxx059, sj1xxx060. I want to setup the config in a way that based on hostname the config decides what relay to connect to.
Is this possible ?
Yep, it is.
One way to do it is to use filters: you set up a filter that matches each colo, and set up your destinations so that each message that matches a given colo filter, goes to the appropriate relay.
Something along these lines:
filter f_colo_sj { host("^sj"); }; filter f_colo_du { host("^du"); }; filter f_colo_da { host("^da"); };
destination d_relay_sj { ... }; ...
log { source(s_local); filter(f_colo_sj); destination(d_relay_sh); flags(final); };
log { source(s_local); filter(f_colo_du); destination(d_relay_du); flags(final); };
...and so on and so forth. The disadvantage here is that you need a filter and a destination for each relay, but with only 3 relays, that's bearable.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Regards, Sagar Naravane
Gergely, Here all three log {} gets executed. What i am basically looking for is a "if..else" or "case" sort of condition where only one of client-relay communication happens based on hostname of client system. Aug 22 02:33:39 sj1slm227 syslog-ng[1166]: Syslog connection established; fd='12', server='AF_INET(10.5.81.89:514)', local='AF_INET(0.0.0.0:0)' *Aug 22 02:36:48 sj1slm227 syslog-ng[1166]: Syslog connection failed; fd='66', server='AF_INET(10.116.104.86:514)', error='Connection timed out (110)', time_reopen='60'* *Aug 22 02:36:48 sj1slm227 syslog-ng[1166]: Syslog connection failed; fd='65', server='AF_INET(10.129.104.21:514)', error='Connection timed out (110)', time_reopen='60'* On Wed, Aug 22, 2012 at 3:01 PM, sagar naravane <sagar.naravane@gmail.com>wrote:
Shouldn't it be
filter f_colo_sj { host("^sj1*"); }; filter f_colo_du { host("^du1*"); }; filter f_colo_da { host("^da1*"); };
Sagar
On Wed, Aug 22, 2012 at 2:55 PM, Gergely Nagy <algernon@balabit.hu> wrote:
sagar naravane <sagar.naravane@gmail.com> writes:
I am a newbie to syslog-ng solution. I am trying to create a common config file to be distributed across client machines in San Jose, Dallas and Dublin colos. Every colo has 1 dedicated relay which forwards log data to a central server
(SJ|DU|DA)_ client --> (SJ|DU|DA)_ relay --> SYSLOG-NG server
the clients located in these colos have sj,du & da in their hostnames eg da1xxx058, du1xxx059, sj1xxx060. I want to setup the config in a way that based on hostname the config decides what relay to connect to.
Is this possible ?
Yep, it is.
One way to do it is to use filters: you set up a filter that matches each colo, and set up your destinations so that each message that matches a given colo filter, goes to the appropriate relay.
Something along these lines:
filter f_colo_sj { host("^sj"); }; filter f_colo_du { host("^du"); }; filter f_colo_da { host("^da"); };
destination d_relay_sj { ... }; ...
log { source(s_local); filter(f_colo_sj); destination(d_relay_sh); flags(final); };
log { source(s_local); filter(f_colo_du); destination(d_relay_du); flags(final); };
...and so on and so forth. The disadvantage here is that you need a filter and a destination for each relay, but with only 3 relays, that's bearable.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Regards,
Sagar Naravane
-- Regards, Sagar Naravane
sagar naravane <sagar.naravane@gmail.com> writes:
Gergely,
Here all three log {} gets executed. What i am basically looking for is a "if..else" or "case" sort of condition where only one of client-relay communication happens based on hostname of client system.
That's what flags(final) is for. It will connect nevertheless, but will only send data when the filter matches. syslog-ng 3.4 might be a tiny bit better in this regard, you can more closely model your requirement there, but as far as I understand, even that would try to connect to all three relays. On the other hand, there may be another way, which works slightly differently: it basically makes syslog-ng call out to a shell script when it starts up, to determine the host name, and set up the configuration according to that. Something like: @module confgen context(destination) name(relay) exec("/path/to/script.sh") destination d_relay { relay(); }; log { source(s_local); destination(d_relay); }; Where the script would look something along these lines: ,---- | #! /bin/sh | set -e | | h=$(hostname) | case h in | sj1*) | relay="sj1-relay.localnet" | ;; | *) | echo "Unknown host: $h!" >&2 | exit 1 | ;; | esac | | cat <<EOF | tcp("${relay}"); | EOF `---- This has the advantage of not requiring a filter, and that all conditional stuff is performed at config load time, not for each and every message. Also, it will only ever connect to one single destination. The disadvantage is that the config isn't entirely contained in syslog-ng.conf, but you use an external script to generate parts of it. Also, the above solution requires syslog-ng 3.3+, while filters work with older versions too. Mind you, upgrading to 3.3 would be strongly recommended anyway :) -- |8]
Script worked like a charm... Thanks Gergely :) Sagar On Wed, Aug 22, 2012 at 3:40 PM, Gergely Nagy <algernon@balabit.hu> wrote:
sagar naravane <sagar.naravane@gmail.com> writes:
Gergely,
Here all three log {} gets executed. What i am basically looking for is a "if..else" or "case" sort of condition where only one of client-relay communication happens based on hostname of client system.
That's what flags(final) is for. It will connect nevertheless, but will only send data when the filter matches.
syslog-ng 3.4 might be a tiny bit better in this regard, you can more closely model your requirement there, but as far as I understand, even that would try to connect to all three relays.
On the other hand, there may be another way, which works slightly differently: it basically makes syslog-ng call out to a shell script when it starts up, to determine the host name, and set up the configuration according to that.
Something like:
@module confgen context(destination) name(relay) exec("/path/to/script.sh") destination d_relay { relay(); }; log { source(s_local); destination(d_relay); };
Where the script would look something along these lines:
,---- | #! /bin/sh | set -e | | h=$(hostname) | case h in | sj1*) | relay="sj1-relay.localnet" | ;; | *) | echo "Unknown host: $h!" >&2 | exit 1 | ;; | esac | | cat <<EOF | tcp("${relay}"); | EOF `----
This has the advantage of not requiring a filter, and that all conditional stuff is performed at config load time, not for each and every message. Also, it will only ever connect to one single destination.
The disadvantage is that the config isn't entirely contained in syslog-ng.conf, but you use an external script to generate parts of it.
Also, the above solution requires syslog-ng 3.3+, while filters work with older versions too. Mind you, upgrading to 3.3 would be strongly recommended anyway :)
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Regards, Sagar Naravane
sagar naravane <sagar.naravane@gmail.com> writes:
Shouldn't it be
filter f_colo_sj { host("^sj1*"); }; filter f_colo_du { host("^du1*"); }; filter f_colo_da { host("^da1*"); };
It's a regex, so .*, and as far as I remember, no, you don't need to fully match the host, a partial match is fine. So if the beginning matches, the filter will match, and ^sj (or ^sj1) is faster than ^sj1.*, since the regexp engine does not have to process the whole string. (I might be wrong, though, in which case ^sj1.* is the correct one) -- |8]
participants (2)
-
Gergely Nagy
-
sagar naravane