how rewrite the HOST macro?
Hello, i want to use syslog-ng to receive and log messages, the goal is to write log messages to: /appli/syslog-ng/logs/server0/messages but server0 can have multiples names: server0 server0e1 server0e2 server0g1 server0g2 server0gt1 etc... how can retrieve the name "server0" only ? Regards, Mobidyc
There are two places syslog-ng gets host names from -- firstly there is a macro (FROM_HOST ??) which is the name of the machine that syslog-ng received the record from and then there is the HOST macro with contains the the name that was in the hostname field of the syslog record. So, as I see it there are two possible ways to map all the different names to one. Firstly you could relay the syslog records through another machine so they all get the same hostname or secondly you may be able to tweak syslog on the source machine to write the records with a the name server0 in them. I know this isn't exactly what you asked but I don't think there is any way of directly fiddling with the hostname, I'm sure Balasz will correct me if I'm wrong. mobidyc wrote:
Hello,
i want to use syslog-ng to receive and log messages, the goal is to write log messages to: /appli/syslog-ng/logs/server0/messages
but server0 can have multiples names: server0 server0e1 server0e2 server0g1 server0g2 server0gt1 etc...
how can retrieve the name "server0" only ?
Regards, Mobidyc
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Thanks for your answer, is there a way to create my own macro? i could pass the HOST macro through sed with program() and indicate a good destination file path. i've around 300 servers to log, it would be difficult to maintain if all is not coherent. Russell Fulton wrote:
There are two places syslog-ng gets host names from -- firstly there is a macro (FROM_HOST ??) which is the name of the machine that syslog-ng received the record from and then there is the HOST macro with contains the the name that was in the hostname field of the syslog record.
So, as I see it there are two possible ways to map all the different names to one.
Firstly you could relay the syslog records through another machine so they all get the same hostname or secondly you may be able to tweak syslog on the source machine to write the records with a the name server0 in them.
I know this isn't exactly what you asked but I don't think there is any way of directly fiddling with the hostname, I'm sure Balasz will correct me if I'm wrong.
mobidyc wrote:
Hello,
i want to use syslog-ng to receive and log messages, the goal is to write log messages to: /appli/syslog-ng/logs/server0/messages
but server0 can have multiples names: server0 server0e1 server0e2 server0g1 server0g2 server0gt1 etc...
how can retrieve the name "server0" only ?
Regards, Mobidyc
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On 8/11/07, mobidyc <mobidyc@gmail.com> wrote:
i've around 300 servers to log, it would be difficult to maintain if all is not coherent.
Why not just make sure that all of the servers have good reverse DNS entries for the IP from which they would source syslog packets, and let "keep_hostname(no); use_dns(yes);" populate $HOST with the DNS lookup results? That will give keep everything coherent and consistent so long as your DNS and/or /etc/hosts have accurate records for all 300 servers. Kevin
K K wrote:
Why not just make sure that all of the servers have good reverse DNS entries for the IP from which they would source syslog packets, and let "keep_hostname(no); use_dns(yes);" populate $HOST with the DNS lookup results?
because it's impossible to change the way the reverse dns are done. the names server0e0 and server0e1 are here because there is multpile networks for the same machine, i can't ask the network team for change the way they use for a software i want to use. i think i need to find another way for dispatch the messages in the good destinations, maybe dynamically create symlinks before the message pass through the log statment but i must be careful with the cpu consumption in this case. Regards
On 8/12/07, mobidyc <mobidyc@gmail.com> wrote:
K K wrote:
Why not just make sure that all of the servers have good reverse DNS entries for the IP from which they would source syslog packets, and let "keep_hostname(no); use_dns(yes);" populate $HOST with the DNS lookup results?
because it's impossible to change the way the reverse dns are done.
the names server0e0 and server0e1 are here because there is multpile networks for the same machine, i can't ask the network team for change the way they use for a software i want to use.
There is one way -- you can override them locally on the server runnign syslog-ng, so your log server believes the reverse DNS for all those aliases all actually resolve to "server0". You can do this with /etc/hosts if your resolver will check /etc/hosts for reverse DNS (e.g. with a flag in /etc/resolv.conf, but the exact mechanism is OS-dependent).
i think i need to find another way for dispatch the messages in the good destinations, maybe dynamically create symlinks before the message pass through the log statment but i must be careful with the cpu consumption in this case.
Sounds like it would be easier to hack the syslog-ng source code to find where it translates IP addresses to names, and just add code to check if the last two characters are 'e' followed by a number, and if so, truncate. Not that anybody here would endorse making your own unique unsupportable fork, but it would solve your problem. Kevin
2007/8/12, K K <kkadow@xxx.com>:
There is one way -- you can override them locally on the server runnign syslog-ng, so your log server believes the reverse DNS for all those aliases all actually resolve to "server0".
You can do this with /etc/hosts if your resolver will check /etc/hosts for reverse DNS (e.g. with a flag in /etc/resolv.conf, but the exact mechanism is OS-dependent).
i will lost all the dynamic part with it but it's a solution.
Sounds like it would be easier to hack the syslog-ng source code to find where it translates IP addresses to names, and just add code to check if the last two characters are 'e' followed by a number, and if so, truncate.
i think it would be the best solution, i've really few knowledge in C but i'm going to try. thanks for you help ;) -- Mobidyc
Excuse me if you've already thought of this but, I assume your servers have multiple NICs on separate networks. Why would the same server forward its syslog messages to your central log server using different NICs? Wouldn't normal routing force it to connect using one NIC all the time? i.e. is this really a problem? Have you seen the same server use different NICs to talk to your log server? If you really are seeing this (and can't fix it at the source) I suppose you could use a filter to look at the messages and define different destinations for each server, using a regexp on the host to determine which to put where. # filter Call another filter rule and # evaluate its value # host host(regexp) Match messages by # using a regular # expression against # the hostname field # of log messages. So if you had server names as you describe (e.g. server0xx, server1xx ... servernxx) you could define destinations and filters for each server like this (pardon if the syntax is a bit off, I haven't tested this) filter f_server0 { host (server0); }; filter f_server1 { host (server1); }; filter f_servern { host (servern); }; destination d_server0 { file("/var/log/HOSTS/server0/$YEAR/$MONTH/$DAY/$FACILITY_$HOST_$YEAR_$MONTH_ $DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; destination d_server1 { file("/var/log/HOSTS/server1/$YEAR/$MONTH/$DAY/$FACILITY_$HOST_$YEAR_$MONTH_ $DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; destination d_servern { file("/var/log/HOSTS/servern/$YEAR/$MONTH/$DAY/$FACILITY_$HOST_$YEAR_$MONTH_ $DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; log { source(s_remote); filter (f_server0); destination (d_server0); }; log { source(s_remote); filter (f_server1); destination (d_server1); }; log { source(s_remote); filter (f_servern); destination (d_servern); }; Hope this helps. Jim Hendrick jrhendri@maine.rr.com GCFW, GCIA, GCIH, GCWN, GCUX
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of mobidyc Sent: Sunday, August 12, 2007 7:48 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] how rewrite the HOST macro?
K K wrote:
Why not just make sure that all of the servers have good
reverse DNS
entries for the IP from which they would source syslog packets, and let "keep_hostname(no); use_dns(yes);" populate $HOST with the DNS lookup results?
because it's impossible to change the way the reverse dns are done.
the names server0e0 and server0e1 are here because there is multpile networks for the same machine, i can't ask the network team for change the way they use for a software i want to use.
i think i need to find another way for dispatch the messages in the good destinations, maybe dynamically create symlinks before the message pass through the log statment but i must be careful with the cpu consumption in this case.
Regards
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/sysl> og-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
2007/8/12, Jim Hendrick <jrhendri@maine.rr.com>:
Excuse me if you've already thought of this but, I assume your servers have multiple NICs on separate networks. Why would the same server forward its syslog messages to your central log server using different NICs? Wouldn't normal routing force it to connect using one NIC all the time?
no, one reason why we use multiple network is that we have at least one network backup for servers. i.e. is this really a problem? Have you seen the same server use different
NICs to talk to your log server?
yes If you really are seeing this (and can't fix it at the source) I suppose you
could use a filter to look at the messages and define different destinations for each server, using a regexp on the host to determine which to put where.
to difficult to maintain, we have often some servers who are added to the network, i could write a shell script for auto-generate a config file but that seems ugly. but apparently, there is only ugly solutions for what i want: i could either: - put statically my servers in /etc/hosts - put statically my servers in the syslog-ng.conf - auto-create a syslog-ng.conf file each days through a crontab - hack the syslog-ng source - pass by program(); and call a script for auto-generate symlink as you can see, there is multiple solutions, i will choose one of them i think. thanks all for your help. -- Mobidyc
You would have to follow up with the author to see if it is implimented yet, but there might be a solution using the regular expression functionality of syslog-ng. *** This has not yet been implimented. *** *** This post is to show how usefull such a feature could be. *** The basic idea is to match a regular expression on the host in the filter section and use the matched substring in the template. Log this to a *second* syslog-ng that does the writing to the files. I know that this syntax was proposed by someone (me) so it is not yet into the current distribution of syslog-ng (Balazs - note to raise the priority :-). In syslog-ng server 1 source network { ... }; filter multi-interface { host("^(.*)e[0-9]+(\..*)?$" fullhost myhost domain); }; template rewrite { template("<PRI> $S_DATE $myhost$domain $MESSAGE"); template_escape(no); }; template raw { template("<PRI> $S_DATE $HOST $MESSAGE"); template_escape(no); }; destination second_syslog_rewrite { ... template(rewrite); }; destination second_syslog_raw { ... template(raw); }; log { source(network); filter(multi-interface); destination(second_syslog_rewrite); flags(final); }; log { source(network); filter(multi-interface); destination(second_syslog_raw); flags(final); }; The you use your second syslog with the normal "HOST" macro because it will never see the hostnames with the "e#" ending, they will have already been truncated. Thinking this through a little more, you could use the filter of filter multi-interface { host("^(.*)(e[0-9]+)?(\..*)?$" fullhost myhost interface domain); }; and then use the macros $myhost$domain in place of all of your $HOST macros and not require the second syslog server/instance at all. Evan. mobidyc wrote:
2007/8/12, Jim Hendrick <jrhendri@maine.rr.com>:
Excuse me if you've already thought of this but, I assume your servers have multiple NICs on separate networks. Why would the same server forward its syslog messages to your central log server using different NICs? Wouldn't normal routing force it to connect using one NIC all the time?
no, one reason why we use multiple network is that we have at least one network backup for servers.
i.e. is this really a problem? Have you seen the same server use different
NICs to talk to your log server?
yes
If you really are seeing this (and can't fix it at the source) I suppose you
could use a filter to look at the messages and define different destinations for each server, using a regexp on the host to determine which to put where.
to difficult to maintain, we have often some servers who are added to the network, i could write a shell script for auto-generate a config file but that seems ugly. but apparently, there is only ugly solutions for what i want:
i could either: - put statically my servers in /etc/hosts - put statically my servers in the syslog-ng.conf - auto-create a syslog-ng.conf file each days through a crontab - hack the syslog-ng source - pass by program(); and call a script for auto-generate symlink
as you can see, there is multiple solutions, i will choose one of them i think.
thanks all for your help.
-- Mobidyc
------------------------------------------------------------------------
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
-- Evan Rempel erempel@uvic.ca Senior Programmer Analyst 250.721.7691 Computing Services University of Victoria
participants (5)
-
Evan Rempel
-
Jim Hendrick
-
K K
-
mobidyc
-
Russell Fulton