[syslog-ng]Feature Request: More flexibility for automatic "log rotation"

[Marc Haber]

Hi,

I am pretty new to this list. Pardon me if this has been discussed
recently, but I'd like to hear the opinion of other syslog-ng users
and the author.

I am responsible for a number of hosts that use aide as a file system
based intrusion detection system. Using aide is quite incompatible
with the conservative way of rotating logs since all rotated files
change their names on each rotation when logrotate is used (moving
foo.n.gz to foo.n+1.gz, compressing foo.1 to foo.2.gz, moving foo to
foo.1 and creating a new foo). In this place, syslog-ng's ability to
expand log file names and thus directly log to files like
syslog.20030911 comes in quite handy. However, some of our hosts
generate very small log files, and having one log file per day would
greatly increase the number of files. Hence, these hosts currently
have their logrotate configured to rotate the log file when it exceeds
the size of 6 MB. I'd like to have this behavior natively in
syslog-ng.

The following proposed changes to the file() driver will allow me to
solve my problems, but they're pretty radical since they change some of
the file() driver's fundamental semantics while preserving backwards
compatibility. A state file is used, which can be regarded as ugly.
Another possibility would be to leave file() as it is and to create a
new destination driver that could probably be called stateful_file().

I am proposing to introduce the following new options for the file
destination driver:

Name               Type   Description
state_file()       string Set the path to the state file for this file()
                          instance
create_new_size()  number After exceeding the size given here, create new file
create_new_freq()  time   Create new file each <time> seconds
create_new_time()  string Create new file on a time matching the
                          expression

If state_file() is empty, nothing changes. This preserves backwards
compatibility of the file() driver.

If state_file() is not empty, its argument is expanded for each
incoming message and the result taken as the path to a file containing
the path to the real log file. After obtaining the path to the real
log file, its size and age are checked. If one of the criteria for
new file creation is matched, the old file is closed, and a new log
file created by expanding the positional parameter to the file()
driver. The new file path is written to the state file, replacing the
old path.

That way, the positional parameter to the file() driver, the name of
the log file, is only expanded when a new file should be created
instead of with each message. The process of expanding a file name on
each message is moved to the state file. Having a state file makes it
easier to continue where we left off after syslog-ng was restarted,
and it makes it easy for a gzipping cronjob to determine if a
non-compressed file in the log directory is still written to by
syslog-ng or if it can be safely compressed.

I am currently inclined to leave it to the local administrator to
specify a log file name mask that will not make new files clash with
old ones, but some algorithms to resolve clashes could be implemented
as well. For example, a Macro CLASHNUM could be implemented which is
guaranteed by syslog-ng to have a value that will make the generated
file name unique. That Macro should be expanded last after all other
expansions have been done since we need to have the other parts of the
file name fixed before we can determine clash or non-clash.

While we're at it, I'd appreciate a link() option that specifies a
file name that the current log is always linked to. This makes it
easier to satisfy processes that want to read from the current syslog
by being able to give them a static file name even if the "real" log
file has a dynamic name like syslog.yyyymmdd.

What do you think about this change?

Thank you very much for sharing your opinion.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Karlsruhe, Germany |  lose things."    Winona Ryder | Fon: *49 721 966 32 15
Nordisch by Nature |  How to make an American Quilt | Fax: *49 721 966 31 29