Caveat: I have not done this (this is just a thought) Could you have syslog-ng send the appropriate logs to a program destination, and use the program to anonymize the data? For example, a Perl script could store the real values in a database / associative array, replacing them with randomized values. Then let the Perl script either write to a [file|pipe|socket] that syslog-ng would listen to and handle as if it were a real log. <log source> --> <syslog-ng> --> <perl script> --> syslog-ng --> <final destination> Using berkeley db, the Perl script could preserve the mapping in a file, so if you needed to it would be a simple function to de-anonymize the logfile. Sounds a lot worse than it would be I imagine. (but then everything is easy for he who does not have to do it :-) Later, Jim On Wed, 2011-12-21 at 14:17 +0100, Balazs Scheidler wrote:

On Wed, 2011-12-21 at 14:31 +0530, Anup Shetty wrote:

...

I am new to syslog-ng and would like some help on the pattern matching and the substitution option. Currently the requirement is to substitute a parameter in the message with a random value in order to anonymize it.

For example:

Dec 31 23:13:25 servername sshd[25218]: Failed keyboard-interactive/pam for user1 from 10.x.x.x port 47325 ssh2

If I create a pattern database for this message and pick out the username using the string and substitute it user1 to say anon1, will I be able to store the original-substituted value pair for this user and use it repeatedly? Would I be able to do it for all the subsequent logs?

To be more clear, an example substitution process that must happen as the logs arrive and the patterns are matched. log with user1 arrives and is substituted by anon1 log with user2 arrives and is substituted by anon2 again log with user1 arrives and is again substituted by anon1 log with user3 arrives and is substituted by anon3 again log with user2 arrives and is again substituted by anon2 . . . . This is required so that once the usernames are substituted for attaining anonymity, there must be a way to reverse them for audit purposes.

you want to do that on-the-fly or during postprocessing?

Right now it is not possible to do with patterndb only as it only extracts information from messages and never changes them, but anonimization has always been a hidden agenda of patterndb, which never materialized.

So - buffering would depend on how you setup the sources / destinations. I think you might preserve the buffering if you used sockets or pipes rather than files. As far as preserving the data, it should be pretty simple to write something that doesn't perturb the data except for the fields you want. something you could play with on the command line would be a place to start for the script. then you could send some logs through it (via head, tail or cat) and see what comes out. And depending on how you parse the modified data you might need to tweak the syslog-ng config to correctly get the IP address you want. It should be preserved in the syslog header though. Like I said in my original post, this is just a rough idea. as many of my professors would say "the rest is left as an exercise for the reader" :-) On Thu, 2011-12-22 at 01:17 +0530, Anup Shetty wrote:

Thanks Jim,

With this process would I not loose syslog-ng's buffer mechanism in case my destination is temporarily unavailable? The other thing I fear would be to loose the IP address of the the source and end up with local address of the relay server if I pass back the logs from the script to a file and then back to the syslog-ng.

Another thought: is it possible to increment/decrement the characters, like in a Ceaser's cipher? So when username pattern it matched, it just increments it by a value defined and passes it on. In such case we would just need to remember the incremented value to regain the original.

so username "adh" becomes "bei" (each character incremented by 1)

On Wed, Dec 21, 2011 at 9:34 PM, Jim <jrhendri@maine.rr.com> wrote: Caveat: I have not done this (this is just a thought)

Could you have syslog-ng send the appropriate logs to a program destination, and use the program to anonymize the data?

For example, a Perl script could store the real values in a database / associative array, replacing them with randomized values.

Then let the Perl script either write to a [file|pipe|socket] that syslog-ng would listen to and handle as if it were a real log.

<log source> --> <syslog-ng> --> <perl script> --> syslog-ng --> <final destination>

Using berkeley db, the Perl script could preserve the mapping in a file, so if you needed to it would be a simple function to de-anonymize the logfile.

Sounds a lot worse than it would be I imagine.

(but then everything is easy for he who does not have to do it :-)

Later, Jim

On Wed, 2011-12-21 at 14:17 +0100, Balazs Scheidler wrote: > On Wed, 2011-12-21 at 14:31 +0530, Anup Shetty wrote: > > I am new to syslog-ng and would like some help on the pattern matching > > and the substitution option. Currently the requirement is to > > substitute a parameter in the message with a random value in order to > > anonymize it. > > > > For example: > > > > Dec 31 23:13:25 servername sshd[25218]: Failed > > keyboard-interactive/pam for user1 from 10.x.x.x port 47325 ssh2 > > > > > > If I create a pattern database for this message and pick out the > > username using the string and substitute it user1 to say anon1, will I > > be able to store the original-substituted value pair for this user and > > use it repeatedly? > > Would I be able to do it for all the subsequent logs? > > > > > > To be more clear, an example substitution process that must happen as > > the logs arrive and the patterns are matched. > > log with user1 arrives and is substituted by anon1 > > log with user2 arrives and is substituted by anon2 > > again log with user1 arrives and is again substituted by anon1 > > log with user3 arrives and is substituted by anon3 > > again log with user2 arrives and is again substituted by anon2 > > . > > . > > . > > . > > This is required so that once the usernames are substituted for > > attaining anonymity, there must be a way to reverse them for audit > > purposes. > > you want to do that on-the-fly or during postprocessing? > > Right now it is not possible to do with patterndb only as it only > extracts information from messages and never changes them, but > anonimization has always been a hidden agenda of patterndb, which never > materialized. >

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-- Thanks and regards, Anup

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq