Anonymisation built into the data flow

Analysing problems required real logs, and testing under repeatable, realistic conditions required real production data.

Data-protection requirements did not allow either to be passed on for further use with protected information in its original form.

I based the solution on three principles:

• Anonymisation as part of the flow — obtaining the data is combined with anonymising it, so protected information does not reach analysis or testing in its original form.
• Access on demand — logs are retrieved, anonymised and prepared for analysis when they are needed.
• Data protection outside production — before production data is restored in a test environment, it is prepared and anonymised, keeping the required level of information protection.

I developed an on-demand process that retrieves logs from cloud services, anonymises them and prepares the output for further analysis.

I also prepared a mechanism for using production data in test environments: preparing the data, anonymising it and restoring it in the test environment.

Value: the logs and data needed for diagnosis and testing are available on demand, without passing on protected information in its original form.