Automated Digital Forensics – Towards acceptance of Machine Learning and Artificial Intelligence-based automation within forensic analysis of image files

Digital devices are the modern evolving zeitgeist. They increasingly dominate every aspect of daily life, and without them it is increasingly difficult to participate in society. As a result of this the Digital Forensic (DF) field is one of the most challenging and ever-present domains within law enforcement. The evidential potential of these devices increases with every new advancement. However, the effect of these devices is that they take longer to examine correctly, they are more complex and there are more of them to examine.
There have been ongoing developments to introduce automation technologies in Artificial Intelligence (AI) and Machine Learning (ML) to attempt to solve this urgent need for increased efficiency. However, these advancements have come with increased concern over the governance of these tools and how this can be effectively used whilst maintaining legal standards and principles. This has been achieved until this point with the requirement for human verification of any use of automation technologies.

This thesis proposes a novel term of ‘True Automation’ to refer to the concept of relying upon an automated software or device and the analysis that it conducts without needing a human-in-the-loop component, in an attempt to maximise the gains in terms of efficiency which can be obtained through the use of automation technologies. In addition, the thesis proposes a novel assessment metric dubbed the Practical Reliability (PR) Metric, for the synthesising of technical accuracy and reliability into a single usable and understandable metric that can be for the rapid and accurate assessment of automation technologies.

Finally, a novel assessment framework for the assessment of automated tools within the image analysis domain is proposed, with accompanying documentation and guidance as to how it can be used to assess tools in a real-world environment.

The requirements for this framework are obtained from a combination of secondary and primary research methods, making use of academic research in the field, as well as the input from serving practitioners in the DF unit of a UK Home Office Police Force.

Automated Digital Forensics – Towards acceptance of Machine Learning and Artificial Intelligence-based automation within forensic analysis of image files

Rigorous testing of both the framework and the metric is conducted using both expert review and technical testing.

Framework testing involves expert validation, review of the thresholds set and technical review. This testing shows that, following appropriate revision, all 8 of the proposed gate thresholds were accepted. Additionally, practitioner acceptance testing indicates that 92% of survey respondents and 73.9% of interview respondents would be willing to use automated tools without human review if they passed the framework.

Metric testing shows that current tool developers have not published the data necessary to conduct meaningful F1 and Matthews Correlation Coefficient (MCC) calculations. Therefore, the metric is tested against 17 image manipulation detection methods. This testing shows scores ranging 0.3240 to 0.4960. All results fall within the expected 0 – 1 range but were below the required passing threshold. However, the produced testing results showed that the required simplified score was achieved, accurately representing the required F1 and MCC values, whilst providing a value which could easily be interpreted by non-technical practitioners.

Following these results, the thesis concludes that the framework is effective and suitable at the task and would be viable for assessment of automation tools, and the metric is effective at simplifying complex model assessment metrics whilst maintaining technical relevance. As a result, it is shown that the adoption of ‘True Automation’ is possible with appropriate safeguarding. However, cultural, legal and ethical considerations remain which can only be addressed by increasing