Evaluation of AI assistants for policing, focusing on productivity outcomes and sharing learning to support forces adopting and evaluating similar tools or scaling the intervention.
| Lead institution | |
|---|---|
| Principal researcher(s) |
Chris Price and Claudia Long
|
| Police region |
West Midlands
|
| Collaboration and partnership |
|
| Level of research |
Professional/work based
|
| Project start date |
|
| Date due for completion |
|
Research context
Following the 2022 Police Productivity Review, the Centre for Police Productivity (CfPP) was established to help forces adopt new approaches and technologies to improve productivity. The Centre is committed to setting the foundations necessary for policing to deliver the 38 million police officer hours identified in the independent policing productivity review. One of the high potential areas identified by the Centre is the use of AI Assistant tools, which could have large productivity savings for policing.
The use of AI Assistants has grown in recent times, due to widely available tools such as CoPilot and ChatGPT. These tools are increasingly being used in the workplace to address challenges such as rising demand, reduced funding and an inexperienced workforce – issues also faced by the Police Service.
As a result, the CfPP has developed CoPA, an AI assistant hosted on a secure Police Service Azure Tenant, using a self-contained version of OpenAI on Microsoft Azure. Therefore, unlike public models, CoPA draws exclusively on nationally agreed sources, including the College of Policing, CPS, and gov.uk, providing operational guidance and quick access to trusted information. It supports legal procedures, evidence handling and case law, reducing search time and enabling faster decision-making.
CoPA is being piloted in two forces and the evaluation will inform lessons for future implementation and roll out.
The aims of the AI assistant evaluation are:
- to understand the comparative accuracy and capability of the CoPA tool and various AI tools in providing advice and support to police officers and staff
- determine the financial and operational costs of implementation
- understand the potential benefits, specifically considering direct efficiencies and impact on operational outcomes
- understand the key implementation issues and lessons for future delivery
- develop standardised measures and guidance for future evaluation of AI assistant tools
Research methodology
The evaluation will assess the use of CoPA in two forces: Greater Manchester Police and British Transport Police. The different evaluation approaches are outlined below.
Accuracy and capability testing
- to compare the accuracy of responses, functional capabilities and cost of various AI assistant tools
Developing a theory of change
- to agree and define immediate and longer-term outcomes, impact measures and benefits
- to inform decisions for the selection of specific use cases to undertake impact evaluation on and determine appropriate evaluation methods
Impact assessment of use cases
- to identify, define and evidence potential operational outcomes and impacts identified in the theory of change
Process evaluation
- to understand implementation conditions in each force
- to understand the context for implementation in force and across policing
- to provide lessons and guidance for future implementation decisions
Cost-benefit analysis
- to consistently determine financial and operational costs associated with use of the CoPA tool
- to consistently quantify the cost impact of any positive outcomes
- to make an assessment of the cost-effectiveness of CoPA