Evaluation and review

With the demands placed on intelligence analysis, time for review and evaluation can be limited, but the benefit of the learning achieved can be significant and therefore, despite demand, this should be an area for investment for analysts and their managers.

Analytical evaluation and review should take the form of two areas – self-evaluation and organisational evaluation and review.

Self-evaluation and review

Analysts are responsible for their own professional development. This includes reflective practice and learning from each piece of work completed. Quality review and peer support, seeking constructive criticism from colleagues and analyst managers help to identify learning opportunities and address weaknesses.

Analysts should undertake continuing professional development and be able to demonstrate learning and development in the role.

Reviewing the effectiveness of products is one of key skills required by an analyst. This review may include:

• assessing the analysis against the terms of reference (TOR)
• identifying differences between the analysis product and the TOR
• confirming the evaluation method
• identifying what could have been achieved by using a different methodology

Based on the results of the evaluation, an analyst can outline potential improvements to the intelligence analysis product. This will be achieved in consultation with the customer by collecting information on the impact of improvements and through assessing the effectiveness of improvements.

Once the analysis has been disseminated, any subsequent action should test the identified inference. The inferences that have been included should be reviewed to establish whether or not they were correct and that resources are being deployed effectively. Once they have been tested, options will be based on the outcomes.

• Inference is assessed to be correct but the problem persists – further direction may be required to address the issue and the intelligence cycle may need to start again.
• Inference is correct and the problem no longer persists – the next stage is to end operational activity and begin a review of all activity to date.
• Inference is assessed to be incorrect and the problem persists – further information collection will need to be initiated and additional analysis tasked.

Direction regarding how information should be communicated to the analyst, to allow for a final review, should not be overlooked. This direction needs to be clearly provided, agreed in the TOR and communicated at the time of dissemination to those required to provide feedback and inform a final review. The plan owner and analyst must, prior to dissemination, agree a timescale for this review, with the first intention being to determine the accuracy of the inference.

Analysts should ensure that their analytical products are available to their colleagues through appropriate means as both a source of good practice and to share knowledge. Analysts, with agreement from the product owner, should share innovative and creative practice, not only locally, but through regional and national forums, such as the National Analyst Working Group, conferences and workshops. Articles in journals and on websites, on Knowledge Hub’s Criminal Intelligence Analysis Community (which is an OFFICIAL online tool with access limited to registered PNN and GSi users) should also be considered so that the wider intelligence analysis community benefits from good analytical practice and knowledge sharing. 

Organisational evaluation and review

Analysts have a key role in contributing to an agency’s organisational memory, whether this is through formal review or a more informal process such as an operational debrief. The information captured needs to be fed back into the continuous learning and development that takes place in forces and partner agencies.

Results analysis

Results analysis is a structured review of action that should be tasked at the outset and documented in the TOR. This enables the development of an efficient system of information collection required for results analysis.

Results analysis evaluates the effectiveness of resource deployment and identifies good practice and issues that hindered the outcome. Once a final review has begun, whether formal or informal, it should concentrate on the value of the information gathered, the processes used to obtain information, the contribution of colleagues who collaborated on the work and the final content of the analysis.

Results analysis is a critical evaluation of all aspects of the operation or initiative. This includes:

• enforcement tactics
• intelligence gathering
• impact of prevention activity
• cost benefit analysis
• cause and effect analysis

It also includes whether the crime or incident levels have changed in the way expected as a result of the operation or initiative. For further information, see National Policing Improvement Agency (NPIA) Practice Advice on Analysis.

For this type of analysis to be successful, the operation or initiative must have specific objectives and a process to measure them by, which are agreed at the outset. Results analysis should be considered in the TOR and may be most useful where a new crime or disorder area is being addressed, a new approach or techniques are likely to be used or where the threat posed by the problem or group of individuals is assessed to be high. Any results analysis carried out before a trial or appeal is subject to the rules of disclosure.

Operational intelligence assessment

Operational intelligence assessment (OIA) is a method of ensuring that medium to long-term investigations remain focused on their original objectives. It is not solely the analyst’s responsibility to complete an OIA.

An operational intelligence assessment:

• helps prevent mission creep
• identifies priorities for the intelligence effort involved in the investigation or operation
• focuses decisions about resources
• guides investigative activities
• verifies that protocols such as the correct authorisations are in place
• highlights diversification from agreed objectives
• aids the review of compliance with Human Rights Act 1998, Regulation of Investigatory Powers Act 2000 and other legislation

See further information on OIAs.

MoRILE

One of the major challenges of analysis is in trying to compare and prioritise different types of problem, and to do so consistently.

The Management of Risk in Law Enforcement (MoRiLE) has been developed by a wide range of practitioners as a structured methodology to support prioritisation, providing a consistent approach to identifying tactical and strategic policing priorities across law enforcement agencies.

MoRILE methodology assesses impact, physical, psychological and financial harm to individuals, the community, public expectation and environmental impact – likelihood, confidence and organisational position, taking account of an organisation’s capacity and capability to address the threat.

Two models of MoRILE currently exist.

• Tactical model is a tool to enable law enforcement agencies to understand their operational and tactical risk and should be used regularly in daily and weekly tasking processes.

As well as the tasking process this could also be included in evaluation and review – for example, what was the initial tactical score, and the score at the conclusion of the operation?

• Thematic MoRILE is a tool to enable law enforcement agencies to understand their strategic risk and should be used to inform the strategic assessment process.

With all law enforcement agencies in the UK using the MoRiLE tactical model, it will facilitate interoperability and the movement of operational and tactical work between law enforcement agencies (LEAs), while prioritising the LEAs’ use of resources. This ensures they are targeted at the issues causing them the greatest risk. Using the thematic MoRILE, it will be possible to establish a national picture of harm.

Evidence-based policing

Analysts have a key role in evidence-based policing which uses the best available research techniques to understand what works and what does not in policing. Evidence-based policing places emphasis on targeting, testing and tracking law enforcement activity. Analysts can contribute to the scientific approach required without subjectivity, to inform decision makers on what works, what doesn’t and examine the causes of this.

The Maryland scale indicates the different levels of study that can be conducted to inform an evaluation where we want to know what impact our activity has had, for instance, in reducing crime or reoffending. The graphic below shows the ‘ladder of evidence’. The scale of a study is not a guarantee of quality – there can be poor studies at level 5. This scale helps define how certain analysts can be of their conclusions.

Ladder of evidence

Level 1 is a one-off measure with no comparison site. If there was any change, we cannot say whether the intervention had anything to do with it.

Level 2 is as level 1 but with a before and after measure. This is a baseline to compare to but again, although we can conclude if there is change, we cannot determine if it is related to the intervention. The influence of other factors cannot be ruled out.

Level 3 has a before and after measure across two sites, one where activity was implemented and another which remains business as usual. Any change is likely to be a result of (intervention) being introduced, rather than other factors.

Level 4 is an extension of level 3 with multiple test and control sites which gives some management over variables that can’t be controlled – sickness, leadership and culture, different sociodemographic characteristics and different crime or demand profiles. The evidence is consistent that the introduction of (intervention) led to a positive effect.

Level 5 is the gold standard with randomised control trials. Use of (intervention) directly resulted in (outcomes), as alternative explanations for the change can be ruled out.

The What Works crime reduction toolkit is a useful tool to identify what works (and does not work) to reduce crime. The toolkit summarises all existing systematic review evidence on a range of crime related topics.