Getting started

Effective analysis starts with Terms of Reference (TOR). Time spent developing the TOR will help the analyst understand their customers and the questions they need answering. Getting this part of the process right will help ensure that analysts are employed in the best possible way, delivering value for money, in support of identified priorities and objectives directed against the most significant causes of threat, risk and harm.

Terms of reference

Before any analytical work is commissioned, there should be some form of discussion and agreement on what work is needed, why it is needed and what the intended outcome is envisaged to be.

This discussion will usually result in the development of TOR, often in the form of a written agreement. TOR may also be known as ‘analyst strategy’, ‘scope statement’ or ‘project initiation document’. Smaller tasks may require the TOR being outlined in a simple email, while larger tasks may require a more detailed written document.

Content for consideration in a TOR includes:

• aims and objectives
• data sources
• scope and methods
• parameters/limitations
• timescales
• reporting mechanism and audience

Generating the TOR can be done in a number of ways. One effective method would be to use some form of brainstorming or mind mapping exercise, such as structured brainstorming. This approach can help identify all aspects of the issue to be addressed and prevent important areas being missed.

Once agreed, the TOR should be retained to be referred to, amended as required and reviewed on completion. It is a useful tool at debriefings, adding to organisational learning and knowledge.

Identifying your customers

All products start with one or more identified key stakeholders who will agree the parameters of the work. At this stage it is also worthwhile identifying any other stakeholders who are likely to make use of the end product. This could range from response officers for a volume crime product to counsel and juries for any products supporting an investigation. Considering the needs of your customers will enable you to ensure that you effectively shape the TOR to meet the needs of your audience.

Identifying the questions to be answered

The most effective analysis will be based around specific requirements or questions that need to be answered or met, as opposed to a more general request for a predetermined product or type of report.

Analysts must identify the specific requirements for each task that they undertake. The analyst can make use of techniques such as refining or classifying the question or issue redefinition.

For further information on refining questions, see Structured Analytic Techniques for Intelligence Analysis and Quick Wins for Busy Analysts  (available via the Knowledge Hub Criminal Intelligence Analysis Community, which is an OFFICIAL online tool with access limited to registered PNN and GSi users). 

These techniques help the analyst get to the heart of what is needed and ensure a good understanding of the problem. This is important to help determine whether they are best placed to support this work and, if so, to select the appropriate data, structure, techniques to support the task.

Part of the value of the analyst role is in diagnosing the actual requirements that the customer has and using their professional skills and knowledge to prescribe the specific support that is needed.

Agreeing parameters

In addition to the core questions to be answered, there are a number of other parameters that need to be agreed. It makes sense to note these so that the analyst and the customer have an understanding on the overall parameters of the task. These may include:

• details of the customer/stakeholder requesting the work
• purpose and objectives of the analysis
• overall scope and methodology of the work
• details of the audience(s) for the work (and the implications for format and delivery)
• details of the analyst(s) who will complete the task
• details of any other people critical to completing the task
• date of the tasking
• deadline and milestones set for the task
• timeframe to be covered by the analysis
• justification for the tasking
• operational name or any other relevant names or reference numbers
• the likely security classification
• sources that have (and haven’t) been consulted – see collection plan
• follow-on tasks that may be required, such as assessment of results
• intended outcomes, what does success look like and how can it be measured

Getting the information you need

Analysis is based on an extensive variety of data, information and knowledge from police, partner and other more widely available sources. This range grows continually as new sources are identified and awareness of the importance of understanding the causes and context of crime and disorder increases.

Collection

Analysts must be able to identify sources of information to support their analytical work. Collection involves identifying information from open and closed sources that are relevant to the TOR.

• Closed sources of information are those with restricted access, for example, police crime recording systems and information available through sharing agreements with partners.
• Open sources of information are those that are widely available, including through the internet. Some may require the user to register or pay a small fee, for example, online news media, academic research and the electoral roll.

Collection plans

A key tool for analysts at the outset of any task is the collection plan. These plans may be structured in a table or spreadsheet.

Collection plans:

• provide a structure for collecting information
• assure focus on the TOR
• assist analysts in identifying relevant sources of information
• monitor the success of information collected

Collection plans will likely contain:

• information required, such as a specific question from the TOR
• potential internal and external information sources and contact details
• date that the source has been contacted or searched
• deadlines for the return of information
• tick box or date to signify that the information has been received

The collection plan will be updated regularly to ensure that information gaps, additional potential sources and possible access difficulties are identified. Updates should also reflect changes made in the TOR.

Analysts should be aware of local procedures for approaching sources including information-sharing protocols and single points of contact (SPOC) for data from partners and regular sources. Communication SPOCs have established information-sharing agreements with communication service providers and are able to request information securely and accurately through the appropriate processes with the respective authorities in place. This ensures that the information is appropriately sourced and the process is fully audited. If there is no established local procedure for accessing information from a source, analysts should ensure that sources are contacted a minimum number of times.

Analysts may also collect data from their customers, capturing their assumptions about the problem or issue they are addressing or identifying their hypotheses about what has happened or what is going to happen, for example. This may involve analytical tools and techniques such as the key assumptions check (KAC), expert judgement capture (available via the Knowledge Hub Criminal Intelligence Analysis Community, which is an OFFICIAL online tool with access limited to registered PNN and GSi users) or hypothesis generation and analysis of competing hypotheses, for example.

One tool that can assist in developing a collection plan is 5WH. This tool involves simply organising data into the categories of What, When, Where, Why, Who and How. This matrix can be used to structure the collection of information and intelligence and helps identify what is already known and where there are gaps in information.

5WH template

Environmental scanning may also be used to help organise your information collection in preparation for your analysis. This may be organised under various mnemonics, including PESTELO (political, economic, social, technological, environmental, legal and organisational) or STEMPLE (social, technological, economic, military, political, legal and environmental).

Collating information on relevant political, economic or technological developments will assist in determining which elements are significant to your analysis. Some might prove to be a driver behind a particular issue or an indicator as to how events might develop. This approach may also be used to capture any questions that you or your customers might seek to answer within the document.

Analysts may use expert judgement capture (available via the Knowledge Hub Criminal Intelligence Analysis Community, which is an OFFICIAL online tool with access limited to registered PNN and GSi users) or structured brainstorming to gather this information from control strategy leads, plan owners, subject matter experts or others and may conduct their own research of available material.

Open sources may be particularly valuable in this regard, while analysts may also use existing force, regional, national assessments that might be available to them.

Collation

Collation is ordering information that has been collected for analysis. It ensures that information is stored appropriately, remains retrievable and is accessible for analysis.

Information can be stored and organised using databases, spreadsheets, tables, maps and charts. These products are not an analytical output – it is the evaluation and subsequent interpretation of collated information that leads to an analytical product.

Collation table

Version control should be put in place during collation in order to:

• provide an audit trail of product development
• record what facts the analyst is aware of at any given point in time
• ensure officers have the most up-to-date information on the right document
• maintain good document housekeeping procedures

Collated information needs to be stored safely, both physically and within electronic systems. It also needs to be accessible to the analyst and colleagues. All documents must be stored according to the Government Security Classification. (See also APP on information management). This means that documents may need to be stored in locked cabinets or within password protected or restricted areas on electronic storage systems. The analyst must also consider the appropriate storage of documents that may be displayed on walls to support briefings.

Documents produced by analysts must be stored in line with appropriate legislation, such as the Data Protection Act 2018. Force policy may dictate that all work is stored on a shared drive. Access to that drive may be restricted. When constructing a system to store a volume of documents, using folders and a naming convention can prove useful.

Documents can be password protected to ensure that they are not amended by anyone other than the author. Analysts may wish to consider converting documents to pdf format to minimise storage space required and prevent documents being altered.

Evaluation

Evaluation is the assessment of the information collected in terms of:

• reliability
• validity of information
• relevance to the TOR
• links to other information

All information must be evaluated before it is used. Some information, such as intelligence, should be evaluated before the analyst accesses it.

An understanding of the following is necessary for effective evaluation:

• the source or the grading of the information
• the content of the information
• how the information informs the outcome of analysis

The results of evaluation influence how the final product is prepared and disseminated, how charts are presented and inferences made.

Information can be evaluated in a range of ways depending on its source. Common sense should underpin any evaluation and analysts should consider whether the information is sensible and/or true.

Analysts must know where information used for analysis has come from. This helps to understand the purpose for which the information was originally collected and the likely impact on its accuracy. New information should be compared with information already collected. This will confirm whether it corroborates other information or identifies anomalies. The analyst should also consider whether the new information is worth collating by asking how it contributes to the analysis.

The timeliness of the information is also important in evaluation. An assessment will have to be made on the impact of the age of the information against the value of waiting for new information. This will depend on the potential delay to the analysis and whether it fits with the time period stated in the TOR. Old information, such as census data, is useful. It can be used as an indicator or to identify long-term patterns and trends on which a prediction of future crime and disorder can be based. It can also be used to provide a context for the current situation.

Analysis must be based on the best possible information available within a defined timeframe. Any issues identified with the information must be captured in the introduction to the final analytical report.

Cognitive bias

The impact of bias

The impact of bias on analysts’ ability to deliver objective, accurate assessment has been widely studied. (Haselton, Nettle, Andrews 2005). Various cognitive biases come into play when analysts embark on a piece of work. They also influence customers and partners across roles and ranks.

Cabinet Office guidance from the Professional Head of Intelligence Analysis (PHIA) recognises that; ‘cognitive limitations cause people to employ various simplifying strategies and rules of thumb to ease the burden of mentally processing information to make judgements and decisions’.

Examples of bias, as defined by the PHIA, include:

• confirmation bias – the tendency to search for or interpret information in a way that confirms preconceptions
• anchoring effect – the tendency to rely too heavily, or ‘anchor’, on one trait or piece of information when making decisions
• loss aversion effect – the tendency for people to strongly prefer avoiding losses than acquiring gains
• bandwagon effect – the tendency to do (or believe) things because many other people do (or believe) the same
• congruence bias – the tendency to test a hypothesis exclusively through direct testing rather than attempting to disprove it by indirect testing in competition with alternatives

We all make assumptions and we tend to act on information that we assume to be true. Effective analysis recognises this and seeks to identify those assumptions held by ourselves and perhaps by our customers as well. Routinely identifying and testing these assumptions can help to avoid analysis from going in the wrong direction.

The role of effective analysis in mitigating bias

Cognitive bias cannot be mitigated simply by being aware of its existence. These cognitive biases can be mitigated to some extent by applying effective structured analytic techniques.

Many analytical techniques are considered to be structured analysis (or alternative analysis). Their processes follow a logical, transparent and coherent structure, allowing analysts and customers to easily interpret what has been done and what is being said. Analysts can help to mitigate the impact of bias by embedding good analytical practice into their daily work. One methodology for this is to adopt the Five Habits of the Master Thinker (Huer and Pherson 2015):

1. Challenge key assumptions

2. Consider alternative explanations

There will rarely be one potential explanation for what has happened but people often try to find one. Effective analysts should therefore seek to generate alternative options or explanations for the issue that they are analysing and they should seek to challenge these hypotheses.

3. Look for inconsistent data

4. Identify key drivers 

We can sometimes embark on analysis without having an effective understanding of what it is that we are looking for, particularly when we are pressed for time. When seeking to explain why something has happened or what might happen next, a key element might be identifying those factors that are potentially driving them. These vary from task to task, but effective analysis would consider these and the outcomes that they might drive.

We not only often want to find what we consider the most likely explanation but we might then not seek to question it. It is important to generate and subsequently test alternative hypotheses to mitigate the potential for the analyst to build their work around one particular hypothesis and fail to consider alternative options or explanations. Effective practice is to challenge each hypothesis rather than to seek to support it.

5. Understand the context

Effective analysis should consider the overall context of the work. Factors such as time pressure, the manner in which the work was tasked and competing priorities might lead to individual tasks not being considered within their own context. Instead, analysts might unconsciously reach back into their previous experience in what they judge to be similar tasks and apply their methodology and even their findings to any fresh task.

Using structure to guide questions asked at the outset of the task, data collection or the structure of the report itself can help the analyst consider each task on its own merits. Employing techniques to help to identify and test assumptions, alternative hypotheses etc. will also enable the individual task to be identified within its individual context and to therefore have a better chance of delivering what is required.