Evaluation of Food Standards pilot: Introduction

The Food Standard Agency (FSA) commissioned ICF to complete an evaluation of phase 1 of the ongoing Achieving Business Compliance (ABC) programme, which aims to modernise the way that food businesses are regulated in England, Wales and Northern Ireland by the FSA and local authorities (LAs). 

This is the final report of the evaluation of the food standards pilot project under the ABC programme. The pilot tested a proposed FSDM that aims to support LAs to target resources more effectively; provide better assurance and more flexibility to LAs; and to help LAs meet their statutory obligations. The proposed model introduced a modernised risk assessment approach, including a new risk assessment scheme, a decision matrix and the development of an intelligence-led approach to LA regulatory activity. The pilot programme was carried out with a number of LAs in England and Northern Ireland , starting in January 2021 and finishing in March 2022.  

The report is organised as follows: 

• the rest of this section introduces the background context for this pilot project. It discusses the ABC programme, the food standards project and summarises the methodology followed by the evaluation study.
• Section 2 summarises the findings of the evaluation, organised by research question.
• Section 3 includes a series of considerations and lessons learned.
• Section 4 closes the report with the conclusions. 
• a Glossary for key terms used in this report is included in Annex 1. 

1.1 The ABC Programme

The ABC programme was initiated in 2020 to build on lessons from the Regulating Our Future (ROF) Programme, which ran from 2016-2019. The programme has undergone a complete restructure establishing a clear vision, objectives and delivery structure for the future, whilst overseeing the delivery of some legacy ROF projects, and beginning the delivery of new projects to support its objectives. 

In a rapidly evolving food sector, there is a need to regulate in a smarter way to make sure that food is safe and is what it says it is. The ABC programme aims to ensure consumers continue to have food they can trust through the development of smarter regulatory approaches which: 

• make it easier for businesses to provide safe and trusted food for consumers
• target regulatory resources at the areas which pose the greatest risk 
• improve compliance across the system by working with and through others, including regulatory partners and influential businesses.

The ABC programme is currently organised into 3 workstreams:

1. Targeted and proportionate regulation - Designing a more targeted and proportionate approach to LA regulation of all food businesses within their remit. Introducing a proposed FSDM, which incorporates a revised risk assessment scheme and more emphasis on intelligence as a driver for LA regulatory activity. Modernising the food hygiene delivery model.
2. Enterprise level approaches - Designing new regulatory models for a set of large businesses that are compliant with regulation and influential in the food chain. Working with LAs, primary authorities and businesses. Developing approaches to assure compliance at a business level rather than an individual premises level. 
3. Assurance of online food sales - Using data and research to build up an evidence picture of the new online food sales landscape. Assessing the potential risks to consumers from buying food online and the regulatory levers available to address these, including working with other regulators, international food safety regulators, government bodies and influential businesses to improve compliance. Identifying and implementing a series of interventions to improve consumer safety when buying food online. 

The intention in all three workstreams is to develop regulatory approaches which are deliverable within existing food law, but the programme will also identify any areas in which legislative change could support the objectives and may make recommendations for change.

For the development of the proposed FSDM, FSA consulted with food businesses and other local and national government agencies. In July 2017, FSA published its plan to reform food regulation, entitled “Why food regulation needs to change and how we are going to do it”. It then launched the reform programme in England, Wales and Northern Ireland. 

1.2 The proposed Food Standards Delivery Model

1.2.1 Background

Food standards regulation aims to ensure food on the UK market meets applicable legal requirements, notably in terms of labelling (including allergens), presentation, composition, permitted/authorised ingredients and levels of additives, contaminants, and residues. 

The Food Law Code of Practice (FLCoP) establishes a framework for the delivery of food standards official controls by LAs. The FLCoP determines the appropriate intervention frequency for food businesses based on the associated risk profiles for different establishments and businesses. LAs must have regard to the FLCoP in the discharge of their duties in relation to food. There are three risk assessment schemes currently in use in relation to food standards, namely the National Trading Standards Risk Assessment Scheme (NTS), the LACORS Trading Standards Risk Assessment Scheme (LACORS) and the Food Standards Intervention Rating Scheme within the FLCoP itself. At the time of writing this report, the most common risk assessment scheme used by LAs is the FLCoP intervention rating scheme.  

The FLCoP intervention rating scheme groups food establishments into Category A (high risk, requires an intervention every 12 months), Category B (medium risk, requires an intervention every 24 months), and category C (low risk, should be subject to an intervention at least once every five years). This allows LAs to prioritise their interventions. Under the current intervention rating scheme, some establishments, due to the nature of their activities, are identified as Category A (high risk) regardless of their level of compliance. Interventions by LAs can take the form of an inspection, partial inspection or audit. 

A series of reports highlighted the need for the current food standards operating model to be reviewed, with FSA Board approving a root and branch review of food standards delivery in December 2018. 

The need for a review of the system was first identified in a report commissioned by FSA that highlighted a series of long standing issues within the food standards intervention scheme (see Table 1.1). These challenges, combined with a general decline in LA resources and a growth in different types of food establishments, led to stakeholders noting that the current FSDM was not fit for purpose. 

Additional reports underpinned the need for changes in the regulatory regime, including a 2018 report by FSA on LA views and experiences of the current framework for regulating food standards, and the 2019 report from the National Audit Office, which examined the effectiveness of the current regulatory arrangements .

The findings from the 2018 report mentioned above were presented to the FSA Board in December 2018, which sanctioned a fundamental review of the current delivery model as it was deemed not fit for purpose and had not kept pace with changes in the way LAs work, such as the use of intelligence. Following these insights, FSA worked in collaboration with LAs and other key stakeholder representatives to develop a new FSDM. 

FSA then started a participatory and interactive process with a number of voluntary LAs to review together the way food standards official controls were delivered by LAs and develop the proposed food standards delivery model described below and tested in the pilot.

1.2.2 The proposed Food Standards Delivery Model

The proposed FSDM incorporates three elements: 

• a single, modernised risk assessment scheme that aims to unify the way that LAs risk assess establishments (a new risk scheme).
• using the risk assessment scheme to identify the appropriate frequency for official control activity based on levels of inherent risk and compliance (decision matrix).
• greater integration of intelligence as a driver of local authority regulatory activity and to inform our national understanding of food standards risk.

The proposed model attempted to address the challenges linked to the existing model as identified in the aforementioned reports. The challenges have been summarised in Table 1.1, together with an indication as to how the proposed model sought to address them. 

Table 1.1 How the proposed models seeks to address previously identified challenges

Challenges identified in current food standards regulatory regime	How the proposed model aims to address these challenges
Inconsistent approaches to regulating food standards across LAs. Different risk rating schemes in use that don’t always reflect the overall level of food business risk.	introduction of a single, uniform risk rating scheme supporting a consistent risk rating approach to food standards by LAs. a risk rating scheme that produces a more accurate assessment of the food business risk profile.
Current model follows an establishment risk-based approach, which is perceived as not the most effective in identifying non-compliances compared to an intelligence-driven approach.	the introduction of intelligence-led action can enable LAs to target their resources effectively and efficiently to emerging issues as required.  LAs can direct resources to food businesses presenting the greatest risk. it develops a proactive approach by using intelligence to identify issues before/as they emerge.  seeks to embed and normalise intelligence as an intrinsic element of regulation by recognising its role in the FLCoP and through the creation of a LA intelligence team within FSA. introduction of a nationally coordinated and directed, intelligence-led sampling programme to further improve market surveillance and our understanding of risk.
Need to target resources more efficiently at the areas of greatest risk.	it better reflects the current context in which LAs operate. It aims to be more dynamic and responsive to emerging risks.  it aims to ensure that LA resources are targeted at the highest risk food establishments.  it provides LAs with greater flexibility in their approach to official controls.   it aims to improve communication, co-ordination, and collaboration between FSA and LAs.

1.2.2.1 The proposed risk scheme

The proposed risk scheme was designed to address known issues with the existing risk schemes, and to modernise the regulatory approach so that it better reflects new food business models and provides a more dynamic and accurate assessment of food business risk. 

For example, the existing FLCoP risk assessment scheme was felt to give too much emphasis to the inherent risk of a food business, failing to adequately recognise the business’s level of compliance. This often resulted in highly compliant businesses being inspected by LAs at a higher frequency than was deemed necessary under the current framework. 

There are two elements to the proposed risk scheme – the Inherent Risk Profile and the Compliance Assessment. Under each of the risk elements sit several risk factors, listed below. Following an official control, food businesses are given a score of 1 (high risk/low compliance) to 5 (low risk/high compliance) for each of the Inherent Risk and Compliance Assessment risk factors. These risk factor scores are then used to inform the frequency of official controls at the food business using the decision matrix.

For the Inherent Risk Profile element, the risk factors (sub-categories) considered were: 

• scale of supply and distribution
• ease of compliance
• complexity of supply chain
• responsibility for information
• potential for product harm

For the Compliance Assessment element, the risk factors (sub-categories) considered were: 

• Confidence in Management (CIM)
• Current compliance level
• Management systems and procedures 

1.2.2.2 The decision matrix

The risk factor scores for each of the two risk elements above are averaged, rounded to the nearest whole number, and then applied to the decision matrix (Figure 1). The decision matrix determines the minimum frequency and the regulatory approach for the LA to apply to businesses. 

For the pilot, the following categories of regulatory approach were introduced:

• Intervention: Regulatory activity including official food controls and other interventions such as education, advice and coaching, information and intelligence gathering. 
• Priority intervention: Where a food business has serious non-compliances they will be subject to more frequent and intense interventions until a satisfactory level of compliance is achieved. 
• Targeted Remote Interventions (TRIs): Where businesses are identified as posing a lower risk, the proposed model recommended that they were subject to remote monitoring (for example, labelling, website or documentary checks) to assess their levels of compliance in place of a physical intervention.
• No Actionable Risk (NAR): Under the piloted risk scheme, food establishments deemed to be low risk and with high levels of sustained compliance would not require a programmed intervention. They would instead be subject to ongoing monitoring by the LA using intelligence and other indicators to determine whether intervention was necessary.

The matrix also included a timeline (number of months) to indicate the minimum intervention frequency expected for each of the categories. Figure 4 shows the decision matrix and the intervention action and frequency that is recommended. With regards to the scores on the matrix, 1 is for the highest risk and 5 the lowest.

Figure 1 Food Standards decision matrix

1.2.2.3 Intelligence-led approach

Intelligence is fundamental to the proposed FSDM. Intelligence-led working is defined by National Trading Standards (NTS) as ‘a business process for systematically collecting, organising, analysing and utilising intelligence and information to guide operational and tactical decisions. Intelligence led aids in identifying, assessing, and managing targets, threats and problems at the local, regional and national level’^(footnote) .

In the context of this report, ‘intelligence’ refers to information that has been subject to a defined evaluation and risk assessment process to assist with regulatory decision making. Where the report references the ‘use of intelligence’ in the context of this evaluation, it describes the way information gathered through intelligence-led working is applied. 

The intelligence function aims to generate a feedback loop whereby strategic food standards priorities are established and an action plan is set (via focused intervention and/or targeted sampling, for example). LAs then deliver the action plan and gather intelligence, with the intelligence data being fed back to FSA to inform the next set of strategic priorities.  

1.2.2.4 Directed sampling

LAs implement a sampling policy and programme as part of their food standards work in line with the FLCoP. Research conducted by FSA in 2020 on food standards sampling showed that this activity had decreased in recent years, due to reduced budgets and resource pressures, as well as differences in the structure and approach to sampling taken by LAs, where individual authorities decide how to distribute their sampling budget. 

As part of the pilot, FSA introduced a directed sampling programme to test a coordinated approach to LA sampling that could be responsive to national intelligence, supported through central LA funding. The programme was intended to supplement, rather than replace, any locally implemented sampling activity undertaken by LAs.

To determine food standards sampling priorities, FSA consulted with wider FSA teams, and used available data (e.g. strategic surveillance data, National Food Crime Unit (NFCU) intelligence, incidents data) to support the need for surveillance or enforcement and/or the identification of intelligence gaps where further knowledge of risk was needed. FSA also considered feedback from local and regional sampling activity to help identify wider food issues.
 
FSA issued sampling criteria to pilot LAs for each phase of the directed sampling programme, which included the products, premises type and analyses to be conducted. The aim of the programme was to utilise intelligence to inform a coordinated and effective sampling strategy. Hypothetical risks and confirmed risks were tested in the directed sampling programme. It was expected that the approach to directed sampling within the pilot would lead to the identification of high proportions of non-compliance, as the sampling had been supported through an assessment of intelligence to identify sampling priorities. This is considered in the following section.

Seven pilot LAs participated in the directed sampling programme and followed the below process:

1. FSA issued LAs with sampling criteria based on intelligence, which outlined the products, premises type/s and specified analyses identified, targeting products with high potential for non-compliance.
2. LAs procured samples and submitted them to their nominated Public Analyst (PA) for analysis. 
3. PAs conducted analysis and issued certificates of analysis to LAs.
4. LAs completed a results spreadsheet for FSA, which was sent with certificates of analysis to FSA.
5. FSA interrogated and analysed the results and outcomes and progressed centralised work to address nationwide issues as appropriate. 

Table 1.2 provides an overview of directed sampling activities and the number of participating LAs. Some LAs were unable to participate in all phases of sampling due to COVID-19 officer redeployment or related restrictions, organisational restructuring, or lack of local availability of specified food samples. LAs were reimbursed for the cost of samples procured and the cost of laboratory analysis of the samples.

Table 1.2 Overview of LA directed sampling participation 

LA directed sampling participation">

Directed sampling phase	Number of participating LAs	Total number of product samples taken	Number of unsatisfactory samples	Sampling priorities
Phase 1  (March to July 21)	4	49	10	Online food businesses, newly registered, unregistered or diversified food businesses
Phase 2  (August to October 21)	6	155	113	Imported food products and unregistered food businesses
Phase 3  (November 21 to February 22)	5	113	43	Vegan products, pork products, weight loss products

1.3 The pilot project

The pilot project started in January 2021 and ended in March 2022, testing the proposed FSDM with 11 LAs (seven of them as part of the pilot and four of them as part of a control group). The timeline of the pilot can be seen in Figure 2 below.

Figure 2 Pilot timeline:

Following a request for expressions of interest to participate in the pilot, LAs were selected by FSA based on their willingness to participate and in consideration of a series of characteristics to ensure a representative sample. The selection criteria were: 

• LAs from England and Northern Ireland^(footnote). 
• LAs using/not using the Register a Food Business (RAFB) service.
• LAs using different risk schemes (FLCoP, NTS, or LACORS).
• LAs delivering food hygiene and standards, and LAs delivering food standards only.
• LAs delivering food standards controls through environmental health/trading standards practitioners. 
• LAs of different authority types for example, county council, unitary authority.

The above criteria ensured that a diverse range of LAs participated in the pilot to avoid bias. While LAs were self-selecting, the criteria above ensured a wide representation of LAs within the pilot. The LAs were also selected on the basis of their ability to conduct inspections and having a variety of types of food businesses registered. The diversity of LAs is described further in the sections below. 

1.3.1.1 Background information on LAs

There are 333 local authorities in England, and 11 in Northern Ireland. In England, there are five categories of LAs^(footnote):

• county councils (24 in total)
• district councils (181 in total)
• unitary authorities (59 in total)
• metropolitan districts (36 in total)
• London boroughs (33 in total)

County and district councils are two tiered and have their responsibility for council services split between them, with food hygiene delivered by district councils and food standards by county councils. Unitary and metropolitan authorities and London boroughs operate under a single tier structure, with those LAs responsible for all services in their area, including food hygiene and standards. There are 163 LAs with responsibility for food standards based on these figures.

1.3.2.1 Selection of participating LAs 

Table 1.3 includes the final selection of 11 LAs^(footnote) that participated in the pilot and their main characteristics, of which 7 piloted the proposed model and 4 were used as the control group. The evaluation team interviewed all 7 LAs testing the model, and 2 of the control LAs. 

Table 1.3 Overview of participants interviewed

Pilot or control	Type of authority	Risk Scheme	Hygiene or standards	RAFB	Participated in the interview (Y/N)
Pilot	Unitary Authority	FLCoP	Both	Yes	Y
Pilot	County Council	Hybrid	Food standards only	No	Y
Pilot	Unitary Authority	NTSB	Both	No	Y
Pilot	Unitary Authority	FLOP	Both	Yes	Y
Pilot	County Council	FLCoP	Food standards only	No	Y
Pilot	Unitary Authority	FLCoP	Both	Yes	Y
Pilot	County Council	NTSB	Food standards only	No	Y
Control	Unitary Authority	FLCoP	Both	Yes	Y
Control	County Council	NTSB	Food standards only	Joining soon	Y
Control	Unitary Authority	FLCoP	Both	N/A	N
Control	Unitary Authority	FLCOP	Both	N/A	N

1.4 Evaluation approach

The evaluation used a mixed methods approach, focusing on capturing LAs’ experience of the proposed model through the entire pilot, to adequately capture nuances in its use and factors impacting its performance. In addition, LAs reported monthly progress and activity to FSA. FSA used this data to analyse, quantitatively, the progress of the pilot and the impact of the new model. This report seeks to answer the following evaluation questions, as agreed with FSA: 

1. How did the proposed model perform compared to the current framework? What worked well and less well?
2. What was the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?
3. What was the effect on resources for each of the stakeholders because of the proposed model?
4. What was the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?
5. What lessons were learnt from the pilot?

The ICF study team developed an evaluation matrix to answer the questions, shown in Table 1.4 below. 

Table 1.4 Analytical approach to answering each evaluation question

Evaluation question	Analytical Approach	Data collection method	Examples of data collected/ indicators
1.    How did the proposed model perform compared to the current framework? What worked well and less well?	Assessment of data to look at whether there was:  Increase in the proportion of non-compliances: -    identified  -    resolved Changes in how LA resources (staff time) are used	Baseline, and two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff Data generated by FSA evaluation plan	LA and FSA perspective on ease of use of the new approach Enablers / barriers to using the new proposed model (IT, skills, resources, location, type of FBO) LA and FSA perspective on the comprehension of the proposed model
2.    What was the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?	Analysis of the responses to the interviews and survey.  Analysis of the outcomes of the online workshop.	Baseline, and two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff	Perceptions by LAs on quality of training received to prepare for proposed model  Opinion of users of the proposed model on ease of communication, frequency, and quality of data FSA perceptions on the proposed model
3.    What was the effect on resources for each of the stakeholders because of the proposed model? Analysis of answers to questions on adequacy and use of resources	Analysis of answers to questions on adequacy and use of resources	Baseline, and two waves of interviews with LAs An online meeting with FSA staff	Opinion on the adequacy of resources for implementing the proposed model Changes to how LA use resources  Changes made by LAs and FSA to adapt to proposed model (costs, staff, IT systems, skills)
4.    What was the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?	Analysis based on outcome indicators from the intervention logic	Baseline, and two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff Data generated by FSA evaluation plan	LA perceptions of impact on the identification of non-compliance and actions to resolve  Frequency and type of intelligence shared Perceptions of consistency under the proposed model LA views on potential unintended consequences
5.    What lessons were learnt from the pilot?	Analysis of how lessons learned have been captured, and used in the context of the ABC programme	Data generated by evaluation plan Interviews with FSA staff  A series of meetings with FSA staff	Adaptation of the pilot using data generated and analysed if needed Use of lessons learned from the pilot to inform the proposed model

The evaluation has been delivered through the following key tasks over three phases of work - the scoping phase, the data collection phase, and the analysis phase.

1.4.1 Phase 1: Evaluation scoping

The scoping phase included: 

• review of existing documentation, outlining the food standards objectives, activities, and outcomes to date. 
• interviews with FSA staff to understand their objectives, activities and progress to date, and to inform definition of the evaluation methodology.  
• a virtual workshop with FSA to develop an initial Theory of Change (ToC) for the ABC programme. This ToC included the intervention logic for the food standards project. The workshop enabled the development of a shared understanding of the ABC programme between FSA and ICF, a reflection on how the food standards project will be delivered, the outcomes it aims to achieve, and the assumptions underpinning it. This provided an opportunity to think how the overarching workstreams operate together and what contextual factors may influence their success. The intervention logic for this pilot project is included in Annex 2 
• definition of the evaluation framework, including developing an intervention logic model, identifying, and defining the questions to be addressed by the evaluation and the tools to be used. 

1.4.2 Phase 2: Data collection 

The data collection phase included: 

• three rounds of interviews with LAs to understand their expectations, their experience with the process including any challenges, as well as to document the changes implemented due to the proposed model. The first round was completed at the start of the programme in January 2021, the second round took place during April and May 2021, and the last round was completed a month after the pilot project had ended in April 2022. Interviews were held with all LAs piloting the proposed model and included two of the control LAs. The interviews followed a semi-structured format and were recorded. 
• Interviews with FSA staff at the end of the pilot, to capture their experience with the process. 
• Quantitative data collected by FSA (as discussed in Annex 3): FSA’s Analytics Unit collected monthly data from all participating LAs (control and pilot groups) in a standardised format. LAs self-reported this data during the pilot period. Data collected included risk scores from every inspection, data on reactive (intelligence led) vs proactive (programmed) interventions, data on the effectiveness of risk assessment model, the amount of intelligence received, sample results and time/resources involved.

1.4.3 Phase 3: Data analysis 

The data analysis phase was continuous through the life of the pilot project. The data analysis phase included:  

• a series of meetings with FSA staff to share the findings collected, ensure a common understanding of the main challenges and enablers, and refine the data collection tools. It also included meetings with FSA analytics team to understand their data collection process to integrate the data into this report. 
• analysis of the evidence collected such as interview responses, workshop outputs, review of documents provided by FSA. The ICF team collated and assessed the evidence based on the themes in this report. 
• analysing and integrating the quantitative data collected by FSA Analytics team.

The analysis of the quantitative data showed that one of the control LAs distorted the overall analysis results. This was because the LA dealt with a higher than usual number of food incidents during the pilot. This resulted in multiple samples being taken by the LA in relation to these incidents, which is a normal practice in this scenario (but it does not occur often), which led to a large number of unsatisfactory results due to the targeted nature of this work (common when there is an incident). For this reason, the team decided to remove this LA from the analysis and the figures in the main body of the report. The data including this LA have been included in Annex 3 for transparency.