Evaluation of Food Standards pilot: Executive Summary

Introduction

The Food Standards Agency (FSA) commissioned ICF to carry out an evaluation of phase 1 of the ongoing Achieving Business Compliance (ABC) programme. The ABC programme aims to modernise the way that food businesses are regulated in England, Northern Ireland, and Wales by FSA and LAs.

The ABC programme aims to:

• make it easier for businesses to provide safe and trusted food for consumers
• target regulatory resources at the areas which pose the greatest risk
• improve compliance across the system by working with and through others, including regulatory  partners and influential businesses.

For more information on the ABC programme please follow this link: Achieving Business Compliance (ABC) programme | Food Standards Agency.

This report presents the findings from the evaluation of one of the projects of this programme: the pilot of a proposed FSDM in England and Northern Ireland. The proposed model introduces a modernised risk assessment approach, including a new risk scheme, a decision matrix and the development of an intelligence-led approach to regulatory activity. 

This study aimed to answer the following questions: 

1. How did the proposed model perform compared to the current framework? What worked well and less well?
2. What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?
3. What has been the effect on resources for each of the stakeholders because of the proposed model?
4. What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences? 
5. What lessons were learnt from the proposed model?

Methodology

The evaluation started with a scoping phase, which included a review of existing ABC documentation, interviews with FSA food standards team, a virtual workshop to develop an initial theory of change for the ABC programme and the intervention logic specific to this proposed model, and the definition of the evaluation framework. 

The next phase was the data collection phase. It included three rounds of interviews with all the LAs using the proposed model during the pilot, plus two LAs from the control group for the pilot. The study also included two interviews with FSA staff, and a series of meetings with the FSA throughout the project. In addition, FSA’s Analytics Unit gathered quantitative data throughout the project that has been integrated into the findings of this study. 

Summary of findings

Findings have been structured around the research questions listed above.

How did the proposed model perform compared to the current framework? What worked well and less well? 

The LAs using the proposed model during the pilot consistently identified the following aspects of the proposed model that worked well: 

• risk scheme and rating:  The proposed risk scheme and model introduced a single, uniform risk rating approach to assess the risk profile of FBOs more accurately. It was dynamic and integrated intelligence for a more targeted and efficient response, enabling LAs to target their resources to food businesses presenting the greatest risk. They were found to be suitable for identifying risk and prioritising resources within LAs. 
• integration/‘fit’ with existing working practices: The proposed model was easy to use, which led to the successful integration of the model into LAs’ existing working practices.
• sampling approach: LAs viewed the directed sampling approach as being effective at identifying non-compliance and gathering intelligence. 

On the other hand, LAs using the proposed model perceived the intelligence element was still a work in progress:

• the intelligence element of the model was not fully developed by the end of the pilot, with participating LAs at different stages of maturity in their use of intelligence as a driver of regulatory activity. 

At the end of the pilot, there was no consistent approach to how FSA and LAs understand what information is considered intelligence; the type of information which needs to be shared between LAs and FSA; the frequency of sharing information between the LAs and FSA; and the most suitable and accessible mechanism to share intelligence. 

LAs found it challenging to carry out all the directed sampling activities as piloted under the proposed model, as it was not always possible to provide sufficient notice of proposed sampling activity given that priorities were identified through analysis of current intelligence. 

In terms of the implementation of the pilot: 

• the support provided by FSA throughout the pilot was well-received and perceived as crucial to the success of the pilot. This level of support was specific to the pilot project.  
• the main barriers to the implementation of the pilot were the lack of clarity with some of the risk scheme terminology (resolved during the pilot), lack of consistency of food business data held by LAs (addressed during the pilot), and a lack of compatibility between LAs MIS and the proposed delivery model during the pilot period. 

What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?

The experience of each stakeholder (LAs and FSA) was overall very positive. 

• LAs decided to join the pilot because they wanted to co-create the proposed model with FSA, influence the change, and to be able to adapt early to it. Their experience during the pilot met these expectations. 
• LAs identified that a main challenge during the pilot was related to staff capacity to be able to test the proposed model, particularly at the beginning (as the pilot start date coincided with the introduction of national COVID-19 lockdown measures) and initial challenges linked to LA MIS. LAs mentioned there was a learning curve when using the proposed model, but that once their staff were aware, the resources were more effectively allocated (as discussed below)

FSA mentioned that regular communication with LAs had been the key to the success of the project. They perceived LAs as being very open and honest, which allowed FSA to adapt and develop the tools and guidance in a way that could best serve LAs and the objectives of the proposed model. As part of the proposed model, the main challenge raised was linked to the intelligence function (as discussed above).

What has been the effect on resources for each of the stakeholders because of the proposed model?

In terms of changes in resources, LAs mentioned they required time to adapt to the proposed model. However, once the LA staff adapted to the implementation of the proposed model LAs perceived that the time required was the same compared to the current framework. 

In terms of effectiveness and efficiency of LA resources, the participating LAs identified that the proposed model allowed them to target resources more effectively. 

What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences? 

The pilot achieved its objectives and had a positive effect on: 

• identification of FBOs that present the greatest risk
• risk-based targeting of interventions 
• proportion of LA resources driven by intelligence

Evaluation of the proposed model did not identify any unintended consequences because of the proposed model.

What lessons were learnt from the proposed model?

The lessons learnt were structured in three areas, as shown in Table 1.1 below: 

Table 1.1 Lesson learned from the pilot