Evaluation of the Food Standards Pilot in Wales: Executive Summary

Introduction

This report presents the findings from the evaluation of the proposed food standards delivery model in Wales. This pilot evaluation builds on the findings from the England and Northern Ireland pilot evaluation.

The FSA piloted a proposed new food standards delivery model between January 2021 and March 2022 in England and Northern Ireland.

The evaluation in England and Northern Ireland found that the new elements introduced by the proposed food standards delivery model worked well. The new risk scheme better identified high-risk food businesses and offered benefits to Local Authorities (LA) such as greater flexibility in how to use the model, the ability to re-score premises and the ability to use intelligence to select the most appropriate intervention. LAs highlighted that the new risk scheme was straightforward to use, and by the end of the pilot, they had already embedded the new way of working into their work practices.

LAs identified some challenges during the pilot with the proposed model. Most were resolved within the pilot and the remaining ones (use of intelligence, use of targeted remote interventions (TRIs), and the identification of non-compliances linked to allergens), were addressed by adapting the proposed model to mitigate them. This adapted model was then subject to a formal consultation and amended further, taking formal responses into account. It was subsequently introduced in Northern Ireland (May 2023) and England (June 2023).

As LAs in Wales were unable to pilot the proposed model at the same time as England and Northern Ireland, the FSA piloted the food standards delivery model between September 2023 and February 2024 in Wales. This pilot built on the lessons learned from the previous pilot and the implementation of the model in England and Northern Ireland.

The evaluation of this pilot builds on the previous report and aims to answer the following questions:

1. How did the proposed model perform compared to the current framework? What worked well and less well?
2. What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?
3. What has been the effect on resources for each of the stakeholders because of the proposed model?
4. What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?
5. What lessons were learnt from the pilot of the proposed model?

Methodology

The evaluation included a data collection and a data analysis phase, with two rounds of interviews with each of the four LAs that participated in the pilot, consisting of two pilot LAs and two control LAs. The study also included interviews with FSA staff, and a series of meetings with the FSA throughout the project. In addition, the FSA’s Analytics Unit gathered quantitative data throughout the pilot that has been integrated into the findings of this study.

Summary of findings

Findings have been structured around the research questions listed above.

How did the proposed model perform compared to the current framework? What worked well and less well?

Overall, the proposed model was fit for purpose and was generally easy to understand and use. Both pilot LAs perceived that the proposed model was an improvement in comparison to the current model. The proposed model provided a more balanced assessment of the food businesses, taking account of both the level of inherent risk and the level of compliance displayed. If the level of compliance is of sufficient concern, the proposed model determines that the next intervention to a business should be a priority intervention. This enables LAs to target their resources towards such visits with the aim of improving compliance. This is more in line with how the two pilot LAs feel interventions and the food standards service should be operating and was motivational for the teams.

The proposed risk scheme was seen as beneficial and the addition of the standalone allergen score was particularly valuable in assisting LAs to give allergens proportional focus within interventions and target resource where the risks are highest.

The proposed decision matrix, which indicates the intervention frequency for businesses, was also seen as beneficial.

LAs had overall positive reflections on the new model. LAs also identified lack of clarity with some terminology which made consistent implementation of the proposed risk scheme challenging at the beginning. These areas were:

• Guidance on the allergen scoring criteria and the thresholds between scores.
• Expanding on the risks of online distribution within the code and guidance.
• Definitions and interpretation of key terminology and phrasing.

What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?

The experience of each stakeholder (LAs and the FSA) was overall very positive.

• LAs decided to join the pilot because they wanted to understand and influence the proposed model with the FSA, and to be able to adapt early to it. Their experience during the pilot met these expectations.

LAs reported that they received all the training and support prior to and during the pilot period that they required. They found the food business “risk assessment” scoring scenario exercise and discussion to be particularly beneficial for increasing understanding and highlighting areas of interpretation.

• There were few challenges reported regarding implementation of the proposed model. 
• One LA highlighted concerns around the database mapping conducted to convert the risk score of the premises on their databases to the new risk assessment scheme of the proposed model.
• The proposed model had no impact on use or sharing of intelligence, sampling activities or use of remote inspections (these were aspects of the England and Northern Ireland pilot but were not a focus under the Wales pilot).
• LAs reported no unintended consequences from the pilot or the proposed model.
• The FSA reported that the proposed food standards model had run as expected and overall, the pilot had been a positive experience. They noted the request for further guidance and clarification on certain areas.

What has been the effect on resources for each of the stakeholders because of the proposed model?

LAs discussed resourcing in terms of:

• The pilot period: it was reported that there was no change in resourcing caused by the pilot itself or the proposed model during the pilot period. One of the two pilot LAs made a resourcing change during the pilot period, this was to support efforts to reduce the backlog of interventions and was not due to the proposed model.
• A potential wider roll out: concerns were raised around how to effectively plan and justify resourcing within annual service plans under the proposed model, particularly in light of external financial pressures on LAs.

It was mentioned that the proposed model would not address the shortage in resources and the backlog in inspections.

What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?

The proposed model achieved its objectives of effectiveness, efficiency and impact:

• Effectiveness: The compliance scoring aspect of the proposed new risk assessment helped LAs target interventions and resource towards businesses that posed the most risk.
• Effectiveness: Quantitative data showed that pilot LAs identified significantly more allergen related issues than the control LAs, showing the value of the proposed standalone allergen information risk factor.
• Efficiency: LAs feel that the model does ensure that the resources available are used most efficiently to target the highest risk food businesses and enables better prioritisation of revisits to ensure rectification of non-compliances.
• Impact: The new shorter (1, 3 and 6 months) and longer (72 and 120 months) frequencies within the proposed risk matrix were identified for a range of food business types during the pilot. Therefore, the risk matrix is allocating intervention frequencies which are better aligned with compliance scores than previously (the inherent risk score appears to have a stronger influence on intervention frequencies under the current delivery model), meaning that intervention frequencies align better with the risks faced. Overall, the number of interventions due increased by 14%. 
• Impact: The pilot and proposed model had very little effect on relationships between LAs and the FSA and no effect on relationships between LAs and businesses.

The evaluation team did not identify any unintended consequences of the pilot or proposed model.

What lessons were learnt from the proposed model?

The lessons learnt were structured in three areas, as shown in the table below: