Evaluation of the Food Standards Pilot in Wales

This report presents the findings from the evaluation of the pilot project that the Food Standards Agency (FSA) has completed to test a proposed food standards delivery model in Wales. The aim of the pilot project in Wales was to test the food standards delivery model recently introduced in England and Northern Ireland to determine its suitability for Wales and identify any unintended consequences.

The model was introduced in England and Northern Ireland following a 15-month pilot (January 2021 to March 2022) and a subsequent consultation period across the two nations. This pilot in Wales builds on the findings from the England and Northern Ireland pilot as well as feedback received during the consultation process.

The evaluation methodology focused on capturing local authorities (LAs) experiences through pre- and post-pilot interviews with the participating LAs, interviews with FSA officials, and quantitative data submitted by the participating LAs and collected by the FSA’s analytical unit. Data was then analysed and triangulated to integrate the qualitative and quantitative findings against the evaluation research questions.

The pilot of the proposed food standards delivery model in Wales has been successful in demonstrating that the proposed model works in the Welsh context, and LAs were supportive of the new proposed allergen risk factor and the new intervention frequencies. The findings show that the proposed food standards delivery model and its core elements, a new risk assessment scheme and associated decision matrix, worked well. Key benefits included greater flexibility in the model, the ability to re-score premises following what is currently referred to as a re-visit and the ability to give a standalone score for allergen procedures. LAs highlighted the proposed risk scheme was straightforward to use and, by the end of the pilot, they had embedded the new way of working into their work practices. The proposed decision matrix, which indicates intervention frequency, was also seen as beneficial. However, LAs did identify areas where clarity in the guidance could be improved to further promote consistency in scoring, e.g. the thresholds between allergen scores.

The process for implementing the pilot was successful. LAs reported that they received all the training and support prior to and during the pilot period that they required. LAs reported no challenges to implementing the pilot and that after a period of adjustment the proposed model was easy to use. The FSA also reported that the pilot had been a positive experience and they had not identified any unintended consequences. LAs reported no impact on resourcing during the pilot period but did raise concerns around how they would undertake annual service planning due to the proposed new shorter intervention frequencies. The quantitative data showed that while the number of premises whose intervention frequency increased (35%) was similar to those whose intervention frequency decreased (40%), the overall number interventions due increased by 14%. There may have been some more conservative scoring during the pilot as staff adapted to the proposed food standards delivery model thus increasing frequency of inspection for some premises. 

Ahead of the roll-out of the proposed model in England and Northern Ireland the FSA have made reflections on the Wales evaluation findings and other feedback relating to the Welsh pilot. The FSA are considering additions to the guidance to increase clarity based on recommendations given in this report. The FSA stated that any additions would require a balance between providing prescriptive guidance and allowing flexibility for officers conducting food standards official controls to use their knowledge and experience to risk rate a business.

Introduction

This report presents the findings from the evaluation of the proposed food standards delivery model in Wales. This pilot evaluation builds on the findings from the England and Northern Ireland pilot evaluation.

The FSA piloted a proposed new food standards delivery model between January 2021 and March 2022 in England and Northern Ireland.

The evaluation in England and Northern Ireland found that the new elements introduced by the proposed food standards delivery model worked well. The new risk scheme better identified high-risk food businesses and offered benefits to Local Authorities (LA) such as greater flexibility in how to use the model, the ability to re-score premises and the ability to use intelligence to select the most appropriate intervention. LAs highlighted that the new risk scheme was straightforward to use, and by the end of the pilot, they had already embedded the new way of working into their work practices.

LAs identified some challenges during the pilot with the proposed model. Most were resolved within the pilot and the remaining ones (use of intelligence, use of targeted remote interventions (TRIs), and the identification of non-compliances linked to allergens), were addressed by adapting the proposed model to mitigate them. This adapted model was then subject to a formal consultation and amended further, taking formal responses into account. It was subsequently introduced in Northern Ireland (May 2023) and England (June 2023).

As LAs in Wales were unable to pilot the proposed model at the same time as England and Northern Ireland, the FSA piloted the food standards delivery model between September 2023 and February 2024 in Wales. This pilot built on the lessons learned from the previous pilot and the implementation of the model in England and Northern Ireland.

The evaluation of this pilot builds on the previous report and aims to answer the following questions:

1. How did the proposed model perform compared to the current framework? What worked well and less well?
2. What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?
3. What has been the effect on resources for each of the stakeholders because of the proposed model?
4. What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?
5. What lessons were learnt from the pilot of the proposed model?

Methodology

The evaluation included a data collection and a data analysis phase, with two rounds of interviews with each of the four LAs that participated in the pilot, consisting of two pilot LAs and two control LAs. The study also included interviews with FSA staff, and a series of meetings with the FSA throughout the project. In addition, the FSA’s Analytics Unit gathered quantitative data throughout the pilot that has been integrated into the findings of this study.

Summary of findings

Findings have been structured around the research questions listed above.

How did the proposed model perform compared to the current framework? What worked well and less well?

Overall, the proposed model was fit for purpose and was generally easy to understand and use. Both pilot LAs perceived that the proposed model was an improvement in comparison to the current model. The proposed model provided a more balanced assessment of the food businesses, taking account of both the level of inherent risk and the level of compliance displayed. If the level of compliance is of sufficient concern, the proposed model determines that the next intervention to a business should be a priority intervention. This enables LAs to target their resources towards such visits with the aim of improving compliance. This is more in line with how the two pilot LAs feel interventions and the food standards service should be operating and was motivational for the teams.

The proposed risk scheme was seen as beneficial and the addition of the standalone allergen score was particularly valuable in assisting LAs to give allergens proportional focus within interventions and target resource where the risks are highest.

The proposed decision matrix, which indicates the intervention frequency for businesses, was also seen as beneficial.

LAs had overall positive reflections on the new model. LAs also identified lack of clarity with some terminology which made consistent implementation of the proposed risk scheme challenging at the beginning. These areas were:

• Guidance on the allergen scoring criteria and the thresholds between scores.
• Expanding on the risks of online distribution within the code and guidance.
• Definitions and interpretation of key terminology and phrasing.

What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?

The experience of each stakeholder (LAs and the FSA) was overall very positive.

• LAs decided to join the pilot because they wanted to understand and influence the proposed model with the FSA, and to be able to adapt early to it. Their experience during the pilot met these expectations.

LAs reported that they received all the training and support prior to and during the pilot period that they required. They found the food business “risk assessment” scoring scenario exercise and discussion to be particularly beneficial for increasing understanding and highlighting areas of interpretation.

• There were few challenges reported regarding implementation of the proposed model. 
• One LA highlighted concerns around the database mapping conducted to convert the risk score of the premises on their databases to the new risk assessment scheme of the proposed model.
• The proposed model had no impact on use or sharing of intelligence, sampling activities or use of remote inspections (these were aspects of the England and Northern Ireland pilot but were not a focus under the Wales pilot).
• LAs reported no unintended consequences from the pilot or the proposed model.
• The FSA reported that the proposed food standards model had run as expected and overall, the pilot had been a positive experience. They noted the request for further guidance and clarification on certain areas.

What has been the effect on resources for each of the stakeholders because of the proposed model?

LAs discussed resourcing in terms of:

• The pilot period: it was reported that there was no change in resourcing caused by the pilot itself or the proposed model during the pilot period. One of the two pilot LAs made a resourcing change during the pilot period, this was to support efforts to reduce the backlog of interventions and was not due to the proposed model.
• A potential wider roll out: concerns were raised around how to effectively plan and justify resourcing within annual service plans under the proposed model, particularly in light of external financial pressures on LAs.

It was mentioned that the proposed model would not address the shortage in resources and the backlog in inspections.

What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?

The proposed model achieved its objectives of effectiveness, efficiency and impact:

• Effectiveness: The compliance scoring aspect of the proposed new risk assessment helped LAs target interventions and resource towards businesses that posed the most risk.
• Effectiveness: Quantitative data showed that pilot LAs identified significantly more allergen related issues than the control LAs, showing the value of the proposed standalone allergen information risk factor.
• Efficiency: LAs feel that the model does ensure that the resources available are used most efficiently to target the highest risk food businesses and enables better prioritisation of revisits to ensure rectification of non-compliances.
• Impact: The new shorter (1, 3 and 6 months) and longer (72 and 120 months) frequencies within the proposed risk matrix were identified for a range of food business types during the pilot. Therefore, the risk matrix is allocating intervention frequencies which are better aligned with compliance scores than previously (the inherent risk score appears to have a stronger influence on intervention frequencies under the current delivery model), meaning that intervention frequencies align better with the risks faced. Overall, the number of interventions due increased by 14%. 
• Impact: The pilot and proposed model had very little effect on relationships between LAs and the FSA and no effect on relationships between LAs and businesses.

The evaluation team did not identify any unintended consequences of the pilot or proposed model.

What lessons were learnt from the proposed model?

The lessons learnt were structured in three areas, as shown in the table below:

Table 1.1 Lesson learnt from the pilot

Area of the proposed model/pilot	Lessons learnt
Implementation of the pilot (found in report section 3.1)	The support provided by the FSA throughout the pilot was well received and crucial to the success of the pilot.  To the extent possible, continue to consider timing of pilots to avoid busy period such as the end of financial year
The proposed new model (found in report sections 2.2, 2.3, 2.4 and 2.5)	The proposed model was fit for purpose and was an improvement in comparison to the current model as it allocated risk ratings to establishments in a more balanced way than before. However, the accompanying guidance to implement the proposed model could be improved. There were few challenges reported in implementing the proposed model.  The FSA reported that the proposed food standards delivery model had run as expected with no unintended consequences identified and overall, the pilot had been a very positive experience.
National roll-out (found in report section 3.2)	Guidance and clarity around terminology used should be provided prior to a roll-out. An improvement in the definitions and descriptors would help with consistency in interpretation.  There should be a continued forum (possibly external to the FSA) for sharing of experiences between LAs to encourage discussion around consistency and interpretation. Specific guidance should be provided to joint-service LAs to help maintain smooth working practices between the Food Hygiene and Food Standards services. This guidance should cover how to compare the Food Hygiene and Food Standards intervention frequency reports.   Clarity needs to be increased around the allergen scoring criteria, by improving the descriptors in the risk assessment. Pilot LAs noted that this particularly relates to scoring of businesses with ‘no allergen risk’ due to the specific products they produce or supply as well as further clarity on the thresholds between each of the allergen risk scores. Support should be provided to LAs during the process of mapping databases from the current model to the proposed new model to reduce the risk of high-risk businesses being reclassified as non-priority due to issues with data entry. (This is already being implemented in England and Northern Ireland).

This report presents the findings from the evaluation of the pilot of the proposed food standards delivery model in Wales. The pilot and the evaluation build on the findings from the England and Northern Ireland pilot evaluation. 

The FSA piloted a proposed food standards delivery model between January 2021 and March 2022 in England and Northern Ireland. The proposed model was then subject to a formal consultation and amended further, taking formal responses into account, and has now been introduced in England and Northern Ireland.

LAs in Wales were unable to pilot the proposed model at the same time as England and Northern Ireland. Building on the previous pilot and implementation of the model in England and Northern Ireland, the FSA piloted the model introduced in England and Northern Ireland as a proposed new food standards delivery model to use in Wales between September 2023 and February 2024. The pilot tested a proposed new model for the delivery of food standards official controls. This introduced a modernised risk assessment approach that aims to support LAs target their resources more effectively; provide more flexibility to LAs; and help LAs meet their statutory obligations. 

This is the final report of the evaluation of the proposed food standards delivery model in Wales. 

The report is organised as follows: 

• The rest of this section (section 1) introduces the background context for this pilot project and discusses the proposed food standards delivery model in Wales. It concludes by summarising the methodology followed by the evaluation study. 
• Section 2 summarises the findings of the evaluation, organised by research question.
• Section 3 includes a series of considerations and lessons learned.
• Section 4 closes the report with the conclusions. 

1.1 Background

This section introduces the background context for this pilot project. It discusses the findings from the England and Northern Ireland pilot evaluation study, describes the proposed food standards delivery model, introduces the Wales pilot project and provides background information on the local authorities in Wales. 

1.1.1 Summary of findings of the food standards delivery model pilot in England and Northern Ireland 

Between January 2021 and March 2022, the FSA tested a proposed food standards delivery model with eleven LAs in England and Northern Ireland (the ‘initial pilot’). The proposed food standards delivery model aimed to support LAs to target resources more effectively; provide better assurance and more flexibility to LAs; and to help LAs meet their statutory obligations. The proposed model introduced a modernised risk assessment approach, including a new risk assessment scheme, a decision matrix and the development of an intelligence-led approach to LA regulatory activity. 

The evaluation of the initial pilot found that the new elements introduced by the proposed food standards delivery model worked well. The new risk scheme better identified high-risk food businesses and offered benefits to LAs such as greater flexibility in how to use the model, the ability to re-score premises and the ability to use intelligence to select the most appropriate intervention. Local Authorities (LAs) highlighted that the new risk scheme was straightforward to use, and, by the end of the pilot, they had already embedded the new way of working into their work practices. 

LAs identified some challenges with the model. Most were resolved within the pilot. Other challenges like the use of intelligence, the use of targeted remote interventions (TRIs) or the identification of non-compliances linked to allergens, were addressed by adapting the proposed model to mitigate them. 

This adapted proposed model was then subject to a formal consultation and amended further, taking formal responses into account, and has now been introduced in England and Northern Ireland.

1.1.2 The proposed food standards delivery model in Wales

The Food Law Code of Practice establishes a framework for the delivery of official food controls by LAs. It determines the appropriate intervention frequency for food businesses based on the associated risk profiles for different establishments and businesses. Each country in the UK has their own Food Law Code of Practice, and while those issued in Wales, England and Northern Ireland are generally similar, there are some country-specific differences. LAs in Wales must have regard to the Food Law Code of Practice (Wales) (hereafter, ‘the Code’) while fulfilling their duties in relation to food (both food hygiene and food standards). 

For food standards, the existing intervention rating scheme groups food establishments into Category A (high risk, requires an intervention every 12 months), Category B (medium risk, requires an intervention every 24 months), and category C (low risk, should be subject to an intervention at least once every 60 months). This allows LAs to prioritise their interventions. LAs will visit all new premises to determine compliance with applicable food law and allocate a risk rating. Interventions by LAs can take the form of an inspection, partial inspection or audit. Under this rating scheme, some establishments, due to the nature of their activities, are identified as Category A (high risk) regardless of their level of compliance.

Additionally, the FSA and LAs can monitor the change in compliance from businesses over time using a metric of ‘non-broadly compliant’ and ‘broadly compliant’ businesses. These metrics are used by the FSA and LAs to get a high-level understanding of how FBOs are operating. The current definition of “broadly compliant” is set out below: 

• Definition under the Code: an establishment that has a score of not more than ten points under both the Level of (Current) Compliance and the Confidence in Management/Control Systems.

“Non-Broadly Compliant” would be any establishment that does not satisfy the above requirements. So, under the Code, if an establishment had a score of more than 10 for either the Level of Current Compliance or the Confidence in Management they would be regarded as non-broadly compliant. 

A series of reports (Research on the modernisation of the risk intervention rating systems for UK food establishments, Food Standards Delivery Review, and Ensuring food safety and standards)  highlighted the need to change the current food standards delivery model under the Code. The reports identified several challenges with the current model, which, combined with a decline in LA resources and a rapid change and growth in the types of food establishments, showed that the current food standards delivery model is no longer fit for purpose. 

The challenges highlighted by the reports were: 

• LAs are taking inconsistent approaches to regulating food standards, as the current risk model doesn’t always accurately reflect the overall level of food business risk. 
• The current model follows an establishment risk-based approach, which is perceived as not the most effective in identifying non-compliances. 
• The current model does not support LAs in targeting their resources towards the areas of greatest risk. 

The FSA Board approved a root and branch review of the food standards delivery in December 2018. Following this decision, the FSA designed the proposed new food standards delivery model in consultation with LAs from the three nations (England, Wales and Northern Ireland). The proposed new model intends to give a more comprehensive reflection of the risks posed by food businesses (see Annex 1 for further details of the proposed model). 

To address the challenges identified above, the proposed model incorporates three elements:

• A modernised risk assessment scheme to unify the way that LAs risk assess establishments (a new risk scheme).
• An assessment scheme using a matrix approach based on levels of inherent risk and compliance to identify the appropriate frequency for official control activity.
• Greater integration of intelligence as a driver of local authority regulatory activity and to inform the national understanding of food standards risk.

Additionally, due to the change in scoring approach the proposed model introduces a new definition for ‘broadly compliant’. The new definition of ‘broadly compliant’ is set out below: 

• Definition under the proposed food standards delivery model: an establishment receiving an overall compliance risk assessment score of 3, 4 or 5.

“Non-Broadly Compliant” would therefore be any establishment that does not satisfy the above requirements. So, under the proposed food standards delivery model, if an establishment had a score of 1 or 2 as an overall compliance risk assessment score they would be regarded as non-broadly compliant. 

Based on the findings from the initial pilot and the evaluation report, the FSA decided to adjust the proposed food standards delivery model. A summary of the changes is presented in Table 1.1.

Table 1.1 Changes to the proposed model following the initial pilot

Lessons learned during the pilot in England and NI	Changes to the proposed model to be tested in Wales
England and NI pilot LAs found it difficult to identify non-compliances linked to allergens as the proposed risk assessment scheme did not include allergens as a separate element to be considered (allergens were embedded as part of other elements).	Introduced a new risk factor on allergens to the proposed risk assessment scheme.
England and NI pilot LAs did not find useful having Targeted Remote Interventions (TRIs) linked to a particular risk score under the matrix. They consider TRIs useful in some circumstances but prefer to consider their use on a case-by-case basis rather than a risk score.	TRIs are not part of the decision matrix anymore. The model changed to allow LAs to use remote interventions if and when they consider them to be appropriate.
The England and NI pilot had a new category called No Actionable Risk (NAR), where no future intervention date would be allocated to the food business. This would mean LAs had to react to intelligence or complaints and visit the NAR business accordingly. After the pilot it was considered that the NAR category did not meet the Official Control Regulations (OCRs) requirement. OCRs require regular official controls to be conducted at an appropriate frequency based on risk.	NAR was replaced by a new intervention frequency of 120 months.
England and NI pilot LAs asked for a way to prioritise the backlog of unrated businesses.	The model introduced a desktop assessment (DA) for LAs to evaluate the inherent risk of new food businesses.  The assessment should occur within 28 days of the business registering or the LA becoming aware of its operation. Information for this assessment is sourced from the documents completed by businesses when registering, such as the Register a Food Business (RAFB) completed by new FBOs (Food Business Operator) to comply with the Hygiene Regulations (Food Hygiene (Wales) Regulations 2006 and Article 6(2) of assimilated Regulation (EC) 852/2004), supplemented by other relevant information (for example from company websites, telephone conversations, emails, or questionnaires.)
There were several challenges identified with the intelligence function during the pilot, such as: what type of data should be shared, how frequently, what mechanisms to use and who should the data be shared with.	The Local Authority Intelligence Coordination Team (LAICT) have adapted their training and engagement with LAs to streamline this element. They have introduced, for example, regular intelligence newsletters that share trends and risks with LAs, and identified the type of data and frequency that they would like LAs to share.

1.1.3 Local Authorities in Wales

There are 22 Local Authorities in Wales, and these are all unitary authorities. Unitary authorities operate under a single tier structure, with LAs responsible for all services in their area, including food hygiene and food standards. 

Food standards official controls in Wales are delivered by a combination of dedicated food standards teams and teams that deliver both food hygiene and food standards controls. For the purposes of this report, the latter will be referred to as joint services. In joint services, food hygiene and food standards controls are sometimes delivered together by the same officer during one visit.  

1.1.4 The pilot project

The pilot project in Wales commenced in September 2023 and ended in February 2024. The pilot tested the proposed food standards delivery model (see Annex 1 for further detail on the proposed food standards delivery model) with four LAs. Two LAs implemented the proposed delivery model (pilot LAs), and two LAs continued to work to the existing model detailed in the published Code (control LAs). 

The aim of the pilot was to:

• demonstrate how the proposed new model applies to the Welsh context, and 
• provide opportunities for LAs in Wales to participate and identify areas where the model can be made more suitable and applicable to the delivery of food standards official controls in Wales.

The proposed new model tested under the pilot does not amend the food standards legal requirements that FBOs must comply with.

The four LAs participating in the pilot project were selected based on their willingness to participate and in consideration of the following characteristics to ensure a representative sample: 

• LAs from North and South Wales.
• LAs delivering food standards controls through a joint service (food hygiene and food standards delivered jointly) and others delivering solely via dedicated food standards teams. 

As the pilot in Wales built on the lessons learned from the initial pilot in England and Northern Ireland, the FSA decided to use a smaller sample of LAs in Wales. The Wales pilot involved only four LAs meaning that the findings in this report cannot be completely representative of the Welsh LA population in total. However, the above criteria ensured that the different types of LA in Wales were represented in the pilot. Statistical tests have been included where possible however due to the small sample size differences between LA groups (control vs pilot) and between LAs more generally may be due to other factors and not necessarily due to the proposed model. 

While all LAs in Wales are unitary, the selection represents the delivery method of food standards official controls in Wales: jointly or separately. Table 1.2 below indicates the characteristics of the LAs participating in the pilot: 

Table 1.2 Local Authority Profile

Pilot / Control	Delivery method	North Wales / South Wales
Pilot	Joint service delivery (food standards and food hygiene)	North Wales
Pilot	No joint service delivery (food standards)	South Wales
Control	Joint service delivery (food standards and food hygiene)	North Wales
Control	No joint service delivery (food standards)	South Wales

1.2 Evaluation objectives and research questions

This evaluation aims to answer 5 questions: 

1. How did the proposed model perform compared to the current framework? What worked well and less well? 
2. What has been the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole?  
3. What has been the effect on resources for each of the stakeholders because of the proposed model? 
4. What has been the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences? 
5. What lessons were learned from piloting the proposed model?

The ICF study team developed an evaluation matrix to answer the questions, shown in Table 1.3  below. 

Table 1.3 Analytical approach to answering each evaluation question

Evaluation question	Data collection method	Examples of data collected / indicators
1. How did the proposed model perform compared to the current framework? What worked well and less well?	Two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff Quantitative data gathered by the FSA	LAs and the FSA perspective on ease of use of the new approach Enablers / barriers to using the new proposed model (IT, skills, resources, location, type of food business) LAs and the FSA perspective on the comprehension of the proposed model
2. What was the experience of each of the stakeholders with respect to specific elements of the proposed model and the proposed model changes as a whole	Two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff	Perceptions by LAs on quality of training received to prepare for proposed model  Opinion of users of the proposed model on ease of communication, frequency, and quality of data The FSA perceptions on the proposed model
3. What was the effect on resources for each of the stakeholders because of the proposed model?	Two waves of interviews with LAs An online meeting with FSA staff	Opinion on the adequacy of resources for implementing the proposed model Changes to how LAs use resources  Changes made by LAs and the FSA to adapt to proposed model (costs, staff, IT systems, skills)
4. What was the overall effect of the proposed model? Did it deliver its objectives? Were there unintended consequences?	Two waves of interviews with LAs Interviews with FSA staff  A series of meetings with FSA staff Data generated by FSA evaluation plan	LAs perceptions of impact on the identification of non-compliance and actions to resolve  Frequency and type of inspections Perceptions of consistency under the proposed model LAs views on potential unintended consequences
5. What lessons were learnt from the pilot?	Data generated by evaluation plan Interviews with FSA staff  A series of meetings with FSA staff	Adaptation of the pilot using data generated and analysed if needed Use of lessons learned from the pilot to inform the proposed model

1.2.1 Methodology

1.2.1.1 Phase 1: Scoping

The scoping phase included: 

• Review of existing documentation, including the updated pilot guide for the proposed new food standards delivery model in Wales, and other relevant documents outlining the activities and outcomes to date. 
• Interviews with FSA staff to understand the pilot objectives, activities and progress to date, and to inform the evaluation methodology. Three interviews were conducted. Interviewees included members of the Wales FSA team, the FSA Local Authority Intelligence Coordination Team (LAICT) and the FSA Analytics Unit. 
• Definition of the evaluation framework, including developing the interview guides and finalising the method.  

1.2.1.2 Phase 2: Data collection 

The data collection phase included: 

• Two rounds of interviews with LAs to understand their expectations, their experience with the process including any challenges, as well as to document the changes implemented due to the proposed model. The first round was completed before the start of the project between June and July 2023 and the final round was completed a few weeks after the pilot project had ended in March 2024. Interviews were held with all four LAs participating in the pilot. The interviews followed a semi-structured format and were recorded. 
• A series of regular meetings with the FSA to share progress updates and discuss any issues which had arisen, as well as defining timelines.
• Interviews with FSA staff at the end of the pilot, to capture their experience with the process. Two interviews were conducted, one with a member of the FSA in Wales and the other with a member of the Regulatory Compliance Division (RCD) leading on the England and Northern Ireland roll out.
• Quantitative data collected by the FSA Analytics Unit (AU). AU collected monthly data (between September 2023 and February 2024) from all participating LAs (control and pilot groups) in a standardised format. LAs self-reported this data during the pilot period. Data collected included risk scores from every inspection, data on reactive (intelligence led) vs proactive (programmed) interventions and compliance scores. For pilot LAs the risk assessment data was provided for both the current model and proposed model. For control LAs the risk assessment data was only for the current model as they did not operate under or see the proposed model.

The pilot LAs carried out 353 interventions (of which, full details were given for 198). The control LAs carried out 225 interventions (of which, full details were given for 184). These figures exclude any premises visited by LAs that were found to be permanently closed. It should be noted that all data shown in this report is for premises that were visited by LAs during the pilot period only, data is not shown for the full set of premises within an LAs area. 

1.2.1.3 Phase 3: Data analysis  

The data analysis phase was continuous through the life of the pilot project. The data analysis phase included:  

• A series of meetings with FSA staff to share the findings collected, ensure a common understanding of the main challenges and enablers, and refine the evaluation data collection tools. It also included meetings with FSA Analytics Unit to understand their data collection process to integrate the data into this report. 
• Analysis of the evidence collected such as interview responses and a review of documents provided by the FSA. Thematic analysis was used to analyse the interview responses in line with the evaluation questions. The ICF team collated and assessed the evidence based on the themes in this report. 
• Analysing and integrating the quantitative data collected by the FSA Analytics Unit. Chi-squared tests were performed at a 5% significance level. 

Limitations:

• The sample size (4 LAs) was small and the length of the pilot (6 months) short. This meant that there were limitations to the representativeness of the data collected (in terms of LA characteristics, mix of premises types, and period of time for the LA to complete interventions). 
  • As mentioned in section 1.1.4, the small sample size of 4 LAs for the pilot project meant that the results were not representative of all LAs in Wales. While statistical tests have been used where possible, it is important to note that differences between LA groups (control vs pilot) and between LAs more generally may be due to other factors and not necessarily due to the proposed model. 
  • The small sample size should also be noted when considering the qualitative findings as there was a limited number of individuals involved in piloting the proposed model.
  • Further, the short timeframe of the pilot (6 months) made it more difficult to run robust statistical analysis with which to draw conclusions. It also meant it was not possible to observe evidence of changes in compliance of targeted premises. 
• This pilot project built on the positive findings and lessons learned from the initial pilot whilst testing the proposed model in the Welsh context. Therefore, the shorter timescale and small sample size were deemed appropriate for this purpose.

Despite these limitations the evaluation team were able to assess the fit of the model in the Welsh context and identify any unintended consequences. 

2.1 Baseline

A baseline round of LA interviews was conducted before the pilot. The purpose of baseline interviews was to compare the changes between the baseline and the end of the pilot, as well as to observe whether the control LAs evolved in a different manner to the pilot LAs. The interviews explored how LAs were currently implementing the existing model, challenges faced and expectations around the pilot.

2.1.1 Approach to current delivery of official controls in Wales

There are two different approaches within LAs to the delivery of official food controls in Wales: those LAs that deliver all food standards official controls through a dedicated food standards team, and those LAs that operate a joint service delivering food hygiene and food standards official controls via a single team.  

• No joint service delivery (food standards): These LAs deliver food standards controls separately to hygiene controls. They explained they assess and identify their food standards interventions following the Code but will often emphasise high-risk premises. Their teams would assess business risk and allocate the interventions at the beginning of the year. Their sampling and project priorities determined on a local or regional basis can also guide the remainder of the work they do. 
• Joint service delivery (food standards and food hygiene): These LAs use the same team to carry out food standard and food hygiene inspections. When they can, they aim to do joint inspections. For example, for medium and low risk premises, LAs would decide whether a food standard inspection is needed or not while reviewing the food hygiene inspection programme. As food hygiene controls are often delivered more frequently (every 6 months) than food standards (high risk being every 12 months), food hygiene inspections often dictate the timings and selection of establishments for food standard inspections. Sometimes, for specific high-risk food standard premises, LAs would prioritise food standards and visit them. The LAs interviewed perceived that, since they started delivering food standards and food hygiene inspections jointly, they are more efficient in their way of working. They mentioned they are able to carry out more interventions than before. However, they also explained that not all officers perceive themselves to be capable to carry out food standards. They said officers from the food hygiene team had a steep learning curve to get the level of technical knowledge necessary to deal with some of the complexities of food standards. 

During the interviews, the LAs discussed specific practices around key areas: 

Identification and resolution of non-compliances

LAs commonly identify non-compliances through inspections, complaints, or sampling activities. High-risk premises are prioritised, with notification to businesses upon finding non-compliance. LAs would recommend remedial actions to follow up on how the business has addressed the non-compliance. These vary based on severity of non-compliance, from simple fixes with photographic evidence to re-visits or prosecution. However, resource constraints and backlogs due to the COVID-19 pandemic have strained inspection capacities.

Use of Intelligence

LAs use intelligence, gathered from various sources, to help identify any potential non-compliance and risk areas. Two of the LAs included in the pilot explained they were already effectively incorporating intelligence into their work. One of these two LAs is the regional intelligence lead for Wales. However, the other two LAs mentioned using intelligence was challenging for them as it was time-consuming to record and share intelligence in line with relevant requirements. They also perceived the information they held or received was not always complete. Overall, the four LAs that participated in the pilot project perceived intelligence-led approaches were beneficial for resource allocation and risk targeting and would like to use more of these approaches in the future.

Sampling activities

LAs explained that food sampling priorities are set on a national, regional, and local basis. This happens annually. At national level, the yearly sampling priorities are selected following criteria that the FSA in Wales determines. Some LAs supplement this information with relevant data from the National Food Crime Unit (NFCU). On a regional basis, LAs emphasised there were some regional differences between North and South Wales when determining sampling priorities. As such, LAs in Wales also coordinate with their regional areas to set priorities. Recently, local budget reductions meant LAs rely more on knowledge sharing and suggestions on what to focus on, working as a region. 

Adaptations to the current model

LAs explained they are making small adaptations to the existing model for better efficiency and flexibility. Some examples of these adaptations included seeking flexibility in business rankings (in terms of high, medium or low risk), identifying improvements to make the process reviews more efficient, and incorporating lessons learned from the pandemic into their practices. However, the four LAs explained they would appreciate the Code having more flexibility to account for regulatory changes and the evolving food sector.

In summary, the four LAs perceived that while the existing food standards delivery model in Wales had demonstrated effectiveness, the current delivery model had some challenges related to its rigidity (e.g. reduced flexibility could lead to LAs to miss allergen issues due to changes in menus or suppliers), not being up to date with regulatory changes (e.g. new types of establishments or ingredients) and the allocation of resources. LAs emphasised they would prefer the delivery model to focus on outcomes rather than mere inspection completion numbers. As such, LAs were excited to test the proposed model hoping it would address some of the challenges identified. 

2.2 Research Question 1: How did the proposed model perform compared to the current framework? What worked well and less well?

The proposed model operated well, was fit for purpose and was generally easy to understand and use. 

Both pilot LAs perceived that the proposed model addressed the limitations identified of the prior model. The proposed model provided a more balanced assessment of the food businesses taking account of both the level of inherent risk and the level of compliance displayed. The proposed model ensured the next official control after significant non-compliance identification are a priority, allowing LAs to target their resources towards those visits. This is more in line with how the LA feels interventions and the food standards service should be operating and was motivational for the team. 

The proposed risk scheme was seen as beneficial and the addition of the standalone allergen score was particularly valuable in assisting LAs to give allergens proportional focus within interventions and target resource where the risks are highest. 
The proposed decision matrix which indicates the intervention frequency for businesses was also seen as beneficial. 

Despite the positive reflections, LAs found some of the terminology lacked clarity which made it challenging to implement the proposed risk scheme in a consistent manner. Key areas where more clarity was expected were: 

• Guidance on the allergen scoring criteria and the thresholds between scores. 
• Expanding on the risks of online distribution within the code and guidance.
• Definitions and interpretation of key terminology and phrasing.

This section presents the findings showing how well the proposed model worked. It assesses whether the model was fit for purpose, its ease of use and any challenges identified in using the proposed model in relation to its application and operation. This section also analyses the context in which the pilot was delivered, assessing the enabling factors and barriers faced during the implementation of the pilot project itself. The effects of the proposed model, namely whether it achieved its objectives, are discussed in section 2.5 of this report. 

2.2.1 Appropriateness of the proposed model (fit for purpose)

Both pilot LAs perceived that the proposed model was fit for purpose, and was an improvement in comparison to the current model as it addressed the limitations identified of the current model. LAs perceived that the proposed model provided a more balanced assessment of the food businesses taking account of both the level of inherent risk and the level of compliance displayed. However, both LAs noted that the accompanying guidance to implement the proposed model could be improved (see section 2.2.3). 

The rest of the section explores the appropriateness of the three elements of the proposed model (the risk scheme, the decision matrix, and the inherent risk desktop assessment). Two of the elements were considered positive, however there were concerns about the inherent risk desktop assessment for new businesses. 

2.2.1.1 The proposed risk scheme

Pilot LAs welcomed the proposed risk scheme. LAs reported that the proposed model allowed for more flexibility than the existing one, enabling them to effectively target resources to higher risk establishments. This is because the proposed model combines the assessment of inherent risk of the establishment with a more comprehensive assessment of the establishment’s compliance (through assessment of four factors instead of the current two). The way the two risk elements are combined is also improved as the proposed model uses a matrix approach to average the scores. It also includes an assurance rule whereby any compliance factor given a score of 1, the lowest level of compliance, results in an overall compliance score of 1 as the averaging process is ignored. This assurance rule better accommodates the high risk posed by non-compliance in the overall scores in comparison to the current model where scores are added together. The proposed risk scheme gives a more balanced assessment, allowing the level of compliance within a business to be a more prominent part of the rating rather than focusing primarily on inherent risk. It allows for more granularity in the risk assessment process. 

While under the current model, LAs consider the risks of allergens and act on all non-compliances found relating to allergens, all four LAs (pilot and control) perceived that, as mentioned in the baseline (section 2.1.1), the current risk scheme did not sufficiently reflect the high risk posed by allergens. 

As such, pilot LAs found the way the proposed allergen risk score was incorporated into the risk scheme particularly beneficial. One pilot LA felt that the standalone allergen score helped to emphasise the importance of the food standards service within overall food safety. 

2.2.1.2    The decision matrix

Overall, it was reported by both pilot LAs that the proposed decision matrix worked well and there were some benefits to the increased number of risk categories and intervention frequencies such as the more accurate prioritisation of risks and ability to reduce resourcing focus on food businesses who are considered low risk and highly compliant. 

The decision matrix was reported to be easy to use. However, there was an adjustment period needed before food standards officers felt comfortable and confident with the new risk levels within the proposed decision matrix. One LA commented that, initially, the proposed decision matrix was challenging for them. This is because the proposed decision matrix contains 10 risk frequencies compared to the current three. They had to switch their way of thinking from the prior ‘A’ high risk, ‘B’ medium risk and ‘C’ low risk to a different fit within the risk categories. A period of transition is expected given the length of time the current model has been in use (see section 2.3.3 for the FSA response and preparation on this). 

Pilot LAs were also concerned about the change in the frequency of visits for some establishments. One LA was particularly concerned about large manufacturers. While large manufacturers are largely compliant, the LA was keen to maintain contact and rapport with the FBOs teams – as one non-compliance (even if unlikely) could have large consequences. Under the current model all manufacturers are classed as high-risk meaning that they should receive an annual inspection from the LA. The proposed model, however, showed that the intervention frequency for large manufacturers only decreased for two manufacturers and increased for four, out of a total of nine manufacturers, as shown in section 2.5.1.5. It should also be noted that the frequencies generated by the risk matrix represent a minimum frequency of intervention and LAs have the flexibility under the Code to bring interventions forwards if deemed appropriate.

2.2.1.3    New businesses: inherent risk desktop assessment

The proposed model also introduces a desktop assessment (DA) to assign new establishments with an estimated inherent risk and assist in prioritising initial inspections of new businesses. The DA uses multiple data sources, including Register A Food Business (RAFB), other documentation that may be used when registering a food business as well as publicly available data sources such as a business’s website. The proposed DA aims to enable more effective prioritisation of new business initial inspections and provide a structure to support LAs with an increasing backlog of unrated new businesses. The mandating of desktop assessments will be considered as part of any potential consultation prior to consideration of providing advice to Ministers regarding a potential wider roll out of the proposed model in Wales. Both pilot LAs were asked to use it during the pilot period. Further details on this mechanism are available in Annex 1. The aim is to prioritise food businesses for initial intervention based on their perceived inherent risk. 

Of the two pilot LAs, one found this new process burdensome as they perceived that it added paperwork without providing additional insights. They mentioned they were already following a similar process internally; however, they explained that the DA took longer for them to complete without adding additional value. They recommended including additional questions in the business registration form to streamline the inherent risk assessment process.

The other LA did not comment on this. 

2.2.2    Ease of use of the proposed model

Both pilot LAs found the proposed model easy to use once they had built up their confidence and understanding. They emphasised that the change is only about frequency of interventions and how they assess risks and compliance, but the interventions remain the same. 

One pilot LA appreciated the increased capacity to prioritise revisits to non-compliant food businesses through the new proposed 1-, 3- and 6-month intervention frequencies. This was identified as a positive feature, as under the current model revisits are scheduled but often given lower priority compared to priority interventions already scheduled. Due to resource constraints, these revisits are often delayed. The proposed model ensures revisits after a non-compliance are identified as a priority, allowing LAs to target their resources towards those visits. This element allows LAs to focus on addressing non-compliances quickly rather than focusing on the number of interventions that have taken place. This is more in line with how the LA feels interventions and the food standards service should be operating and was motivational for the team. 

The proposed model was viewed as being compatible with the way that LAs perceived food standards should operate, as it identified the major risks and enabled the LAs to prioritise revisits to seek to resolve non-compliances. 

2.2.3 What were the enablers (what worked well) and the barriers (what worked less well) for implementation of the proposed model?

The FSA support was a key enabling element in the success of the pilot. The support received from the FSA before and during the pilot was reportedly very useful to LAs. The pilot opened a line of informal communication between the FSA and the LAs, where they could collaborate to solve issues together. 

The FSA were identified as being responsive to communication. LAs felt they had a ‘point of contact’ where their queries would be addressed. More detail on training and support provided can be found in section 2.3.1.2. One LA reported conducting their own in-house consistency exercises to help provide further reassurance. 

Despite the benefits identified and the positive feedback around the proposed model and the FSA’s support both pilot LAs identified one main barrier to consistent implementation and areas for improvement – around terminology used. There were a number of areas within the risk scoring guidance which were open to interpretation and further definitions and guidance are needed, namely: 

• Scoring of premises which pose ‘no allergen risk’ in the view of LAs (e.g. only selling pre-packed food or producing single-ingredient foodstuff like honey). Clarity was needed on how to give an allergen score for premises who have ‘no allergen risk’ and therefore no allergen information or process.
• Clarity around thresholds between the scores for the allergen information risk score (for example between a score of 1 and a score of 2). It was unclear for the LAs what the tipping point was between these two scores. The LAs felt that any level of non-compliance on allergens poses a high-level of risk to consumers. This clarity is needed to avoid food businesses being scored more harshly than necessary, impacting their intervention frequency. 
• One LA identified that online distribution is not sufficiently discussed within the current guidance. This was particularly the case for the wording ‘scale of supply and distribution’ and whether online distribution should be considered within this. The LA highlighted that although often online distribution is at a local level the online format means that distribution could be national or international. The LA had noted a substantial difference in how their officers were scoring the inherent risk of FBOs who distribute online.
• Definitions and increased clarity around the wording ‘wide range’ and ‘limited range’ within the phrase ‘Establishments responsible for producing or labelling a wide range [or limited range] of foods’ within the Ease of Compliance criteria. The LA gave the example of a manufacturer who produces 25 different ready meals, are they considered to be producing a wide range or would they need to produce a range of different products in addition to ready meals to be wide range.
• Better links to existing definitions and increased clarity around key words such as ‘local’ to ensure consistency in approach and confidence in the continuity of definitions between the current model and the proposed model. 

Both pilot LAs reported that the proposed model was fit for purpose but that increased clarity and guidance covering the above identified issues would increase their confidence in using the proposed model and the consistency of risk assessment and management across Wales. This was under the context of limited resource capacity meaning that if food businesses were not identified as needing priority intervention they could end up having limited contact with the LA. 

2.3 Research Question 2: What has been the experience of each of the stakeholders with respect to specific elements of the proposed model, and the proposed model changes as a whole?

The experience of each stakeholder (LAs and FSA) was overall very positive. 

• LAs decided to join the pilot because they wanted to understand and influence the proposed model with FSA, and to be able to adapt early to it. Their experience during the pilot met these expectations. 
• LAs reported that they received all of the training and support prior to and during the pilot period that they would have liked. They found the food business “risk assessment” scoring scenario exercise and discussion to be particularly beneficial for increasing understanding and highlighting areas of interpretation.
• There were few challenges reported regarding implementation of the proposed model.
• One LA highlighted concerns around the database mapping conducted to convert the risk score of the premises on their databases to the new risk assessment scheme of the proposed model. 
• The proposed model had no impact on use or sharing of intelligence, sampling activities or use of remote inspections (these were aspects of the initial pilot but were not the main focus under the Wales pilot). 
• LAs reported no unintended consequences from the pilot or the proposed model.
• The FSA reported that the proposed food standards model had run as expected and overall, the pilot had been a positive experience. They noted the request for further guidance and clarification on certain areas. 

This section analyses the experience of the different stakeholders engaged in the pilot and their perspective on the new proposed model. The section addresses the reasons for LAs to join the project, considers LAs’ attitudes towards the pilot and finally summarises the FSA experience of the pilot.

2.3.1 LA experience

2.3.1.1 LA motivations for joining the pilot

LAs decided to join the pilot project for a number of reasons. All LAs stated that the decisive factors for joining were their willingness to influence the model to ensure it fits their needs, and the capacity to anticipate and adapt to the changes as soon as possible. LAs appreciated having the opportunity to start early and work on this new proposed model with the FSA. This shows that all LAs participating in the pilot, whether testing the proposed model or in the control group, had already identified some challenges with the prior model, were willing to try the proposed one, and to collaborate with the FSA to ensure a smooth transition. They all voluntarily expressed an interest in participation after an invitation from the FSA was sent out to all LAs in Wales. 

2.3.1.2 Preparation for implementing the proposed model

The FSA supported LAs prior to and during the implementation of the pilot. Section 2.2.3 discussed that LAs found this support to be one of the enablers of the pilot process. Preparation activities carried out included: 
 

• An in-person contact day in Cardiff with the FSA to gain understanding of the proposed model, including a pilot LA from the initial pilot explaining their experience and an exercise where everyone individually rated a food business scenario and then discussed the differences in their scoring. This highlighted some of the interpretation issues (discussed in section 2.2.3). 
• FSA visit day at the LA premises where the FSA spoke with all staff involved in the pilot and answered any queries. The LA team leads really appreciated this approach as it meant questions could be asked directly and avoided all of the information being passed to the lead officer and then cascaded down to the rest of the team.
• Regular contact with the FSA team to respond to queries.

Both LAs reported receiving all of the training and support that they would have liked. They found the scenario exercise from the contact day particularly useful. As a result, one of the pilot LAs decided to implement a consistency training with their food standards team, which included a scoring of a scenario exercise (this is discussed further in section 2.3.1.3 below). The FSA team member running the training felt that it went well, there was good engagement from the LAs in the activities and many clarification questions were asked at this stage. They also felt that the scenario exercise was particularly useful.

Support from the FSA continued throughout the pilot in the form of monthly joint catch-up meetings with the two pilot LAs and some individual meetings. These meetings facilitated sharing of experiences and learning from all parties, noted as a benefit by the two pilot LAs. The pilot LAs reported that having one point of contact within the FSA had been beneficial and useful as it ensured clearer messaging and allowed for more open sharing of issues. 

Online meetings were held with the control LAs prior to the start of the pilot to establish and discuss the data submission aspects that the control LAs would be required to complete. The control LAs were also offered regular meetings but did not take up this offer as they did not require any additional support from the FSA as they were operating as normal under the current model.

2.3.1.3 Implementing and working with the proposed model

Both pilot LAs trained some, or all, of their officers to work with the proposed model. One pilot LA provided additional support to their team for the implementation of the pilot:  

• Setting up of a Microsoft Teams site to share guidance and provide space for their team to ask questions.
• Fortnightly catch ups with the LA team to discuss how the pilot model was working.
• Running of ‘consistency exercise sessions’ where the LA team could discuss any interpretation issues or queries to ensure a level of consistency across the team for scoring of premises. These sessions reduced in frequency through the pilot period showing an increase in confidence in implementation of the proposed model. The LA found these exercises to be very useful and are considering running a session annually regardless of the outcome of the pilot.

2.3.1.4 Challenges to implementing the proposed model

Overall, both pilot LAs reported not having any major challenges with implementing the proposed model. They both noted that the team required training and an adjustment period, which they expected. Once the confidence and understanding was built there were no major questions. 

One LA identified issues with the initial mapping of their database (Initial exercise conducted by the FSA to define the new intervention frequencies for each establishment at the start of the pilot.) and resulting initial risk scores. This LA team were very familiar with their ‘A’ rated premises so expressed concern when not all these were flagged as priority interventions after the mapping to the proposed model was completed. The LA gave one example where a manufacturing site had not been rated a priority due to its high level of compliance, but they considered it to be higher risk as it produces allergen sensitive products and distributes them widely. The mapping system used for the pilot primarily used the overall business type e.g. ‘manufacturer’ to map food businesses to the new proposed risk matrix and therefore did not consider any contextual information. However, this is only a risk during the initial set up stages of the proposed model as the systems are converted. Once outliers have been identified and interventions resume this risk decreases. The FSA response to this ahead of the England and Northern Ireland roll out of the proposed model can be found in section 3.2.1. This shows a risk of the current database mapping system which only considers the food business type.

The LA noted that for their LA, due to having a smaller and more experienced team, knowing which premises should be a priority was easy but that for other LAs, with many more premises to cover, the initial mapping may lead to rating some premises wrongly. The FSA has noted this risk and is reflecting on how to mitigate it (see section 3.2.1).

2.3.2 Other aspects of food standards operations

The Welsh pilot of the proposed food standards delivery model did not include a directed focus on piloting use of intelligence, directed sampling or targeted remote interventions (elements tested in the initial pilot, see section 1.1.1 and Table 1.1). There was potential for directed sampling and intelligence tasking to be issued but these did not materialise throughout the pilot. The LAs were able to use remote interventions as and when they considered they were appropriate. 

These elements were discussed with LAs to understand if the proposed model had had any unintentional impact in these areas. It was found that the pilot did not impact on levels of sampling, remote interventions or use of intelligence for all four of the LAs. 

2.3.2.1 Intelligence

There is a very wide range of understanding and use of intelligence across the four LAs. Interview data showed that intelligence is typically gathered through reports on the intelligence database system (IDB), reports from the FSA NFCU, complaints from other sources, sampling data, discussions within the regional sampling groups, discussions at the food standards expert panel, and the export/import control groups. 

Baseline data showed that the two pilot LAs used and shared less intelligence than the two control LAs before the pilot started. Data from the pilot period shows that the control LAs submitted a total of 14 intelligence reports to IDB and the pilot LAs submitted a total of 4 reports. However, to put this in context one control LA is the regional intelligence coordinator for Wales meaning that they already submit above average volumes of intelligence as part of their day-to-day intelligence role. The other control LA specified in their baseline interview that this year they were focusing on improving their use and sharing of intelligence. On the other hand, the two pilot LAs noted that they struggle to upload intelligence onto IDB due to resource constraints and lack of confidence with the IDB system. It is therefore likely that the difference in intelligence reporting during the pilot period between the pilot and control LA was as a result of the specific LAs pre-existing relationship with intelligence rather than being caused by the proposed model.

LAs noted in interviews that they do act on intelligence reports where they are relevant.  

One LA stated that more clarity is needed in some intelligence reports sent and received by LAs to make sure they include all necessary details to ensure that LAs can use intelligence to guide their work and support them in identifying the likelihood of the risk or threat identified. All four LAs indicated that they would like to use intelligence more in the future. 

2.3.2.2 Sampling

Qualitative data from interviews shows that the four LAs are involved in regional sampling groups and follow the regional sampling plans. LAs confirmed in interviews that this set up had not changed during the pilot period and many of the sampling plans were set prior to and covered the whole pilot period. 

The proposed model did not have an impact on sampling for the four LAs.

2.3.2.3 Remote interventions

All four of the LAs showed hesitancy around the use of remote interventions (RI). No RI’s were conducted as part of the pilot but there were some examples of where LAs were beginning to use RI style interventions. One control LA gave the following example: directing a non-qualified officer to check that a simple non-compliance had been rectified (dependent on type of non-compliance) or do a selected ‘C’ rated food business intervention (e.g. clothing retailers with pre-packed sweets at the counter) using a self-assessment questionnaire, this example is under the current model and was not used during the pilot period.

Joint service LAs noted that when joint interventions were needed, they would always have to be done in-person to satisfy the Food Hygiene Rating Scheme (Food Hygiene Rating (Wales) Act 2013). Single service LAs were also hesitant as they felt that in-person interventions were a better way to guarantee an accurate scoring and that FBOs may not mention something that they do not see as relevant but that the trading standards officer could find if they were in-person at the premises. The responsibility for protecting the public from risk impacts upon trading standards officer’s confidence in implementing RIs rather than in-person interventions. Confidence is the main barrier to use of RIs. 

When asked in interviews where LAs felt RIs could be appropriate or where they may wish to use them in the future each LA did identify examples of where they could be used. Examples given included checking on updated risk assessments or an allergen matrix via email. The two pilot LAs identified that RIs could be used with the short frequency interventions but they stressed that this would depend on the nature of the non-compliance identified. 

2.3.3 The FSA experience (Food Standards Agency reflections and response)

2.3.3.1 Food Standards Agency reflections on the pilot

The FSA reflections on the implementation of the pilot were positive. The staff involved in the implementation and working regularly with the LAs shared that the regular communication with LAs had been the key to the success of the project. 

The FSA perceived that a challenge could be LAs reactions to changes in frequency of interventions, especially a potential increase in the number of interventions (this perception was based on engagement with LAs across Wales prior to the pilot). However, after risk scores were mapped to the proposed model the new risk frequencies showed that this was not the case (See section 2.5.1.5).  

Another challenge identified by the FSA was the administrative burden for LAs caused by the extra data submission spreadsheets for the pilot monitoring (this included additional data submissions and the use of a scheduling spreadsheet which acted as a temporary management information system (MIS) to guide the work during the pilot, this will not be part of a wider roll out). The FSA noted that both pilot and control LAs in the Wales pilot were funded to submit this data and the extra data submissions would not be part of any potential roll out in Wales. Additionally, the FSA recognised that the proposed risk rating scheme and the new frequencies would take LAs some time to adjust to, especially as the current model has been in place for a long period of time.  

The FSA also noted the selection of a shorter time period (six months instead of a year) was the right balance of testing without additional burden. As the FSA had already tested the effectiveness and amended the proposed model in England and Northern Ireland, the focus of this pilot evaluation was to ensure the refined proposed model worked in the Welsh context, and to test the new elements of the proposed model such as the allergen score.

2.3.3.2 Food Standards Agency reflections on the proposed model and on a potential further rollout

The FSA reported that the proposed food standards delivery model had run as expected. The LAs had adapted well to the proposed model with fewer clarification questions asked of the FSA during the pilot period than expected. The LAs did not identify any unintended consequences of the pilot or the proposed model.

The FSA identified that the interpretation and consistency issues (discussed in section 2.2.3) were the biggest challenge to effective implementation of the proposed model. 

The FSA acknowledges that a period of transition will be needed to ensure LAs can acclimatise to the proposed new model if rolled out. The FSA have developed a suite of training packages to help LAs become familiar with the new model (these are currently being delivered to LAs in England and Northern Ireland following introduction of the new model there).

Throughout the pilot period the Welsh FSA pilot team shared feedback with the FSA’s Achieving Business Compliance (ABC) team to ensure the recommendations are reflected on the England and Northern Ireland wider roll out (and on the potential roll out of the proposed model in Wales). The FSA Welsh pilot team and ABC team have also been working closely with the performance management team and the Audit team to ensure other FSA services appropriately take account of any upcoming changes (and the pilot period itself).  This reflects the FSAs awareness that the proposed model would take some time for LAs to adjust to. This has led to the development of a new set of KPIs relating to the proposed model which will be introduced if the proposed model is rolled out in Wales.

In light of these conversations, the FSA has noted the request for more guidance on: 

• Scoring of premises with no allergen risk.
• Difference between scores (particularly between a 1 and a 2) for allergen criteria (this has been covered in the England and Northern Ireland guidance and associated training).
• Definitions on specific phrasing used (e.g. online and wide vs limited range has been covered in the England and Northern Ireland guidance). 
• Implementation of the proposed model for LAs operating joint services, explaining the scheduling of interventions through the comparison of intervention frequency reports for food standards and food hygiene. Verbal guidance was provided during the pilot period, but written guidance is identified as beneficial.
• Service planning due to the need to have a more dynamic team with the proposed shorter intervention frequencies (this was also a key point from the initial pilot and non-prescriptive guidance has been developed on this). 

The FSA mentioned that the proposed model would require a balance between having prescriptive guidance and allowing flexibility for food standards officers to use their knowledge and experience to risk rate a food business. Further, the training and preparation for a full roll out would be different to the pilot preparation (see section 3.2.1). 

2.4 Research Question 3: What has been the effect on resources for each of the stakeholders because of the proposed model?

LAs discussed resourcing in terms of: 

• The pilot period: it was reported that there was no change in resourcing caused by the pilot or the proposed model during the pilot period. One LA made a resourcing change during the pilot period, and this was to support efforts to reduce the backlog of interventions not due to the proposed model. 
• A potential wider roll out: concerns were raised around how to effectively plan and justify resourcing within annual service plans under the proposed model, particularly in light of external financial pressures on LAs. 

It was mentioned that the proposed model would not address the shortage in resources and the backlog in inspections.

The answer to this question focuses on the changes made by the LAs and the FSA to adapt to the proposed model. This includes changes linked to the implementation of the pilot, and changes to resources linked to the delivery of the proposed model. 

2.4.1 Resourcing the pilot

The pilot and proposed model did not affect the resourcing needs of the LAs. Across the four LAs resources have changed very little during the pilot process. The pilot leads within the LAs have stayed consistent (although one has had a change in job title with no impact on the food standards role). 

Only one LA experienced any change in resources during the pilot period. The LA brought in a consultant in December 2023 to assist with solving the backlog of interventions. This consultant did participate in interventions under the pilot model but the LA emphasised that this change in resource would have happened regardless of the pilot and was not caused by the proposed model.

The FSA’s role in supporting the LAs in the preparation and throughout the pilot was as expected. The FSA received fewer queries than anticipated. 

2.4.2 Resourcing for a potential wider roll out of the proposed model

Both pilot LAs expressed concerns around resourcing for the potential future roll out of the proposed model due to the potential for increases in inspection frequencies and concerns around service planning (however, they did not require additional resources during the pilot). 

The joint service pilot LA were particularly concerned about how they would create service plans to help make decisions and justify resource needs for the coming year. This was in the context of local authority spending cuts and tight budgets. 

This LA explained that under the current model the shortest food standards intervention frequency was 12 months so the LA could work out how many ‘A’ rated priority premises there were and then look at how many food hygiene interventions were needed that year to align them. However, the proposed new model may require some one, three and six month visit frequencies so the LAs feel that they will be less able to accurately forecast how many interventions will be required throughout the year. While this is a reasonable worry, as shown in the section below (section 2.5.1.5) the reality is that not that many establishments were categorised with higher risk rating frequencies. 

While joint service LAs aim to align their food standards and food hygiene interventions wherever possible to make most efficient use of resources this is not foreseen in the Code. The joint service LA was concerned that the proposed shorter intervention frequencies would reduce opportunities for alignment, therefore increasing the number of intervention visits required of the LA. Data from the joint service pilot LA showed that under the proposed model only 12% of food standards inspections were due before the food hygiene inspection. Therefore, this data shows that the proposed model is likely to have a limited impact on how many standalone food standards inspections the joint service LAs will need to conduct.

When discussing the resources required, LAs explained that they valued better targeting of their resources thanks to the proposed model. However, they perceived that the same number of staff would be required to deliver the proposed model, which is as expected. The main change would be that the proposed model would enable LAs to be more effective at prioritising food standards work and identifying and resolving non-compliances with food law. 

In view of a potential roll out of the proposed model the FSA would need to consider internal resourcing for supporting LAs in adapting to the proposed new model. The FSA noted that this support is likely to be guided by the number of different MIS providers used by LAs and support provided in a group setting. 

2.5 Research Question 4: What has been the overall effect of the proposed model? Did it deliver its objectives? Were there any unintended consequences?

The proposed model achieved its objectives of effectiveness, efficiency and impact:

• Effectiveness: The compliance scoring aspect of the proposed new risk assessment helped LAs target interventions and resource towards businesses that posed the most risk.
• Effectiveness: Quantitative data showed that pilot LAs identified significantly more allergen related issues than the control LAs, showing the value of the proposed standalone allergen information risk factor. 
• Effectiveness: Quantitative data showed that the proposed risk decision matrix allowed premises to be categorised across a wider range of intervention frequencies, taking into account the business inherent risk and the assessed level of compliance. Overall number of interventions due increased by 14%. 
• Efficiency: LAs feel that the model does ensure that the resources available are used most efficiently to target the highest risk food businesses and enables better prioritisation of revisits to ensure rectification of non-compliances.
• Impact: The proposed new shorter (1, 3 and 6 months) and longer (72 and 120 months) frequencies within the proposed risk matrix were identified for a range of food business types during the pilot. Therefore, the risk matrix is allocating intervention frequencies which are better aligned with compliance scores than previously (the inherent risk score appears to have a stronger influence on intervention frequencies under the current delivery model), meaning that intervention frequencies align better with the risks faced.
• Impact: The pilot and proposed model had very little effect on relationships between LAs and FSA and no effect on relationships between LAs and businesses.

Through analysis of the qualitative interview data and quantitative data the evaluation did not identify any unintended consequences of the pilot or proposed model.

This section summarises the effects of the proposed model, namely whether it achieved its objectives:

• Effectiveness: Identifying food businesses that pose the greatest risk and targeting interventions more effectively. 
• Efficiency: Resources are better targeted by prioritising interventions in establishments that pose the greatest risk.
• Impact: LAs have a greater degree of flexibility to use intelligence and respond to where the highest risk of non-compliance is.

The evaluation team also assessed whether there had been any unintended consequences caused by the proposed model. This section draws together data supplied to the FSA by LAs engaged in the pilot and qualitative data from LA post-pilot interviews. It should be noted that the FSA quantitative data was only available for businesses which received an intervention during the pilot period and did not cover the full set of premises within the LAs area. Additionally, the short time span of the pilot and small sample number of LAs presented limitations to analysing the quantitative data. This has made it difficult to reach definitive conclusions on some aspects of the proposed model, as explained in the sections below. 

2.5.1 Effectiveness

The proposed model has performed as expected, allowing LAs to better identify and prioritise high-risk food businesses for interventions. The proposed approach to combining the inherent risk and compliance in a way which emphasises the compliance score offered a more balanced approach to identifying risk and allocating intervention frequencies. This was identified as a positive element because businesses are not rigidly limited to always being in the same risk category.

The rest of the section reviews the impact of the proposed model on changes in non-compliance and changes in frequency of interventions.

2.5.1.1 Comparison in identification of non-compliant businesses

A key finding of the initial pilot was that the proposed model was able to identify and prioritise businesses with lower compliance levels for interventions through the scoring system (with the assurance rule) and the new intervention frequencies.

To understand how the proposed model impacted on which businesses were identified for intervention within the pilot period and whether these businesses were more likely to be non-broadly compliant (NBC), the difference in businesses rated NBC during the pilot period by both the pilot LAs and the control LAs can be compared. This analysis compares compliance under the current Code definition of “broadly and non-broadly compliant”. The definition of “broadly and non-broadly compliant” can be found in annex section A1.2. 

When looking at how the proposed model performed in Wales under the current definition of NBC the pilot LAs had an NBC rate of 43% and the control LAs had an NBC rate of 18%. A chi-squared test was performed on the difference between the control and pilot LAs and this was significant at the 5% significance level. This showed that the premises visited in the pilot LAs were more likely to be NBC than for the control LAs. However, the differences in NBC rate between the two pilot LAs and the two control LAs was bigger than the difference between the two groups as a whole. Therefore, local contexts have also made a large contribution to the NBC rate. This is shown in Figure 2.1 below.

These figures exclude interventions where a risk assessment was not carried out and businesses that were found to be permanently closed.

Figure 2.1 The proportion of broadly and non-broadly compliant businesses - using the Code definition identified by the control and pilot LAs during the pilot period (source: data collected by the FSA during the pilot period)

2.5.1.2 The allergen information risk factor

The allergen information risk factor was added to the three other factors which make up the compliance score as a result of feedback from the initial pilot where it was felt that a more direct focus on allergen practices was needed. 

Data gathered by the FSA during the pilot period showed that the pilot LAs reported significantly more allergen related issues than the control LAs (A chi-squared test was performed on the difference between the control and pilot LAs at the 5% significance level). There were 55 mentions (28% of premises that had interventions) of allergen issues in the pilot group (26 of which were for non-compliant businesses) and 24 (13% of premises that had interventions) mentions in the control group (14 of which were for non-compliant businesses). This aligns with the qualitative data where pilot LAs said that the proposed model gave a more proportional focus on allergens and helped to specifically highlight where allergens and related practices were an issue (see section 2.2.1.1). 

Both pilot LAs reported that they were expecting to identify more non-compliances than they found during the pilot period due to the new allergen information risk factor. They noted that inconsistencies in interpretation of the allergen scoring guidance may have had an impact on the levels of non-compliance found, see section 2.2.3 for further details on the allergen guidance. 

However, the quantitative data showed that, when considered in isolation, the allergen information compliance factor had a small impact on the overall compliance rating. Out of 334 premises, which had interventions where a risk assessment score was given, the addition of the allergen information risk factor, overall led to the compliance score increasing for 19 businesses (5.7%) and decreasing for one (0.3%). Three of these premises received an allergen information risk factor score of one, of these two premises also had at least one other compliance factor of one meaning the overall compliance score did not change. Due to the assurance rule, the third business overall compliance score did change due to the allergen information risk factor score (see Annex 1 for more information on the assurance rule).

2.5.1.3 Interventions resulting in follow-up action

Under the proposed model it would be expected that there was an increase in follow-up action (at least in the short term) due to the model being expected to more effectively identify and prioritise high-risk premises for intervention. 

However, the quantitative data showed that significantly more premises received follow up action from the control LAs (44%) than the pilot LAs (34%). A chi-squared test was performed on the difference between the control and pilot LAs and this was significant at the 5% significance level. When minor actions such as ‘advice letters’ and ‘advice & guidance’ were excluded the difference between the control LAs and pilot LAs was no longer significant. This suggests that the control LAs used minor follow up actions more often than the pilot LAs. It should be noted that (similarly to the level of “non-broadly compliant” premises) the differences between the two pilots and the two controls were larger than the difference between the two groups. Therefore, it is unclear how the proposed model impacted on follow up action. 

Figure 2.2 below shows the follow up action rates for each participating LA. An action was counted if one or more of the following categories of actions were carried out: 

• Action in relation to non-compliance
• Advice letter
• Advice and guidance 
• Business advice/support, 
• Further compliance checks,
• Intelligence report issued to IDB
• Notified FSA
• Notified Primary Authority
• Ongoing investigation
• Referral - Local Authority
• Referral - Other
• Revisit
• Request for 3rd party action (e.g. ISP/selling platform action)
• Sampling activity
• Share information/intel
• Voluntary surrender
• Warning letter
• Warning notice given at the time
• Website takedown

Figure 2.2 The proportion of interventions that had a follow up action (source: data collected by the FSA during the pilot period)

Bar chart to compare the proportion of interventions that required follow up action for each pilot and control LA. Pilot 1 had 74% of interventions with no follow up action, 21% where advice was given and 5% where significant follow up action was required. Pilot 2 had 59% of interventions with no follow up action, 11% where advice was given and 30% where significant action was required. Control 1 had 42% of interventions where no action was required, 36% where advice was given and 20% where significant action was required. Control 2 had 66% of interventions where no action was required, 23% where advice was given and 10% where significant action was required.

2.5.1.4 Change in ratio of proactive and reactive interventions

During the pilot period, data shows that there was no significant difference in the ratio of proactive and reactive interventions between the pilot and control LA groups: 

• Pilot: 333 (94%) proactive: 20 (6%) reactive
• Control: 199 (88%) proactive: 26 (12%) reactive

A chi-squared test was performed on the difference between the control and pilot LAs at the 5% significance level. Figure 2.3 below shows the proportions of proactive and reactive interventions by each participating LA.

An intervention is classed as reactive when it is prompted by receipt of intelligence (such as a complaint), whereas an intervention is proactive if it is part of the planned programme of official controls which is determined based on the outcome of the previous official control (this data excludes advice and information gathering for Primary Authorities as well as interventions where data is not known). However, the proposed model also now defines revisits as proactive interventions meaning that over a longer period than the 6-month pilot (where there was limited time available for revisits to take place) it would be expected that the proposed model would cause LAs to increase the number of interventions classed as proactive. 

This data is likely influenced by one of the control LAs being the regional intelligence coordinator in Wales who have a higher-than-average use of intelligence. It should also be noted that the Welsh pilot, unlike the initial pilot, did not include a directed focus on piloting use of intelligence, therefore it was less likely that a change in ratio would be identified in comparison to the initial pilot findings.

Figure 2.3 The proportion of proactive and reactive interventions for the control and pilot LAs. (source: data collected by the FSA during the pilot period)

Bar chart to compare the proportion of proactive and reactive interventions for each pilot and control LA. Pilot 1 had 100% proactive interventions. Pilot 2 had 92% proactive and 9% reactive interventions. Control 1 had 75% proactive and 25% reactive. Control 2 had 99% proactive and 1% reactive.

2.5.1.5    Changes in frequency of intervention

Pilot LAs were asked to risk rate food businesses under both the current and proposed food standards delivery models. This enabled direct comparison of how intervention frequencies (the time between interventions) would vary between the current and proposed models.

The proposed model introduced a higher range of intervention frequencies, going from three options (12, 24 and 60 months) to ten options in the proposed model (1, 3, 6, 12, 24, 36, 48, 60, 72, and 120 months). Some LAs were concerned about the implications of both the new shorter intervention frequencies and the new longer intervention frequencies in terms of resourcing and the ability to keep in regular contact with food businesses.

The FSA quantitative data showed that the proposed model recategorised the intervention frequency for 75% of premises. Figure 2.4 shows how those businesses were recategorised. 

The recategorisation led to the number of interventions due (scheduled for an intervention within the next 12 months) increasing overall by 14%. This was due to the number of businesses with new shorter-term frequencies exceeding the number of establishments with longer frequencies. It is the intention of the model that in practice premises would not remain on such short intervention frequencies but would resolve issues and improve compliance levels as a result of the increase inspection frequency for poorly compliant businesses. This would then reduce the required intervention frequency, within the year as business compliance improves.

Figure 2.4 Comparison of how business’s intervention frequency change under the proposed model compared to the current model - n= 328 as not all interventions were scored under the current model by the pilot LAs. (source: data collected by the FSA during the pilot period)

A bar chart showing overall how the intervention frequencies for businesses changed under the proposed model. 35% had a shorter time period until next intervention, 25% had the same time period (no change), and 40% had a longer time period until next intervention.

To understand how the workload of the pilot LAs changed, analysis was done to compare the intervention frequency for food businesses under the current model with the intervention frequency generated by the decision matrix under the proposed model (using scoring from the same intervention during the pilot). The figures below compare how the intervention frequency changed for premises under the proposed model in comparison to the current model. There is a figure for each of the current frequencies: 

• 12 months (12 months = current Category A premises, high risk) (Figure 2.5), 
• 24 months (24 months = current Category B premises, medium risk) (Figure 2.6), 
• 60 months (60 months = current Category C premises, low risk) (Figure 2.7).

Figure 2.5 Comparison of intervention frequencies generated by the proposed model for all premises categorised as a 12-month intervention frequency (Category A) under the current Code - n=13. (source: data collected by the FSA during the pilot period)

A bar chart showing the percentage of businesses with different intervention frequencies under the proposed model who were previously categorised as a 12 month intervention frequency. 15% of these premises increased in frequency to 3 month frequencies, 23% increased to a 6 month frequency, 31% stayed at a 12 month frequency, 8% decreased frequency to 36 months, 15% decreased to 48 month frequency, 8% decreased to 60 months.

Figure 2.6 Comparison of intervention frequencies generated by the proposed model for all premises categorised as a 24-month intervention frequency (Category B) under the current Code - n=144. (source: data collected by the FSA during the pilot period)

A bar chart showing the percentage of businesses with different intervention frequencies under the proposed model who were previously categorised as a 24 month intervention frequency. 5% of these premises increased in frequency to 6 month frequencies, 14% increased to a 12 month frequency, 28% stayed at a 24 month frequency, 22% decreased frequency to 36 months, 14% decreased to 48 month frequency, 8% decreased to 60 months, 5%decreased to 72 months and 3% decreased in frequency to 120 months.

Figure 2.7 Comparison of intervention frequencies generated by the proposed model for all premises categorised as a 60-month intervention frequency (Category C) under the current Code - n=171. (source: data collected by the FSA during the pilot period)

A bar chart showing the percentage of businesses with different intervention frequencies under the proposed model who were previously categorised as a 60 month intervention frequency. 2% of these premises increased in frequency to 12 month frequencies, 4% increased to a 24 month frequency, 20% increased to a 36 month frequency, 31% increased to a 48 month frequency, 22% stayed at a 60 month frequency, 21% decreased frequency to 72 months and 9% decreased in frequency to 120 months.

Data showed that the proposed risk decision matrix allowed premises to be categorised across a wider range of intervention frequencies, taking into account the business inherent risk and the assessed level of compliance. The proposed model categorised a varied range of food business types under the newly proposed shorter intervention frequencies (1, 3 and 6 months), and the new longer intervals (72 and 120 months). In addition to these longer/shorter frequencies businesses may fall into 36 or 48 month interventions depending on their risk rating. See Annex 2: Table A2.1 to see a breakdown of how each food business type was categorised across the 10 intervention frequencies under the proposed model. 

This range of food business types and lack of pattern suggests that the proposed model is categorising based on both inherent risk and compliance scores rather than mainly by inherent risk under the current model. This should then enable LAs to target their resource where it will be most effective at reducing non-compliances.

2.5.1.6 Takeaways

Prior to the pilot period, the pilot LAs expressed concern that there could be increases in intervention frequency for many of their takeaway premises, as the LAs felt takeaways were one of their most high-risk categories of premises. LAs were concerned that this increase would put a considerable strain on resources. Under the current model, LAs felt that the risk posed by takeaways was not fully appreciated as most takeaways may have an intervention frequency of 24 months and then for non-compliances 12 months, therefore it was felt that the proposed model was likely to increase frequencies. However, qualitative interview data shows that both pilot LAs reported that the change in intervention frequencies for takeaways was not as evident as expected. Analysis of the quantitative data showed that 25 (38%) out of 65 takeaways receiving interventions during the pilot had an increase in intervention frequency, 23 (35%) out of 65 takeaways had no change in intervention frequency, and 17 (26%) out of 65 had a decrease in intervention frequency (See Annex 2: Table A2.2 for a breakdown of change in intervention frequency for takeaways). However, the overall number of interventions due for takeaways increased by 35% (this compares to 9% for all other premises types considered together). This range of variation likely shows that the changes in intervention frequency were caused by the level of compliance found on inspection rather than the inherent risk posed by the takeaway trade.

2.5.1.7 Manufacturers

One pilot LA expressed concern that the pilot would reduce intervention frequencies for large manufacturers. Although these manufacturers were often broadly compliant the LA officer was concerned about the impact of a lack of contact with the FBO and the nature of risks posed by such large distribution channels. An example of this is given in section 2.3.1.4. 

Data collected by the FSA showed that two out of nine manufacturers receiving an intervention by a pilot LA during the pilot period had an increase in their intervention frequency, three out of nine had no change and four out of nine had a decrease in their intervention frequency (See Annex 2: Table A2.3 for a breakdown of change in intervention frequency for manufacturers). While it is acknowledged these are based on a small number of premises, when considered alongside the conclusions around intervention frequency change for takeaways, the data shows that the proposed model is more effective at differentiating the appropriate intervention frequency based on inherent risk and compliance levels in combination.

2.5.2    Efficiency

Section 2.4 considers the resource needs for the pilot itself and a potential wider roll out of the proposed model. As stated previously, the proposed model will not alleviate any pre-existing issues relating to resourcing of food standards teams, nor was it intended to. LAs feel that the model does ensure that the resources available are used most efficiently to target the highest risk food businesses and enables better prioritisation of revisits to ensure rectification of non-compliances. The quantitative data shown in section 2.5.1 also supports this as only 25% of interventions remained unchanged meaning that 75% of interventions changed, likely to be more in line with the level of risk they present.

2.5.3    Impact

Whilst the quantitative data around non-compliances was inconclusive (see section 2.5.1.1) the qualitative evidence showed that the proposed risk scheme allowed officers to assess the risk of food businesses more accurately (in the opinion of food standards officers) and determine their intervention frequency based on the officers’ professional assessment of risk posed by the food business. There is less potential for a food business’s compliance level to impact the frequency of the next intervention under the current system as it is primarily driven by the inherent risk of the business. Under the proposed risk scheme officers can use their knowledge of a food business’s previous performance and management to assess the risk posed by these businesses and determine the appropriate intervention frequency.

Further, quantitative data suggested that the proposed model generates the expected impact of assisting LAs to more effectively target resource where there is most risk and more granular scoring of food businesses through the proposed compliance scoring criteria. This is shown in section 2.5.1.5 where intervention frequencies for all food business types as well as for takeaways and manufacturers (food business types specifically of concern to the pilot LAs identified in interviews) appears to have spread fairly evenly across the proposed new intervention frequencies. Additionally, the introduction of the assurance rule means that if a food business has a poor level of compliance in one of the compliance factors (therefore given a score of 1), the overall compliance risk is given a score of 1 leading to a short intervention frequency of 1, 3, 6 or 12 months. This will ensure LAs prioritise that next intervention to ensure any non-compliances are rectified. 

2.5.3.1    Impact of the pilot on relationships

In terms of the relationship between the FSA and LAs, one pilot LA and one interviewee from the other pilot LA felt that the pilot had had no impact on their relationship with the FSA. However, the other interviewee from the pilot LA reported that the pilot had made them feel ‘a little more positive towards the FSA’. 

Both LAs reported an increase in frequency of communication with the FSA and found the FSA to be communicative and available when needed. 

Both pilot LAs felt that the proposed model did not (and would not) change any relationships with FBOs. They noted that some FBOs are more aware of when the LA visits than others but any change is due to the individual rather than the proposed model. 

2.5.4    Unintended consequences

There were no unintended consequences identified during the pilot period. 
 

3.1 Implementation of the pilot

The implementation of the pilot highlighted that: 

• The support provided by the FSA throughout the pilot was well received and crucial to the success of the pilot. This level of support was specific to the pilot and any support under a potential roll out would likely be conducted in a different format to accommodate the larger number of LAs involved as well as the number of different MIS systems involved. Learning from the Welsh pilot has already been fed into preparation of the support for the roll out of the model in England and Northern Ireland. This feedback can be found in sections 2.2.3 around further guidance needs and 2.3.1.2 regarding feedback on preparation and support activities within the pilot. Overall, support was important to: 
  • Ensure consistent understanding and interpretation of the terms used to define the proposed risk scheme and decision matrix.
  • Quickly resolve any queries or provide support around any feedback LAs had during the testing of the proposed model.
  • Build confidence across LAs and help embed the pilot and model into the LAs work practices.
  • Create peer to peer learning opportunities for the two pilot LAs where they were able to share examples of best practice between themselves and present their experiences to a wider audience. 

The two pilot LAs also noted feedback on two practical elements of the pilot itself:

• More consideration should be given to the timing of pilots in the future. One LA reported finding the end of the pilot particularly stressful as it ended a month before the end of the financial year which increased the administrative burden during an already busy time.
• Both LAs noted that a longer pilot period would have helped to show the impact of the proposed model more as they felt that 6 months wasn’t long enough to see the changes in non-compliance. As this pilot builds on the findings of the initial 18-month pilot in England and NI it is likely that given a longer time period the Welsh LAs would have seen similar patterns in non-compliance.

3.2 Lessons ahead of a potential roll out of the proposed model 

The evaluation has also illustrated some lessons about how the proposed model operates and what needs to be considered ahead of a potential roll out. 

The LAs reported that they would be happy to continue with the proposed model and found it beneficial. However, they both felt additional training is needed. One in particular felt that much more guidance was needed from the FSA about how to run the joint service intervention frequency reports. The LA also identified that they would require further guidance on creating alternative enforcement questionnaires, suitable to different premises for use when food standards interventions are needed without food hygiene, to be beneficial (see section 2.2.1.3).

Both LAs emphasized that, if rollout of the proposed model was to take place, then the FSA auditors would need to accommodate the time needed for LA teams to adapt to implementing a new scheme. There is likely to be a slowdown in activities whilst confidence and experience are built up and LAs were looking for reassurance that they would not be penalised for this whilst they adapted. 

Both pilot LAs suggested that regional or database based LA support groups could be useful if a wider roll out was to take place. One pilot LA would also recommend their consistency exercises (developed by the LA) as they found it to be beneficial for their team bonding as well as for the quality of their service.

3.2.1 Reflections from the England and Northern Ireland process 

There are a number of reflections to be considered in a potential roll out of the proposed model in Wales, following the experience of the roll out in England and Northern Ireland: 

• Training and support could follow a similar approach – in England and Northern Ireland it will be provided within LAs’ MIS provider-based groups. 
  • Each group could have a single point of contact at the FSA who will provide support and training materials, and access to an online space (subject to data protection) on the Knowledge Hub where LAs can discuss issues and solutions. 
  • Additional support could include attendance of the FSA at regional food standards group meetings to give up-dates on the roll out and provide an opportunity to ask questions. 
  • Consistency exercises could be offered following roll out, where officers have had an opportunity to better familiarise themselves with the model. 
• The process for mapping LA databases will change when the proposed model is rolled out in England and Northern Ireland, due to both the algorithm used and the data quality within some LA systems, leading to some food businesses not being rated correctly (as discussed in 2.3.1.4). This is only relevant once, as it will form the basis of the new interventions. LAs will then update the risk ratings based on their interventions and intelligence received. 
  • To reduce this risk the new process will ask LAs to identify key businesses of concern prior to mapping. These can be assessed individually to ensure they received an accurate score. 
  • LAs will be asked to validate the scores of 10% of their mapped database to ensure accuracy. 
  • These two actions could help reduce the risk of establishments not being rated correctly as well as be an opportunity for LAs to familiarise themselves with the new model. 
• The FSA also noted that MIS providers could be involved in the data mapping process, helping to promote a smooth transition. MIS providers have been regularly engaged by the FSA to keep informed of the pilot and progress.

3.2.2 Reflection from the evaluation team

The evaluation team would recommend the following during a potential roll out (some of these may have already been taken into account by the FSA team):

• Facilitate a shared understanding of the terminology used in the proposed risk scheme and decision matrix, together with practical guidance illustrated with examples from the LAs in the pilot. LAs mentioned that some of the descriptions initially used for the proposed risk scheme were not very clear. LAs suggested they could benefit from re-wording or some examples. An improvement in the definitions and descriptors would help with consistency in interpretation between LAs or officers during a national roll-out. 
• Continue to provide a forum for sharing of experiences between LAs. This could be in the form of regional or MIS based discussion groups or online discussion threads. This could be coupled with consistency exercises facilitated by the FSA to encourage discussion around consistency and interpretation.
• Ensure that specific guidance is provided to joint-service LAs to maintain smooth working practices between the food standards and food hygiene services. This guidance should cover how to compare food standards and food hygiene intervention frequency reports. 
• Explain how allergens are assessed and integrated in the proposed model more clearly, by improving the descriptors in the risk rating. This particularly relates to scoring of businesses with no allergen risk due to the specific products they produce or supply as well as further clarity on the thresholds between each of the allergen risk scores.
• Provide support to LAs during the process of mapping databases from the current model to the proposed new model to reduce the risk of high-risk businesses being reclassified as non-priority due to issues with data entry. LAs should be supported in understanding how the mapping process will work and how they should go about verifying if the mapping has been successful to promote confidence in the new model and its set up processes.

The Welsh food standards pilot project has been an important step in evidencing the suitability of using the proposed food standards delivery model in Wales ahead of a potential roll out across Wales. The evaluation found that there were no unintended consequences of the proposed model when implemented in the Welsh context. 

The evidence gathered in this evaluation has built on the evaluation of the initial pilot of the food standards delivery model in England and NI. It has also contributed valuable evidence towards the England and Northern Ireland roll out, particularly on aspects of guidance and on the impact of the allergen risk score. 

LAs clearly stated that they found the proposed model to be more effective at targeting their resource to high-risk businesses and the proposed model gave LAs flexibility on how to target their resources. This is supported by quantitative data on the change to intervention frequencies. The addition of the specific allergen information risk criteria and additional guidance around allergens within other risk criteria has been seen as beneficial and allows for a stronger, more proportional, focus on allergens and allergen practices within interventions. However, it should be noted that more clarity and guidance would be needed on the various aspects discussed throughout this report to support a potential roll out, ensure sufficient support to joint service LAs and maintain a level of consistency in scoring across LAs in Wales.

There may be concern over the 14% increase in interventions, but it must be recognised this data has been obtained from just two LAs over a 6-month period and there was variation between the two pilot LAs. It is expected that as LAs become familiar with the new risk assessment scheme, some of the risk scores may change. The FSA will continue to monitor the evidence gathered as the model is implemented across England and Northern Ireland to determine whether the intervention frequencies require amending.

The following description comes from the FSA presentation documentation and the England and Northern Ireland Food Standards Pilot Evaluation final report. 

The proposed model introduces a new food standards intervention risk rating scheme that seeks to provide an accurate assessment of the potential risk posed by a food business. It incorporates three elements (Figure A1.1): 

• A single, modernised risk assessment scheme that aims to unify the way that LAs risk assess establishments (a new risk scheme).
• Using the risk assessment scheme to identify the appropriate frequency for official control activity based on levels of inherent risk and compliance (decision matrix).
• Greater integration of intelligence as a driver of local authority regulatory activity and to inform our national understanding of food standards risk.

Figure A1.1 The three elements of the proposed new food standards delivery model

Diagram depicting the three elements of the Food Standards Delivery Model: the New risk rating scheme, the Decision Matrix and Intelligence.

A1.1 Proposed risk assessment scheme

The proposed risk scheme (Table A1.2) was designed to address known issues with the existing risk schemes, and to modernise the regulatory approach so that it better reflects new food business models and provides a more dynamic and accurate assessment of food business risk. 

For example, the existing Food Law Code of Practice risk assessment scheme was felt to give too much emphasis to the inherent risk of a food business, failing to adequately recognise the business’s level of compliance. This often resulted in highly compliant businesses being inspected by LAs at a higher frequency than was deemed necessary under the current framework. 

The proposed model’s intervention risk rating scheme takes into account the ‘inherent risk’ associated with the business and the level of current and, when appropriate, sustained compliance demonstrated by the business following a ‘compliance assessment’.

The inherent risk profile gives an indication of the risk associated with a particular food establishment and the compliance assessment evaluates the food business’s performance. The Inherent Risk Profile comprises of 5 separate factors, while the Compliance Assessment comprises 4.

Table A1.2 Inherent Risk Profile and Compliance Assessment risk sub-categories

Risk element	Risk factor
Inherent Risk Profile	Scale of supply and distribution
Inherent Risk Profile	Ease of compliance
Inherent Risk Profile	Complexity of supply chain
Inherent Risk Profile	Responsibility for information
Inherent Risk Profile	Potential for product harm
Compliance Assessment	Management systems and procedures
Compliance Assessment	Allergen information
Compliance Assessment	Current compliance level
Compliance Assessment	Confidence in Management (CIM)

Inherent Risk

• Scale of supply - This factor considers the number of consumers likely to be at risk if the establishment fails to comply with food standards legislation. The greater the number of customers the greater the potential impact of any non-compliance.
• Ease of compliance - This factor considers the volume and complexity of food standards law that applies to the establishment, and with which it has a responsibility to ensure compliance.  
• Complexity of supply chain - The complexity of a food establishment’s supply chain increases risk as there is greater potential for problems with the foods and products they use, which in turn enter the supply chain.  Recognition as well to potential consequences if and when carrying out a potential product recall.  
• Responsible for information - The risk increases where an establishment is responsible for providing information about its products to its customers. There is potential for human error in compiling or communicating the product information which must be provided to consumers, as well as the opportunity for misleading claims or food labelling breaches to be included. 
• Potential for product harm - This risk factor considers the extent to which consumers may suffer harm, whether physical or financial, and to which legitimate establishments may be disadvantaged, by the supply of food which is not compliant. For example, foods aimed at particular groups (medical foods, free-from) and high value foods increasing the incentive for substitution/adulteration.

Compliance Assessment

• Management systems/procedures – This factor considers internal/external quality management systems & whether assurances are in place, how these are implemented and verified.  Proportionate to the size, scale and nature of the establishments.
• Allergen Information – This factor considers how well the business controls the aspects of allergen management and provision of allergen information applicable to them.
• Compliance level – This factor considers the current level of compliance with food law as witnessed during the intervention.
• Confidence in Management – This factor considers the likelihood of whether a business will be compliant or not given their history of compliance, attitude to compliance, management systems in place (including allergens) etc.

Each of those nine factors is given a score of 1 to 5, with 1 being poor and 5 being good. (9 factors scored in total). For both categories, the scores attributed to each of their factors is averaged to provide the respective inherent risk and compliance assessment scores.

Definitions of “broadly compliant”

Due to the change in the scoring approach the proposed model uses a different definition of “broadly compliant” and “non-broadly compliant”. Whether a premises has a status of broadly compliant or non-broadly compliant has no impact on the frequency of interventions or the burden placed on LAs. These terms are used by the FSA and LAs to get a high-level understanding of how FBOs are operating and how compliance is changing over time. The two different definitions of “broadly compliant” are set out below: 

• The current definition under the Food Law Code of Practice (Wales): an establishment that has a score of not more than ten points under both the Level of (Current) Compliance and the Confidence in Management/Control Systems.
• The definition under the proposed Food Standards Delivery Model: an establishment receives an overall compliance risk assessment score of 3, 4 or 5.

“Non-Broadly Compliant” would be any establishment that does not satisfy the above requirements. So, under the Food Law Code of Practice, if an establishment had a score of more than 10 for either the Level of Current Compliance or the Confidence in Management they would be regarded as non-broadly compliant. For the proposed Food Standards Delivery Model, an establishment would be "non-broadly compliant" if it received an overall compliance risk assessment score of 1 or 2.

The assurance rule

A principle of the proposed model establishes a rule where, if one of the risk factors of the compliance assessment has been identified as a significant non-compliance, and so given a low score of 1, the overall compliance assessment score will be 1 regardless of the performance of the other risk factors. This rule provides assurance within the proposed model that significant non-compliance will attract more frequent official controls.

The two scores are then used in conjunction with the Decision Matrix to determine the frequency of interventions for that establishment.

A1.2 Proposed decision matrix

The proposed decision matrix (see Figure A1.2 below) follows a more graduated approach than the current model so that those businesses posing the highest risk are subject to an official control at a higher frequency than those posing a lower risk, thus enabling LAs to focus their resources on those establishments posing a greater risk to public health and consumer protection.

Figure A1.2 The proposed decision matrix

Table mapping risk and compliance levels. Arrow along the inherent risk axis points to a scenario of an inherent risk score of 5 and a compliance assessment score of 1 which is an example of a Very low hazard. Arrow along the inherent risk axis points to a scenario of an inherent risk score of 1 and a compliance assessment score of 1 which is an example of a Serious hazard. Arrow along the compliance assessment axis points to a scenario of an inherent risk score of 1 and a compliance assessment score of 1 which is an example of Serious non-compliance. Arrow along the compliance assessment axis pointing to a scenario of an inherent risk score of 1 and a compliance assessment score of 5 which is an example of High and Sustained compliance.

Those food businesses posing the most severe risk will be subject to a more intensive frequency of official controls and those presenting the very lowest level of risk subject to less frequent controls. This will enable LAs to focus their resources on establishments requiring prompt action to safeguard public health (allergens) or consumer protection. Those businesses posing the greatest risk are deemed to require a “Priority Intervention”, as seen in the bottom left-hand area of the matrix.

Priority interventions are currently Category A businesses with 12-month intervention frequencies, however under the new model priority interventions cover all businesses with an intervention frequency of 6 months or less. 

When undertaking a priority intervention, officers should focus on those areas of concern and rescore the establishment on that basis, therefore a partial inspection/intervention may be sufficient, with the intention of working towards a more compliant establishment and a less intensive intervention frequency.

If following an inspection, an establishment’s intervention frequency is determined to be a priority intervention, corrective action should be taken to address the non-compliances. This could involve a revisit (focussed inspection) the following day or week. This will be on a case-by-case basis and dependent on the non-compliances found and should be in line with the hierarchy of enforcement, and in accordance with the Code.

Within that intervention frequency it is expected that the establishment would be re-risk rated to reflect any improvements made. Ideally the intervention frequency will have improved (i.e. moved out of the priority intervention frequency category). So, under the proposed new model, food standards officers can re-risk rate a premises following a focussed inspection – what the FSA currently regard as a re-visit is actually a focused inspection/focused audit.

A1.3 Remote interventions

The purpose of Remote Interventions (RI) is to allow LAs to remotely assess levels of food law compliance and verify activities at food establishments as appropriate and as determined by the LA in line with the Official Controls Regulation. Rather than a physical inspection, the LA can assess an establishment’s level of compliance without visiting it. Establishments in this category will typically have high levels of compliance and lower inherent risks.  

RIs will involve the use of a variety of approaches and techniques by LAs to monitor and verify the food establishment’s activity and compliance with food law. It is for LAs to determine when an RI is an appropriate official control.

The principles of an RI are as follows:

• Confirm the business is still trading, under the same Food Business Operator.
• Focus on the aspects of the establishment that can be assessed remotely to determine their ongoing levels of food law compliance. 
• Review the inherent risks of the establishment so the inherent risk scores can be amended if required.

Use of RIs does not mean LAs should avoid having direct contact with food establishments. The LA may decide to engage with establishments through various channels such as phone, email or letter, to obtain the evidence they need to complete the RI.

Under an RI, the establishment can be asked to provide documentation that will allow the LA to assess their levels of compliance. An RI could involve one or more of the following:

• A review of products on the food establishment’s website 
• A review of product labelling
• A check of traceability records
• Sampling to verify the compliance of products
• Product specification checks
• Steps taken to ensure the accuracy of information and requirements of food law
• The provision of food management system information to support due diligence checks. 

A1.4 Intelligence use

The proposed new model aims to formalise the use of intelligence. Chapter 4.3.2 of the current Code states that when new information becomes available that might suggest the nature of a food business’s activities has changed, or the level of compliance has deteriorated, LAs can reconsider the appropriateness of the current intervention rating, decide whether it is appropriate to undertake further activities (inspection, partial inspection, audit, investigate further), revise the intervention rating and then record the adjustment and the justification for doing so.

As a result, the proposed new model seeks to reaffirm to LAs that intelligence can initiate a re-assessment of the risk posed by a business and a review of the date of the next official control.

A1.5 New businesses: inherent risk desktop assessment

The desktop assessment of the inherent risk should be used to prioritise the initial inspections of new food businesses. For new businesses, a desktop assessment of the establishment’s Inherent Risk should be carried out within 28 days of the business registering, or from when the LA becomes aware the business is trading – whichever is the sooner.

It is envisaged that information provided through the Register a Food Business (RAFB) process should provide sufficient detail to enable the inherent risk profile to be assessed – looking at aspects such as the type of business (e.g. manufacturer, retailer, caterer), their position in the supply chain (e.g. wholesalers, selling online) and their trade activities (importing or exporting).

LAs should also use any other relevant information to help inform the assessment. This could be obtained by a search of the company’s online profile, telephone conversations, exchange of emails or questionnaires.

For businesses deemed to have a high inherent risk, so given a score of 1 or 2, their initial inspection should be undertaken within 1 month of the business opening or, if it had commenced trading prior to registering, within a month of the desktop assessment being carried out – whichever is the sooner. For businesses deemed to pose a moderate inherent risk, so given a score of 3, they should receive their initial inspection within 2 months. Those assessed as being low risk, so given an inherent risk score of 4 or 5, should be inspected within 3 months. At the initial inspection, a more accurate inherent risk assessment can be obtained alongside a compliance assessment to determine the frequency of future interventions.

Initial inspections should be prioritised so that those with a lower inherent risk do not cause undue delays to the delivery of official controls at higher risk/non-compliant businesses. It is acknowledged that some businesses are not permitted to trade until they have received an inspection, and these may have a low inherent risk – in such instances it would be acceptable to prioritise such businesses over relatively more risky businesses to prevent undue burdens/barriers to trade. 

 

Table A2.1 Intervention frequency (months) identified under the proposed model by business type. Numbers show the number of businesses recategorised under the proposed model. n= 327

Business type	Intervention frequency under proposed model (months)
	1	3	6	12	24	36	48	60	72	120
Catering premises	0	0	0	0	2 (light green)	9 (dark green)	6 (green)	4 (green)	9 (dark green)	5 (green)
Distributors / Transporters	0	1 (light green)	0	0	0	1 (light green)	0	0	0	0
Hotel / Guest House	0	0	0	0	1 (light green)	1 (light green)	0	4 (green)	0	0
Importers / Exporters	0	0	0	2 (light green)	1 (light green)	1 (light green)	0	0	0	0
Manufacturers / Packers	0	0	1 (light green)	2 (light green)	1 (light green)	2 (light green)	0	3 (green)	1 (light green)	0
Mobile food unit	0	0	0	0	0	0	0	1 (light green)	0	0
Primary producers	0	0	0	0	0	0	0	0	0	0
Pub / Club	0	0	0	1 (light green)	0	10 (dark green)	3 (light green)	4 (green)	1 (light green)	1 (light green)
Restaurant / Cafe / Canteen	0	0	2 (light green)	3 (light green)	14 (dark green)	16 (dark green)	13 (dark green)	9 (dark green)	8 (green)	1 (light green)
Restaurants and caterers - other	0	0	0	0	1 (light green)	1 (light green)	3 (light green)	2 (light green)	0	0
Retailers - other	0	1 (light green)	1 (light green)	0	1 (light green)	4 (green)	0	0	6 (green)	6 (green)
School / College	0	0	0	0	0	0	0	1 (light green)	0	1 (light green)
Small retailer	0	0	3 (light green)	5 (green)	9 (dark green)	14 (dark green)	25 (dark green)	3 (light green)	12 (dark green)	6 (green)
Supermarket / Hypermarket	0	0	1 (light green)	1 (light green)	0	2 (light green)	1 (light green)	8 (dark green)	0	0
Takeaway	0	0	2 (light green)	15 (dark green)	20 (dark green)	6 (green)	8 (green)	9 (dark green)	4 (green)	1 (light green)

Different shades of green have been used to show the differing levels of allocation to each intervention frequency across each row (by business type). The darker the green the higher the number of that business type allocated under a specific intervention frequency.

Table A2.2 The change in intervention frequency for takeaways receiving an intervention during the pilot period (n=65)

Intervention frequency under current model (months)	Intervention frequency under proposed model (months)
	1	3	6	12	24	36	48	60	72	120
12	0	0	0	1 (light green)	0	0	0	0	0	0
24	0	0	2 (light green)	11 (dark green)	16 (dark green)	4 (green)	5 (green)	3 (light green)	1 (light green)	0
60	0	0	0	3 (light green)	4 (green)	2 (light green)	3 (light green)	6 (green)	3 (light green)	1 (light green)

Different shades of green have been used to show the differing levels of allocation to each intervention frequency. The darker the green the higher the number across the whole table.

Table A2.3 The change in intervention frequency for manufacturers receiving an intervention during the pilot period (n=9)

Intervention frequency under current model (months)	Intervention frequency under proposed model (months)
	1	3	6	12	24	36	48	60	72	120
12	0	0	1 (light green)	2 (light green)	0	0	0	1 (light green)	0	0
24	0	0	0	0	0	1 (light green)	0	1 (light green)	0	0
60	0	0	0	0	0	1 (light green)	0	1 (light green)	1 (light green)	0

Different shades of green have been used to show the differing levels of allocation to each intervention frequency. The darker the green the higher the number across the whole table.

Background

1. Please can you introduce yourself and your role?

• Prompts: length of time in the role, amount of time they spend on standards, what else they do and how they split the time.

2. Confirm key characteristics - type of authority, if use RAFB (register a food business), food standards delivery method.

Existing process for Food Standard delivery model

Prompt for interviewer: Explain to pilot LAs that all these questions relate to before changes which have occurred due to the pilot

3. Can you describe how food standard controls are delivered within your local authority?  [prior to the use of the pilot? – for pilot group]

• Probe for their views on the number of interventions they were delivering; use of resources; number of non-compliances identified, number of FTEs 

4. What do you currently know about the use of intelligence? 

• Do/ how do you incorporate intelligence into the existing process for the food standard delivery model? NOTE – The food standard delivery model as set out in the current Food Law Code of Practice (Wales)
• To what extent was intelligence used in guiding your work before? Can you please explain how you used intelligence (who do you generally share intelligence with / receive intelligence from, type of tools used, IT, resources used, non-compliance cases detected, efficacy/quality of intelligence)? 
• How do you keep up to date with intelligence data and trends on compliance with food law requirements in your area (e.g., hygiene, safety, composition, adulteration, authenticity)?
• Is there any other type of data that you share with or receive from other stakeholders on a regular basis (e.g. FSA, FBOs, PA)? How often? Please give examples
• Do you have any ideas about how incorporating intelligence into the delivery model could be beneficial?

5. How do you normally identify and address non-compliances?

6. What was your local authority approach to food sampling? How did you prioritise products to sample? 

7. In your view, how well does the existing delivery model work?  What does it not do well?

• What are the enablers to delivering food standards controls? And what are the current challenges around conducting food standards controls that your LA is facing? 

8. How has your approach to food standards official controls evolved in recent years? 

• Probe: changes into how you inspect FBOs, changes in use of resources

Pilot model 

9. How did your LA find out about the food standards review and development of the new delivery model? Probe for any engagement/promotional activities by the FSA?

10. Why did you volunteer to participate in the food standards pilot / evaluation? What are your expectations for the pilot? (depending on pilot/control group)

11. For pilot - What type of engagement have you had with the FSA up to this point in relation to the project? Probe for trainings, meetings, working group, other support

• How useful have these been in preparing you for the pilot?
• To what extent have you had the information and instruction you need on how to implement and use the new delivery model?  
• How could the training and engagement activities delivered by the FSA be improved? Prompt for any gaps/need for additional training or support on a specific topic or area

12. For pilot - Have you made any changes internally to accommodate participating in the pilot e.g. new role, training, equipment, database - how easy has this been?  Which ones? What support did you receive to do this, if any?   

13. For pilot - Going forward, how do you think the new model will change the way you are currently operating?

• Probe for: New infrastructure, resources (staff), training; Use of intelligence; Relationships- FBO, regional intelligence team, the FSA
• Use of intelligence, non-compliance identification and resolution
• Relationship and communication with - FBOs, regional intelligence team, the FSA, other LAs 
• Sampling activities 

14. How, if at all, do you think the new model could alleviate the challenges your LA faces? 

15. What benefits do you think the new operating model will bring? 

• Prompts: for LA? And to FBOs? And to consumers? Other stakeholders? 
• Probe for non-compliances; understanding of FBOs and using resources more efficiently; public trust in the system; better targeting of resources, time savings, trust in supply chain.

16. Do you anticipate any challenges when implementing the new delivery model?

• Do you have any concerns around the new delivery model?
• Probe for resources, infrastructure, data management / technology barriers to implementation (potential solutions?)  

17. Do you anticipate any risks associated with the new delivery model?

• Probe for solutions and specific examples

18. If time, ask: What are the main changes expected for FBOs because of the new model?

Close

19. Is there anything you would like to add on this pilot which we have not already discussed?

20. Can we include your organisation name as one of the stakeholders interviewed?

Background

1. Please can you introduce yourself and your role within your authority? 

• Length of service
• Change in role – amount of time they spend on standards – whether this is consistent
• Whether role was different through the pilot

2. Can you explain, in general terms, the changes your LA has made to accommodate the pilot?

• How have you used the management information system (MIS) to accommodate work carried out during the pilot
• New staff / Staff reallocation
• Developed new processes
• Held any additional training sessions (internally? Training with the FSA)

3. How easy did you find it to adapt to the new pilot and make the changes required? (excluding temporary scheduling spreadsheets and additional data submissions)

• What worked well – probe emails, online training (including support received by the FSA)
• What more was needed – probe frequency, format

Pilot Process and Challenges

We’d like to understand more about how you have experienced the journey of the pilot from the beginning to its end. We have identified high level categories of the implementation of the process. 

• Support received to participate
• Using the proposed new model
• Identifying non-compliance
• Impacts of participation

4. Please can you talk us through your experience of the pilot, in the steps outlined above, and highlight any stages of the process that were difficult for your organisation?

• Probe for preparation before the pilot started (trainings, new risk model)
• Probe for ease of use of the risk rating scheme and intelligence use
• Probe for understanding of the new approach
• Probe for changes in number of non-compliances identified and resolved, consistency of the approach 
• Probe for changes in sampling, if any
• Probe for resources, skills, capacities

5. What were the specific challenges you faced whilst implementing the pilot?

• Probe: mention challenges discussed in the last interview and check how they overcome them
• Practical challenges
  • Ease of use
  • Staff
  • Data gathering
  • MIS systems
  • Computer issues/training needs
• Cultural challenges
  • Organisational adjustment to working
  • Staff buy-in

6. How were the challenges overcome?

• What did you struggle with the most?
• Are any of these challenges likely to be experienced by other LAs? Why? 

7. What helped you, and your team, work with the proposed new model being piloted?

8. How do you think this experience differed for the following:

• Staff
• Food businesses (FBOs)

9. How do you think the proposed new risk model identifies FBOs that present the greatest risk? Is it more effective / consistent when compared to the prior model?

• Do you feel the model is consistent in identifying and resolving non-compliances?
  • Identifying issues before/as they emerge
  • Responding to future risks 
  • Resolving non-compliances

10. Thinking about the process: 

• Is there anything throughout the process that has worked well?
• Any areas that have worked less well? Why? How were this overcome?

11. Please can you describe how your LAs delivery of food standard controls has changed as a result of being part of the pilot?

• Number of interventions being delivered
• TRIs
• Non-compliances identified
• Overall staffing resources

Use of Intelligence

We’d like to understand more about how your LA has been using intelligence

12. How does your organisation typically gather intelligence? 

• How is this recorded?
  • Ease of uploading data?
• How is this intelligence communicated?
  • Communicated to whom?
  • Accessed by whom?

13. Has the way that your LA uses intelligence changed throughout the pilot?

• How? 
• Definition of intelligence – is there a broader classification? 
• Type of intelligence shared
• Data updating/sharing
• Resource planning 
  • Operating pro-actively?
  • Staff
  • Prioritising FBOs presenting greatest risk
• Identifying non-compliance
• Communication with the FSA/LAs

14. How has your use of data changed throughout the pilot?

• Frequency of collection
• Quality of data being collected
• Communication of data 

15. How has your use of sampling changed throughout the pilot?

• Changed in what way?
• More effective?

Remote Interventions

We’d like to understand more about how your LA has been using RIs. 

16. Did you conduct any RIs during the pilot? (if no- do RIs have a place, are they useful)

17. How effective have you found RIs?

• Why/why not
• Staff perception of effectiveness
• Workload/ease of completion 
• FBOs experience/feedback
  • Which businesses are more suited to RIs?
  • How FBOs receive them? 
• Identifying non-compliance

18. What are the main benefits of RIs?

• Targeting resources/saving staff time 
• Gathering data

19. How will you use RIs in the future?

• Resource planning
• Guiding 

Reflecting on the Pilot

20. Did you capture any learning from the pilot? If yes, how was this captured and shared? 

• Format – (e.g. training sessions or team meeting updates) and perceived effectiveness
• Audience – whether within organisation or across LAs (peer to peer sharing?)

21. How has the pilot impacted on relationships with the FSA?

• More frequent communication?

22. How do you think the new risk scheme will impact relationships with FBOs?

23. Has there been any unintended consequences of the pilot? Please explain with examples?

24. How prepared do you feel to continue with the proposed new model without the FSA support?  

• Any support needed? What kind?

In the Future

25. What is your overall perspective on the pilot? 

26. What impact do you think the pilot will have on identifying and addressing non-compliances?

• Allergens
• Sampling
• Intelligence use

27. How do you think the implementation of the pilot will affect relationships:

• with other LAs
• with FBOs
• with the FSA

28. What could have helped the pilot to have been more effective?

29. Do you have any recommendations/suggestions on how other LAs can adapt to the new system?

• Best practices?

Close 

30. Is there anything you would like to add on this pilot which we have not already discussed? 
 

Background

Purpose: To identify any changes if the person has changed

1. Please can you introduce yourself and your role within your authority?

• Length of service
• Change in role – amount of time they spend on standards – whether this is consistent
• Whether role was different through the last 6 months

2. Please can you describe how your delivery of food standard controls has changed since the last interview (if it has changed)? [September- February 2023/24]

• Overall staffing resources

3. If there have been any changes, why were these introduced? [probe for conversations with other LAs, not enough capacity, other reasons?]

Non-compliance identification

4. How are you currently identifying non-compliances? [probe for specific examples to illustrate] 

• Is this effective?
• What can be improved?
• Consistent approach?

5. Are you carrying out Remote Interventions (RIs)?

• If so, in what circumstances?
• How effective do you think they are?
• Effect on different types of FBO
• Staff perception of use

6. What do you think are the main benefits of RIs? [probe for specific examples to illustrate] 

• Targeting resources/saving staff time 
• Gathering data

7. How will you use RIs in the future? [probe for specific examples to illustrate] 

• Resource planning
• Guiding 

8. How is sampling used to guide your work? [probe for specific examples to illustrate] 

• How has this changed since the last interview? 
• Are you conducting more/less sampling? Please explain 

Use of Intelligence

9. How does your organisation typically gather intelligence? [explain what intelligence is if not clear] [probe for specific examples to illustrate] 

• How is this recorded?
  • Ease of uploading data?
• How is this intelligence communicated?
  • Communicated to whom?
  • Accessed by whom?

10. Has the way that your LA uses intelligence changed since the last interview?

• Why/Why not?
• Definition of intelligence – is there a broader classification? 
• Type of intelligence shared
• Data updating/sharing
• Resource planning 
• Uses to identify non-compliance

11. What tools are used to share intelligence? 

• Is this the same as before?

12. What do you use intelligence for? 

• How does this influence work planning?

13. What are the barriers to using intelligence?

Barriers/Enablers

14. What is the main challenge in the delivery of food standards controls today? 

• How can the FSA address this?
  • Wider support/collaboration needed 
• How are you addressing this?

15. Has there been a change in your relationships with the establishments since the last interview?

16. Are you carrying out any practices which you think could be considered best practice?

• How are these shared?

Close

17. How could the FSA better support the delivery of Food Standards?

18. Is there anything you’d like to add to the discussion?
 

Introduction

1. What has been your involvement in the Food Standards pilot in Wales? Was this as expected? Why/why not?

• Prompt: training and guidance for LAs, following up on data needs.

Implementation of the Food Standards pilot 

2. Could you provide an overview of how the Food Standards pilot was rolled out?

• What training was provided? 
• Any other support provided to the LAs by the FSA throughout the 6-month period? 
• Did the training and support occur as expected? Do you think LAs understood the changes and felt supported? 
• Is there anything you would like to keep/change about this when considering wider roll out? 
• Would you have done something different? 

3. On reflection to what extent do you think the pilot and control LAs participating in the pilot provided a good basis for testing the model? 

• Were there any aspects not covered by these 4 LAs that you think it would have been good to cover? (example: urban vs rural)     

4. To what extent do you think the Food Standards pilot achieved what it set out to achieve?

• Prompt: improve the risk rating of businesses, improve accuracy in the assessment of compliance, better prioritisation for interventions through the new decision matrix

Outcomes and impacts

5. What are your overall reflections on the pilot model now that you had seen it being tested in a Welsh context?

6. During the scoping phase you defined success as: 

• The LAs understanding the risk ratings and how to implement them.
• Identifying non-compliance during visits and achieving resolution.
• There being no fear or overwhelm from LA caused by the new pilot model and the possibility for more high priority interventions in a shorter timeframe.
• To what extent do you think these success criteria have been met? Why/why not? Please provide examples? 

7. What do you think the benefits of the Food Standards pilot were for…

• the FSA?
• LAs?
• Businesses involved

8. And do you think there were any disadvantages/risks of the Food Standards pilot for…

• the FSA?
• LAs?
• Businesses involved

9. How has the pilot impacted on the FSA’s relationships with the LAs?

10. How do you think the new risk scheme has impacted relationships with LAs and FBOS?

11. Is there anything you would change about the model in view of potential further roll out? 

12. Did you capture any learning from the pilot? If yes, how was this captured and shared? 

• Format – (e.g. training sessions or team meeting updates) and perceived effectiveness
• Audience – whether within organisation or across LAs (peer to peer sharing?)

Challenges and risks for further role out

13. In the scoping interview you suggested that changing LAs mindsets around the new model and frequency of interventions will be the biggest challenge to the pilot: was this as big of a challenge as originally thought?  Why do you say that? 

14. What were the LAs main challenges? How were they overcome?

15. What were the main challenges for the FSA? How were they overcome?

16. One LA raised that they were concerned about how challenging it will be to train their Environmental Health colleagues on the new Food Standards model, these colleagues have just gotten used to the current model. How do you think this has been accommodated for within the pilot?

• How do you think the proposed model will work for joint services where 
  • Food Standards interventions are sometimes carried out by Environmental Health officers who are not qualified in food standards? 
  • now they are not able to do the interventions jointly with hygiene interventions? 

17. A risk was raised about the ability to plan annual resourcing needs for the LA under the new model, particularly for joint services. Do you see this as an issue, is it something the FSA are considering? 

Feasibility of wider roll out for the FSA

18. What change in resource has the Food Standards pilot required within the FSA? 

• Prompt: resource, skills, time

19. How easy did you find to adapt to the pilot model and make the changes required?

20. Given your experience of the pilot to what extent would the FSA be ready to roll out the pilot model wider? 

• Prompt: resource, skills, relationship with LAs, promotion of the scheme.
• What would need to change to help prepare the FSA? 

Considerations for the evaluation 

21. What are your hopes for the application of the study findings?

Close

22. Do you have any other comments you would like to make about the Food Standards pilot, or the evaluation?