Annex D: how the FSA uses evidence

Case study 1: evaluating the use of remote assessments by local authorities for regulating food businesses (2021)  

As part of its response to the coronavirus pandemic the FSA advised local authorities that they may use remote assessments in some circumstances to determine areas to target during a subsequent onsite visit and in other circumstances to inform the need for an onsite visit to assess and address potential public health risks.  

These remote assessments can take a variety of forms, for example, a phone call, a video call, or exchange of information online.  

The FSA commissioned ICF to undertake a short evaluation of local authorities and food business operators experience of using remote assessments. This was to help inform the FSA’s thinking about future regulatory practice and to ensure the most efficient and effective use of local authority resources.   

ICF took a qualitative approach to the evaluation. They first conducted a scoping and desk research phase followed by 20 interviews with local authorities.  

The evaluation found that remote assessments were perceived as a helpful tool in the context of the pandemic and for use with low-risk businesses and for Food Standards Controls (for example, assessing menus, labelling). It also highlighted that there were concerns about a perceived potential for food businesses to conceal information and falsify/mask problems with their businesses.  

Evidence from this evaluation provided the FSA with valuable insight on the viability of using remote assessments with food businesses. It subsequently informed the decision to conduct further research on the use of remote reassessment in the scenario in which a business requests to be re-rated if they did not achieve the top food hygiene rating and if they have made the required improvements.  

More information on this evaluation is available on the FSA’s website. 

Case study 2: evaluations of recalls and withdrawals 

Between 2016 and 2017, the FSA and Food Standards Scotland (FSS) undertook a review of the withdrawal and recall system in the UK food retail sector, to identify if improvements were needed to enhance the current system.  

This system redesign aimed to increase consumer awareness of the recall process, outline clear recall roles and responsibilities (for Food Business Operators, local authority enforcement officers, consumers and the FSA) and increase legislative compliance among food business operators (FBOs). The system redesign resulted in the creation of a package of tools, including UK guidance on Traceability, Withdrawals and Recalls, best practice guidance on communicating food recalls to consumers, a template point of sale notice and a Root Cause Analysis (RCA) package. 

RSM UK Consulting LLP (RSM) was commissioned jointly by FSA/FSS in 2021 to conduct a process evaluation to explore the programme processes and the partnership approach used as part of these processes, and the success (or otherwise) of achieving: clear and distinct roles/ responsibilities in the new system; consistent and accessible information provided to consumers, and cross industry sharing of approaches and impact; increased public awareness of food recalls and actions they need to take; and, commitment to continuous system improvement. 

The evaluation involved a desk review of existing programme documents and data to understand the original evidence base and rationale for change, interviews with external stakeholder reference group members, case studies with FBOs and enforcement agencies involved in recent recalls, exploration of hypothetical scenarios to glean learning on the ability of the redesigned recalls system, focus groups with consumers to explore consumer awareness of product recalls, and secondary data analysis to establish baseline and explore implementation for the post system redesign.  

The final report of findings and recommendations is due to be delivered to the agency in August 2022. This evaluation is expected to inform provision of new and tailored guidance for FBOs, communications work to educate and raise awareness of the recalls procedures and resources amongst businesses and consumers, and, an improved system for sharing Root Cause Analysis findings.  

Learnings from the evaluation will also be shared across programmes within the FSA to support learning across workstreams.  The evaluation report will also recommend that the Agency use the successful Recalls system redesign approach, with clearly designed workstreams and regular engagement with key stakeholders, for any future projects that require partnership working.  

Case study 3: evaluation of the Food Hygiene Rating Scheme and the Food Hygiene Information Scheme

In 2010/11 the FSA introduced two schemes intended to provide consumers with information about the hygiene standards of food premises so that they can make informed decisions about where to buy food and eat away from home: The Food Hygiene Rating Scheme (FHRS) in England, Wales and Northern Ireland and the Food Hygiene Information Scheme (FHIS) in Scotland (which now comes under the responsibility of FSS). The schemes aim to improve food hygiene standards among food businesses, which are expected to respond to public demand for higher standards, and their overarching goal is to reduce the incidence of food-borne illnesses in the UK population.  The schemes are FSA/local authority partnership initiatives which provide consumers with information about hygiene standards in food premises at the time they are inspected to check compliance with legal requirements. The FHRS rating or FHIS result given to the business reflects the inspection findings. Under the FHRS, businesses are given one of six ratings on a numerical scale from ‘5’ (very good hygiene standards) at the top to ‘0’ (urgent improvement required) at the bottom. Under the FHIS, businesses are given either a ‘Pass’ result or an ‘Improvement required’ result. 

In 2011, the FSA and FSS commissioned the Policy Studies Institute (PSI) to evaluate these programmes. The overall aim of the evaluation was to assess whether the FHRS and FHIS were operating as intended; whether the schemes improved food hygiene standards at food premises and ultimately contributed to a reduction in food-borne illnesses.  

It had two main strands:  

1. A process evaluation, which took place between autumn 2011 and summer 2013 and collected data from the perspectives of local authority food safety team staff, food business operators and consumers. The process evaluation took a case study approach in which fieldwork took place within a sample of UK local authorities. Data collection included interviews with local authority officers and food business operators, a survey of food business operators and focus groups with consumers. 
2. An impact evaluation which focused on those local authorities that launched the FHRS or FHIS during the 2010/11 financial year and tested the causal effect of the FHRS/FHIS on two sets of outcomes: i) compliance with food hygiene standards and ii) food-borne illnesses. The study used a difference-in-differences (DID) methodology: outcomes for local authorities that introduced the FHRS/FHIS (in financial year 2010/11) were compared to outcomes for local authorities that did not. The difference between the outcomes observed for the two groups of local authorities provided an estimate of the causal effect of the FHRS/FHIS. Impacts were observed in the first and second years after local authorities launched a national scheme (early adopters), financial years 2011/12 and 2012/13. 

To support the evaluations, theories of change were developed to set out the scheme logic and assumptions underpinning behaviour changes for each stakeholder group (local authorities, food business operators and consumers). These theories of change served as the conceptual framework for the evaluation. 

The process evaluation validated the theory of change, in the most part, with the UK-wide implementation of the schemes on target at the point the evaluation reported. All but one LA was operating or had committed to run the schemes by the end of the evaluation. That said, there were variations in how the schemes were implemented across the nations with the marketing of the schemes varying a great deal in Northern Ireland and Wales.   

The impact evaluation found that the FHRS had a statistically significant positive impact on food hygiene standards; the FHIS scheme was not shown to have had a statistically significant impact on compliance but that the trends were broadly the same as in the rest of the United Kingdom. Business compliance had also improved and there had been a significant decrease in the volume of poorly performing premises due to FHRS. However, due to serious data limitations it was not possible to derive reliable impact estimates testing the effect of the FHRS/FHIS on the incidence of food-borne illnesses. 

The evaluation provided valuable evidence for FSA to inform and support the take up of FHRS by local authorities, becoming mandatory in Wales in 2013 (and extended to business to business in 2014) and in Northern Ireland in 2016. It has been used to build an evidence case in support of England introducing a mandatory scheme. The evaluation also enabled the FSA to take targeted actions to support Local Authorities operating the schemes, encourage businesses to comply and support consumer awareness and engagement. Learnings were also circulated internally to ensure colleagues could reflect these in their workstreams.