Efficacy of Withdrawals and Recalls: Evaluation methodology

Listed below are the six elements in our evaluation approach for this final report:

1. Desk review of FSA/FSS programme documentation
2. Interviews with ESRG members
3. FBO case studies
4. Hypothetical scenarios
5. Consumer focus groups
6. Analysis of FSA/FSS data

5.1 Desk review of FSA/FSS programme documentation

The evaluation team reviewed approximately 100 separate pieces of programme documentation provided by the FSA/FSS, including: 

• Guidance on Food Traceability, Withdrawals and Recalls within the UK Food Industry Working Group papers and meeting notes for each workstream; 
• Terms of References (TOR) for each working group; and 
• Root Cause Analysis (RCA) guidance for LAs and Industry. 

The purpose of this desk review was to understand the original evidence base and problem statement/rationale for change, as well as the processes used to design the programme. The documents were provided by the FSA/FSS Incidents & Resilience Team, and a gap analysis was conducted by the evaluation team to identify any additional documents. 

5.2 Interviews with ESRG members

First set of interviews: November-December 2021 

Between November and December 2021, eleven interviews were conducted with members of the ESRG, including representatives from local authorities (x3), FSA/FSS (x6) and industry bodies (x2). The purpose of these interviews was to explore ESRG members’ perceptions on how effective the processes have been in developing the new system, as well as the effectiveness of its implementation in delivering the planned outcomes. ESRG members were sampled based on levels of involvement, region, workstream and stakeholder type (e.g. consumer, industry or local authority representative). The topic guide can be found in Appendix D. 

Second set of interviews: January-March 2022 

Between January and March 2022, another seven interviews were conducted with ESRG members. These interviews were designed to understand how well ESRG members considered the current withdrawals and recalls system to respond to new and emerging food trends. The topic guide can be found in Appendix E. 

5.3 FBO case studies: combining findings from interviews with FBOs and associated enforcement officers 

To assess the efficacy of system reform implementation and capture the experiences and views of FBOs and enforcement agencies involved in recent recalls, nine real-life anonymised case studies were developed. These case studies involved a review of FSA/FSS documentation, followed up by in-depth virtual interviews with affected FBOs and relevant enforcement authorities. The topic guide can be found in Appendix F. 

The following sample of case studies was produced: 

5.4 Hypothetical scenarios: combining findings from interviews with ESRG members and enforcement officers 

To glean learning on the ability of the redesigned system to address withdrawals and recalls relating to new and emerging trends in the food sector, we undertook seven interviews with ESRG members and eight interviews with enforcement officers. The topic guides can be found in Appendix E and Appendix H. 

5.5 Consumer focus groups 

To explore consumer awareness of product recalls, we conducted five virtual focus groups comprised of four-eight participants in each, including those who have and who have not experienced a food product recall since 2019. Participants were sampled using the criteria outlined in the table below, to ensure that the sample was representative of consumer shopping habits (eg from all four nations, across age groups and those purchasing food for families and those shopping for themselves). The topic guide can be found in Appendix G. 

5.6 Secondary data analysis

In order to establish a baseline, a review was undertaken of FSA/FSS datasets prior to the rollout of reforms (March 2018 – March 2019). This review included: 

• FSA/FSS incident data 
• RCA data 
• FSA/FSS web and social media data 
• Public Attitudes Tracker survey data (until 2019) and Food and You 2 (post 2019) 
• FSA Micro and Small Business Tracker survey data 

5.7 Limitations for each evaluation stage

Desk review

• Limited documents were available for some workstreams (eg workstream three). 

ESRG member interviews 

• Despite repeated invitations, no consumer group representatives were available for interview, so their views were unable to be included in this evaluation. 
• Members from England, Northern Ireland and Scotland were interviewed; however, no member from Wales was available. 
• We were unable to secure an interview with the lead of the industry led Workstream 3(trade to trade work stream) 
• Some ESRG members had changed roles since the first set of interviews, and were unable to participate in the second set of interviews. 

Case studies

• There were no withdrawals that would have easily added value to the learnings of the evaluation, so instead case studies focused on recalls. 
• There were no suitable incidents within agriculture and fisheries, and catering and hospitality which were originally suggested in our case study sampling framework. 

Hypothetical scenarios 

• Despite repeated invitations, no ESRG consumer group representatives were available for interview, so the views of these groups cannot be included in this evaluation 

Data analysis

• The desk review was limited to recall data, as withdrawal data was regarded as too complex by the FSA/FSS project team to extract from the FSA/FSS system. 
• Web analytics and social media data was unavailable from the FSS for the period 2021/22, so no comparison was possible 
• There were some differences in routinely collected data from the FSA and FSS, meaning that comparisons were not possible in some instances. 
• As products are sold in all four nations, there could be cases of incidents which are duplicated across both FSA and FSS datasets.