Final report for the efficacy of withdrawals and recalls evaluation
Appendix C: Evaluation framework
The evaluation framework presents the evaluation questions mapped against potential sources of data/ information.
The evaluation framework forms the basis for all evaluation activities and directly informs the development of research tools (such as interview guides). The following sources of evidence are to be used:
- desk review of documentation (provided by the FSA);
- secondary data (provided by the FSA, FSS and any partners eg. Local enforcement agencies);
- interviews with ESRG members;
- interviews with enforcement officers and wider stakeholders (eg. Industry body);
- interviews with FBOs (sampled to represent companies who have/ have not experienced a product recall or withdrawal under the new system);
- focus group with consumers; and
- case studies with FBOs and enforcement agencies.
Evaluation framework
The evaluation framework below was discussed with the steering group at the evaluation workshop.
Table 12: Evaluation framework
Objective 1: Efficacy of the internal programme process
| Evaluation question/theme | Research questions or metrics | Data source/method |
|---|---|---|
| 1. How effective was the system redesign? | A) Strength of evidence base/how did it inform system re-design? i) Evidence base and problem statement (for example, consumer insights). Was the evidence sound, what were the recommendations and was best practice relevant to UK? ii) How were insights and pilot approaches with industry, used to inform design/implementation? |
|
| 1. How effective was the system redesign? | B) Governance/management structures for the programme of work: were they effective in system re-design? i) Were the governance/management structures/partnership approach fit for purpose? What worked well/less well? |
|
| 1. How effective was the system redesign? | C) What were the objectives for reform (4 ESRG workstreams)? i) Discuss objectives for the 4 workstreams and overarching aims. ii) What was the purpose/intention of each workstream? |
|
| 2. How effective is the system delivery? | A) Delivery progress of internal programme process and partnership approach i) What worked well/less well? ii) Were approaches piloted to assess effectiveness before rolling out? iii) How were tools, guidance and processes decided on? iv) Were industry/enforcement authorities/consumer groups consulted/engaged with effectively? v) Could any improvements have been made? |
|
| 2. How effective is the system delivery? | B) Have reform objectives been met in the design of the new system and the ‘package’ for FBOs/LAs? (Nb. This question is focused on design not implementation). i) A withdrawal and recall system founded on a clear and distinct set of roles and responsibilities, agreed, and commonly understood by all participants ii) Information to consumers is consistent and accessible, based on proven best practice and underpinned by cross industry sharing of approaches and impact iii) The public are aware of the recall process and what actions they should take iv) Feedback loops and a philosophy of continuous improvement amongst all stakeholders underpins the withdrawal and recall system |
|
Objective 2: Efficacy of the new system
| Evaluation question/theme | Research questions or metrics | Data source/method |
|---|---|---|
| 1. Are roles/responsibilities in the new system clear and distinct? | A) Regulators/industry awareness and understanding of ‘the package’ i) Describe the relevant incident and processes followed in providing support and reporting Root Cause Analysis ii) Did you use the package and what actions were advised? iii) Do you agree that the current withdrawal and recall system is founded on a clear and distinct set of roles and responsibilities, agreed, and commonly understood by all participants? |
|
| 2. Is information provided to consumers, and cross-industry, sharing of approaches and impact, consistent and accessible? | A) Industry use of ‘the package’ i) What are the most useful elements of the package and why? ii) What are your views on the new guidance and materials for recalls and withdrawals available to FBOs, on the FSA/FSS website? iii) Do you know where to access these and are using the package regularly to support FBOs? iv) Do you think that industry is making good use of the new guidance and package of tools/support? |
|
| 2. Is information provided to consumers, and cross-industry, sharing of approaches and impact, consistent and accessible? | B) Industry use of root cause analysis (RCA) and success in cause identification. i) Is RCA being routinely conducted? ii) How successful is RCA in finding a cause and are findings shared with industry bodies? iii) Are any further improvements needed? iv) Has your organisation completed RCA e-learning training? If yes, was it useful? If no, why? |
|
| 2. Is information provided to consumers, and cross-industry, sharing of approaches and impact, consistent and accessible? | C) Sharing RCA learning with wider industry and impacts i) Have any RCA reports resulted in learning that has helped other businesses avoid the same problems? How has this worked in practice? ii) If learning has not been shared, why not? iii) What more could be done to share learning in industry? |
|
| 2. Is information provided to consumers, and cross-industry, sharing of approaches and impact, consistent and accessible? | D) Impacts attributed to new system i) What difference has ‘the package’ made to timeliness of notices, consistency of information, targeting of consumers, under what circumstances and why? ii) What was the process/mechanism by which the whole package and individual elements led to or contributed to outcomes (process tracing approach)? iii) Have some elements of the package been more impactful than others? iv) How did the package (or elements of it) lead to positive outcomes? |
|
| 3. Has public awareness of food recalls and actions they need to take been increased? | A) Increased awareness of food recalls and actions required For consumers: i) How aware were you of product recall procedures prior to the incident? ii) What are your preferred channels of information for food recalls (news sources, in store notices, social media, email, letter)? iii) How might technology be used to inform you about a product recall in future? For FBOs: i) Do you agree that the public are more aware of the recall process and what actions they should take? ii) Have you seen increased numbers of returns after incidents? |
|
| 4. Is there commitment to continuous system improvement? | A) Commitment to improve delivery through continuous learning by delivery agencies i) Which agencies did you engage with, what were their roles/responsibilities, and were they aware of these? ii) How well did the agencies carry these roles/responsibilities? |
|
| 4. Is there commitment to continuous system improvement? | B) Data collected/monitored and by which agencies. i) What data is collected and analysed? What are the indicators for success? ii) How is it used for system improvements? |
|
| 5. Other impacts and learning | A) Have there been any unintended outcomes arising from system change? i) Have there been any positive unintended effects? ii) Have there been any negative unintended effects? What is needed to address these? |
|