Review of FSA Social Science
Review of FSA Social Science: Annex 2c Assessment of Value of the FHRS
Review of FSA Social Science report Annex 2c, an assessment of research outputs using GSR code and the FSA QAT assessing research reports checklist.
Outputs: Value of the FHRS: Businesses, Consumers and Local Authorities
Authors: Ipsos
Date: May 2023
Assessment of Value of the FHRS: Businesses, Consumers and Local Authorities using GSR code
Rigorous and impartial
| Rigorous and impartial | Rating | Comments |
|---|---|---|
| Based on sound methodology and established scientific principles | High | IPSOS and FSA researchers worked closely together on this project to establish the best and most feasible combination of online workshops, deliberative engagement, in-depth interviews and scenario setting to elicit views of the businesses, consumers and local authorities about the value and working on the FHRS. This generated surveys with appropriate sample sizes and representativeness that capture the diversity of businesses, consumers and LAs. There is good evidence of careful methodological planning by IPSOS and the FSA researchers. |
| Quality assured | Medium | This survey was quality assured internally by FSA social researchers at the commissioning, analytical and reporting stages. There was no external quality assurance of these three surveys. This is regrettable given the importance of independent peer review of research outputs. Other research outputs of the FSA social science team have been independently peer reviewed (footnote 1). |
| Based on best design, given constraints | High | Given that the aim of this assessment was to “to understand in more detail how Local Authorities (LAs), businesses and consumers feel about the current FHRS”, and that it had to capture views of the FHRS in England, Wales and NI, the chosen qualitative design was appropriate and well thought-through. Sample sizes were also appropriate for qualitative research with over-sampling where necessary. The range of methods (online workshops, in-depth interviews, scenario setting etc) was also well chosen. |
| Conclusions are clearly and adequately supported by data | High | The conclusions are presented clearly and succinctly and are adequately supported by the data. |
Relevant
| Relevant | Rating | Comments |
|---|---|---|
| Anticipates future policy issues as well as addressing current ones | High | This qualitative survey contributes to the FSA’s responsibility for food safety across England, Wales and NI. It provides an evidence base for current and future discussion of policy issues surrounding the FHRS. |
| Answers clear and researchable questions | High | This survey does indeed address clear and researchable questions about the value of the FHRS to businesses, consumers, the local authorities, and possible areas of change for the regulatory approach of the FSA |
| Contributes to all stages of the policy and delivery process | High | This survey certainly informs how the current FHRS is operating in practice and how it might be modified going forward. By doing so from the perspective of businesses, consumers and local authorities it has the potential to contribute to the appraisal, implementation and delivery processes of the FHRS. |
| Delivers solutions that are viable, actionable and represent value for money | Medium | This survey does not deliver solutions. That is not really the purpose of the survey. However, it does identify and test empirically views on some potential changes to the FHRS (for example, using third-party independent audits and internal audits, the use of remote inspections, reduced physical inspections, assessing supermarkets and other large or multi-site businesses as a whole business). These are viable and actionable policy options, for which the value for money would need to be assessed separately using economic appraisal methods. |
Accessible
| Accessible | Rating | Comments |
|---|---|---|
| Published | High | These three reports on the FHRS have been published by the FSA. |
| Data made available where possible | Medium | There is a great deal of summary data in each of the three sub-reports (businesses, consumers, local authorities) on the responses to the online workshops and in-depth interviews. These are generally well presented with key themes, concepts and principles having been identified. There is a summary of the sampling methods used for the Consumers’ survey (Appendix 1), but not for the businesses or local authorities’ surveys. Greater detail about sampling and the wider methodology was readily forthcoming from the FSA social research team on request. It would have been advisable to make these background methodological details publicly available on the FSA website, preferably in the form of a Technical Report, as is the case with some of the other research outputs of the FSA social research team. |
| Clear and concise | High | Given the breadth of data collected across three samples (businesses, consumers, local authorities) the three reports on the Value of the FHRS are both clear and concise. The Executive Summary and the Summary Conclusions are also well presented in a clear and concise manner. |
| Related to existing work in field | Low | Apart from the statement at the beginning of each report indicating that “as part of its work on the Achieving Business Compliance (ABC) programme, the FSA wanted to understand in more detail how Local Authorities (LAs), businesses and consumers feel about the current Food Hygiene Rating Scheme (FHRS)” there is little or no link to other relevant research work on the FHRS. This does not really lessen the value of these three reports, but it would have been useful to indicate the other related FSA research projects on the FHRS (e.g. the ‘FHRS Consumer Attitudes Tracker’ surveys, and the ‘Qualitative research to explore consumer attitudes to food sold online’). |
Legal and ethical
| Legal and ethical | Rating | Comments |
|---|---|---|
| Complies with relevant legislation | High | These reports comply with GDPR legislation. |
| Complies with GSR ethical guidelines | High | These three surveys do comply with GSR ethical guidelines. |
FSA QAT Assessing Research reports checkllst, FSA Food Hygiene Rating Scheme Surveys: Checklist 2: Assessing research reports
Q1. Title, lead author and year
Value of the FHRS: Business, Consumers and Local Authorities. Lead author: IPSOS May 2022.
Q2. Has a clear research need been outlined?
Yes – fully - The research need for these three surveys was expressed in terms of the FSA wanting to a) understand in more detail how Local Authorities (LAs), businesses and consumers feel about the current FHRS, and b) capture consumer views on potential changes to the regulatory approach.
Q3. Has a precise research question/aim been specified?
Yes – fully - For each of the three surveys (businesses, consumers, local authorities) clear and precise questions have been made from the outset.
Q4. Is the research design…
Cross-sectional - Comparative (across England, Wales and Northern Ireland).
Q5. Is the research method…
Qualitative
Q6. Is there a good match between the research question/aim, research design and research method?
Yes – fully - The research design and qualitative methods are a good match with the aims of the research and the questions addressed. The sections on methodology and sampling indicate an appreciation of how to undertake a qualitative survey.
Q7. Is the study population and setting specified?
Yes – fully - The study population includes the populations of England, Wales and Northern Ireland. The samples are well designed to capture these different populations and their distribution by key demographic factors.
If Q5 = Qualitative, go to Q8a. If Q5 = Quantitative, go to Q8b. If Q5 = Both, go to Q8a and Q8b.
Q8b. Is the sampling method…
Purposive sampling. Other Thematic Analysis.
Go to Q9.
Q9. Is the sampling method appropriate for addressing the research question?
Yes – fully - The samples for each survey – of businesses, consumers and local authorities - are purposively selected to reflect these entities and different characteristics of each (e.g. type and size of business; demographic factors; local authority areas).
If Q5 = Qualitative, go to 9a. If Q5 = Quantitative, go to 9b. If Q5 = Both, go to Q9a and Q9b.
Q9a. Is the sampling method appropriate for addressing the research question?
Yes Fully - The sample sizes are appropriate for qualitative research, and there is some over-sampling of businesses with low FHRS score (1-3) that represent a small proportion of businesses.
Go to Q10.
Q10. Are the research instruments valid and reliable?
Yes – fully.
If Q5 = Qualitative, go to Q11a. If Q5 = Quantitative, go to Q11b. If Q5 = Both, go to Q11a and Q11b.
Q11b. Is the analytical approach…
Thematic analysis
Got to Q12.
Q12. Is there a good match between the analytical approach, the research method and the research question?
Yes – fully - The analytical approach has been designed and executed to identify the themes, concepts and principles of respondents’ experiences of the FHRS. It has done this well.
Q13. Has a relevant checklist from the EQUATOR Network been used in the reporting of the results?
Yes – partly - Not explicitly, though the reporting of these three surveys follows closely the structure of the EQUATOR Network guidance document for qualitative research (O'Brien B C, Harris I B, Beckman T J, et al, 2014).
Q14. Have descriptive data on the characteristics of participants been presented?
Yes – partly - There is less descriptive data on the characteristics of participants in the final reports of these three surveys than in the background methodology responses provided by the contractors for the FSA research team. It would be good practice to make available these methodological details perhaps in the form of a separate Technical Report.
Q15. Have two or more researchers been involved in the analysis process (e.g., through double coding)?
Cannot say. This information is unavailable
Go to Q16.
Q16. Is there consistency between the data presented and the themes?
Yes – fully - The themes, concepts and principles underlying respondents’ views and experiences of the FHRS are fully consistent with the data presented
Go to Q17.
Q17. Have similarities and differences between participants been explored (e.g., negative cases)?
Yes – fully - The analysis of the qualitative data has identified positive and negative themes in respondents’ views and experiences of the FHRS. It has also shown that these different views and experiences reflect the background characteristics of respondents (e.g. type and size of business, urban and rural setting, etc)
Go to Q18.
Q18. Did participants provide feedback on the findings (i.e., member checking)?
No - There is no indication that the analysis and findings of these three reports have been fed back to the businesses, consumers or local authorities involved. Further information on this would be helpful.
Go to Q21.
Q21. Has generalisability been considered in the interpretation of the results?
Yes – fully - The reports have revealed generalised findings within the context of a qualitative survey, for example, identifying themes, concepts and principles that were revealed across nations and sub-groups of the population. There was also attention given to where there was a diversity of views and experiences about the FHRS. The balance between generalisability and context specificity has been presented rather well.
Q22. Has causality been considered in the interpretation of the results?
Not applicable - Causality is hardly appropriate with a non-experimental or non-quasi experimental design. Thematic analysis is what is offered and what is appropriate.
Q23. Has uncertainty been considered in the interpretation of the results?
Yes – partly - The results of these three surveys have been reported with due caution. Uncertainty is indicated to some extent given the diversity and the context specificity of the findings, but this is perhaps best expressed as caution rather than uncertainty.
Q24. Has a clear study conclusion been presented?
Yes – fully - All three studies that make up this overall view/review of the FHRS has presented clear conclusions.
-
The Acqua Book (HM Treasury, 2015:27) indicates that “a formal peer review or audit should also be accounted for in the time and resource needs of the analysis and the impact on the wider programme should be understood and managed.” The Food Hygiene Rating Scheme is a major and central programme of the Food Standards Agency that carries high business risk, in terms of both food safety and the financial survival of firms, and high complex analysis. Consequently, it warrants external peer review of the highest order.