Review of FSA Social Science
Review of FSA Social Science: Annex 2b Assessment of FSA Small and Micro FBO Tracking Survey Wave 3 2021
Review of FSA Social Science Annex 2b an assessment of research outputs using GSR Code and the FSA QAT Assessing research reports checklist.
Outputs: FSA Small and Micro FBO Tracking Survey Wave 3 2021
Authors: IFF Research
Date: March 2022
Assessment of FSA Small and Micro FBO Tracking Survey Wave 3 2021 using GSR code
Rigorous and impartial
| Rigorous and impartial | Rating | Comments |
|---|---|---|
| Based on sound methodology and established scientific principles | High | This survey was undertaken after relevant and careful cognitive testing and sample design/planning. Considerable effort went into sample preparation – for example, research to undertake telephone lookups. using external data suppliers (Market Location and REaD Group) and internal desk research. Weighting of the sample was also undertaken carefully and appropriately. |
| Quality assured | Medium | This report was quality assured by FSA internal researchers and external consultants. FSA research reports are quality assured via peer reviewer comments directly on draft reports and discussions in meetings rather than with a separate report of comments. |
| Based on best design, given constraints | High | The survey design was appropriate and built upon previous waves of the FBO Tracker, with some additional questions (with the agreement of the FSA). |
| Conclusions are clearly and adequately supported by data | High | The conclusions are laid out clearly at the end of the report, and a summary of the conclusions is presented in the Executive Summary. In both case the conclusions are adequately supported by the data. |
Relevant
| Relevant | Rating | Comments |
|---|---|---|
| Anticipates future policy issues as well as addressing current ones | Medium | This report certainly addresses how recent and current policy issues of the FSA, and the wider UK Government, have impacted on small and medium sized businesses. It does not really anticipate future policy issues. |
| Answers clear and researchable questions | High | This research report addresses, and answers, clear and researchable questions about:
|
| Contributes to all stages of the policy and delivery process | Medium* | Whilst this report does not directly contribute to all stages of the policy and delivery process it does ‘unpack’ the evidence on how UK’s exit from the European Union (EU), the regulatory structures of the FSA, and trust in the FSA has developed since the 2019 Tracker survey. This may have implications of FSA policy and practice issues. |
| Delivers solutions that are viable, actionable and represent value for money | Low | This report presents clear conclusions about the findings of the survey. It does not venture into delivering solutions or establishing value for money. This is not the responsibility or role of the survey company (IFF) who undertook this survey. It does not appear to have been in the survey specification. |
Accessible
| Accessible | Rating | Comments |
|---|---|---|
| Published | High | The Small and Micro Food Business Operator (FBO) Tracking Survey: Wave 3, 2021 Technical Report is published and available at: https://doi.org/10.46756/sci.fsa.sty242 |
| Data made available where possible | High | Summary data are available in this report and the accompanying Technical Report. These are presented clearly and extensively. |
| Clear and concise | High | Although this report overall is not really concise (approximately 100 pages) it is clearly presented. Also, the individual chapter are reported concisely. |
| Related to existing work in field | High | This report builds upon previous waves of this annual tracking survey that were conducted in 2018 and 2019. Hence, it relates to existing work in the field, and it extends this earlier work with additional questions. |
Legal and ethical
| Legal and ethical | Rating | Comments |
|---|---|---|
| Complies with relevant legislation | High | This report seems to comply with relevant legislation that outlines the FSA’s regulatory role, and its law enforcement function on food crimes and food hygiene. |
| Complies with GSR ethical guidelines | High | This survey has been undertaken in compliance with the GSR ethical guidelines. |
* External contractors are not always in a position to “anticipate future policy issues as well as addressing current ones”, “contribute to all stages of the policy and delivery process” and “deliver solutions that are viable, actionable and represent value for money”. Hence, a low or medium score reflects the limitation of using the GSR Self-Assessment tool for assessing the quality of research outputs.
FSA QAT Assessing Research reports checkllst, FSA Small and Micro FBO Tracking Survey Wave 3 2021: Checklist 2: Assessing research reports
Q1. Title, lead author and year
FSA Small and Micro FBO Tracking Survey Wave 3 2021, Author: IFF Research, March 2022
Q2. Has a clear research need been outlined?
Yes – fully - A clear research outline is presented about the implications of the EU Exit on small and micro enterprises, attitudes towards FSA regulation, and trust in the FSA.
Q3. Has a precise research question/aim been specified?
Yes – fully - The survey has been designed to address questions about the EU Exit on small and micro enterprises, attitudes towards FSA regulation, and trust in the FSA.
Q4. Is the research design…
Longitudinal
Q5. Is the research method…
Quantitative
Q6. Is there a good match between the research question/aim, research design and research method?
Yes – fully - This survey addresses the research aims using an appropriate design and research methods.
Q7. Is the study population and setting specified?
Yes – fully - This survey clearly specifies that the population in question is small and micro Food Business Operators (FBO) in England, Wales and Northern Ireland.
If Q5 = Qualitative, go to Q8a. If Q5 = Quantitative, go to Q8b. If Q5 = Both, go to Q8a and Q8b.
Q8b. Is the sampling method…
Stratified sampling
Go to Q9.
Q9. Is the sampling method appropriate for addressing the research question?
Yes – fully - The samples selected allows sub-group analyses by types of FBO in the constituent countries of England, Wales and Northern Ireland.
If Q5 = Qualitative, go to 9a. If Q5 = Quantitative, go to 9b. If Q5 = Both, go to Q9a and Q9b.
Q9a. Is the sampling method appropriate for addressing the research question?
Yes Fully
Q9b. Has a sample size calculation been conducted?
Yes – fully - The sample sizes in England, Wales and Northern Ireland have been calculated for statistical power and are adequate to undertake analysis of different types of FBO in England, Wales and Northern Ireland. Details are presented in the text and Annex 2 of the report
Go to Q10.
Q10. Are the research instruments valid and reliable?
Yes – fully - The research instruments have been developed using good cognitive fieldwork amongst a small subset of businesses. A total of 700 ‘mainstage 1’ interviews were conducted via a Computer-Assisted Telephone Interviewing (CATI) methodology. The data was weighted to be representative of the in-scope micro and small FBO population across England, Northern Ireland and Wales at the time of sampling. This represents good fieldwork planning and development.
If Q5 = Qualitative, go to Q11a. If Q5 = Quantitative, go to Q11b. If Q5 = Both, go to Q11a and Q11b.
Q11b. Is the analytical approach…
Time series analysis
Q12. Is there a good match between the analytical approach, the research method and the research question?
Yes – fully - The analytical approach builds upon Waves 1 and 2 of the Tracker Survey and it has included additional questions to address policy and business issues since the 2019 Tracker (and the COVID pandemic).
Q13. Has a relevant checklist from the EQUATOR Network been used in the reporting of the results?
Yes – partly - This survey does not explicitly use the EQUATOR Network checklist for surveys, but it certainly follows most of the requirements of this checklist.
Q14. Have descriptive data on the characteristics of participants been presented?
Yes – fully - This survey provides descriptive details of the different participants amongst the service activities sector, larger businesses, sole traders and those with a higher FHRS rating certificate.
If Q5 = Qualitative, go to Q15. If Q5 = Quantitative, go to Q19. If Q5 = Both, go to Q15 and Q19.
Q19. Have descriptive data on exposures/interventions and potential confounders been presented?
Yes – partly - This is a single wave report on a longitudinal survey and not an experimental or quasi-experimental study. It does investigate the consequences of the UK exit from the EU and, to some extent the COVID-19 pandemic, but it does not attempt statistical modelling of other potential confounders. This would require a different research design and analytical approach.
Go to Q20.
Q20. Have unadjusted and adjusted point estimates and confidence intervals been presented alongside statistical significance?
No -The survey results are presented mainly as percentages of respondents’ different responses. There are no unadjusted and adjusted point estimates and confidence intervals, though the results are based on weighted data. The distribution of responses are often classified as “most likely”, “more likely than average” or “particularly likely”.
Go to Q21.
Q21. Has generalisability been considered in the interpretation of the results?
Yes – fully - The survey has been designed to enable findings to be generalisable across different FBO businesses and the different countries (England, Wales and Northern Ireland). At the same time the authors are careful to note where and when caution should be taken when interpreting these results due to small base size.
Q22. Has causality been considered in the interpretation of the results?
No
Not applicable
Causality is hard to establish with survey data that does not have counterfactual samples. This survey recognises this and merely reports descriptive percentages of responses to a wide range of questions. The report does not go beyond analysis that is possible with survey data only.
Q23. Has uncertainty been considered in the interpretation of the results?
Yes – partly - This survey acknowledges where the results are sometimes based on small sub-samples and, hence, where the results are uncertain. It does not over-interpret the findings of the survey.
Q24. Has a clear study conclusion been presented?
Yes – fully - The Conclusions section of this report is clear and cautious. It is balanced and indicates where the survey responses are mixed in terms of the consequences of (for instance) UK exit from the EU and the resultant impacts on business activity
Revision log
Published: 8 August 2023
Last updated: 28 June 2024