Review of FSA Social Science: Annex 2b Assessment of FSA Small and Micro FBO Tracking Survey Wave 3 2021

Outputs: FSA Small and Micro FBO Tracking Survey Wave 3 2021

Authors: IFF Research

Date: March 2022

Assessment of FSA Small and Micro FBO Tracking Survey Wave 3 2021 using GSR code

Rigorous and impartial

Relevant

Accessible

Legal and ethical

* External contractors are not always in a position to “anticipate future policy issues as well as addressing current ones”, “contribute to all stages of the policy and delivery process” and “deliver solutions that are viable, actionable and represent value for money”. Hence, a low or medium score reflects the limitation of using the GSR Self-Assessment tool for assessing the quality of research outputs.

FSA QAT Assessing Research reports checkllst, FSA Small and Micro FBO Tracking Survey Wave 3 2021: Checklist 2: Assessing research reports

Q1. Title, lead author and year
 FSA Small and Micro FBO Tracking Survey Wave 3 2021, Author: IFF Research, March 2022
Q2. Has a clear research need been outlined?  
Yes – fully -  A clear research outline is presented about the implications of the EU Exit on small and micro enterprises, attitudes towards FSA regulation, and trust in the FSA.
Q3. Has a precise research question/aim been specified?  
Yes – fully - The survey has been designed to address questions about the EU Exit on small and micro enterprises, attitudes towards FSA regulation, and trust in the FSA.
Q4. Is the research design… 
Longitudinal 
Q5. Is the research method… 
Quantitative 
Q6. Is there a good match between the research question/aim, research design and research method?  
Yes – fully - This survey addresses the research aims using an appropriate design and research  methods.
Q7. Is the study population and setting specified?  
Yes – fully -  This survey clearly specifies that the population in question is small and micro Food Business Operators (FBO) in England, Wales and Northern Ireland.
If Q5 = Qualitative, go to Q8a. If Q5 = Quantitative, go to Q8b. If Q5 = Both, go to Q8a and Q8b.

Q8b. Is the sampling method… 
Stratified sampling 
Go to Q9. 
Q9. Is the sampling method appropriate for addressing the research question? 
Yes – fully - The samples selected allows sub-group analyses by types of FBO in the constituent countries of England, Wales and Northern Ireland.
If Q5 = Qualitative, go to 9a. If Q5 = Quantitative, go to 9b. If Q5 = Both, go to Q9a and Q9b.
Q9a. Is the sampling method appropriate for addressing the research question?
Yes Fully
Q9b. Has a sample size calculation been conducted? 
Yes – fully -  The sample sizes in England, Wales and Northern Ireland have been calculated for statistical power and are adequate to undertake analysis of different types of FBO in England, Wales and Northern Ireland. Details are presented in the text and Annex 2 of the report
Go to Q10.
Q10. Are the research instruments valid and reliable?  
Yes – fully -  The research instruments have been developed using good cognitive fieldwork amongst a small subset of businesses. A total of 700 ‘mainstage 1’ interviews were conducted via a Computer-Assisted Telephone Interviewing (CATI) methodology. The data was weighted to be representative of the in-scope micro and small FBO population across England, Northern Ireland and Wales at the time of sampling. This represents good fieldwork planning and development.

If Q5 = Qualitative, go to Q11a. If Q5 = Quantitative, go to Q11b. If Q5 = Both, go to Q11a and Q11b.
Q11b. Is the analytical approach… 
Time series analysis 
Q12. Is there a good match between the analytical approach, the research method and the research question?
Yes – fully -  The analytical approach builds upon Waves 1 and 2 of the Tracker Survey and it has included additional questions to address policy and business issues since the 2019 Tracker (and the COVID pandemic).
Q13. Has a relevant checklist from the EQUATOR Network been used in the reporting of the results?  
Yes – partly -  This survey does not explicitly use the EQUATOR Network checklist for surveys, but it certainly follows most of the requirements of this checklist.
Q14. Have descriptive data on the characteristics of participants been presented?
Yes – fully - This survey provides descriptive details of the different participants amongst the service activities sector, larger businesses, sole traders and those with a higher FHRS rating certificate.
If Q5 = Qualitative, go to Q15. If Q5 = Quantitative, go to Q19. If Q5 = Both, go to Q15 and Q19.
Q19. Have descriptive data on exposures/interventions and potential confounders been presented?
Yes – partly -  This is a single wave report on a longitudinal survey and not an experimental or quasi-experimental study. It does investigate the consequences of the UK exit from the EU and, to some extent the COVID-19 pandemic, but it does not attempt statistical modelling of other potential confounders. This would require a different research design and analytical approach.
Go to Q20.
Q20. Have unadjusted and adjusted point estimates and confidence intervals been presented alongside statistical significance? 
No -The survey results are presented mainly as percentages of respondents’ different responses. There are no unadjusted and adjusted point estimates and confidence intervals, though the results are based on weighted data. The distribution of responses are often classified as “most likely”, “more likely than average” or “particularly likely”.
Go to Q21. 
Q21. Has generalisability been considered in the interpretation of the results?  
Yes – fully -  The survey has been designed to enable findings to be generalisable across different FBO businesses and the different countries (England, Wales and Northern Ireland). At the same time the authors are careful to note where and when caution should be taken when interpreting these results due to small base size.
Q22. Has causality been considered in the interpretation of the results? 
No 
Not applicable
Causality is hard to establish with survey data that does not have counterfactual samples. This survey recognises this and merely reports descriptive percentages of responses to a wide range of questions. The report does not go beyond analysis that is possible with survey data only.
Q23. Has uncertainty been considered in the interpretation of the results? 
Yes – partly  - This survey acknowledges where the results are sometimes based on small sub-samples and, hence, where the results are uncertain. It does not over-interpret the findings of the survey.
Q24. Has a clear study conclusion been presented?  
Yes – fully - The Conclusions section of this report is clear and cautious. It is balanced and indicates where the survey responses are mixed in terms of the consequences of (for instance) UK exit from the EU and the resultant impacts on business activity