Review of FSA Social Science - Chapter 2 An Appraisal of FSA Social Research Outputs

Introduction

The social science team produces an extensive library of research outputs ^{(footnote 1)} that includes one-off studies for specific programmes or projects, as well as reports of regular surveys and evaluations of the FSA’s broad programme of activities. 
The research outputs reviewed below were published between January and September 2022. They include a range of social science approaches and methods: quantitative surveys, qualitative studies, a controlled evaluation, a literature review, and an ethnographic and iterative qualitative study:

• Food and You 2: Wave 4 Survey.
• FSA Small and Micro FBO Tracking Survey Wave 3. 
• The Value of the FHRS: Business Strand, Consumer Strand, Local Authority Strand.
• The FSA Online Supermarket Trial
• Kitchen Life 2 Literature Review.
• Psychologies of Food Choice - Public views and experiences around meat and dairy consumption

Approach

Each of the above research outputs was reviewed and assessed using the GSR Self-Assessment template of the Government Social Research Code - People and Products (2018) and the recently published FSA Quality Assurance Toolkit (ACSS, 2023). 

Research Output 1: Food and You 2 Wave 4 (FSA, 2022a)

Food and You 2 is a biannual ‘Official Statistic’ survey commissioned by the FSA. The survey measures self-reported consumers’ knowledge, attitudes and behaviours related to food safety and other food issues amongst adults in England, Wales, and Northern Ireland. 

Fieldwork for Food and You 2: Wave 4 was conducted between 18th October 2021 and 10th January 2022. A total of 5,796 adults from 4,026 households across England, Wales, and Northern Ireland completed the ‘push-to-web’ survey.

Table 2.1 indicates that the combined assessment scored ‘High’ on all of the dimensions of the GSR Self-Assessment Tool except: ‘contributes to all stages of the policy and delivery process’ (‘Medium’ score) and ‘delivers solutions that are viable, actionable and represent value for money’ (‘Low’ score) ^{(footnote 2)}. Annex 2A presents the peer reviewer’s comments for all of these ratings.

The Food and You 2, Wave 4 report fully met most of the quality criteria of the FSA’s Quality Assessment Toolkit (Table 2.2). There were two criteria that were partly met. Annex 2A presents the peer reviewer’s comments for all of these ratings.

Table 2.1 Food and You 2 Wave 4 Technical report and Key findings (combined assessment) Assessment using the GSR Self-Assessment tool:

Rigorous and impartial

Based on sound methodology and established scientific principles	Quality assured	Based on best design given constraints	Conclusions are clearly and adequately supported by data
High	High	High	High

Relevant

Anticipates future policy issues as well as addressing current ones	Answers clear and researchable questions	Contributes to all stages of the policy and delivery process	Delivers solutions that are viable actionable and represent value for money
High	High	Meduim	Low

Table 2.2 Food and You 2 Wave 4 Technical report and Key Findings Assessment using the QAT checklist:

Question	Answer
Q1 Title, lead author and year Food and You 2 Wave 4 Key Findings 2022	-
Q2 Has a clear research need been outlined?	Yes fully
Q3 Has a precise research question/aim been specified?	Yes fully
Q4 Is the research design experimental, cross-sectional, longitudinal case study?	Longitudinal
Q5 Is the research method qualitative, quantitative both?	Quantitative
Q6 Is there a good match between the research question/aim research design and research method?	Yes fully
Q7 Is the study population and setting specified?	Yes fully
Q8a Is the sampling method purposive, convenience, theoretical, other, not specified?	Not applicable
Q8b Is the sampling method, simple random, stratified sampling, quote sampling, convenience sampling, other, not specified?	Stratified
Q9 Is the sampling method appropriate for addressing the research question?	Yes fully
Q9a Is the target sample size justified?	Yes fully
Q9b Has a sample size calculation been conducted?	Yes fully
Q10 Are the research instruments valid and reliable?	Yes fully
Q11a Is the analytical approach, thematic, grounded theory, framework analysis, other not specified?	Not applicable
Q11b Is the analytical approach chi-square test, correlation test, t-test or analysis of variance, linear regression, logistic regression, survival analysis, time series analysis, meta-analysis, other, not specified?	Time Series Analysis
Q12 Is there a good match between the analytical approach, the research method and the research question?	Yes fully
Q13 Has a relevant checklist from the EQUATOR Network been used in the reporting of the results?	Yes fully
Q14 Have descriptive data on the characteristics of participants been presented?	Not applicable
Q15 Have two or more researchers been involved in the analysis process (for example, through double coding)	Not applicable
Q16 Is there consistency between the data presented and the themes?	Not applicable
Q17 Have similarities and differences between participants been explored (for example, negative cases_	Not applicable
Q18 Did participants provide feedback on the findings (for example, member checking)	Not applicable
Q19 Have descriptive data on exposures/interventions and potential cofounders been presented?	Partly met
Q20 Have unadjusted and adjusted point estimates and confidence intervals been presented alongside statistical significance?	Partly met
Q21 Has generalisability been considered in the interpretation of the results?	Yes fully
Q22 Has causality been considered in the interpretation of the results?	Not applicable
Q23 Has uncertainty been considered in the interpretation of the results?	Yes fully
Q24 Has a clear study conclusion been presented?	Yes fully

Food and You 2: Wave 4 Strength of the research output

• the conclusions of the ‘Key Findings’ Report have been clearly presented in the Executive Summary and they are well supported by the data presented in the main body of the report. The report has also been presented in a clear and well-structured way. 

• the presentation of the ‘Technical terms and definitions’ section (page 94) is helpful and explains the technical terms and issues very well.
• the Key Findings report indicates many areas where the public’s awareness of food content, the food supply chain and policy measures, such as the Food Hygiene Rating Scheme, is considerable. This may be seen as a success for the work of the FSA over the past two decades. 
• the technical report is an excellent record of how Wave 4 of the Food and You survey was developed conceptually and methodologically. Its attention to technical detail is excellent. 
• the methodology (sampling strategy, sample size, power, weighting and sub-group analysis, attention to deprivation analysis and local authority contextual sensitivity etc.) are all good.
• the contractor and the FSA clearly had a good and collaborative relationship in developing this survey. The FSA’s social researchers and the FSA Advisory Group were active participants in this collaborative relationship and in influencing the survey. This is commendable.
• the push-to-web methodology has been used well, as has the sequential mixed-method approach of this survey.

Issues for considerations

The range of questions and issues to be addressed by the survey was ambitious, albeit relevant for understanding food choice and food behaviour. The FSA and its external stakeholders might have been a little more discriminating about the number of themes and questions to be asked in one survey. 

Concluding assessment

Overall, this is a high quality technical report that enhances the strength and value of the Food and You 2 survey (Wave 4) and the evidence base for decision making by the FSA and other health promotion agencies.

Research Output 2: FSA Small and Micro FBO Tracking Survey Wave 3 (FSA, 2021)

The FSA Small and Micro Food Business Organisations (FBO) Tracking Survey, Wave 3, is survey of small and micro Food Business Operator (FBO) sites in England, Wales and Northern Ireland. This research output represents the third wave of an annual tracking survey with the following aims:

• to gain insight, and understand the implications of UK’s exit from the European Union (EU) on small and micro enterprises
• to ‘unpack’ attitudes towards regulation and deepen insights and knowledge of small and micro enterprises
• to measure trust in the FSA and extent to which the FSA is considered a modern, accountable regulator

Fieldwork was conducted in November and December 2021 and comprised 700 interviews with small (10-49 employees) and micro (fewer than 10 employees) FBOs.

Table 2.3 indicates that the FSA Small and Micro FBO Tracking Survey Wave 3 scored ‘High’ on ten of the dimensions of the GSR Self-Assessment Tool. The remaining four dimensions scored ‘Medium’ (N=3) or ‘Low’ (N=1). Annex 2B presents the peer reviewer’s comments for all of these ratings.

Table 2.3 FSA Small and Micro FBO Tracking Survey, Wave 3, 2021 - Assessment using the GSR Self Assessment Tool ^{(footnote 3)}

Rigorous and impartial

Based on sound methodology and established scientific principles	High
Quality assured	Medium
Based on best design given constraints	High
Conclusions are clearly and adequately supported by data	High

Relevant

Anticipates future policy issues as well as addressing current ones	Medium
Answers clear and researchable questions	High
Contributes to all stages of the policy and delivery process	Medium
Delivers solution that are viable, actionable and represent value for money	Low

Accessible

Published	High
Data made available where possible	High
Clear and concise	High
Related to existing work in field	High

Legal and Ethical

 

Complies with relevant legislation	High
Complies with GS ethical guidelines	High

The FSA Small and Micro FBO Tracking Survey Wave 3, 2021 fully met most of the quality criteria of the FSA’s Quality Assessment Toolkit (Table 2.4). There were three criteria that were partly met and one technical detail that was not met (see below). Annex 2B presents the peer reviewer’s comments for all of these ratings.

Table 2.4 FSA Small and Micro FBO Tracking Survey, Wave 3, 2021 Assessment using the FSA's Quality Assessment Toolkit (QAT)

Question	Answer
Q1 Title and year, FSA Small and Micro-FBO Tracking Survey Wave 3 2021	-
Q2 Has a clear research need been outlined?	Yes fully
Q3 Has a precise research question/aim been specified?	Yes fully
Q4 Is the research design experimental, cross-sectional, longitudinal, case study?	Longitudinal
Q5 Is the research method, qualitative, quantitative, both?	Quantitative
Q6 Is there a good match between the research question/aim, research design and research method?	Yes fully
Q7 Is the study population and setting specified?	Yes fully
Q8a Is the sampling method; purposive, convenience, theoretical, other, not specified?	-
Q8b Is the sampling method; simple random sampling, stratified sampling, quote sampling, convenience sampling, other, not specified?	Stratified
Q9 Is the sampling method appropriate for addressing the research question?	Yes fully
Q9a Is the target sample size justified?	Yes fully
Q9b Has a sample size calculation been conducted?	Yes fully
Q10 Are the research instruments valid and reliable?	Yes fully
Q11a Is the analytical approach thematic grounded theory, framework analysis, other, not specified?	Not applicable
Q11b Ia the analytical approach chi-square test, correlation test, t-test or analysis of variance, linear regression, survival analysis, time series analysis, meta-analysis, other, not specified?	Time series analysis
Q12 Is there a good match between the analytical approach, the research method and the research question?	Yes fully
Q13 Has a relevant checklist from the EQUATOR Network been used in the reporting of the results?	Partly met
Q14 Have descriptive data on the characteristics of participants been presented?	Yes fully
Q15 Have two or more researchers been involved in the analysis process (for example through double coding)?	Not applicable
Q16 Is there consistency between the data presented and the themes?	Not applicable
Q17 Have similarities and differences between participants been explored (for example negative cases)?	Not applicable
Q18 Did participants provide feedback on the findings (for example, member checking)?	Not applicable
Q19 Have descriptive data on exposures/interventions and potential cofounders been presented?	Partly met
Q20 Have unadjusted and adjusted point estimates and confidence intervals been presented alongside statistical significance?	No
Q21 Has generalisability been considered in the interpretation of the results?	Yes fully
Q22 Has causality been considered in the interpretation of the results?	Not applicable
Q23 Has uncertainty been considered in the interpretation of the results?	Partly met
Q24 Has a clear study conclusion been presented?	Yes fully

FSA Small and Micro FBO Tracking Survey, Wave 3, 2021, Strengths of the Research Output

• this survey was undertaken after relevant and careful cognitive testing and sample design/planning. Considerable effort went into sample preparation. 
• weighting of the sample was also undertaken carefully and appropriately. Details of how the sample and sub-samples were drawn, weighted and coded are available in the accompanying Technical Report. 
• the conclusions are laid out clearly at the end of the report, and in the Executive Summary. In both cases the conclusions are adequately supported by the data. 
• although this report overall is not concise (approximately 100 pages) it is clearly presented. Individual chapters are reported concisely and well.

Issues for consideration

This report was quality assured by FSA’s internal researchers and external consultants using comments in track changes only. A separate peer reviewers’ report, preferably by external reviewers, would be of greater scientific value and transparency.

It seems inappropriate for external contractors’ reports to be assessed against the criteria of ‘anticipates future policy issues as well as addressing current ones’ and ‘contributes to all stages of the policy and delivery process’, given that they have little or no opportunity to do this. The same applies to the criterion ‘delivers solutions that are viable, actionable and represent value for money’. These dimensions of the GSR Code should normally be provided by FSA’s social researchers and/or policy colleagues.

Concluding assessment

Overall, the FSA Small and Micro FBO Tracking Survey Wave 3 report can be considered high quality in terms of design, conduct and reporting, and it provides robust data on the key questions being addressed by the survey.

Research output 3: Value of the Food Hygiene Rating Scheme (FHRS): Businesses, Consumers and Local Authorities (FSA 2023a, 2023b, 2023c)

The FSA is responsible for food safety across England, Wales, and Northern Ireland. As part of its work on the Achieving Business Compliance (ABC) programme, the FSA wanted to understand in more detail how Local Authorities (LAs), businesses and consumers feel about the current Food Hygiene Rating Scheme (FHRS). In addition, the FSA wanted to capture consumer views on potential changes to the regulatory approach. 

This research report provides findings from four reconvened online discussion groups conducted with consumers in England, Wales, and Northern Ireland.

Table 2.5 indicates that the Value of the Food Hygiene Rating Scheme outputs scored ‘High’ in all but four of the dimensions of the GSR Self-Assessment Tool. Three of these four dimensions received a ‘medium’ score and one was rated a ‘low’. Annex 2C presents the peer reviewer’s comments for all of these ratings.

Table 2.5 Value of the Food Hygiene Rating Scheme (FHRS): Assessment using the GSR Self Assessment Tool ^{(footnote 4)}

Based on sound methodology and established scientific principles	High
Quality assured	Medium
Based on best design given constraints	High
Conclusions are clearly and adequately supported by data	High

Relevant

Anticipates future policy issues as well as addressing current ones	High
Answers clear and researchable questions	High
Contributes to all stages of the policy and delivery process	High
Delivers solution that are viable, actionable and represent value for money	Medium

Accessible

Published	High
Data made available where possible	Medium
Clear and concise	High
Related to existing work in field	Low

Legal and Ethical

Complies with relevant legislation	High
Complies with GS ethical guidelines	High

The Value of the Food Hygiene Rating Scheme (FHRS) fully met most of the quality criteria of the FSA’s Quality Assessment Toolkit (QAT) for qualitative studies (Table 2.6). Three of the QAT quality criteria were partly met (see below) and one was unmet (‘providing feedback on the findings for example, member checking’). Annex 2C presents the peer reviewer’s comments for all of these ratings.

Table 2.6 Value of the Food Hygiene Rating Scheme (FHRS): Assessment using the FSA's Quality Assessment Toolkit (QAT)

Question	Answer
Q1 Title and year. Value of the FHRS business, consumer, and local authorities LAs 2022	-
Q2 Has a clear research need been outlined?	Yes fully
Q3 Has a precise research question/aim been specified?	Yes fully
Q4 Is the research design experimental cross-sectional, longitudinal case study?	Cross-sectional and comparative
Q5 Is the research method, qualitative, quantitative, both?	Qualitative
Q6 Is there a good match between the research question/aim, research design and research method?	Yes fully
Q7 Is the study population and setting specified?	Yes fully
Q8a Is the sampling method; purposive, convenience, theoretical, other, not specified?	Purposive and thematic.
Q8b Is the sampling method: simple random sampling, stratified sampling, quota sampling, convenience sampling, other, not specified?	Not applicable
Q9 Is the sampling method appropriate for addressing the research question?	Yes fully
Q9a Is the target sample size justified?	Yes fully
Q9b Has a sample size calculation been conducted?	Not applicable
Q10 Are the research instruments valid and reliable?	Yes fully
Q11a Is the analytical approach, thematic grounded theory, framework analysis, other , not specified?	Thematic
Q11b Is the analytical approach: chi-square test, correlation test, t-test or analysis of variance, linear regression, survival analysis, time series analysis, meta analysis, other, not specified?	Not applicable
Q12 Is there a good match between the analytical approach, the research method and the research question?	Yes fully
Q13 Has a relevant checklist from the EQUATOR Network been used in the reporting of the results?	Partly met
Q14 Have descriptive data on the characteristics of participants been presented?	Partly met
Q15 Have two or more researchers been involved in the analysis process (for example through double coding)?	Information unavailable
Q16 Is there consistency between the data presented and the themes?	Yes fully
Q17 Have similarities and differences between participants been explored (for example negative cases)?	Yes fully
Q18 Did participants provide feedback on the findings (for example, member checking)?	No
Q19 Have descriptive data on exposures/interventions and potential cofounders been presented?	Not applicable
Q20 Have unadjusted and adjusted point estimates and confidence intervals been presented alongside statistical significance?	Not applicable
Q21 Has generalisability been considered in the interpretation of the results?	Not applicable
Q22 Has causality been considered in the interpretation of the results?	Not applicable
Q23 Has uncertainty been considered in the interpretation of the results?	Partly met
Q24 Has a clear study conclusion been presented?	Yes fully

Value of the Food Hygiene Rating Scheme (FHRS) - Strengths of the Research output

This research was undertaken using a combination of qualitative research methods including workshops, deliberative engagement, in-depth interviews and scenario setting. These are appropriate methods for addressing the central questions of the study and they were undertaken well. 

Each interview study addresses clear and researchable questions about the value of the FHRS to businesses, consumers, the local authorities, and possible areas of change for the regulatory approach of the FSA.

Sample sizes were appropriate for qualitative research with over-sampling where necessary. A summary of the sampling methods used for the consumers’ study is provided (in Appendix 1 of the report), but not for the businesses’ or local authorities’ surveys. No reason was given for this exclusion.

The FSA’s social research team was actively engaged in seeking more information and detail on methodology from the contractor during the commissioning process. The forthcoming additional information improved the design of these studies considerably. This is good practice.

The conclusions are presented clearly and succinctly and are adequately supported by the data. Subgroup analyses by type and size of businesses, demographic variation and different countries (England, Wales and Northern Ireland) are provided and are generally well presented.

The three interview studies provide a robust evidence base for current and future discussion of policy issues surrounding the FHRS. The reports identify areas where there is consensus about the effective and beneficial working of the FHRS, and where there is disagreement.

Issues for consideration

The amount of detail on the methodology used for these three studies was very limited in the publicly available reports. They were readily provided for this review on request. Given the high profile of the FHRS it would have been beneficial had these details been made available in the form of a separate Technical Report, as is the case with most other research outputs of the FSA social research team. 

Concluding assessment

Overall, these three studies represent high quality research that provide a valuable evidence base on how the FHRS is working, and how it might be developed in future.

Research Output 4: The FSA Online Supermarket Trial (FSA, 2023d)

This research output reports on a clustered randomised trial, with a matched pairs design, to test whether Food Business Organisation (FBO) staff would make customers feel more confident that they could identify ingredients that they want to avoid consuming, feel more comfortable to ask about ingredients on a future visit, and increase consumers’ perceptions of food safety regarding food and drink sold at the given chain. The clusters were branches of a national FBO. Participants were customers who entered the FBOs between 28th March 2022 and 30th June 2022, who placed a food order at the till, and who chose to complete a voluntary survey about their experience.

Table 2.7 indicates that the FSA Online Supermarket Trial research scored ‘High’ on nine of the fourteen quality criteria of the GSR Self-Assessment Tool. Two of the dimensions of the GSR Self-Assessment Tool were rated as ‘medium’ (see above) and three were rated as low. As noted in footnote 6, external contractors are not always in a position to meet these three quality criteria given that they may have little or no role in these policy making activities. Annex 2D presents the peer reviewer’s comments for all of these ratings.

Table 2.7 The FSA Online Supermarket Trial: Assessment using the GSR Self Assessment Tool ^{(footnote 5)}

Rigorous and Impartial

Based on sound methodology and established scientific principles	High
Quality assured	Medium
Based on best design given constraints	High
Conclusions are clearly and adequately supported by data	High

Relevant

Anticipates future policy issues as well as addressing current ones	Low
Answers clear and researchable questions	High
Contributes to all stages of the policy and delivery process	Low
Delivers solution that are viable, actionable and represent value for money	Low

The FSA Online Supermarket Trial fully met all but one of the quality criteria of the FSA’s Quality Assessment Toolkit for experimental trials (Table 2.8). The only criterion that was not ‘fully met’ was the use of the EQUATOR Network in the reporting of the results. Although a relevant checklist from the EQUATOR Network was not used, the reporting of the results was well structured in accordance with these guidelines. Annex 2D presents the peer reviewer’s comments for all of these ratings.

Table 2.8 The FSA Online Supermarket Trial: Assessment Using the FSA’s Quality Assessment Toolkit (QAT)
 

Question	Answer
Q1 Title and year. Testing the impact of overt and covert ordering interventions on sustainable consumption choices: a randomised controlled trial on an online supermarket 2022	-
Q2 Has a clear research need been outlined?	Yes fully
Q3 Has a precise research question/aim been specified?	Yes fully
Q4 Is the research design experimental, cross-sectional, longitudinal case study?	Experimental
Q5 Is the research method, qualitative, quantitative, both?	Quanitative
Q6 Is there a good match between the research question/aim, research design and research method?	Yes fully
Q7 Is the study population and setting specified?	Yes fully
Q8a Is the sampling method; purposive, convenience, theoretical, other, not specified?	Not applicable
Q8b Is the sampling method: simple random sampling, stratified sampling, quota sampling, convenience sampling, other, not specified?	Quota sampling
Q9 Is the sampling method appropriate for addressing the research question?	Yes fully
Q9a Is the target sample size justified?	Yes fully
Q9b Has a sample size calculation been conducted?	Yes fully
Q10 Are the research instruments valid and reliable?	Yes fully
Q11a Is the analytical approach, thematic grounded theory, framework analysis, other , not specified?	Not applicable
Q11b Is the analytical approach: chi-square test, correlation test, t-test or analysis of variance, linear regression, survival analysis, time series analysis, meta analysis, other, not specified?	Linear regression
Q12 Is there a good match between the analytical approach, the research method and the research question?	Yes fully
Q13 Has a relevant checklist from the EQUATOR Network been used in the reporting of the results?	Partly met
Q14 Have descriptive data on the characteristics of participants been presented?	Yes fully
Q15 Have two or more researchers been involved in the analysis process (for example through double coding)?	Yes fully
Q16 Is there consistency between the data presented and the themes?	Not applicable
Q17 Have similarities and differences between participants been explored (for example negative cases)?	Not applicable
Q18 Did participants provide feedback on the findings (for example, member checking)?	Not applicable
Q19 Have descriptive data on exposures/interventions and potential cofounders been presented?	Yes fully
Q20 Have unadjusted and adjusted point estimates and confidence intervals been presented alongside statistical significance?	Yes fully
Q21 Has generalisability been considered in the interpretation of the results?	Yes fully
Q22 Has causality been considered in the interpretation of the results?	Yes fully
Q23 Has uncertainty been considered in the interpretation of the results?	Yes fully
Q24 Has a clear study conclusion been presented?	Yes fully

The FSA Online Supermarket Trial - Strengths of the Research Output

• this is a well-planned, well-implemented and well-reported randomised controlled trial of the impact of overt and covert ordering interventions on sustainable consumption choices.
• the study concludes that “analysis of the control condition showed that the positioning of products had no effect on (consumer) choices.”
• the three-arm, between-subjects design with randomisation is an appropriate research design for the central research question of the study. 
• careful attention has been given to sample size and power, appropriate quota sampling and linear regression analysis that is both appropriate and well presented. There is also good attention to participants’ socio-demographic characteristics.

Issues for consideration

This trial addresses consumer choice under simulated online supermarket conditions and, as the authors note, this simplifies many of the real-life considerations. The authors also note that “more field research in real supermarket environments is required to establish the external validity of the effects of pop-ups on behaviour” (page 40).

Concluding Assessment

Notwithstanding the simulated nature of this trial this research output presents a solid evidence base on how consumers behave in response to different presentations of sustainable consumption choices. This is a high-quality report of a well-planned, well-implemented and well-reported randomised controlled trial. 

Research Output 5 - Kitchen Life 2 Literature Review (FSA, 2022b)

The aim of the Kitchen Life 2 project is to identify the key behaviours relating to food safety that occur in domestic and business kitchens, as well as the factors that may reduce the likelihood to enact recommended food safety & hygiene behaviours. This literature review seeks to identify existing key behaviours, actors, triggers and barriers in domestic and business kitchens and develop successful behavioural interventions and risk assessment models.

Table 2.9 indicates that the Kitchen Life 2 Literature review scored ‘High’ on seven dimensions of the GSR Self-Assessment tool and ‘Medium’ on five dimensions. Two dimensions (‘based on best design, given constraints’ and ‘data made available where possible’) were rated as low. Annex 2E presents the peer reviewer’s comments for all of these ratings. 

Table 2.9 Kitchen Life 2 Literature Review. Assessment using the GSR Self-Assessment Tool ^{(footnote 6)}

Rigorous and Impartial

Based on sound methodology and established scientific principles	Medium
Quality assured	Medium
Based on best design given constraints	Low
Conclusions are clearly and adequately supported by data	Medium

Relevant

Anticipates future policy issues as well as addressing current ones	High
Answers clear and researchable questions	High
Contributes to all stages of the policy and delivery process	Medium
Delivers solution that are viable, actionable and represent value for money	Medium

Accessible

Published	High
Data made available where possible	Low
Clear and concise	High
Related to existing work in field	High

Legal and Ethical

Complies with relevant legislation	High
Complies with GS ethical guidelines	High

Assessment using the FSA's Quality Assessment Tool (QAT)

The FSA’s Quality Assessment Toolkit does not include a checklist for assessing the quality of literature reviews. It does provide links to the ‘Preferred Reporting Items for Systematic Reviews and Meta-Analyses’ (PRISMA) guidelines and checklist via the Equator Network. The ‘issues for consideration’ below are based on the PRISMA guidelines.

Kitchen Life 2 Literature Review - Strengths of the Research Output

• the review has a clear and researchable question and is an informative literature review. It is presented clearly and reasonably concisely and is also easy to read. 
• the ‘Recommendations’ section summarises the key findings and ‘take home’ messages clearly and well. These are supported by the data that were collected, but given the somewhat narrow search procedures (see below) there may be some risk of bias.
• this review focuses on key behaviours relating to food safety in domestic and business kitchens within the context of the FSA’s policy response to the COVID pandemic. Its ‘key recommendations’ about what people need to know, and should do, have considerable implications for current and future policy issues.
• this review identifies a number of activities in domestic and business kitchens that may require action to improve food hygiene. 
• this review was quality assured internally by the FSA social research team and by members of the Advisory Board on Social Science. 

Issues for consideration

• Literature reviews are no longer considered a “sound methodology [based on] established scientific principles” (GSR 2018). Given that this review had over eight months to be completed the best design would have been a narrative systematic review following PRISMA procedures and guidelines. Alternatively, it could have used Rapid Evidence Assessment methodology which would have enhanced its search and analytical approach.
• the review uses a limited search string and only two electronic databases (Scopus and Web of Science). It also does not appear to have used double screening or double data extraction. 
• this review does not include a ‘flow diagram’ of how the initial search yields were reduced to the finally included studies. There is no list of excluded studies with the reasons for their exclusion. There is no attempt to rate the included studies in terms of their strength of evidence.
• these limitations do introduce a risk of bias to the review.

Concluding Assessment

Despite having used a form of evidence review no longer considered “sound methodology and [based on] established scientific principles” this review provides some valuable insights into “the key behaviours relating to food safety that occur in domestic and business kitchens, as well as the factors that may reduce the likelihood to enact recommended food safety & hygiene behaviours”. Its contribution to the evidence base on Kitchen Life would have been enhanced had it followed established and up-to-date scientific principles for reviewing evidence.

Research Output 6: Psychologies of Food Choice - Public views and experiences around meat and dairy consumption (FSA, 2022c)

This research output explores UK public views and experiences around meat and dairy consumption, including key drivers of participants’ chosen dietary approach. It presents findings drawn from qualitative remote ethnography research with 24 UK people conducted during July and August 2021, plus 9 peer-to-peer interviews conducted by our main sample participants with their friends and family.

Table 2.10 indicates that this study scored ‘High’ on nine of the criteria of the GSR Self-Assessment Tool (see above). Five of the criteria were rated as ‘Medium’. Annex 2F presents the peer reviewer’s comments for all of these ratings. 

Table 2.10 Psychologies of Food Choice, Public views and experiences around meat and dairy consumption Assessment Using the GSR Self-Assessment Tool 

Rigorous and Impartial

Based on sound methodology and established scientific principles	High
Quality assured	Medium
Based on best design given constraints	High
Conclusions are clearly and adequately supported by data	High

Relevant

Anticipates future policy issues as well as addressing current ones	Medium
Answers clear and researchable questions	High
Contributes to all stages of the policy and delivery process	Medium
Delivers solution that are viable, actionable and represent value for money	Medium

Accessible

Published	High
Data made available where possible	Medium
Clear and concise	High
Related to existing work in field	High

Legal and Ethical

Complies with relevant legislation	High
Complies with GS ethical guidelines	High

The Psychologies of Food Choice report fully met most of the quality criteria of the FSA’s Quality Assessment Toolkit for qualitative research (Table 2.11). The only criterion that was not ‘fully met’ was the use of the EQUATOR Network in the reporting qualitative research. The reporting that is provided in the report, however, does meet many of these standards. Annex 2F presents the peer reviewer’s comments for all of these ratings. 

Table 2.11 Psychologies of Food Choice - Assessment using the FSA's Quality Assessment Toolkit (QAT)

Question	Answer
Q1 Title and year, Psychologies of Food Choice Public views and experiences around meat and dairy consumption 2021	-
Q2 Has a clear research need been outlined?	Yes Fully
Q3 Has a precise research/question aim been specified?	Yes fully
Q4 Is the research design experimental, cross-sectional, longitudinal, case study?	Ethnographic and behavioural
Q5 Is the research method, qualitative, quantatitve both?	Qualitative
Q6 Is there a good match between the research question/aim, research design and research method?	Yes Fully
Q7 Is the study population and setting specified?	Yes Fully
Q8a Is the sampling method purposive, convenience, theoretical, other, not specified	Purposive sampling
Q8b Is the sampling method simple random sampling, stratified sampling, quote sampling convenience sampling other not specified?	Quota sampling
Q9 Is the sampling method appropriate for addressing the research question?	Yes fully
Q9a Is the target sample size justified?	Yes fully
Q9b Has a sample size calculation been conducted?	Not applicable
Q10 Are the research instruments valid and reliable?	Yes fully
Q11a Is the analytical approach, thematic grounded theory, framework analysis, other, not specified?	Thematic
Q11b Is the analytical approach chi-square test, correlation test, t-test or analysis of variance, linear regression, logistic regression, survival analysis, time series analysis, meta-analysis, other not specified?	Not applicable
Q12 Is there a good match between the analytical approach, the research method and the research question?	Yes fully
Q13 Has a relevant checklist from the EQUATOR network been used in the report of the results?	Partly met
Q14 Have descriptive data on the characteristics of participants been presented?	Yes fully
Q15 Have two or more researchers been involved in the analysis process (for example, through double coding)?	Yes fully
Q16 Is there consistency between the data presented and the themes?	Yes fully
Q17 Have similarities and differences between participants been explored (for example negative cases)?	Yes fully
Q18 Did participants provide feedback on the findings (for example, member checking)?	Yes fully
Q19 Have descriptive data on exposures/interventions and potential cofounders been presented?	Not applicable
Q20 Have unadjusted point estimates and confidence intervals been presented alongside statistical significance?	Not applicable
Q21 Has generalisability been considered in the interpretation of the results?	Yes fully
Q22 Has causality been considered in the interpretation of the results?	Not applicable
Q23 Has uncertainty been considered in the interpretation of the results?	Fully met
Q24 Has a clear study conclusion been presented?	Yes fully

Psychologies of Food Choice - Strengths of the research output

• this study shows the value of ethnographic and mixed methods qualitative research in identifying the drivers of behaviour and the triggers to eating differently.
• the use of the ‘capability-opportunity-motivation behaviour (COM-B)’ model is welcome because it addresses the need for behavioural insights in changing people’s food consumption.
• the study concludes that a one-size-fits-all policy approach is inappropriate for understanding how to influence food consumption behaviour. 
• the study was undertaken to a reasonably high degree of quality. It has identified some of the sources of the uncertainty in the relationship between drivers and behavioural outcomes.

Issue for Consideration

Given the length of the published report the authors were unable to present detailed transcript data or field notes, which are important features of ethnographic and mixed qualitative methods. These could have been provided in a separate Technical Report. The report does, however, provide good summary data on fieldwork observations and participants’ responses in interviews and groups.

This study did not intend to contribute to all stages of the policy and delivery process. It does, however, indicate that the policy and delivery processes need to be more nuanced and contextualised.

Concluding assessment

This report provides important information on some of the things that need to be in place to change people’s food consumption behaviour. It has been undertaken to a reasonably high standard. There may be more data and insights available from this study that could not be included in this report given length limitations.

Chapter summary

Six research outputs of the FSA social science team have been reviewed using the GSR Self-Assessment tool and the FSA’s Quality Assessment Toolkit (QAT). Each report is of a high standard, though some issues for consideration have been identified. 

The criteria used by the GSR Self-Assessment tool assess research outputs at a high level of generality and do not identify details about methodology or the quality of analysis at a sufficiently granular level. The GSR Self-Assessment tool also requires an assessment of whether research outputs “anticipate future policy issues as well as addressing current ones”, “contribute to all stages of the policy and delivery process” and “deliver solutions that are viable, actionable and represent value for money”. Whilst these are important features of research and analysis for government not all research has the opportunity to do this, especially where research is contracted externally. 

The FSA’s Quality Assurance Toolkit provides a more appropriate and sensitive tool to assess the quality of social research. This reinforces the recommendation (Chapter 3) that, going forward, the FSA’s Quality Assurance Toolkit should be used as the main means of assessing the quality of social science research at the FSA, and in other government departments and agencies.