Review of FSA Social Science
Review of FSA Social Science: Annex 2D Assessment of Research outputs
Review of FSA Social Science report Annex 2d, an assessment of research outputs using GSR code and the FSA QAT assessing research reports checklist.
Outputs: Testing the impact of overt and covert ordering interventions on sustainable consumption choices: a randomised controlled trial on an online supermarket.
Authors: Kantar Public's Behavioural Practice
Date: May 2022
Assessment of Research outputs using GSR code
Rigorous and impartial
| Rigorous and impartial | Rating | Comments |
|---|---|---|
| Based on sound methodology and established scientific principles | High | This trial is well designed with good attention to sample size, appropriate comparison groups (three-way), valid and reliable indicators and other research instruments, and is well reported with appropriate cautious interpretation. |
| Quality assured | Medium | This report was quality checked internally by two FSA social researchers and one external expert. The format of the QA process consists of reviewers’ comments using track changes within the margins of the final report. A separate QA report, preferably by two external reviewers, would be more rigorous and appropriate. |
| Based on best design, given constraints | High | RCT design is the best approach to establish the net effects of an intervention over a counterfactual. Hence this was the best design. Given the budget constraints (mentioned by the contractors) this trial was undertaken to a high standard. |
| Conclusions are clearly and adequately supported by data | High | The conclusions of the trial are presented clearly as ‘Key Findings’ in the Executive Summary (page 6), and they are adequately supported by data. It would have been a good idea to have the ‘Key Findings’ presented at the end of the full report. |
Relevant
| Relevant | Rating | Comments |
|---|---|---|
| Anticipates future policy issues as well as addressing current ones | Low | The report locates the topic of overt and covert ordering interventions on sustainable consumption choices within the context of the UK government’s National Food Strategy (Dimbleby, 2021) and the need “to understand how interventions in online shopping environments affect consumer choices in relation to the sustainability of products”. Apart from this very brief mention of the policy context there is no anticipation of future or current policy issues. This does not appear to have been part of the trial’s specification. |
| Answers clear and researchable questions | High | This report poses the central research question as “how interventions in online shopping environments affect consumer choices in relation to the sustainability of products”. More specifically it asks “whether a specific choice architecture intervention – displaying products in an ascending order of their carbon footprint – in an online supermarket environment can shift consumer choices towards more sustainable options compared to when products are randomly ordered”. |
| Contributes to all stages of the policy and delivery process | Low | By indirect implication the findings of this report might contribute to how food policy and delivery might be developed, but the contribution is implicit and opaque. It does not contribute to all stages of the policy and delivery process. |
| Delivers solutions that are viable, actionable and represent value for money | Low | The report is on the technical aspects and key findings of the trial. It does not address solutions that are viable, actionable and represent value for money. |
Accessible
| Accessible | Rating | Comments |
|---|---|---|
| Published | High | This report of the online supermarket trial was published in May 2022. It has a good and readable Executive Summary as well as a full report of its methods and findings. |
| Data made available where possible | High | Summary data tables are fully presented in the body of the report and in its Appendices. |
| Clear and concise | High | Given the complexity and detail of the intervention, and the trial methodology and procedures, the report can be considered and concise. Technical details are presented relatively clearly and the Executive Summary is very clear and concise. |
| Related to existing work in field | Medium | There is a brief mention of “the few existing studies, which were based on behaviour in bricks-and-mortar environment using hard-copy menus” and to studies of prompting people to make healthier food decision using pop-ups. Otherwise, there is little or no reference to related to existing work in the field. |
Legal and ethical
| Legal and ethical | Rating | Comments |
|---|---|---|
| Complies with relevant legislation | High | These reports comply with GDPR legislation. |
| Complies with GSR ethical guidelines | High | These three surveys do comply with GSR ethical guidelines. |
* External contractors are not always in a position to “anticipate future policy issues as well as addressing current ones”, “contribute to all stages of the policy and delivery process” and “deliver solutions that are viable, actionable and represent value for money”. Hence, a low or medium score reflects the limitation of using the GSR Self-Assessment tool for assessing the quality of research outputs.
FSA Quality Assurance Toolkit - Online Supermarket Trial: Checklist 2: Assessing research reports
Q1. Title, lead author and year
Testing the impact of overt and covert ordering interventions on sustainable consumption choices: a randomised controlled trial. Kantar Public’s Behavioural Practice. May 2022
Q2. Has a clear research need been outlined?
Yes – fully - This research has been undertaken to address the need for evidence on how consumers respond to different ways of presenting information about the sustainability of food products. This is in response to the fact that online supermarkets constitute an increasingly large share of grocery shopping, 12.6% of grocery sales were made online in March 2022 compared with just 8.0% three years ago.
Q3. Has a precise research question/aim been specified?
Yes – fully - The central research question was how do interventions in online shopping environments affect consumer choices in relation to the sustainability of products. Specifically, do overt, covert or no ordering of information about the sustainability of food products make a difference to consumer choice?t.
Q4. Is the research design…
Experimental.
Q5. Is the research method…
Quantitative
Q6. Is there a good match between the research question/aim, research design and research method?
Yes – fully - The three-arm between-subjects design with randomisation is an appropriate research design for the central research question of the study.
Q7. Is the study population and setting specified?
Yes – fully - The study population is specified as online grocery shoppers who are aged over 18 in England, Wales, and Northern Ireland. They are then selected by population characteristics of age, gender and ethnic group (White, Asian, Black, Mixed, Other).
If Q5 = Qualitative, go to Q8a. If Q5 = Quantitative, go to Q8b. If Q5 = Both, go to Q8a and Q8b.
Q8b. Is the sampling method…
Quota sampling.
Go to Q9.
Q9. Is the sampling method appropriate for addressing the research question?
Yes – fully - The authors note “as no official statistics were available on the specific demographic breakdown of online grocery shoppers in the targeted areas, we used quotas plus screening questions to get a sample close to a representative sample of the target group”.
If Q5 = Qualitative, go to 9a. If Q5 = Quantitative, go to 9b. If Q5 = Both, go to Q9a and Q9b.
Q9a. Is the sampling method appropriate for addressing the research question?
Yes Fully - The authors note that “the panel provider sent out new invites to potential participants in batches until the planned sample size was reached”. Large sample sizes were collected that were appropriate for three-way randomisation.
Go to Q10.
Q9b. Has a sample size calculation been conducted?
Yes fully - Sample size was calculated based on a power simulation, run using a logistic regression model. The authors used a power of 0.999 to detect a difference of 8%, and a power of 0.843 to detect a difference of 5%.
Q10. Are the research instruments valid and reliable?
Yes – fully - Six products in each product category were used because this study focuses on ordering of products, namely the position effects, and six products should give enough variation in terms of position of products. Given a fixed budget, there is a trade-off between the number of products in each category and the number of categories. participants were asked about their environmental concern, attitudes towards nudges, normative attitudes towards shopping sustainably, whether they eat meat, and demographics.
If Q5 = Qualitative, go to Q11a. If Q5 = Quantitative, go to Q11b. If Q5 = Both, go to Q11a and Q11b.
Q11b. Is the analytical approach…
Linear Regression.
Got to Q12.
Q12. Is there a good match between the analytical approach, the research method and the research question?
Yes – fully.
Q13. Has a relevant checklist from the EQUATOR Network been used in the reporting of the results?
Partly met - Although a relevant checklist from the EQUATOR Network was not used the reporting of the results was well structured and relevant.
Q14. Have descriptive data on the characteristics of participants been presented?
Yes – fully - Baseline demographic characteristics are presented fully in Appendix D.
Q19. Have descriptive data on exposures/interventions and potential cofounders been presented?
Yes - fully - Descriptive data on exposures/interventions and potential confounders have been presented in the ‘Procedure’ section and in Appendix B (Products Lists).
Go to Q20.
Q20. Have unadjusted and adjusted point estimates and confidence intervals been presented alongside statistical significance?
Yes - fully - Confidence intervals (CI) are reported around Odd Ratio (OR) throughout the data analysis.
Go to Q21.
Q21. Has generalisability been considered in the interpretation of the results?
Yes – fully - The authors note that “the results still come from an online experiment completed by panellists, which potentially threatens the ability to generalise from our results to the real-life situation we are studying (external validity) and which could be better dealt with using a field trial”. They also mention potential threats to generalisability at various points in the analysis and discussion sections of the report.
Q22. Has causality been considered in the interpretation of the results?
Yes fully - A causal pathway linking inputs to activities, outcomes and longer-term outcomes has been developed. This is commendable. Caution is advised by the authors about interpreting the ‘real world’ external validity and causality of the findings given the trial’s simulated nature.
Q23. Has uncertainty been considered in the interpretation of the results?
Yes – fully - The authors note: We used logistic mixed-effects models, which included separate error terms for participant and product category, allowing us to incorporate additional uncertainty in the estimates of intervention effects associated with variation between participants and categories.
Q24. Has a clear study conclusion been presented?
Yes – fully - The report concludes with a list of ‘Key Findings’ and an overall conclusion that “There was no effect of the covert ordering intervention on the probability of choosing more sustainable products versus less sustainable products, compared with the control arm (OR = 0.97, 95% CI 0.88-1.07, p-value = 0.533).