Review of FSA Social Science - Executive Summary

Executive Summary

Objectives of the Review

1. To assess the contribution that the FSA social science team makes to the FSA and its mission, and to identify what it does well, areas for improvement, and make recommendations for CPD. 
2. To assess the seven principles of the GSR Code for People and Products and the use of the GSR Self-Assessment tool to appraise social science outputs.

Approach 

A thorough review was undertaken using the following procedures:

1. Stakeholders’ perspectives of the FSA’s social science team.
2. Appraisal of the FSA’s social science research outputs.
3. Appraisal of the GSR code self-assessment and an external peer review.
4. An online survey of the technical skills of FSA’s social researchers.
5. A group interview with FSA’s social researchers.

FSA’s Stakeholders’ Perspectives (Chapter 1)

Stakeholders’ views on the work of the FSA’s social science team were explored by undertaking ten online in-depth interviews, five with internal stakeholders and five with external stakeholders. The topics covered in these interviews are presented in Annex 1 and mirror those used in the group interviews with the social science team.

The social science team is highly valued by most internal and external stakeholders. The team’s qualitative analysis and their evaluation of the equality issues surrounding food policy and food insecurity were highly valued, as were its exploration of consumers’ attitudes and perceptions of food products. The ‘lived experience’ research of the team was also highly valued. The team was applauded for having “really stepped up in response to the cost of living crisis”, giving policy teams evidence that helped them understand the lived experience of food insecurity.

Some concerns were expressed about the technical skills of some members of the team, especially in quantitative methods and evidence synthesis approaches, though this was a minority view. Other issues that could be improved upon are the consistency in the effectiveness at commissioning and managing social research, the management of external contractors and identifying the “big messages” of research findings. 

An Appraisal of FSA Social Research Outputs (Chapter 2)

Six recent research outputs were reviewed and appraised by the author of this review. These were:

• Food and You 2: Wave 4 Survey.
• FSA Small and Micro FBO Tracking Survey Wave 3. 
• The Value of the FHRS: Business Strand, Consumer Strand, Local Authority Strand.
• The FSA Online Supermarket Trial.
• Kitchen Life 2 Literature Review.
• Psychologies of Food Choice-Public views and experiences around meat and dairy consumption.

Each research output was appraised using the GSR Code for Products and the FSA Quality Assurance Toolkit (QAT).

Each report was found to be of a high standard in terms of methodology used, the design, execution and reporting of the study, and the role of the FSA social science team in terms of project management.

Some issues for consideration about the documentation of research outputs were identified. Not all published research outputs have a separate technical report providing in-depth details of how research projects are designed, samples and research instruments are selected, or how analysis will be undertaken. Such details were readily available from the social science team on request. 

External peer reviewing and quality assurance of research outputs is “undertaken on a case-by-case basis, subject to the pace, complexity and purpose of the work” ^{(footnote 1)}. Whilst a degree of proportionality is necessary when working to tight timelines, external peer review and quality assurance is one of the hallmarks of good science and should be common practice.

Self-Assessment using the GSR Code (Chapter 3)

The FSA social science team undertook a self-assessment of how well it meets the seven key principles of the GSR Code for Self-Assessment. These were peer reviewed by the author of this review.

A traffic light rating scale is provided for assessing performance against the GSR Code’s criteria, with ‘green’ representing ‘strong’ performance, ‘amber/green’ (‘development area/strong’), ‘amber’ indicating ‘development area’ and ‘red’ identifying ‘serious concerns’.  

There was agreement between the peer reviewer and the self-assessment that the ‘outward facing’ professional standard should be rated as ‘green’. 

The peer reviewer concurred with team’s self-assessed ‘amber/green’ ratings against four of the standards of the GSR Code (rigour and impartiality, relevance, legal and ethical, performing role with integrity). 

Two of the professional standards (accessibility, appropriately skilled) were rated as ‘green’ by the team and ‘amber/green’ by the reviewer. 

Assessing the GSR Assessment Code 

The GSR Assessment Code aims to ensure that its professional standards are met in all GSR’s products and people. It does this at a rather high level of generality and the indicators do not really capture the quality of research outputs at a sufficiently granular level.

The social science team found the GSR Assessment Code unwieldy and the scoring categories insufficiently nuanced. The tool was also seen as conflating standards, such as ‘rigour’ and ‘impartiality’, when assessing the methodological quality of research. 

The peer reviewer also found the Self-Assessment Code difficult to use. The assessment criteria and their associated indicators are not always well-aligned, nor sufficiently detailed to assess the work and outputs of government social research, especially in terms of assessing the rigour of social science methodology. 

The Self-Assessment Tool also requires all research products to be assessed in terms of their implications and solutions for policy and delivery. This is often beyond the scope of most external contractors and therefore scores ‘low’ or ‘medium’. 

FSA Quality Assurance Toolkit (QAT)

This recently published toolkit, developed by the FSA’s Advisory Committee on Social Science (ACSS) and University College London (UCL), provides a comprehensive means of ensuring that social science outputs are procured, managed, assessed and validated in a structured and rigorous way. The toolkit covers quantitative and qualitative social research methods and provides checklists for each approach. It also provides links to the EQUATOR Network ^{(footnote 2)} for additional resources.

The FSA Quality Assurance Toolkit (QAT) was used in addition to the GSR Self-Assessment Code to appraise the research outputs of the FSA social science team (Chapter 2). On the basis of this it is strongly recommended that going forward the Quality Assurance Toolkit be used as the main means of assessing the quality of social science research at the FSA and across government.

The Technical Skills of the Social Science Team (Chapter 4)

These were assessed by self-reports of ability to meet the requirements of the GSR Technical Skills Framework (2022). The framework outlines the technical expertise expected of members of the GSR profession, and how they apply for each grade of researcher.

Overall, the team has considerable self-confidence in its ability to meet most of the requirements of the Framework. Most of the technical skills required by the Framework were self-rated as ‘good ability’ or ‘very good ability’ by most of the members of the team. It was not possible for this review to independently verify these self-assessments of technical knowledge and skills.

There were a few methodological skills (ROs and SROs) and some leadership and coaching skills (PROs) for which professional development would be appropriate. 

The Social Researchers’ Perspectives (Chapter 5)

The group interview with the entire social science team identified the following:

Commissioning Social Research

The team thinks it does well in terms of the following aspects of commissioning social research:

• Identifying the policy issues requiring social research and clarifying the objectives and anticipated outcomes of a policy or programme. 
• Maintaining a productive relationship with the FSA’s procurement team.
• Keeping abreast of innovative research methodologies from academic and commercial research communities via seminars, workshops and training courses, and discussions with suppliers about new and upcoming research methods.
• Liaising with the FSA’s Areas of Research Interest (ARI) steering groups to link with project technical leads and policy colleagues. 

The team thinks there are the following challenges in terms of commissioning social research:

• Getting early involvement from policy colleagues in developing research objectives and questions.
• Managing expectations of policy colleagues regarding how long it takes to procure, deliver and quality assure research to high standards.
• Resolving different stakeholders’ opposing demands or conflicting needs for social research.

Managing Social Research

The team thinks it does well in terms of the following aspects of managing social research:

• establishing how research is going to be used and adapting the research specification and procurement approach accordingly.
• keeping contracts on track, developing good relationships with contractors and working collaboratively with a range of stakeholders.
• managing the wide range of stakeholders with which the FSA engages, including policy makers across government, contractors, academics, food suppliers, consumers, third sector organisations and local authorities across England, Wales and Northern Ireland.
• quality control of research outputs.

The team thinks it could improve upon the following aspects of managing social research:

• having more time to understand policy issues in greater detail, design the research, and deliver higher quality outputs. Being involved early would allow the team to respond more effectively. 
• improving research dissemination and establishing the impact of research outputs. Impact can only be partially assessed by how often research outputs are read, referred to and used for decision making. Impact also requires monitoring and evaluating the uptake of the FSA social research team’s outputs and identifying their effects on dietary and hygiene-promoting behaviour.

GSR Code Standards: People 

The social science team thinks it does well in terms of performing with integrity and being outward facing. The team gave examples of how it meets these two professional standards of the GSR Self-Assessment Code.

The team also provided evidence that its recruitment is aligned with the GSR recruitment protocol (GSR, 2023) and that most of its members are ‘badged’. It has also recruited staff from outside the civil service which brings considerable experience and different perspectives to the team. 

The FSA is seen by the social science team as being very supportive of staff undertaking professional development and training. CPD is actively encouraged and undertaken across the team.