Review of FSA Social Science
Review of FSA Social Science - Chapter 3: Self-Assessment using the GSR code
The GSR Code “sets out seven key principles that all GSR members must adhere to in order to ensure research and analysis that is scientifically rigorous, relevant and valued”.
The GSR Code “sets out seven key principles that all GSR members must adhere to in order to ensure research and analysis that is scientifically rigorous, relevant and valued” (GSR, 2018). It provides four standards for assessing the products of GSR research and three standards for assessing people:
GSR Code Products
- rigour and impartiality
- relevance
- accessibility
- legal and ethical
CSR Code People
- performing the role with integrity
- appropriately skilled and continuously developed
- outward facing
Each standard has GSR assessment criteria, and indicators are provided for assessing an organisation’s performance against these criteria (Table 3.1). A traffic light rating scale is provided for assessing performance against the GSR Code’s criteria, with ‘green’ representing ‘strong’ performance, ‘amber/green’ (‘development area/strong’), ‘amber’ indicating ‘development area’ and ‘red’ identifying ‘serious concerns’. Additionally, assessors are required to provide details about any issues or risks, and examples of good practice in their organisation.
The full GSR Code for Assessment template, including a self-assessment by the FSA social science team, is available at Annex 3A.
Table 3.1 GSR Code for Self-Assessment
| GSR Code for Products: Standards | GSR Assessment Criteria | Indicators Used |
|---|---|---|
| Rigor and impartiality | a. Based on sound methodology and established scientific principles b. Quality assured c. Based on best design, given constraints d. Conclusions are clearly and adequately supported by data |
|
| Legal and ethical | a. Complies with relevant legislation b. Complies with GSR ethical guidelines |
|
| Relevance | a. Anticipates future policy issues as well as addressing current ones b. Answers clear and researchable questions c. Contributes to all stages of the policy and delivery process d. Delivers solutions that are viable, actionable and represent value for money |
|
| Accessibility | a. Published b. Data made available where possible c. Clear and concise d. Related to existing work in field |
|
| Performing role with integrity | a. Make best use of available resources b. Give appropriate methodological and impartial evidence-based advice, challenging where appropriate. |
|
| Outward facing | a. Establish effective links with the external research community b. Actively collaborate with policy/delivery colleagues c. Actively collaborate with other analytical professions within and across departments |
|
| Appropriately skilled and continuously developed | a. Recruited and promoted in line with GSR recruitment protocol b. Committed to continuous professional development in line with the CPD requirements |
|
Peer review of the self-assessment of FSA’s social science
This peer review was undertaken by the author of this report. A summary of the peer reviewer’s ratings on the GSR Code for Products and People, and those of the social science team, is presented at Annex 3B. There was agreement between the peer reviewer and the self-assessment that the ‘outward facing’ professional standard should be rated as ‘green’.
The peer reviewer concurred with team’s self-assessed ‘amber/green’ ratings against four of the standards of the GSR Code (rigour and impartiality, relevance, legal and ethical, performing role with integrity). Two of the professional standards (accessibility, appropriately skilled) were rated as ‘green’ by the team and ‘amber/green’ by the reviewer.
No standards were rated by the peer reviewer or the social science team as amber (‘development area’) or red (‘serious concerns’).
Rigour and impartiality
The peer reviewer agrees with the amber/green self-rating of the team on this standard. Project specifications and proposals are well reviewed internally. The evidence provided by the team to support this is valid.
The team notes that “external reviewing and QA is undertaken on a case-by-case basis, subject to the pace, complexity and purpose of the work”. Whilst the latter caveat is appreciated, and a degree of proportionality is necessary when working to tight timelines, external peer review and quality assurance are two of the hallmarks of good science. This is undertaken to some extent by the ACSS. It would enhance the quality and optics of the peer review and assurance process if all published research outputs were also peer reviewed by a wholly independent source.
The self-assessment notes that “engaging colleagues in exploratory/experimental work” and “accessing good research suppliers at pace and within government procurement processes” is a challenge. This may affect the rigour with which these methods are undertaken. That said, the contractors that the team uses to undertake such work have excellent reputations for planning and conducting experimental and quasi-experimental methods, and for delivering in a timely manner.
The social science team self-rated this standard amber/green, which is appropriate for this standard.
Relevance
There is a good balance between the team’s short term (operational) and longer term (strategic) research and analysis. For example, the waves of the FSA’s tracker surveys are undertaken in a timely manner and help to inform policy and delivery issues with good quality real-time data. The horizon scanning work by the social science team in 2020 and 2021 (FSA, 2020; Pierce and Mumford, 2020) is rightly claimed as evidence of its contribution to strategic analysis.
The GSR Self-Assessment tool’s indicators for ‘relevance’ include “anticipates future policy issues as well as addressing current ones”, “contributions to all stages of the policy and delivery process”, and the delivery of “solutions that are viable, actionable and represent value for money”. Whilst these are important features of research and analysis for government, not all social research has the opportunity to do this.
It is hard to see how external contractors can meaningfully make these contributions unless they are deeply embedded in the decision-making process of government. This makes it all the more important for the social science team to provide the link between research findings and their potential for policy making and delivery.
The self-assessment notes that “turning dissemination into measurable impact” is challenging. This might require linking the findings of FSA’s research outputs on consumers’ reported behaviour with data on food consumption, nutrient intake and the nutritional status of the general population.
The team also claims, quite rightly, to have good liaison with policy and delivery stakeholders. This should provide opportunities for knowledge translation and knowledge transfer from research outputs to potential policy and practice.
The social science team self-rated this standard amber/green, which is appropriate for this standard.
Accessibility
The publication of social science research outputs is timely, readable and informative. The different types of presentational format are well documented in the self-assessment and are generally good. There is a focus on reporting and analysing the findings of research rather than detailed accounts of the methodology. This is good for the general audience for whom it is intended. Summary data and sub-group presentations are presented well in published outputs.
The above approach, however, does require some further details to be available about the methodological and technical aspects of published research. This is important for reviewing and quality assuring the approach taken, and for transparency.
Technical Reports are frequently, but not always, published by the team. Making available technical details of how research has been conducted should be common practice if the scientific quality of research outputs is to be assured.
More detailed information on methodology, data and analysis was readily made available from the social science team for this review on request, which is good practice and commendable.
The social science team self-rated this standard as green (‘strong’ performance). A rating of amber/green would be more appropriate for this standard.
Legal and ethical
The team follows guidance on commissioning, managing and conducting research well, including attention to GSR ethical guidelines. Data security is handled well and in accordance with civil service requirements. Data sharing is generally good and readily available, notwithstanding the above observation about the consistent availability of technical reports.
Procurement procedures have been identified as a challenge for some contractors and the team. Through good working relationships with procurement colleagues the social science team manages this well. There is room for improvement in the research procurement arrangements, but these are cross-government requirements beyond the control of the social science team. This suggests that the procurement arrangements for government research should be reviewed with the aim of having separate procedures and requirements from those of general government procurement.
The social science team self-rated this standard amber/green, which is appropriate for this standard.
Performing role with integrity
The team certainly meets the GSR criteria for a green rating for being open, fair and honest and considering the added value of a project before undertaking new research. The claim in the self-assessment that the social science team “works with policy stakeholders to understand the existing evidence base” was substantiated by the stakeholders’ interviews.
Identifying existing and emerging research is currently undertaken by the team using literature reviews and rapid evidence assessments. There is some room for improvement in terms of using up-to-date methods of evidence synthesis such as systematic reviews and evidence gaps maps. This involves identifying and using existing systematic reviews and other evidence synthesis products (footnote) as well as commissioning or undertaking some (rapid) reviews in-house. This may require some professional development training.
Sharing of information gained by gathering existing evidence is generally good. Knowledge management also requires good storage, file management and retrieval of evidence. The team has improved this in recent months.
The social science team self-rated this standard amber/green, which is appropriate for this standard.
Appropriately skilled and continuously developed
All indications are that the team is recruited and promoted in line with GSR protocols. Recruitment of researchers from the external research and evaluation community has enhanced the team’s experience and expertise.
There is some imbalance of grades in the current structure of the SR team (six PROs, six SROs, 3 ROs) which gives the appearance of the team being top-heavy. That said, there is evidence of most routine research tasks being undertaken by PROs and SROs as well as ROs. Future recruitment might focus on adding more ROs to the team.
Some technical skills might require development (see Chapter 4) including quantitative skills and methods of evidence synthesis. Staff receive CPD opportunities in line with GSR guidelines. Procurement requirements sometimes limit the choice of CPD opportunities, but otherwise talent management is good.
The social science team self-rated this standard as green (‘strong’ performance). A rating of amber/green would be more appropriate for this standard.
Outward Facing
The self-assessment of the team’s links with the external research community, other government analysts and policy delivery colleagues is fair and accurate. This was confirmed in the interviews with internal and external stakeholders and is well documented in Chapter 2.
The social science team self-rated this standard as green, which is appropriate for this standard.
Summary of the peer review
This peer review generally concurs with the self-assessment by the social science team using the GSR Self-Assessment Code. Some differences in rating the team’s performance against the Self-Assessment Code’s standards have been made, and some areas of improvement have been suggested.
Assessing the GSR Assessment Code
The GSR Assessment Code aims to ensure that the professional standards are met in all GSR’s products and people. It does this at a rather high level of generality and the indicators do not really capture the quality of research outputs at a sufficiently granular level.
Prior to this review, most of the social science team was not aware of the GSR Self-Assessment Code. It does not appear to have been routinely used in other government departments either, according to interviews with the social science team and both internal and external stakeholders.
Most of the team used the tool for the first time to undertake the FSA self-assessment for this review. They found it unwieldy and ‘clunky’, and the Red, Amber, Green scoring categories insufficiently nuanced. The tool was also seen as conflating standards, such as ‘rigour’ and ‘impartiality’, when assessing the methodological quality of research.
The peer reviewer for this report also found the Self-Assessment Code difficult to use. The GSR assessment criteria and their associated indicators are not always well-aligned, nor sufficiently detailed to assess the work and outputs of government social research, especially in terms of assessing the rigour of social science methodology.
Despite its aim to assess whether research products are “based on sound methodology and established scientific principles” the Assessment Code does not indicate how this is to be determined in terms of technical detail. This has been remedied by the new FSA Quality Assurance Toolkit [QAT] (FSA, 2023).
FSA Quality Assurance Toolkit (QAT)
This toolkit provides a comprehensive means of ensuring that social science outputs are procured, managed, assessed and validated in a structured and rigorous way. It has been co-created by the FSA’s Advisory Committee on Social Science (ACSS) and University College London. This involved focus groups with FSA staff and “multiple rounds of feedback” as well as the piloting of several study protocols, research reports, and tender specifications (ACSS, 2023).
The ‘Guidance’ section of the QAT has 3 parts:
- Part 1 contains guidance for producing, assessing, and procuring research.
- Part 2 contains guidance for research management and dissemination (primarily relevant for producing and procuring research).
- Part 3 contains additional guidance for procuring research.
The toolkit covers quantitative and qualitative social research methods and provides checklists for each approach (see Chapter 2). It also provides links to the EQUATOR Network (footnote) for additional resources for procuring, assessing and reporting research, and for assessing other study types.
Guidance and checklists for undertaking systematic reviews of evidence are not included in the QAT framework, but links are provided via the Equator Network to the ‘Preferred Reporting Items for Systematic Reviews and Meta-Analyses’ (PRISMA) guidelines and checklist.
The FSA Quality Assurance Toolkit (QAT) was used along with the GSR Self-Assessment Code to appraise the research outputs of the FSA social science team (Chapter 2). On the basis of this it is strongly recommended that going forward the Quality Assurance Toolkit be used as the main means of assessing the quality of social science research at the FSA and across government.
The GSR Self-Assessment code could potentially be used to assess the broader dimensions of professional standards, such as the accessibility, legal and ethical issues and the recruitment and professional development dimensions.
Summary
In addition to peer reviewing the self-assessment of FSA’s social science team (using the GSR Code and Self-Assessment Tool) this chapter has also appraised the structure and applicability of the GSR Self-Assessment Tool itself. It is concluded that the tool operates at a rather high level of generality and the indicators do not really capture the quality of research outputs at a sufficiently granular level.
The recently published FSA Quality Assurance Toolkit (QAT) has also been reviewed and appraised. It is strongly recommended that going forward the Quality Assurance Toolkit be used as the main means of assessing the quality of social science research at the FSA and across government.