Evaluation of the PATH-SAFE programme - Evaluation framework

The chapter comprises two distinct but interrelated frameworks: a process evaluation framework and an outcome evaluation framework. Presented in tabular form, each framework outlines evaluation questions (EQs), looking to assess progress made towards the PATH-SAFE ToC components. For each EQ, the frameworks contain indicators and proposed data sources that will be used to collect evidence to enable us to answer the EQ. Alongside process and outcome evaluation frameworks, the chapter also provides further detail on the data collection methods and analytical approaches that will be used for each type of evaluation. Impact feasibility is not included in this chapter as Section 3.3 describes the approach that will be undertaken to conduct the feasibility assessment. We have not ascribed indicators and data sources for the ToC impacts at this point as a result. 

4.1.    Process evaluation framework 

As highlighted in Chapter 3, the process evaluation will seek to understand the extent to which PATH-SAFE’s programme governance and resourcing has been fit for purpose and assess the mechanisms of actions across the four WSs. This assessment will be done through the lens of the principles of relevance and coherence based on the OECD criteria. Table 1 below sets out the process evaluation framework. Each WS’s activities (A) and outputs (O) are assigned key EQs which will be assessed through the relevant indicators and data sources listed. The last column lists the methodology that will be undertaken for answering each EQ.  Sections 4.1.1 and 4.1.2 describe in more detail the data collection tools and analytical approaches that will be used to undertake the process evaluation. 

Table 1. Process evaluation framework

4.1.1.    Process evaluation data collection methods

As shown in Table 1, the process evaluation will rely on three main methods of data collection: document review, desk research, and key informant interviews. These data collection methods are described in more detail below.  

Document review

We will conduct a review of PATH-SAFE management information such as business case bids, initial design documentation, and governance and monitoring requirements/criteria to further develop our understanding of PATH-SAFE programme processes. Documents to be reviewed will also include programme WS specific documentation such as WS project briefs (noting any changes in scope and delivery), latest highlight reports, and latest documentation for a given month/quarter from the Data Advisory Group (DAG), Shared Outcomes Fund, Scientific Advisory Group (SAG) and the Strategic Board. We will also review WS activity/technical reports where appropriate and available. This will be undertaken at both the interim process evaluation and the final process evaluation stages to assess the extent to which the intended outputs have been delivered. 

Desk research

We will review the AMR national action plan and the NBN documents to assess alignment with PATH-SAFE in more detail as helpful context of the process evaluation. We will also undertake a high-level grey literature search to map out key pathogen surveillance initiatives across Europe and the devolved nations in the UK to create a robust assessment of surveillance mechanisms and infrastructure already in place in the agriculture/environment sectors. 

Key informant interviews

Alongside document review, data on how the programme has been received by key delivery partners, government stakeholders and any other end users, as well as experience of engagement and incorporation of views into WSs, will be collected primarily through key informant interviews. To inform the process evaluation at the interim stage, we will conduct interviews with: 

• Up to four central operational staff at FSA 
• Up to 10 delivery partners including academics across WSs 1-4
• Up to 15 end users/key government stakeholders across DEFRA, UKHSA, FSS, EA, DHSC and Public Health Wales and NI, etc.

To inform the final process evaluation (and the outcome evaluation) we will conduct interviews with:

• Up to three central operational staff at FSA
• Up to six delivery partners across WSs 1-4
• Up to 10 key government stakeholders/end users

Interviewees will be selected based on the PATH-SAFE stakeholder database, using a purposive sampling approach to ensure representation across WSs, government departments and types of end users. This will be done in consultation with the PATH-SAFE central team at FSA. Interview topic guides and analysis coding will be guided by the evaluation questions as specified in Table 1. We will also complement interviews through engagement with PATH-SAFE central and delivery teams at bi-weekly meetings and attendance at monthly Delivery Board meetings. 

4.1.2.    Process evaluation analysis

Data collected through the methods above will be brought together and triangulated against our process evaluation framework to create an understanding of how processes supported and/or created barriers in delivery of PATH-SAFE. In addition, to create an exemplified picture of effectiveness of PATH-SAFE processes, we propose to develop two case studies based on existing data collection methods highlighted with a potential for deeper dives into the proposed topics via interviews and documentary reviews. 

Case studies

Case studies will be selected purposively and will be used to tease out instances in which processes have worked exceptionally well, or to highlight examples where things haven’t gone as expected, highlighting opportunities for improvement. This will be determined through consultation with the programme team and considering the data emerging during the interim process evaluation. We propose to develop two process case studies. 

Given the central importance of cross-government engagement, we suggest focusing one case study on exemplifying good practice of an instance where cross-government collaboration has worked particularly well (this could be at central programme or at WS level). The case study will not only centre on what worked well but also look to identify enabling factors and levers for change that could be applied across the rest of the programme. 

We propose to focus the second case study on data sharing enablement, given its importance across multiple WSs and the programme as a whole. We will again look to exemplify good practice of where data sharing has been enabled or an agreement put in place and go further to identify what catalysed the process and what barriers remain to be addressed. 

4.2.    Outcome evaluation framework

As described in Chapter 3 Evaluation approach, the outcome evaluation will provide an assessment of the extent to which the outcomes outlined in the ToC have been realised. This will be a theory-led approach and will utilise CA to validate central claims made about the programme’s success, utilising the evidence collected against key outcomes and the key EQs (see section 3.4.1 and 4.2.2 for more info). Within the outcome evaluation framework, most outcomes listed are broadly mapped to the key WSs that are likely to contribute towards them, but some are at a programme-level, to which all WSs are anticipated to contribute. All outcomes have been assigned key EQs which will be assessed through the relevant indicators and data sources listed. The last column lists the methodology that will be undertaken for answering each EQ. Section 4.2 describes in more detail the data collection tools and analytical approaches that will be used to undertake the outcome evaluation.
 
Table 2. Outcome evaluation framework

4.2.1.    Outcome evaluation data collection methods

As indicated in Table 2, the outcome evaluation will draw on a wide range of sources underpinned by four main methodologies: documentary review, desk research, key informant interviews, and a workshop.  

Documentary review

Analysis of key activity reports and papers from meetings of the SAG, DAG, shared outcomes fund, and the strategic board will be analysed to assess the extent to which outcomes have been realised. More focus will be placed on direct WS reports to provide a sense of progress towards intended outcomes at the WS level.

Desk research

Desk research will be conducted on Google Scholar to assess grey literature outputs that can be attributed to PATH-SAFE. We will do this for the first 100 hits through a targeted search. In addition, an assessment of publications of academic papers, strategy and policy documents will be conducted through a search on bibliographic data platforms to assess what publications PATH-SAFE has enabled, if any, which will provide an understanding of PATH-SAFE knowledge generation and wider influence. 

Key informant interviews

To further strengthen our understanding of the extent and mechanism of outcome realisation, we will conduct key informant interviews. Please note that these will be the same set of interviews that are proposed for the final process evaluation stage to reduce burden on interview respondents. We foresee speaking to the same set of stakeholders and given the parallel timelines of the final process and the outcome evaluation, these set of interviews will look to assess both process and outcome EQs. As mentioned in section 4.1.1, we will conduct the following interviews for the final process and outcome evaluations:

• Up to three central operational staff at FSA
• Up to six delivery partners across WSs 1-4
• Up to 10 key government stakeholders/end users

As with the process evaluation interviews, topic guides will be developed based on the key EQs in Table 1 and Table 2, and all interviews will follow a semi-structured format. 

Workshop

Assessment of step changes or any improvements made on high level outcomes of ‘improvements in surveillance capabilities and mechanisms’ and ‘awareness and engagement across government departments and key decision/policy makers’ will be more appropriately gleaned through a large workshop/group exercise (with up to 15 participants) undertaken with the relevant stakeholders. The central programme team will be key in determining the most appropriate mix of stakeholders to engage in this exercise.  

4.2.2.    Outcome evaluation analysis

The evidence from the methodologies outlined above will be triangulated to develop a holistic understanding of the difference PATH-SAFE has made. This will be crucially underpinned by undertaking a contribution analysis exercise (detailed below) and development of two case studies exemplifying a select component of a given outcome. 

Case studies

Case studies will be selected purposively and will be used to tease out instances where tangible examples of progress can be seen towards outcome realisation and/or to highlight examples where things haven’t gone as expected, or where outcomes have been significantly delayed, highlighting key barriers. This will be determined through consultation with FSA and considering the data emerging during the early phase of the outcome evaluation. We propose to develop two outcome case studies. 
We propose to focus one case study on showcasing an example (if available) of PATH-SAFE influencing a nationally linked operation/strategy (for example, the NBN), and focus on the enablers of influence and the nature of the influence to understand its importance. We propose to focus the second case study on an example of a novel tool or framework for testing/surveillance that has been developed and assess its value to improvement of surveillance. 

Contribution analysis

As mentioned before, CA is a method for assessing causal claims that provides a framework for capturing progress towards aims through testing working hypotheses and establishing a case to explain the contribution made by PATH-SAFE and its projects over alternative hypotheses. The six steps involved in CA are as follows:

1. Set out the cause-effect issue to be addressed.
2. Develop the postulated ToC and risks to it, including other influencing factors.
3. Gather the existing evidence on the ToC.
4. Assemble and assess the contribution claim, and challenges to it. 
5. Gather new evidence from the implementation of the intervention.
6. Revise and strengthen the contribution story. 

At this stage of the evaluation, Steps 1, 2 and 3 have been completed. Based on our understanding of what PATH-SAFE is aiming to achieve and the ToC underpinning the evaluation, we propose three main contribution claims for the programme as an output of Step 4. These are hypotheses that are central to the programme and can be interpreted as high-level and holistic outcomes of the programme. 

• The processes established in PATH-SAFE programme lead to cross-government collaboration on FBP and AMR surveillance because of increased transparency and engagement across departments through the work on interrelated WSs.
• The development of the data platform in PATH-SAFE leads to easier data sharing across government departments because of data sharing agreements put in place and extent of user engagement carried out.
• The collective outputs of the WSs in PATH-SAFE leads to establishment of a nationally connected and improved FBP and AMR surveillance approach because of multilocation sampling, novel testing tools and an interconnected data platform.

We plan to utilise the process and outcome evaluation evidence holistically (i.e., evidence from interviews, workshops and case studies) to address Step 4 in assessing the body of evidence to validate the contribution claim. We will then create an overarching narrative (i.e., the contribution story) relative to the strength of the evidence that makes a qualitative judgement on whether the contribution claims stand or whether an alternative hypothesis exists for what caused the change to occur. The alternative hypothesis in particular will be tested through interviews and a workshop, and will be derived from the external initiatives listed in Section 2.2.2. The contribution story will identify any gaps in the evidence or weak links, where we will look to alternative sources of data and revise the contribution narrative accordingly. The contribution narrative will ultimately rest on the collective evidence surfaced through the process and outcome evaluations.