Evaluation of the PATH-SAFE programme - Evaluation approach

We propose to conduct three types of assessments: a process evaluation; an outcome evaluation based on contribution analysis methodology; and an impact feasibility assessment using an adapted context mechanisms and outcomes framework. The sections below provide more information on the aims of each assessment. 
 
Figure 4. Evaluation approach

3.1.    Process evaluation

The process evaluation establishes how the programme is working, whether it is progressing as intended, and identifies any lessons learned that can be applied to programmes that are still ongoing as well as their future iterations. It will use the Organisation for Economic Cooperation and Development (OECD) evaluation criteria of relevance and coherence, assessing if the intervention is doing what it should (i.e. incorporating needs of stakeholders and considering the context), and whether it is compatible with other interventions (carried out by programme partners or other actors) that predate it or are in development within the same field. 
Our process evaluation is based on the ToC, focussing on the inputs, activities, and the resulting outputs of the PATH-SAFE programme and its WSs. The process evaluation will consider the mechanisms and structures in place leading to the delivery of outputs, which are primarily governance arrangements, cross-government collaboration, delivery barriers and enablers, links with existing surveillance and monitoring approaches, and end user engagement. Given the programme is a pilot and looking to build on existing capabilities as well as generate new ones, the process indicators will be focussed on considering ‘the extent’ to which activities have created step change and resulted in the anticipated outputs rather than looking to quantify processes.

3.2.    Outcome evaluation

The outcome evaluation will be focussed on whether the programme and its WSs have realised the changes expected at a given point in time and determine how  the changes may or may not have occurred. The goal of the outcome evaluation is not to attribute outcomes exclusively to PATH-SAFE but rather to provide evidence-based explanations of whether and how the programme contributed to the outcomes of interest alongside other external factors through undertaking contribution analysis (CA) (see Section 3.4.1 Contribution analysis). Given the start of the programme in early 2022, most outcomes will likely not have emerged at the time when the evaluation is being conducted and concluded.  Therefore, the contribution claims assessment will look to focus on iterative trends and leading indicators of progress. The outcome evaluation will use the lens of the OECD evaluation criteria of effectiveness in assessing if PATH-SAFE is on the path towards accomplishing its objectives. 

3.3.    Impact feasibility assessment

The impact feasibility assessment is an exercise to determine how to best evaluate the longer-term impact of PATH-SAFE. The assessment of outcomes based on the CA methodology will provide us with a useful baseline of impact and whether the contribution claims being tested are realistic or feasible. It will clarify which impacts remain relevant for the programme and what methodologies and indicators may be useful to consider. We will adapt and use the context, mechanisms and outcomes (CMO) framework, usually used in a realist evaluation approach (see section 3.4.2 Context, Mechanism, Outcomes Framework), to develop projections of impact. The purpose of utilising a CMO-style framework is not to actually conduct an impact evaluation using the CMO which would duplicate the work of the CA analysis, but rather to use the CMO in a novel way to create hypothesis of what the future outcomes/impacts might be, what the potential mechanisms of action and the context for it might be. This will be entirely based on the knowledge amassed from the process and the outcome evaluation, which will culminate in the CA. The outputs of the CA will inform the CMO style projections/hypotheses. The study team will reflect on the PATH-SAFE context (i.e. the external environment) to assess its potential effect on outcomes yet to be realised, and also consider the mechanisms in place in the PATH-SAFE programme (uncovered during the process and outcome evaluation) that are contributing and could continue to contribute to realising the anticipated outcomes and impacts. We will not be undertaking a CMO evaluation but rather utilising the framework for considering appropriateness of PATH-SAFE future outcomes and impacts and their potential realisation pathways which can inform a future measurement approach. 

3.4.    Methodological frameworks

The theory-based approach being utilised is underpinned by the programme ToC discussed in Section 2.1. Further to that, the outcome evaluation will be analysed through the framework of contribution analysis to assess PATH-SAFE’s contribution to outcomes and impacts, while the impact feasibility assessment will be undertaken using an adapted CMO framework. 

3.4.1.    Contribution analysis

To help attribute causality in a programme of this size and complexity, this theory-based evaluation will use the CA methodology on the data collected. CA is a method for assessing causal claims that examines the contribution of an intervention to observed results. It provides a framework for capturing progress towards aims at a relatively early stage through testing working hypotheses and establishing a case to explain the contribution made by PATH-SAFE and its projects over alternative hypotheses. Determining contributions requires qualitative methodologies (for example, deciding whether the relevant evidence has been identified, or if it is sufficient to discard alternative hypotheses), but is informed by both quantitative and qualitative evidence from all the methods undertaken throughout the evaluation. We will place greater weight on findings stemming from multiple data sources to assess the added value and true contribution of PATH-SAFE to the outcomes anticipated and realised. See Chapter 4 for further details on how this will be done. 

3.4.2.    Context, Mechanism, Outcomes Framework

The impact feasibility assessment will be conducted through utilising the CMO framework. The CMO framework will be used to create a projection of how the outcomes and impacts of PATH-SAFE may arise, as anticipated, based on the ToC. This projection will rely on abductive reasoning and the evidence gathered during the evaluation on identifying contextual factors and trends, as well as identified mechanisms of actions within PATH-SAFE. This assessment will allow us to iterate on the ToC and develop a realistic measurement approach for a longer-term and/or follow-up evaluation of PATH-SAFE. As mentioned above, the data gathered during the evaluation culminating in a CA will inform the basis of the CMO projection exercise. Although we are not undertaking a CMO based realist evaluation, the use of this framework provides a useful and structured template for impact feasibility assessment.

An illustrative example of a projection for PATH-SAFE utilising CMO is depicted in Figure 4 below. When assessing one of the anticipated impacts of PATH-SAFE, preventing the increase in foodborne illness, the evaluation of PATH-SAFE could help identify the mechanism through which this could occur. In this instance, work of WS3 could result in identification and development of onsite diagnostics for FBP and AMR which, if adopted, could help decrease the incidence of foodborne illness. This change might be possible if the technology in question is scalable and can be commercialised. This is an entirely hypothetical projection and will need to draw on the PATH-SAFE evaluation for validity. The next step after creating the projections would be to develop recommendations to modify outcomes and impacts in the original ToC (if required) and to propose methods for conducting future-focussed evaluations. 
 
Figure 5. Illustrative example of CMO analysis

3.5.    Limitations of the evaluation approach

The approach and methodologies outlined in the chapters above will provide a wide-ranging set of data and evidence around the ambitions of the PATH-SAFE programme, and whether these ambitions have been achieved. However, our approach to the evaluation is also subject to a number of important limitations. 

Firstly, whilst our approach aims to be comprehensive and cover different impacts of the PATH-SAFE programme, the lack of counterfactuals to compare the programme against poses a significant limitation. The programme being a pilot means that there are new outputs being developed such as the creation of a new genomic database and a pilot surveillance infrastructure. However, a potential mitigation of these limitations is to understand what was already in place preceding PATH-SAFE and to position the outputs of PATH-SAFE as building on existing capabilities.  

Second, our evaluation approach focuses mainly on the PATH-SAFE programme with a limited role for analysing the interactivity with external programmes of work in this space such as the AMR national action plan or the EU Farm to Fork strategy. On a similar note, the lack of international programme assessments means that it is more difficult to position the programme in a broader/international context. Lastly, developments in industry are not factored into the programme itself, so the evaluation has also not included them. This is a blind spot in understanding the state of play in terms of surveillance. 

Third, given that much of the anticipated impact of the PATH-SAFE programme will only emerge over a lengthy time horizon, the evaluation will not be able to capture its outcomes nor its long-term impacts in full. Ideally, the evaluation would involve a long-term follow up and assessment of PATH-SAFE to track these impacts of the programme. What we are proposing is a step in this direction, setting out a range of indicators that can be used to assess whether the programme is on track to achieve longer-term desired outcomes and impacts. Additionally, the evaluation will provide recommendations on a future-focussed evaluation approach to further the assessment of longer-term outcomes and impacts. 

Finally, as the programme is at pilot stage, and our data sources are limited and reliant upon the programme data availability itself, the possibility of low availability of baseline data due to project delays could be a challenge, limiting the range of data available across our evaluation timeline. This lack of data will need mitigation and caveats as the evaluation progresses.