Evaluation of the PATH-SAFE programme - Analytical framing

We have chosen a theory-based approach for this evaluation. Whilst experimental evaluation approaches usually measure the effect of an intervention in comparison to a counterfactual group, hence assessing the causal relationship between an intervention and its effects, they do not uncover why the intervention worked or not and how, if it did. A theory-based evaluation addresses these questions and considers the complexity within which an intervention is being delivered. The combination of complexity of the external environment, large and disparate areas of focus for the WSs, and lack of a counterfactual makes a theory-based approach the most useful and feasible for the PATH-SAFE evaluation. Given the use of a theory-based approach, the evaluation of PATH-SAFE is underpinned by a ToC described below, which is the foundational structure used to develop the evaluation framework presented in Chapter 4. 

2.1.    PATH-SAFE ToC

A ToC, read from left to right, is a programme theory that hypothesises the intended change an intervention is likely to bring about. It assumes a causal relationship between the intervention activities and its outputs and outcomes. An integral part of conducting a robust evaluation, a ToC helps articulate how various programme inputs and activities are expected to work, as well as identify the strength of the evidence that underpins them. However, the further one moves to the right-hand side of the ToC and the longer term the outcomes and impacts become, the effect or the contribution of the intervention becomes diluted and direct causality is less attributable. To account for this dynamic programme environment and complexity, an iterative and participatory approach to refining the ToC was followed, involving key stakeholders and triangulation with desk research. This approach is also in alignment with the Magenta book guidelines on handling complexity in policy evaluation while developing the logic model for a ToC.

A ToC serves two broad purposes: 

• It clarifies for stakeholders the role they can play in accomplishing the goals of the intervention through articulating a shared understanding of the aims of the intervention in question and how these will be achieved.
• It also functions as a tool, a base framework upon which to map evaluation questions, indicators, and data sources by displaying the logic through which the performance of the intervention can be assessed (see Chapter 4. Evaluation framework).

FSA, in conjunction with programme partners, developed a ToC for the PATH-SAFE programme that was shared with RAND Europe. This ToC adopted the standard logic model approach of modularly stating inputs, activities, outputs, outcomes and impacts of the programme. After documentary review, consultation with subject-matter experts, and a validation workshop with the central team, we revised the original ToC by providing more specificity in the outputs and outcomes of the programme and a clear linkage between these and the four WSs. 

As an example, the original output listed for WS1, “Pilot FBP/AMR genomic data system using exemplar species”, was divided into two separate outputs that differentiated between the delivery of the database itself and its ability to integrate with other data systems (see ‘Outputs’ column in Figure 2.) Similarly, the original list of five outcomes were expanded and further nuanced to take into account the anticipated changes realised through the four WSs as well as changes achieved by the programme holistically building on the individual outcomes of the WSs. For instance, WS2 has been expanded to contribute to distinct outcomes. This includes the original outcome focused on understanding source attribution of FBP and AMR, with an added focus on infection threat as well as an explicit mention of international entry points in the newer version. 

Additional outcomes focused on bringing together key stakeholders and decision makers to engage with key evidence, and contribution to ‘One Health’ goals and ambitions for public health have been added to signify the holistic change anticipated at the programme-level. The revised ToC is presented in Figure 2, and will be edited following the impact feasibility assessment (see Section 3.3). The activities and outputs are intended to be carried out and delivered by March 2024 when the current phase of funding completes. The outcomes and impacts are anticipated to be realised over the medium (2-5 years) to long-term (5-10 years). 

Figure 2 PATH-SAFE programme ToC

2.2.    Factors influencing the ToC 

This section outlines the main assumptions that need to be fulfilled for the ToC to be realised and the external factors to be aware of in the evaluation. These are key factors that could impact the programme’s delivery, so the evaluation needs to refer to these in its methodological approach. The original assumptions underpinning the ToC, provided by the central programme management team, were modified based on the documentary review and desk research conducted, when revising the ToC. Furthermore, a list of external factors was developed, supported by desk research, that could affect the delivery of PATH-SAFE and hence impact the ToC and the ensuing evaluation. The list of assumptions and external factors presented is not exhaustive and, where appropriate, will be revisited at the conclusion of the evaluation. 

2.2.1.    ToC assumptions 

The ToC is underpinned by a range of assumptions about the expected behaviour of key entities across the PATH-SAFE programme, which in turn affect the realisation of the intended outputs and outcomes of the intervention. These assumptions cover the actions of end users, stakeholders, and the programme itself. We have identified the following as relevant for the PATH-SAFE programme: 

• end users know about and engage with programme outputs facilitated through a strong engagement strategy. 
• key collaborations, with stakeholders needed for programme delivery, are established and maintained at the programme and project level.
• datasets, surveillance systems, and innovations are fit for purpose and functional to track AMR and FBP.
• further funding covers running costs of legacy products and financial input continues until projects draw to a close with a plan for infrastructure maintenance.
• there is use of systems and frameworks produced from the programme across the agrifood landscape in the UK.
• programme activities are commissioned and awarded on time and as intended.
• programme funds activities that align with the aims of PATH-SAFE.

2.2.2.    ToC external factors

The implementation of the PATH-SAFE programme is taking place in an environment that includes exogenous shocks such as the UK’s exit from the European Union (EU) and Covid-19, and other relevant initiatives in the broader ecosystem of pathogen surveillance and detection. Exogenous shocks can result in significant changes to the economic and societal landscape within which PATH-SAFE operates. EU exit, for example, could impact which pathogens are selected for surveillance due to divergent policies on AMR and crop technologies between the UK and the EU. Covid-19 and the ongoing war in Ukraine, other shock variables, have already compressed resource availability and the ability to deliver projects as intended. Beyond shocks, we also include relevant initiatives that could impact PATH-SAFE delivery or affect its intended outcomes and impacts by either accelerating or hindering them (for example, the UK national action plan for AMR). These activities can help us assess where the contributions of PATH-SAFE are unique and where they form part of a larger effort across the UK and international agri-tech sector. They will also show us whether PATH-SAFE is compatible with other interventions that predate it and are in development (see Section 3.2).

The external factors identified so far are as follows: 

Shocks

• external events (for example, Covid-19, war in Ukraine) may impact resource availability across the program, impacting ability to deliver as intended.
• EU exit’s effect on UK and EU divergence on AMR policies and crop technologies could impact how, when, and on what pathogens the surveillance is conducted and also contribute to PATH-SAFE impacts. 

Relevant initiatives that may impact PATH-SAFE outputs and outcomes

• UKHSA investment into another surveillance platform, developing data linkage pipelines with NHS hospital episode statistics, could fortify or detract from PATH-SAFE impacts.  
• UK National Action Plan for AMR entailing reduction in use of antibiotics in livestock will impact AMR surveillance datasets and mapping. 
• investment of $1 billion by industry to set up the AMR Action fund to bring 4 new antibiotics to market by 2030 will potentially have an impact on AMR reduction but this is outside the timelines of PATH-SAFE. The lead candidates are BV100, BV 200, BV300, and BVL-GSK098. 
• work carried out by Centre for Genomic Pathogen surveillance, enabling genomic data for surveillance of AMR tracking in the UK and globally could be complementary to or overlapping with PATH-SAFE. 
• WHO’s Global Genomic Surveillance Strategy 2022-2032 for pathogens with pandemic and epidemic potential, which aims to facilitate connectivity between different disease control programs and surveillance networks, has the potential to improve the PATH-SAFE outputs. 
• Centre for Pandemic Preparedness developing a global early warning system to detect new infectious disease threats by bolstering surveillance and sequencing capacity could provide an opportunity for learning or to contribute to a global agenda thus realising PATH-SAFE outcomes and impacts. 
• EU Farm to Fork Strategy (2020): Its objective is the reduction by 50% of the overall EU sales of antimicrobials for farmed animals and in aquaculture by 2030. It will impact AMR surveillance and mapping and will need to have coherence against PATH-SAFE systems. 
• climate change sector policy impact on use of genetically engineered/genetically modified editing in the agrifood sector could impact how, when, and on what pathogens the surveillance is conducted and also contribute to PATH-SAFE impacts. 
• a new National Biosurveillance Network (NBN) is about to enter a discovery phase and will form part of Pillar 3 “Detect” of the new Biological Security Strategy. The aim of the discovery phase is to understand biosurveillance capabilities and then develop a biosurveillance ‘pilot’ business case, covering all biological threats. The NBN could offer legacy opportunities for PATH-SAFE if FBP and associated AMR are deemed to be within scope.