Evaluation of the PATH-SAFE programme - Introduction

The report is intended for HM Treasury as the programme sponsor and delivery partners of the programme, primarily Food Standards Agency (FSA), Food Standards Scotland (FSS), Department for Environment and Rural Affairs (DEFRA), UK Health Security Agency (UKHSA), Department for Health and Social Care (DHSC), and the Environment Agency (EA). The findings of the evaluation will help the delivery partners manage PATH-SAFE adaptively and assess the impact created through the programme on foodborne pathogen and antimicrobial resistance surveillance. The report is structured as follows: 

• The remainder of this introduction describes the context for the PATH-SAFE programme, the goals and structure of the programme, the aims of the evaluation and its limitations, and our approach to developing this evaluation framework report.
• Chapter 2 presents the analytical framing for the evaluation comprising the PATH-SAFE theory of change (ToC).
• Chapter 3 describes our overarching evaluation approach through which we will collect evidence to assess the PATH-SAFE programme. The evaluation approach will comprise three types of evaluation: a process evaluation; an outcome evaluation; and an impact feasibility assessment.
• Chapter 4 presents the process and outcome evaluation frameworks through which the evaluation of the PATH-SAFE programme will be operationalised. Presented in tabular form, each framework comprises key evaluation questions (EQs), derived from the PATH-SAFE ToC, which our evaluation will seek to answer, alongside indicators and proposed data sources. The chapter builds on the frameworks and provides further detail on the methodology that will be undertaken to conduct the evaluation.
• Chapter 5 shows the evaluation timelines and key deliverables. 
• Chapter 6 outlines the risks to the evaluation and our mitigations in place. 
• The Annexes to the report contain additional information on the four workstreams (WSs) and sub streams of the PATH-SAFE programmes (Annex A), which is useful context for the evaluation frameworks, and more detailed information on the activities that led to the development of the evaluation framework report (Annex B) and the process and outcomes evaluation frameworks (Annex C). 

1.1.    Background

Foodborne diseases pose a major public health risk to the UK population, and creates a significant burden on our health services and economy. The majority of human disease is caused by a handful of pathogens that, in most cases, enter the food chain from farmed animals or the environment. In addition, these foodborne pathogens can also develop antimicrobial resistance (AMR) due to the overuse of antimicrobials in food production systems. Since they are transmissible to humans via the food chain, causing illness and disease, antimicrobial-resistant foodborne pathogens are a global health issue affecting countries of all economic levels. The combined threat of foodborne and AMR pathogens creates a crucial risk for the food chain as well as the environment and need to be investigated and monitored holistically.

For AMR specifically, as its disease and economic burden rises globally, there is significant demand for new and emerging diagnostic and identification technologies to reduce its spread together with a coordinated and collaborative regulatory approach. According to the World Health Organisation (WHO), antimicrobial resistant infections are estimated to cause around 700,000 deaths each year, a figure which is expected to rise to 10 million deaths by 2050. They also pose a significant economic threat that could cost a cumulative 100 trillion dollars of economic output within the same timeframe. In January 2019, the UK government published a 5-year action plan to tackle AMR, developed as a cross-governmental effort in collaboration with a range of stakeholders across academia, industry, and professional bodies. The action plan included measures to improve the development and access to diagnostics, and adopted a ‘One Health’ approach, setting out commitments that cut across human and animal health, as well as food production and the environment. A coordinated, multi-sectoral ‘One Health’ response to surveillance of foodborne pathogens and AMR is possible with the research and development of new and safe antimicrobials, diagnostics, vaccines, waste management tools, as well as the availability of affordable and quality versions for both animals and humans. The ambitions of the UK government-funded PATH-SAFE programme integrate cohesively into this approach.

1.2.    The PATH-SAFE Programme

PATH-SAFE is a pilot programme that aims to develop comprehensive surveillance of foodborne pathogens (FBPs) and AMR in all four nations of the UK. It proposes to create a national genomic surveillance infrastructure, through improved and novel uses of data and technology creating a blueprint for national surveillance. Through a range of parallel proof of concept pilot workstreams, it will demonstrate how this infrastructure will support a national roadmap for One Health surveillance across the UK.

The PATH-SAFE programme is funded by HM Treasury through the Shared Outcomes Fund.  This Fund was established to pilot projects to test innovative ways of working across the public sector. It has had two rounds of funding, in 2019 and 2020, with a £200 million allocation in each round. PATH-SAFE was funded in round 2 with a funding allocation of £19.2 million which will last until 2024. It is a cross-governmental collaboration across the FSA, FSS, DHSC, DEFRA, UKHSA, and the EA. 

The PATH-SAFE programme consists of four workstreams (WSs,), which are summarised below, and are being delivered through multiple partners from the government departments and their agencies, centres and directorates mentioned above as well as devolved administrations in Wales and Northern Ireland and a variety of academic institutes. Further details on the WSs and their sub streams can be found in the Annex.

WS1: Establish a curated, national foodborne disease genomic data platform
This WS’s key ambition is to work with academic colleagues and major ‘big data’ stakeholders to create a ‘user-friendly’ platform for the rapid interrogation and of genomic data.  They will build easy to use reporting capabilities to create powerful, but easily understood, interfaces that can be used by decision makers (for example, epidemiologists or other public health professionals).  A key element of the data platform development will be allowing the integration of sample data with other existing data sources to create new knowledge.  

A distinct project of the WS is WS1b which is focussed on understanding source attribution, infection threat and level of AMR of E. coli. in Scotland using whole genome sequencing, with samples isolated from a range of reservoirs across Scotland.  

WS1 is led by FSA and delivered by a consortium of government and academic partners (Project 1a) and FSS (Project 1b) and is due to finish in March 2024. 

WS2: Develop a pilot infrastructure for regular, multi-location sampling 
The WS will develop a pilot infrastructure to provide high granularity whole genome sequencing (WGS) data from regular, multi-location sampling of wastewater and food products to capture AMR and FBP data. This is being done across multiple projects and multiple settings (i.e. milk laboratory, sheep abattoirs etc.). 

WS2 is led by Defra and delivered by the Centre for Environment, Fisheries and Aquaculture Science (CEFAS) (Project 2a); Animal and Plant Health Authority (APHA)/Veterinary Medicines Directorate (VMD) (Project 2b); Public Health Agency Northern Ireland (PHA NI) (Project 2c) and FSA (Project 2d) and is due to finish in March 2024. 

WS3: Understand the feasibility of using portable diagnostics as inspection tools
The WS will investigate the technology readiness levels of existing and new portable diagnostics. The results of these studies will inform options for in-field testing and/or development. The co-design of applications with end-users (for example, policy teams/inspectorates, operational staff) will be critical to ensure real-world applicability. The WS will also undertake a pilot study investigating the feasibility of using wastewater approaches developed in response to the Covid-19 pandemic with complimentary diagnostic technology (for example, Loop-mediated isothermal amplification or LAMP) to understand Norovirus outbreaks in a contained setting. WS3 is led by FSA and delivery by Fera Science Ltd. (Project 3a) and UKHSA (Project 3b) and is due to finish in March 2024. 

WS4: Develop a pilot environmental AMR surveillance system
The overall aim in WS4 is to create an evidence-based understanding of the nature and extent of AMR in the environment and the drivers that influence this. This pilot will deliver an agreed and tested methodology for environmental AMR surveillance, as well as an environmental information technology (IT) platform that aims to enable a scaled-up surveillance programme to be undertaken. This IT platform will be designed and developed so that it will have the capability to integrate AMR surveillance data collected from animals so that the ambition of having a UK ‘One Health’ surveillance system for AMR can be realised.  WS4 is led by DEFRA (with EA and VMD) and UKHSA and is due to finish in June 2023. 
 
Figure 1. PATHSAFE WSs

1.3.    Evaluation of PATH-SAFE 

In November 2022, FSA commissioned RAND Europe to undertake the evaluation of the PATH-SAFE programme. There are three main objectives of the evaluation:

• Design and framing: to design a programme ToC for the programme, articulating the change that is intended to be achieved by PATH-SAFE. This includes outlining assumptions, external factors and managing risks associated with the programme.
• Formative process evaluation:  to utilise bespoke evaluation methods for assessing the processes underpinning the programme and delivering the outputs of the four WSs to identify areas for learning and improvement.
• Understanding impact: to identify and prioritise outcome indicators and data sources to validate the outcomes of the ToC and develop an assessment of PATH-SAFE’s contribution. 

The main evaluation questions to be addressed in the process and outcome evaluations are listed below and can also be found in Table 1 and Table 2:

• How appropriately resourced has PATH-SAFE been throughout the stages of inception, design and implementation?
• How effective and appropriate is the governance in place to support delivery of PATH-SAFE?
• How is cross-government interaction being enabled/conducted?
• How is PATH-SAFE linked to existing/developing surveillance programmes?
• To what extent have relevant end users been engaged and how has have their needs been incorporated into the design of the database?
• How has data interconnectivity and interoperability been considered in designing the platform?
• What existing and novel analysis technologies are being utilised?
• What is the extent of data collection and curation?
• How (if at all) are new capabilities being generated to improve surveillance?
• How is data being accessed/ shared across relevant stakeholders and departments?
• To what extent is the technology readiness level (TRL) assessment approach valuable for identification of relevant technology?
• How is LAMP assessment feeding into TRL mechanisms for FBP diagnostics?
• What is being learnt and incorporated from existing AMR surveillance systems and tools?
• How is connectivity between the WS4 AMR environment platform and WS1a being considered?
• How is evidence being aggregated across the multiple departments involved in WS4 delivery?
• How has PATH-SAFE (if at all) enabled a community of practice and decision makers to come together to inform and act on surveillance of FBPs and AMR?
• How and to what extent has PATH-SAFE evidence (if at all) contributed to national policies and frameworks for improved public health?
• Has data access and use for FBP and AMR been enabled and improved across government departments?
• To what extent has the platform supported use of relevant metadata and historic isolates for comparative assessments and risk profiles of FBP?
• How has the collective source detection efforts and use of novel technology translated to (if at all) improved surveillance of FBP and AMR?
• To what extent have the pilot efforts been able to exemplify practice and enhance national surveillance capability?
• What kind of strategies and operations have been enhanced, enabled and influenced enabled (if at all) through the surveillance activities?
• Have the tools identified been useful for end users? Can they be utilised?
• To what extent have gaps been identified to further development of onsite rapid FBP detection?

The PATH-SAFE evaluation will look to understand existing surveillance mechanisms that precede the programme, which will be useful context for assessing the additionality of PATH-SAFE. The evaluation project will run from November 2022 to June 2024 

1.4.    Evaluation framework report

This document presents the evaluation framework report for PATH-SAFE. The purpose of the report is to detail our evaluation approach, including evaluation questions as well as data collection and analysis methods to guide the process and outcome evaluations of PATH-SAFE. We outline the key activities undertaken by the evaluation team to inform the development of this report in Annex B.  

1.5.    Developing the Process and Outcomes evaluation frameworks

The evaluation frameworks were developed systematically by reviewing and refining the EQs initially provided by FSA (developed with the programme partners) and developing new additional ones. The EQs were mapped to all key outputs and outcomes of the ToC to ensure coherence. Against each EQ, process and outcome indicators were developed which were then mapped against relevant data sources and the data collection methodology. We list the key activities undertaken by the evaluation team to inform the development of these frameworks in Annex C.