Good Practice Regulatory Change: Measuring Success

The wider literature around measuring success 

The literature describes how consistently changing regulation and inspection frameworks, alongside external political pressures, can make it difficult to see how effective regulation has been (Ozga et al., 2013). The OECD (2021) found that less than one-quarter of member states systematically assess whether regulations achieve their objectives. They argue that governments need to develop more ‘adapt and learn’ approaches that can help provide learning and ensure changes correspond to the latest evidence and science.  

This reflects insights from the NAO (2017) which highlight how the benefits of regulation are not easily quantifiable. The NAO (2017) has argued that regulators need good performance measurement to enhance accountability and demonstrate value for money, even if this is something they find challenging to do. Their good practice guide on performance measurement by regulators (NAO, 2016) sets out the importance for a regulatory framework to give a balanced picture on where the needs of providers and consumers may conflict, the costs of collecting performance information on providers, and on the ability of regulators to influence their intended outcomes. They have also published a corresponding maturity model that can help an organisation to assess their own performance against good practice criteria across several stages: developing a performance measurement framework, reporting performance information and using performance information. 

Findings from the interviews 

Ongoing monitoring and data platforms 

Regulators emphasised the value of monitoring performance throughout the life of a programme. By capturing data on a continued basis, interviewees described how they were able to use this to gain buy-in and provide evidence that they were achieving their goals. This was seen as important for building relationships with both internal and external stakeholders, particularly where there was scepticism about the reasons behind a proposed change. This data could also be used as part of regular meetings with senior leaders or wider stakeholder groups. 

This was often easier to achieve for programmes involving a greater use of technology or data-driven approaches that can provide real-time intelligence and insight into what is happening across a sector. In contrast, programmes built on changing a culture or delivering a new strategy could be harder to measure. Interviewees also described challenges in upskilling teams to understand how to use new platforms to measure performance and to ensure the accuracy of data being collected. 

“We carried out a project on data accuracy. As part of that, we developed a dashboard to show how it is improving over time. Any elements we can report on, we add them into the dashboard. It is important to show the progress, it helps to get buy in, particularly if you have a programme set up without lots of permanent staff and you are relying on input from other teams within the organisation. Anything that helps buy-in and helps others see the benefits or the differences being made.”

One interviewee described how they have built automated processes into their relationships with regulated bodies. They see ongoing data into how organisations are interacting with consumers and if something has not been resolved satisfactorily, the regulator will get involved. This provides a good indicator of whether an organisation is getting small things wrong, which can suggest there are larger problems. By flagging concerns early, the regulator can check an organisation is getting the basics right and provide support to ensure larger problems do not arise, as well as tracking the success of the new regulatory approach. 

Similarly, interviewees described how continued engagement with stakeholders could help to provide ongoing insight into how a change programme was progressing. By building new relationships that are more collaborative or proactive, organisations were able to have more real-time, honest conversations that could help to inform an approach. 

Measuring impacts

Developing an approach for measuring outcomes was seen as important but was not always done systematically. Regulators used a range of methods to measure outcomes including: staff surveys, analysis of financial data, impact assessments, complaints data, the number and results of enforcement cases, accident reporting and Value for Money (VfM) assessments. Measures tended to be quantitative, with regulators describing how they translated specific regulatory outcomes into quantifiable indicators of change. However, there was recognition that this could result in a focus on measuring outputs, such as the number of businesses engaged in a programme, rather than outcomes. There were few mentions of establishing formal evaluations of the programme from the start. 

Interviewees described a number of challenges in measuring the outcomes of change programmes:

• It can take a considerable length of time after introducing changes to be able to measure the outcomes. For example, changes to protect employees against long-term health conditions can take decades to materialise in the data.  
• External changes to the policy landscape in fast-moving sectors, such as the technology sector, were reported to make it difficult to determine whether intended changes to regulatory programmes were successful.
• It is often difficult to determine a direct link between changing regulations and behaviour change. For example, accident reporting can help to show a reduction or increase in incidents, but this does not provide information about near misses. 

In response, interviewees emphasised the value of having proxy measures or indicators for the outcomes of a change programme. This can help to provide insight into the success of measures at an earlier stage. 

“When you are right at the beginning, it is important to make sure you are not just focusing on the outputs of change but doing some sort of impact assessment and making sure you include things like behaviour, mindset, attitude so you can start thinking about the challenges to measuring this.”

Regulators also stressed that financial indicators, and the final costs of the programme should not be the only method of measuring how successful a project is. It is important to balance the financial indicators with other measures that can assess whether the project has achieved its desired outcomes. These other measures could include internal staff surveys and feedback from stakeholders in the industry.