Consistency in regulatory work

Background

Consistent regulation is essential to achieving optimal policy outcomes on a national scale, yet it is also notoriously difficult to achieve.

Difficulty arises from the wide range of factors that may generate inconsistencies in controls: individual-level factors, such as the level of experience of individual inspectors; organisational-level factors, such as the structure and working habits of teams within enforcing authorities; or macro-level factors, such as institutions and broad societal factors.

Moreover, inspectors and authorities need to adapt general rules to the variety of circumstances on the ground. Regulatory practice should not be uniform but rather reflect objective differences between different regulated individuals or businesses. As a consequence, however, it is not easy to draw a line between ‘normal’ variations (resulting from objective differences between regulatees on the ground) and ‘inconsistencies’.

This particular research programme aimed to improve the evidence base on which the FSA’s efforts towards consistency could be built and revised. It is highly innovative, as very little analytical work of any significant scope has been published on consistency in regulation. It is therefore one of the first co-ordinated and multi-disciplinary attempts to better measure and explain consistency and inconsistency through the use of analytical methods.

Research Approach

Research Approach Study 1: Consistency and cognitive influences on the expert judgement of Environmental Health Officers: An experimental study

The project investigated psychological factors that lead Environmental Health Officers (EHOs) to make inconsistent judgements about catering outlets. It draws on research from the fields of ‘cognitive biases’ and ‘naturalistic decision making’ to understand decision making on food hygiene. These fields of study examine how variations in decision-making arise from heuristics (rules and habits that influence decision-making) and cognitive bias (errors that arise from a heuristic being applied incorrectly). To investigate variance in decision-making by EHOs and the extent to which psychological factors explain this variance, decision-making on food hygiene was studied through a series of experiments.

Based within a food and beverage outlet at the London School of Economics and Political Science (LSE) campus, two ‘simulated kitchen’ environments were created, and EHOs were asked to inspect and assess them. Scenarios were designed to examine the impact of three specific biases upon the decision making of EHOs: anchoring, confirmation, and overconfidence. The simulated kitchens replicated aspects of a real catering environment (e.g. equipment, facilities food preparation, catering staff, food storage), and contained a number of positive (e.g. food storage) and negative informational cues (e.g. out-of-date records) that were expected to influence ratings in food hygiene (environmental health) inspections.

To assess whether an ‘anchoring bias’ influences assessments of food safety, the team examined whether the assessments of the experimental catering facilities by inspectors are in-part determined by information on a previous inspection.

To assess if ‘confirmation bias’ influences assessments of food hygiene, the team examined whether the decision-making of EHOs is influenced by an early informational cue presented during an inspection.

To assess ‘over-confidence bias’, the team examined whether high or low confidence in EHOs (as measured by a post-study survey on self-assessed confidence and confidence on a number of knowledge items) results in differential assessments of catering facilities.

Research Approach Study 2: Analysis of LAEMS data to inform a study on inconsistencies in the delivery of official food safety controls

Data from the Local Authority Enforcement and Monitoring System (LAEMS) was analysed to identify potential inconsistencies between local authorities in the delivery of official controls, in particular with regards to the Confidence in Management scoring of food establishments and enforcement activity. This was to inform the selection of local authorities for the third study: Consistency in the delivery of official food safety controls: the role of organisational-level factors.

Research Approach Study 3: Consistency in the delivery of official food safety controls: the role of organisational-level factors

The study aimed to explore how organisational-level factors impact on the delivery of official controls by local authorities, and how they might contribute to inconsistencies in (i) the scoring of Confidence in Management (CIM) in food businesses, and (ii) enforcement responses to non-compliance with food hygiene inspections.

Thirteen factors were identified for exploration in the research. They were categorised into three broad themes (management practices, structure, and communications and engagement). From the outset, the Agency was interested in distinguishing variations in the delivery of official controls that might be justified or acceptable (e.g. targeted interventions), from other variations that could be considered ‘true’ inconsistencies. The latter were the core focus of this research. The study has contributed to clarifying that distinction for the field of food hygiene controls by looking at numerous specific examples arising from the data.

In order to achieve this, a small-scale, comparative approach was adopted, based on a programme of qualitative research with five pairs of comparable English local authorities, representing different categories of urban-rural classification. The local authorities were selected by the FSA on the basis of the statistical analysis of the LAEMS data. Between four and six in-depth interviews were conducted with individuals occupying various roles in each authority, to gather feedback from a range of perspectives (totalling 49 interviews across the case studies). The interview findings fed into a comparative analysis process that explored the role of each organisational-level factor on inconsistencies in CIM scoring and enforcement action.

Results

Results Study 1: Consistency and cognitive influences on the expert judgement of Environmental Health Officers: An experimental study

The experimental results suggest that inspectors are not susceptible to anchoring bias on previous ratings, nor to confirmation bias triggered by the order in which information was presented to them. Yet, considerable variance in ratings was found, and this was primarily explained by differences in scoring confidence in management. Through post-study surveys, Environmental Health Officers (EHOs) reported basing inspection decisions on a range of cues (e.g. chef knowledge, documentation), and this was particular to each EHO. This, along with limitations in the study design, likely explains the lack of experimental effect in this study.

Future research and training may wish to focus more upon the patterns and cues that EHOs use to make food safety decisions, rather than the generic biases that may influence these. Moreover, future research may wish to leave the laboratory, and focus on heuristics and biases in the real world. Finally, despite recognising some of the limitation of laboratory-style environments, they may be appropriate for training inexperienced EHOs

Results Study 2: Analysis of LAEMS data to inform a study on inconsistencies in the delivery of official food safety controls

There is considerable variation between local authorities in England in the Confidence in Management (CIM) scores awarded to food establishments. This includes the fact that food establishments in more urban local authorities tend to be awarded worse CIM scores than those in more rural local authorities.

Much of this variation remains after the following differences between local authorities are taken into account:

• the other compliance scores awarded (which assess level of compliance in terms of food handling/preparation and hygiene of the food premises);
• intrinsic risk scores (which assess the intrinsic risk associated with the type of food activity the establishment is involved in, e.g. it may be involved in high risk food activities, or a large number of consumers may be put at risk were something to go wrong); and
• types of food establishment (e.g. school/college canteen, take-away, supermarket/hypermarket, food manufacturer)

While this indicates possible systematic differences in how local authorities apply the CIM score, it may be the case that at least some of this variation can be explained by other characteristics of individual food establishments which are not recorded in LAEMS, or other factors.

Results Study 3: Consistency in the delivery of official food safety controls: the role of organisational-level factors

The study found that management practices within local authorities may result in inconsistencies observed within the case study pairs, notably: ‘management attitudes towards compliance’, ‘perceptions towards official guidance’ and, to a lesser extent, ‘task allocation’ and ‘staff skills and development’. ‘Engagement with FBOs’ may also lead to inconsistencies.

The research also shows that there is a temporal dimension to the influence of some organisational-level factors, because local authorities are dynamic entities and consequently their characteristics (including policies, structures and activities) tend to change over time. For example, the research did not find evidence that the ‘use of external contractors’ has contributed to inconsistent use of official controls in the recent past, because local authorities have tended to manage contractors closely or avoid using them altogether. However, the findings show that this has not always been the case.

Finally, insight into the role that inter-authority collaboration plays in the consistent use of official controls is a particularly pertinent finding from this research. It has two main influences: (1) where present, the use of shared processes and policies to enforce food safety regulation has promoted regional consistency in the delivery of official controls between local authorities; (2) conversely, consistency training delivered before 2015 appears to have had limited impact because it did not provide clear enough direction on the use of official guidance in different circumstances.