Framework for the assessment of uncommissioned third-party evidence

Our approach to science and evidence

We are proud of our transparent use of science and evidence to inform our advice and recommendations. As well as funding and conducting our own research, independent evidence is sent to us from a variety of sources. For example, this can be from food business operators, members of the public or those who represent them, and most commonly arrives during our consultations.

When new advice on food safety is required, our risk analysis process will be used to assess the risk, consider the measures that could be taken, and communicate the risk and necessary actions with others.

Our use and interpretation of scientific evidence and analysis is informed by the input, scrutiny and challenge of independent experts, for instance through our Scientific Advisory Committees (SACs).

Uncommissioned evidence

Sometimes evidence may be sent to us by a member of the public, industry representative, consumer group or others, outside of our usual research and consultation processes. Such uncommissioned evidence might be submitted with a variety of motivations and aims, such as filling a perceived gap in knowledge or suggesting a change relevant to a policy or legislation.

Evidence required to place a regulated product on the market, is the subject of specific guidance and fall outside the consideration of this introductory framework.

When we receive evidence, we will:

• be transparent about how the evidence is assessed and used to develop our evidence base, policy recommendations and risk communication;
• assess evidence in its proper context using the principles of quality, trust and robustness;
• seek to minimise bias in our assessments of evidence by using professional protocols, our SACs, peer review and/or multi-disciplinary teams;
• be open and transparent about the conclusions we have reached about any evidence submitted to us.

Guidelines on the assessment of uncommissioned third-party evidence

The guidelines below have been developed by our advisory Science Council and are intended to be used by anyone seeking to submit evidence to us – both those directly performing studies, and those choosing and evaluating existing evidence to support a position.

The guidelines outline our expectations concerning the standard of uncommissioned evidence that we receive and provide guidance on how its strengths and weaknesses will be assessed. The Science Council has aimed for the guidelines to be as accessible as possible, though some of the underlying scientific concepts may require or benefit from supplementary reading.

The guidelines are organised according to the core principles of quality, trust and robustness. Consideration of how any evidence to be submitted performs against these principles will help in preparing a useful contribution to the relevant body of evidence.

The guidelines are not exhaustive as our work covers a broad range of disciplines and areas of interest, including but not limited to microbiology, toxicology and behavioural and social science. They should be employed as appropriate to the type of science forming the evidence submission. Links to some of the organisations that provide detailed guidance for specific areas and disciplines are provided below and throughout the page in the relevant section:

• UK Accreditation Service
• International Organization for Standardization
• European Committee for Standardization
• International Programme on Chemical Safety
• UK Statistics Authority
• National Institute for Health & Care Excellence

How your submitted evidence will be used

The decisions that we make on food safety are based upon a broad body of evidence, that may span a range of constituent or other legitimate factors. When we receive new evidence, we will consider it in the context of the body of evidence that has already been used to inform a policy or decision. This will inform the assessment of whether any action, including changes to the existing position may be needed.

In new, emerging or rapidly developing areas, a decision may need to be taken based upon limited evidence. We will use the best available evidence to make this decision, recognise where there are gaps or limitations in knowledge, and will be open to change as new evidence becomes available. This recognises that in some instances, to protect consumers, incomplete or indicative preliminary evidence might need urgent consideration.

Quality

Evidence should be reliable and relevant to the question at hand. Clearly defining the context of the study and the question originally asked can help to identify if the evidence is relevant. Using well-recognised methods and data analysis can help to ensure it is both relevant and reliable. If a novel method is used, a clear explanation of why it has been used and what advantage it brings is important. Data and analysis should be clearly presented, with a narrative that directly links them to the conclusions within the study.

Clarity

• All evidence sent to us should be clearly laid out, outlining the study approach, the data collected, and analysis performed
• If evidence has been collated from several sources this should be clearly indicated, and the method used for its collation and integration described
• Precise language should be used to describe the aims of the study or research question, relating this to the study design and conclusions
• Methods should be described in enough detail that they could be independently reproduced – including the controls, reference standards and quality assurance measures used. This includes both study methods and methods for data analysis
• A clear statement should be provided describing how data (all direct outputs from a study, including both quantitative and qualitative results, and digital images used to support analysis and conclusions ) were cleaned (i.e. the detection and removal of incorrect or corrupt data points, duplicates or empty fields, and ensuring consistency of units and formatting), processed and analysed, and why such approaches were taken. Providing the underlying data wherever possible, offers opportunity for its independent assessment, which can improve confidence in the conclusions reached
• The conclusions of a study must be based on the evidence presented, with a clear narrative linking the data and analysis to those conclusions

Relevance

• To assess the relevance of a study to a particular issue, we will look at the context of the original study and the question(s) it was designed to answer. As key information about the way the study was conducted will be used to assess this, the clarity and transparency of the evidence are therefore important
• The study design and the methods used should be justified with reference to the original question or hypothesis – including how potentially confounding variables (any additional variable that influences both the supposed cause and supposed effect) were controlled for
• Consider the relevance of the study population, specimen or substance to the target population, specimen or substance. This is important when considering the biological relevance of a study and its conclusions. For example, the work on Biological Relevance and Statistical Significance led by the Committees on Toxicity, Carcinogenicity and Mutagenicity will explore this in further detail
• If the study is qualitative, a comprehensive description of the context of the work should be included. For example, the culture, livelihood, community, socio-economic status and environment of participants
• Statistical analysis is essential in scientific studies. Studies should include a clear outline of the methods used, and why they were chosen, with an explanation of what question the analysis aimed to answer. Statistical point estimates and confidence intervals are recommended alongside significance testing
• Where the evidence relates to a new method, outline the context in which the method should be used and why. Where relevant, make clear the advantages and drawbacks relative to validated methods

Reliability

• Where possible, methods recommended by national and international bodies such as the Food and Agriculture Organisation of the United Nations (FAO), the Organisation for Economic Co-operation and Development (OECD) and the Codex Alimentarius Commission, and recognised by national and international standardisation bodies should be used
• Good governance should be practiced when performing research. Refer to best practice guidelines such as the OECD’s Principles of Good Laboratory Practice
• Whether routine or not, all methods used should be referenced. If a standard method has been adapted, the study should state why and describe the differences. If a new method is proposed, a description of how it differs from the standard method(s), and where possible a comparative study should be provided
• All evidence must include consideration of uncertainty- the estimated sum of the limits in knowledge (we include limitations to apparatus, experimental techniques, models and study designs, as well as essential unpredictability). Where possible this should be quantified using recognised methods. If the uncertainty is associated with an expert judgement or population sample, state whether it is qualitative or quantitative, and how it was discerned
• Variability (the inherent heterogeneity between individuals or groups, or over time or space) must also be considered, and where possible quantified. Where variability has been controlled for in a study, consider if this affects generalisability to the target population, specimen, or substance
• If mathematical models are used, the results of the sensitivity analysis performed should be provided, stating which parameters were tested, which were not and why

Trust

Transparency and impartiality are key in providing confidence that evidence is trustworthy. Evidence that is shared transparently will include access to underlying data, a clear explanation of the methods used and why, and the limits to the evidence provided. This includes stating uncertainties, variability and assumptions, indicating where results differ from comparable investigations and where there is dissenting opinion among experts. Any evidence and its assessment are at risk of bias, but this can be mitigated by ensuring that sources of bias are recognised, peer review is performed, and challenge is built into the assessment process.

Transparency

• Openness and transparency are core principles of the way we work; evidence submitted to us should also demonstrate these principles as far as reasonably possible
• In addition to clearly presenting all relevant data and associated analysis, access to the raw and omitted data from the study, including negative results, should be provided wherever possible. If this is not possible, state why. We acknowledge that there may be legitimate commercial confidences and will respect these as far as reasonably possible
• Known gaps in the evidence should be stated and limitations to models or study designs outlined. This includes assumptions on what is or is not important for the question being asked, and therefore what has been included or excluded from the study or model design
• Consider alternative hypotheses and make comparisons to the published body of research on the area, stating where results differ or where there is disagreement in expert opinion
• Clearly indicate when evidence is compiled from a range of sources. Reference all sources and state the method used to compile the evidence, for example, using widely accepted guidelines for evidence synthesis such as meta-analysis and systematic review procedures

Impartiality and bias

• Increased risk of bias reduces the confidence in the outputs of a piece of evidence
• All potential sources of bias should be clearly described, considering each stage of the study and any actions taken to mitigate them should be stated. The sources of bias and appropriate mitigating actions will be dependent upon on the type of study being performed
• Where data are omitted from a study report, or where analysis is restricted to one or more subsets of the data available, this should be stated, with reasoning provided. If evidence is from a range of sources, the way in which sources were chosen or omitted should be given. This is consistent with the provision and reference to underlying data
• Where expert judgement is used, state why, how the experts were chosen and the initial question that was asked of them. Any underlying data or evidence that the judgement is based upon should be provided, and a statement of uncertainty should be included with the judgement
• If the evidence used is not published in a peer-reviewed journal, any independent critical review that has been performed should be described.
• In all instances, sources of funding and conflicts of interest must be stated

Robustness

For evidence to be robust, a broad body of evidence should be considered from several perspectives, with each piece of evidence weighed based on its quality and trustworthiness. The body of evidence will be made more robust if the pieces of evidence are reproducible using the same and different methods. If an outcome is consistently observed when tested using different methods and populations, this provides confidence that the outcome itself is robust.

Consistency

• Describe how tests were replicated and the extent of any variation in the observed results
• The clarity and transparency of a study, as well as the use of standard methods, reference standards and quality control methods can help ensure that a study can be repeated by other researchers
• If several independent studies are performed repeating the same or similar tests and gaining the same or similar outcomes, this will increase confidence in the outcome
• The robustness of an outcome can be tested by varying parameters within the study, and by using different methods to test the same relationship or outcome (triangulation). This may be done in a single study, or by comparing the outcomes of several studies

Adequacy

• Explain the importance of the evidence with reference to the broader body of evidence to which it contributes. Consider whether evidence highlights any gaps in the existing body of evidence, and how much it increases the understanding of a new or emerging area
• Different types of evidence may need to be combined for a comprehensive assessment of an issue to be undertaken. For example, as described in the Committees on Toxicity and Carcinogenicity’s guidelines on the synthesis and integration of epidemiological and toxicological evidence. Consider the other types of evidence that are required when assessing an issue and explain how your evidence relates to them
• The adequacy of a piece of evidence will vary depending on the type of study and the question being asked. However, criteria such as the magnitude of any effect, the power of a study, and its applicability to the target population, specimen or substance may be considered
• Significance testing is often used to indicate the magnitude of a result, but it is not by itself sufficient to indicate that a piece of evidence is strong or will translate to an important real-world impact. Consider the relevance of the study and the statistical test to the decision or policy that the evidence is being used to address