Familiarise yourself with different types of evidence, sources where users find evidence and the role of knowledge brokers

They also access evidence from brokers, as detailed further below. Table 1 below details some of the key types of evidence utilised by diet shift actors, along with some pros and cons, and insights from diet shift evidence users (that participated in the research project). 

Table 1 Key types of evidence utilised by diet shift actors

Source: Authors

Where do different evidence users source their evidence from?

There are no hard and fast rules on which users get evidence from where. But the insights below, drawn from both the evidence-use literature and directly from evidence users themselves, provide some pointers. Many evidence users will pull in multiple sources at once. 

Policymakers

• prefer a wide range of sources of information, combining their own experience with information ranging from peer reviewed scientific evidence and the ‘grey’ literature, to public opinion and feedback from consultation
• parliamentary committees cite academic research much less frequently than government, private sector and not-for-profit organisations
• local government officials rely more often on evidence from government, third sector organisations and think tanks than from universities ^{(footnote 2)}  

Commercial practitioners

• use peers and networks as an important evidence sources ^{(footnote 3)} 
• get evidence - including lay knowledge - from suppliers and internet sources (particularly SMEs in the hospitality sector) 
• of a larger size are likely to use a range of sources but evidence synthesis reports and webinars are seen as very useful
• are likely to use peer learning from conferences and bodies such as IGD and Kantar 
• have concerns about physical access to evidence, especially peer-reviewed journals

Third sector organisations

• get evidence from academia, other NGOs, and international sources
• often don’t have systems in place around evidence-use ^{(footnote 4)}
• can be constrained by funding requirements, including reporting, which shapes the types of evidence or evaluation employed
• use population-level data such as on demographics and income to justify the need for a specific programme or practice, and especially for funding applications
• use media as a source of knowledge on what is ‘trending’/ what people care about/ what is topical; and then compare that with government strategies and the ‘ethos’ of their organisation to select which actions to pursue
• may have an inherent suspicion of government commissioned research, particularly at grassroots community level, where it may be felt there is a hidden agenda behind the evidence

“It depends on what audience we're wanting to speak to. So we can pull in evidence at a national level, we can pull in academic evidence. So a story about a person to an elected member is much more powerful than what we'd say 'gold standard' evidence base. So it needs to be a combination of the two because you're trying to pull on different levers and use different players in the system because they all have an influence. It's like a whole-systems approach to evidence gathering. You need different bits to speak to different people. I think academics work really really well when you're talking to directors of public health and senior policymakers. Less so I think at a local level. I think, you know as I've said, stories make a difference. The experience of local organisations make a difference. It depends really.” – Regional Public Health Network

Understanding the role of credibility (and how to demonstrate it)

Evidence and knowledge that is seen as credible, or comes from a trusted source, is much more likely to be considered valuable and ultimately be adopted / implemented into policy and practice. Credibility and trust are particularly important for evidence users addressing healthy sustainable diets, due to a general perception that the evidence related to it is unclear and often inconsistent. Trusted sources could be:

• an individual or organisation that has a direct relationship with the practitioner (through a partnership, network or collaboration); or
• an individual or organisation that has a reputation for being independent, credible and / or respected. 

Credibility concerns both the evidence generated and the disseminator. For that reason, credibility stretches across both generation and translation. 

Understanding which evidence sources are credible can be challenging for users, and different evidence sources are associated with different credibility issues. For example, evidence from reports by campaigning organisations may be viewed as a mixture of science and ideology, which needs to be “cut through to get to the truth” ^{(footnote 5)}.  Different evidence users also have different perspectives of what credibility is. For some it may be about scientific credibility, whereas for others real-world credibility is more of a priority. An example is the International Panel for Climate Change, which is hailed around the world as an example of robust scientific endeavour, but has also been criticised for focusing on scientific evidence at the expense of the kind of real-world evidence required to improve policy and practical action on climate change ^{(footnote 6)}.  These different perspectives on credibility are also relevant to how evidence-generators ‘frame’ their evidence: for example, in terms of translating uncertainty or complexity, or making policy recommendations. There are a series of steps which users can work through to evaluate an evidence source including:

• currency – when was the information published or posted and has it been updated at any point
• relevance – does the information relate to the users’ topic or answer their research questions? What is the intended audience and academic level?
• authority – legitimacy of author/s
• accuracy – is the information supported by evidence (references, research data) and can the information be verified in another source?
• purpose – is the information fact, opinion or propaganda. How objective is the information and are there any political, ideological, religious, cultural or personal biases evident in the source? ^{(footnote 7)}

Academic evidence is often associated with being more credible, because it goes through a robust process of peer review, though some evidence users are not always sure about the independence of this process, when particular journals are supportive of a certain school of thought. One of the ways that evidence users are recommended to ensure credibility of academic research, and avoid ‘cherry picking’ of evidence based on an author’s own bosses or interests, is to focus on systematic reviews. Systematic reviews of evidence aim to be exhaustive, and cover all of the available evidence on a particular issue or question as possible, using explicit methods, and may screen studies for quality ^{(footnote 8)}.  However, systematic reviews in the field of diet shift are relatively rare (discussed in Principle 1). Academic evidence-sources also suffer from a range of general barriers such as lack of access, which may be due to them being behind a paywall, or because users don’t have the capability to understand them. 

“Most of food guidance I think is generally met with a little bit of derision. People say, ‘well one day they said butter’s bad for you and then the next day it’s good for you’... there’s a lot of inconsistency.” – Food Bank Manager

“Organisations with good reputations tend to go down better than, say, quite strident campaigning organisations, which wouldn't necessarily go down quite as well with politicians.” – Regional Public Health Network

Evidence users would like an independent body to signpost and curate the evidence, and the International Panel For Climate Change is seen by some as a good model which could be applied to food more specifically. In the absence of such a body, existing knowledge brokers and other types of intermediaries who have a reputation with a particular evidence user group are the next best option. 

Methodology as a shorthand for credibility

Improving the credibility of your evidence can be achieved by ensuring the methods used to produce it are robust and clearly explained. Both policy and practice evidence users report that methodology can be an important signifier of credibility. Papers which are useful to policymakers are “explicit about methodologies, limitations and weaknesses” ^{(footnote 9)}.  Evidence users are also concerned about relevance, robustness (of generation, analysis and interpretation of evidence) and generalisability.

“This may sound obvious to writers from some scientific traditions but, for example, in many social sciences, very limited methods may be outlined in reputable journals. The technical part of any policy team should be trying to assess the strength of each bit of evidence used, whether via formal grading system as used in medical guidelines or more informally. Doing this without methodologies laid out is nearly impossible ^{(footnote 10)}.”  – Chief Medical Officer and former chief scientific advisor Sir Chris Whitty

Similarly, third sector evidence users examine methodology to determine how credible a piece of evidence is. 
“We look at the credibility of evidence and also when we commission we take a good look at the methodology behind the evidence. For example when looking at Life Cycle Analysis, has the team take into account systems boundaries, or doing consumer work, how many people and what type of people in which demographics?” – Large International NGO

Conversely, lack of robustness can mean evidence is weaponised to avoid taking action; those in favour of continuing with the status quo may argue the evidence isn’t ‘good’ enough to justify change. While you can’t stop people from arguing about methodology, being clear, open and confident in your methods can pre-empt any disputes. 

“...people make statements, and sort of broad statements, and it's very important that they're well-evidenced. Not necessarily for the people who are going to pick up and want to push it forward, but for the people who are going to try to stop it. If you do not evidence what you're talking about very well, it's almost handing your opposition a tool by saying 'well they've said this but it's come from here and that's just naff, you know, it's not on.” – Regional Public Health Network

Relationships as a route to trust

Building relationships is often recommended as a way of developing trust between evidence generators and users, increasing engagement and project credibility ^{(footnote 11)}, and researchers are encouraged to build diverse networks and contacts by taking advantage of informal channels such as coffee, lunchtime seminars and distributing research PDFs via email ^{(footnote 12)}.  Putting in the effort early on to build these relationships and sustaining them over time enables researchers to direct experience with the practical decision-making process and allows them to adapt to their audience more effectively ^{(footnote 13)}.  Knowledge brokers, such as What Works Centres and other brokers, can also support building trust and connecting generators and users. 

“We decided to partner with a particular research programme because we both trust them and they have got a strong track record in food systems work. They work differently to other researchers we have come across. They take time to understand our challenges and always regularly update us on the progress of the work. Also they give us time to ensure we can participate by giving us longer lead times to gain internal support and sign off. It is more of an equal partnership.” – Food Retailer

However, the networks that policymakers operate within can influence which evidence they access, and what kind of evidence they consider useful or valid. This can be problematic for evidence generators outside those favoured networks, or with different perspectives to the current paradigm ^{(footnote 14)}.  Relationship-building activities also require a major investment and skills, which some evidence generators, in particular researchers earlier on in their career, may not have access to ^{(footnote 15)}.  

Importance of evidence brokers

Brokers are intermediaries between evidence generators and evidence users. Brokers therefore play an important role in translating and disseminating evidence for users. Including these actors in your dissemination strategy could improve the chances your evidence reaches users, and in a format they find accessible and credible. 

Brokers are used by all different user groups, who have their own particular types and favoured organisations. Brokers are a particularly important part of how academic evidence and expertise enter policy. For example, surveys have revealed that policy officials, including at national and local government levels, and Parliament, use brokers such as media organisations and think tanks more often than going directly to academics. Parliamentary committees cite government, private sector and not-for-profit organisations much more often than academic evidence, and local government officials rely more often on evidence from government, third sector organisations and think tanks than from universities ^{(footnote 16)}.  The Parliamentary Office for Science & Technology is an important dedicated source of evidence for UK parliament ^{(footnote 17)}, along with parliamentary clerks and librarians. Evidence users, particularly those looking for ways to ensure the credibility of evidence, other than accessing via peer-reviewed journals, may prefer to access it through learned societies (such as the Royal Society, Royal Statistical Society, British Academy and others) ^{(footnote 18)}. 

“One of my favourite evidence reports was the Future Farming and Environment Evidence Compendium, which brought together a range of evidence sources including academic, quality think tanks and select committee reports to provide some very useful data analysis on the state of the food and farming sector in the UK. This allowed us to look at where the weak areas of our food system existed." – Former Policymaker

For practitioners, whether they be professional or commercial, their relevant professional body is an important source of evidence. Such bodies can play a role in synthesising, translating and disseminating evidence for their networks, based on what they judge their members require to take effective action on the ground. There are many different food industry trade associations which play a role in both producing and translating evidence for their members. Networks relevant in the third sector include: Sustain: the Alliance for Better Food and Farming; the Sustainable Food Places Network; the Trussell Trust and the Independent Food Aid Network. 

Practical example: Participant Recommendations

The following are examples, directly sourced from evidence users, of what has worked well in their experience:

• “We like the launch of a report offering new insights that has synthesised complex evidence that is combined with a webinar. You don’t have to read the full report you can just jump onto a one hour webinar to get the evidence summary and new insights. A good example was the launch of the OECD report Making Better Policies in Food Systems, which is over 200 pages long. They launched the report and in partnership with academic group N8 Agrifood presented a webinar with insights from responders and for a retailer it was so useful the whole webinar. This approach saves us a lot of time.” ~ Food Retailer
• The British Nutrition Foundation was mentioned by several research participants, as producing useful summaries, by a team of trained professionals. “They also organise really good webinars online where you can jump on for an hour and really bring yourself up to date quickly.” ~ Food Retailer
• Joint working between a UK Government Department and the SysRisk research team (one of the COVID-19 grants) has built a new way of co-creating research and a new protocol tool to help other government departments identify systemic risks. Trust was built initially by co-designing the research proposal and then there was an equal partnership in the research process which built further trust. For example, both parties would present at research workshops with stakeholders. (Systemic Environmental Risk: process to appraise interventions for complex risks Final Report and Presentation)

Checklist

• are you aware of the pros and cons of different types of evidence?
• have you factored in that users source different types of evidence and do you understand why they use it?
• do you know which kinds of evidence are seen as credible by different users?
• can you utilise knowledge brokers or other intermediaries to add credibility to your evidence?
• have you demonstrated credibility through using and detailing methods which are robust and clearly explained?
• are you working to establish trusted relationships with users and are the resources required for this available to you?
• have you identified the evidence brokers that can be used to reach particular actors? 
• is it possible to disseminate your evidence via a trusted scientific body?