Neidio i’r prif gynnwys
English Cymraeg
Guiding Principles for translating evidence on diet shift for people in the real world

Familiarise yourself with different types of evidence, sources where users find evidence and the role of knowledge brokers

There are multiple types of evidence which policymakers and practitioners use, and different users have a tendency to favour, or find useful, particular types.

Diweddarwyd ddiwethaf: 16 November 2022
Gweld yr holl ddiweddariadau
Diweddarwyd ddiwethaf: 16 November 2022
Gweld yr holl ddiweddariadau

They also access evidence from brokers, as detailed further below. Table 1 below details some of the key types of evidence utilised by diet shift actors, along with some pros and cons, and insights from diet shift evidence users (that participated in the research project). 

Table 1 Key types of evidence utilised by diet shift actors

Evidence type Details Pros and Cons Insight from evidence users
Academic Research conducted to create new knowledge 
Includes new primary research or synthesis of existing research
Follows a scientific method.
Research findings are often published scientific journals, following a peer-review process 
Often accessed via direct relationships with experts/ academics rather than peer-reviewed journals 
For example: partnerships, advisory groups, commissions, collaborations or networks 
Pros
Peer review process offers reassurance of quality
Cons
May be difficult to access due to paywalls 
Technical jargon used may be difficult to understand
May address only part of a problem (requiring additional synthesis)
Sometimes academics are proponents of a particular school of thought
May not be quick or responsive enough for practice
May fail to clarify actions, for example 'so what' and how can this work for me? 
“Although there is a lot of academic evidence out there, most companies will not be resourced to read academic literature / follow debates and so the actions that companies need to take are not clear”. – Food Industry Representative
"[what is considered credible] is going to vary with audience but, generally speaking, high quality academic papers. And if you're talking about policymakers, it's going to be named -- if somebody's coming from the Lancet it's going to be taken quite seriously for example." – Public Health Representative
“Our main sources of information, I would say, are the academic published literature on food systems quite broadly, so looking across production and consumption.” – NGO 
Reports by non-academic organisations, such as governments, non-governmental organisations, professional bodies  May be labelled ‘grey literature’ (to distinguish it from academic research)
Definition of grey literature: "Information produced on all levels of government, academia, think tanks, business and industry in electronic and print formats not controlled by commercial publishing, for example, where publishing is not the primary activity of the producing body. (footnote 1)"  
Pros
Speedier review process means likely to be in the public domain quicker
May produce evidence on niche or emerging research areas that are not (yet) addressed in academic publications (due to longer process)
Cons
May combine evidence with ideology of the organisation 
Grey literature sources can vary hugely in terms of quality
Not always subject to same peer-review rigour
“There are some good reports from campaign organisations for example, WWF but you have to understand these type of reports are often a mixture of good science and ideology. You have to cut through this to get to the truth so we do use a range of sources from UKRI, Innovate UK etc. where the evidence has been sanitised for the end user.” – Food Retailer 
“We bring in organisations like Sustain. So we've worked with Sustain a couple pieces of work and they have got a really good reputation." – Regional Public Health Network
"Reports that come from PHE [are seen as credible]." – Regional Public Health Network
“The sources of evidence we have used have been reports such as WWF Live well Sustainable Diets Report and the EAT Lancet Report. We also use Kantar Data for our own retailer brand, which measures what customers are buying so we can also measure the journey of our shoppers also towards healthy sustainable diets. We also go to conferences and therefore use a combination of internal (Kantar data) and external evidence.” – Food Retailer
“We also take an active interest in reports coming from other NGOs. We have frequent conversations with lots of NGOs, to keep abreast of work coming out of that sector.” – NGO 
Organisational Research Evidence created by an organisation, primarily for its own usage.
Includes focus groups, rapid evidence reviews, customer surveys, trials of interventions.
Common in the private/ commercial sector, and third sectors
Includes formal evaluations and other methods organisations use to monitor their success
Pros
Faster for organisations
Important for confidential projects that may involve a new innovation where there are trademark issues
Cons
Organisation specific and may involve a commercial advantage, or not be deemed appropriate for external audience, meaning much is not published
Evaluations may be biased 
-
Intermediaries such as media organisations and think tanks Media is a source of knowledge to identify what is trending/ what people care about / what is topical Pros
May alert users to evidence from research projects which they would not have noticed via academic publishing outlets
May provide synthesis function
Cons
May have own (biased) agenda and frame the evidence in a particular way
“Media is a source of knowledge to identify what is ‘trending’/ what people care about / what is topical; which can then be compared to current government strategies and the ‘ethos’ of your organisation to select which actions to pursue.” – NGO 
“One of my slides tomorrow is a picture of a local newspaper – the Hull Daily Mail – of a very obese kid in 2010. And, you know, front page of the Hull Daily Mail.” – Local Food Partnership
Published Database Statistics Databases of statistics

Pros

Credible

Free to use

Useful to triangulate data with primary insights

Cons

Users may not have skills to access and interpret the data 
May not include the right kind of data/evidence required (for example the right level of granularity on location, demographics)

“We use evidence to support us to make the case for need…such as National Child Measurement Program data… sort of national statistic on school meals or something like that, so evidence that is actually telling us that the case is there to do something, do something in a certain way, make changes.” – NGO 
Lay evidence/Tacit knowledge

Although not strictly an evidence source, lay knowledge is an important evidence source for users
The ‘common sense’ justification of feeding people healthy food is also a key driver of practice decisions

Pros
Can lead to some quick wins regarding the need to make clear changes 
Cons
Likely to be influenced by personal biases
Unlikely to be representative
May not take a holistic view, meaning potential unintended consequences 
For example: considering the impact of a particular health action have on sustainability? 
 
“You’ve got an obese mom, you’ve got an obese dad, you’ve got both of them obese, you know the risk of that kid is going to be strongly likely to become overweight and obese. You know, what can we do to stop that happening? Because that’s where you can put in the intervention that works. Surely. You know, it’s kind of the bleeding obvious. I’m a layman, not an academic, but that’s the sort of evidence I’d say was available...” – Local Food Partnership

Source: Authors

Where do different evidence users source their evidence from?

There are no hard and fast rules on which users get evidence from where. But the insights below, drawn from both the evidence-use literature and directly from evidence users themselves, provide some pointers. Many evidence users will pull in multiple sources at once. 

Policymakers

  • prefer a wide range of sources of information, combining their own experience with information ranging from peer reviewed scientific evidence and the ‘grey’ literature, to public opinion and feedback from consultation
  • parliamentary committees cite academic research much less frequently than government, private sector and not-for-profit organisations
  • local government officials rely more often on evidence from government, third sector organisations and think tanks than from universities (footnote 2)  

Commercial practitioners

  • use peers and networks as an important evidence sources (footnote 3) 
  • get evidence - including lay knowledge - from suppliers and internet sources (particularly SMEs in the hospitality sector) 
  • of a larger size are likely to use a range of sources but evidence synthesis reports and webinars are seen as very useful
  • are likely to use peer learning from conferences and bodies such as IGD and Kantar 
  • have concerns about physical access to evidence, especially peer-reviewed journals

Third sector organisations

  • get evidence from academia, other NGOs, and international sources
  • often don’t have systems in place around evidence-use (footnote 4)
  • can be constrained by funding requirements, including reporting, which shapes the types of evidence or evaluation employed
  • use population-level data such as on demographics and income to justify the need for a specific programme or practice, and especially for funding applications
  • use media as a source of knowledge on what is ‘trending’/ what people care about/ what is topical; and then compare that with government strategies and the ‘ethos’ of their organisation to select which actions to pursue
  • may have an inherent suspicion of government commissioned research, particularly at grassroots community level, where it may be felt there is a hidden agenda behind the evidence

“It depends on what audience we're wanting to speak to. So we can pull in evidence at a national level, we can pull in academic evidence. So a story about a person to an elected member is much more powerful than what we'd say 'gold standard' evidence base. So it needs to be a combination of the two because you're trying to pull on different levers and use different players in the system because they all have an influence. It's like a whole-systems approach to evidence gathering. You need different bits to speak to different people. I think academics work really really well when you're talking to directors of public health and senior policymakers. Less so I think at a local level. I think, you know as I've said, stories make a difference. The experience of local organisations make a difference. It depends really.” – Regional Public Health Network

Understanding the role of credibility (and how to demonstrate it)

Evidence and knowledge that is seen as credible, or comes from a trusted source, is much more likely to be considered valuable and ultimately be adopted / implemented into policy and practice. Credibility and trust are particularly important for evidence users addressing healthy sustainable diets, due to a general perception that the evidence related to it is unclear and often inconsistent. Trusted sources could be:

  • an individual or organisation that has a direct relationship with the practitioner (through a partnership, network or collaboration); or
  • an individual or organisation that has a reputation for being independent, credible and / or respected. 

Credibility concerns both the evidence generated and the disseminator. For that reason, credibility stretches across both generation and translation. 

Understanding which evidence sources are credible can be challenging for users, and different evidence sources are associated with different credibility issues. For example, evidence from reports by campaigning organisations may be viewed as a mixture of science and ideology, which needs to be “cut through to get to the truth” (footnote 5).  Different evidence users also have different perspectives of what credibility is. For some it may be about scientific credibility, whereas for others real-world credibility is more of a priority. An example is the International Panel for Climate Change, which is hailed around the world as an example of robust scientific endeavour, but has also been criticised for focusing on scientific evidence at the expense of the kind of real-world evidence required to improve policy and practical action on climate change (footnote 6).  These different perspectives on credibility are also relevant to how evidence-generators ‘frame’ their evidence: for example, in terms of translating uncertainty or complexity, or making policy recommendations. There are a series of steps which users can work through to evaluate an evidence source including:

  • currency – when was the information published or posted and has it been updated at any point
  • relevance – does the information relate to the users’ topic or answer their research questions? What is the intended audience and academic level?
  • authority – legitimacy of author/s
  • accuracy – is the information supported by evidence (references, research data) and can the information be verified in another source?
  • purpose – is the information fact, opinion or propaganda. How objective is the information and are there any political, ideological, religious, cultural or personal biases evident in the source? (footnote 7)

Academic evidence is often associated with being more credible, because it goes through a robust process of peer review, though some evidence users are not always sure about the independence of this process, when particular journals are supportive of a certain school of thought. One of the ways that evidence users are recommended to ensure credibility of academic research, and avoid ‘cherry picking’ of evidence based on an author’s own bosses or interests, is to focus on systematic reviews. Systematic reviews of evidence aim to be exhaustive, and cover all of the available evidence on a particular issue or question as possible, using explicit methods, and may screen studies for quality (footnote 8).  However, systematic reviews in the field of diet shift are relatively rare (discussed in Principle 1). Academic evidence-sources also suffer from a range of general barriers such as lack of access, which may be due to them being behind a paywall, or because users don’t have the capability to understand them. 

“Most of food guidance I think is generally met with a little bit of derision. People say, ‘well one day they said butter’s bad for you and then the next day it’s good for you’... there’s a lot of inconsistency.” – Food Bank Manager

“Organisations with good reputations tend to go down better than, say, quite strident campaigning organisations, which wouldn't necessarily go down quite as well with politicians.” – Regional Public Health Network

Evidence users would like an independent body to signpost and curate the evidence, and the International Panel For Climate Change is seen by some as a good model which could be applied to food more specifically. In the absence of such a body, existing knowledge brokers and other types of intermediaries who have a reputation with a particular evidence user group are the next best option. 

Methodology as a shorthand for credibility

Improving the credibility of your evidence can be achieved by ensuring the methods used to produce it are robust and clearly explained. Both policy and practice evidence users report that methodology can be an important signifier of credibility. Papers which are useful to policymakers are “explicit about methodologies, limitations and weaknesses” (footnote 9).  Evidence users are also concerned about relevance, robustness (of generation, analysis and interpretation of evidence) and generalisability.

“This may sound obvious to writers from some scientific traditions but, for example, in many social sciences, very limited methods may be outlined in reputable journals. The technical part of any policy team should be trying to assess the strength of each bit of evidence used, whether via formal grading system as used in medical guidelines or more informally. Doing this without methodologies laid out is nearly impossible (footnote 10).”  – Chief Medical Officer and former chief scientific advisor Sir Chris Whitty

Similarly, third sector evidence users examine methodology to determine how credible a piece of evidence is. 
“We look at the credibility of evidence and also when we commission we take a good look at the methodology behind the evidence. For example when looking at Life Cycle Analysis, has the team take into account systems boundaries, or doing consumer work, how many people and what type of people in which demographics?” – Large International NGO

Conversely, lack of robustness can mean evidence is weaponised to avoid taking action; those in favour of continuing with the status quo may argue the evidence isn’t ‘good’ enough to justify change. While you can’t stop people from arguing about methodology, being clear, open and confident in your methods can pre-empt any disputes. 

“...people make statements, and sort of broad statements, and it's very important that they're well-evidenced. Not necessarily for the people who are going to pick up and want to push it forward, but for the people who are going to try to stop it. If you do not evidence what you're talking about very well, it's almost handing your opposition a tool by saying 'well they've said this but it's come from here and that's just naff, you know, it's not on.” – Regional Public Health Network

Relationships as a route to trust

Building relationships is often recommended as a way of developing trust between evidence generators and users, increasing engagement and project credibility (footnote 11), and researchers are encouraged to build diverse networks and contacts by taking advantage of informal channels such as coffee, lunchtime seminars and distributing research PDFs via email (footnote 12).  Putting in the effort early on to build these relationships and sustaining them over time enables researchers to direct experience with the practical decision-making process and allows them to adapt to their audience more effectively (footnote 13).  Knowledge brokers, such as What Works Centres and other brokers, can also support building trust and connecting generators and users. 

“We decided to partner with a particular research programme because we both trust them and they have got a strong track record in food systems work. They work differently to other researchers we have come across. They take time to understand our challenges and always regularly update us on the progress of the work. Also they give us time to ensure we can participate by giving us longer lead times to gain internal support and sign off. It is more of an equal partnership.” – Food Retailer

However, the networks that policymakers operate within can influence which evidence they access, and what kind of evidence they consider useful or valid. This can be problematic for evidence generators outside those favoured networks, or with different perspectives to the current paradigm (footnote 14).  Relationship-building activities also require a major investment and skills, which some evidence generators, in particular researchers earlier on in their career, may not have access to (footnote 15).  

Importance of evidence brokers

Brokers are intermediaries between evidence generators and evidence users. Brokers therefore play an important role in translating and disseminating evidence for users. Including these actors in your dissemination strategy could improve the chances your evidence reaches users, and in a format they find accessible and credible. 

Brokers are used by all different user groups, who have their own particular types and favoured organisations. Brokers are a particularly important part of how academic evidence and expertise enter policy. For example, surveys have revealed that policy officials, including at national and local government levels, and Parliament, use brokers such as media organisations and think tanks more often than going directly to academics. Parliamentary committees cite government, private sector and not-for-profit organisations much more often than academic evidence, and local government officials rely more often on evidence from government, third sector organisations and think tanks than from universities (footnote 16).  The Parliamentary Office for Science & Technology is an important dedicated source of evidence for UK parliament (footnote 17), along with parliamentary clerks and librarians. Evidence users, particularly those looking for ways to ensure the credibility of evidence, other than accessing via peer-reviewed journals, may prefer to access it through learned societies (such as the Royal Society, Royal Statistical Society, British Academy and others) (footnote 18)

“One of my favourite evidence reports was the Future Farming and Environment Evidence Compendium, which brought together a range of evidence sources including academic, quality think tanks and select committee reports to provide some very useful data analysis on the state of the food and farming sector in the UK. This allowed us to look at where the weak areas of our food system existed." – Former Policymaker

For practitioners, whether they be professional or commercial, their relevant professional body is an important source of evidence. Such bodies can play a role in synthesising, translating and disseminating evidence for their networks, based on what they judge their members require to take effective action on the ground. There are many different food industry trade associations which play a role in both producing and translating evidence for their members. Networks relevant in the third sector include: Sustain: the Alliance for Better Food and Farming; the Sustainable Food Places Network; the Trussell Trust and the Independent Food Aid Network. 

Practical example: Participant Recommendations

The following are examples, directly sourced from evidence users, of what has worked well in their experience:

  • “We like the launch of a report offering new insights that has synthesised complex evidence that is combined with a webinar. You don’t have to read the full report you can just jump onto a one hour webinar to get the evidence summary and new insights. A good example was the launch of the OECD report Making Better Policies in Food Systems, which is over 200 pages long. They launched the report and in partnership with academic group N8 Agrifood presented a webinar with insights from responders and for a retailer it was so useful the whole webinar. This approach saves us a lot of time.” ~ Food Retailer
  • The British Nutrition Foundation was mentioned by several research participants, as producing useful summaries, by a team of trained professionals. “They also organise really good webinars online where you can jump on for an hour and really bring yourself up to date quickly.” ~ Food Retailer
  • Joint working between a UK Government Department and the SysRisk research team (one of the COVID-19 grants) has built a new way of co-creating research and a new protocol tool to help other government departments identify systemic risks. Trust was built initially by co-designing the research proposal and then there was an equal partnership in the research process which built further trust. For example, both parties would present at research workshops with stakeholders. (Systemic Environmental Risk: process to appraise interventions for complex risks Final Report and Presentation)

Checklist

  • are you aware of the pros and cons of different types of evidence?
  • have you factored in that users source different types of evidence and do you understand why they use it?
  • do you know which kinds of evidence are seen as credible by different users?
  • can you utilise knowledge brokers or other intermediaries to add credibility to your evidence?
  • have you demonstrated credibility through using and detailing methods which are robust and clearly explained?
  • are you working to establish trusted relationships with users and are the resources required for this available to you?
  • have you identified the evidence brokers that can be used to reach particular actors? 
  • is it possible to disseminate your evidence via a trusted scientific body?