Developing rapid and effective communications testing: background and methodology

Background 

The FSA wanted to identify features that make their communications most effective. This report presents the learnings from the pilot, which can be used to aid the development of future communications. 

This report is split into five sections: 

1. Executive summary
2. How the pieces of communication landed: including initial reactions and engagement.
3. Reputation indicators: exploring how exposure to materials impacts awareness, familiarity, favourability, and trust in the FSA.
4. Topic specific indicators: exploring the impact exposure to materials has on people’s familiarity and attitudes towards the specific topics covered by them.
5. Testing different best before/use by dates messages: findings from the AB testing using Ipsos DUEL. 

Methodology 

A total of 43 pieces of communications were tested; 27 existing FSA messages using Ipsos Creative Testing (26 through i:Omnibus and one through FastFacts); and 16 different executions of the same message using Ipsos DUEL. The communications tested covered a mix of topics, formats, objectives, channels and stages of message development (see this report’s annex for exact details). 

Ipsos piloted the following tools: 

• i:Omnibus: Ipsos’ standard surveying approach using our online panel to access of a representative sample of 2,000 adults aged 16-75 in the UK. 
• FastFacts: This tool is very similar to the i:Omnibus approach, using the same questionnaire. However, it is more flexible, as it is a stand-alone survey as opposed to an Omnibus and provides quicker result to test during a crisis comms, for example. 
• Ipsos DUEL: A standardised survey tool which aims at measuring front of mind preferences between pieces of material and infer what specific words/ messages/images land best. Participants quickly pick between different stimuli and were asked to select the ones that they found most helpful in knowing when milk is safe to drink.  

Sample

• testing through i:Omnibus was split between two waves of fieldwork. The first wave achieved a sample of 2,243 adults aged 16-75 in the UK, the second wave achieved a sample of 2,246. These samples were weighted to be representative of the UK population by key demographics. Each piece of material was shown to approximately 170 adults. 
• one piece of content was tested through FastFacts. This survey went to 500 adults in the UK aged 18-65, representative by age, gender and region. 
• 16 pieces of content were tested through Ipsos DUEL. This survey went to 150 in the UK aged 18-65.

Note on interpreting the data 

Points to consider when reading this report:

• all pieces of content were shown to the general public as a whole (as opposed to targeting different demographic groups).
• as this testing took a split sample approach, the sample sizes are small (outlined above). 
• averages presented from the i:Omnibus and FastFacts testing are based across all 27 pieces tested through these methods unless stated otherwise.
• shifts shown through the i:Omnibus and FastFacts testing are calculated by subtracting the pre percentages from the post percentages (for example, percentage points). Participants were asked key metrics before being exposed to the pieces of communication (i.e. “pre measures”) and then again afterwards to measure its impact (“post measures”).
• where differences are highlighted, they have been statistically tested. Where we comment on “shifts” in opinion before and after being shown materials, we have focused on the pieces which saw the largest/smallest shifts for the relevant measure that is being reported. 
• data tables are provided in the appendix. These show the largest/smallest shift for each measure.