Consumer Insights Tracker Technical Report: Strengths and Limitations

Strengths of approach

The FSA’s project team carefully weigh up the strengths and limitations of all methodologies before deciding the most appropriate methodology for an individual piece of research.

The approach for the Consumer Insights Tracker offers several strengths including:

• Reliable, representative tracking: A large, nationwide online panel, enables data to be gathered that is representative of the population aged 16 years and over in England, Wales and Northern Ireland, to track changes over time and to collect a wide range of demographic information. A boosted sample of 100 is collected in Northern Ireland which allows for the comparison of each country across England, Wales and Northern Ireland.
• Speed of delivery: An online panel approach means that the survey can be administered to participants quickly, particularly in comparison to other modes such as telephone or face-to-face methods. As participants are pre-screened for eligibility for the panel, this means that surveys can be administered quicker than methods that require recruitment and screening processes. In addition, online surveys can be completed at any time of day (according to respondent preference).
• Cost effectiveness: The combination of speedier delivery and a less resource-intensive approach by eliminating the need for human interviewers contribute to online surveys being significantly more cost effective than other survey methodologies. In addition, because YouGov already holds many of the key demographic indicators on panellists, this saves questionnaire space (and therefore survey cost) by not needing to ask these questions.
• Flexibility in survey design: Online surveys provide greater flexibility in terms of the types of questions which can be included. Questions involving images, videos or audio clips can be included, which is not always possible when using a telephone methodology (for example).
• Reduction in certain forms of bias: As online surveys do not require an interviewer to be present, they can help to reduce the effects of social desirability bias among respondents. Sampling bias can also be reduced online, compared to face-to-face approaches, where the interviewer may be more restricted by location when conducting the survey, although other forms of sampling bias may be present in online surveys (see limitations below). Additionally, using a bespoke survey approach means respondents are only asked the questions in the Consumer Insights Tracker, reducing certain types of bias associated with approaches where respondents are asked to complete multiple surveys.  
• Exclusion period: Participants are excluded from taking the survey again for 6 months. Having a long exclusion period means they are less likely to become familiar with survey questions, or to experience survey fatigue. It also allows up to 6 months of data to be combined, to provide a large base size for further analysis opportunities without the risk of duplicated participants.

Limitations of approach 

Some of the limitations of the approach include:

• Opt-in sampling bias: Panellists on opt-in panels can be more engaged in current issues, which means they are more motivated to opt-into panel surveys. This means precise measures for some attitudes and behaviours reflecting the total population can be difficult to obtain. It is therefore beneficial to regularly validate and compare the results of online opt-in surveys using more robust methodologies such as random probability surveys. The FSA often use multiple methodologies to understand the complexities of a single topic.
• Representation: Some groups are underrepresented in online opt-in panels. This is particularly true of those who do not have internet access (who are excluded) or those who are not comfortable going online, who also tend to be older and in lower socio-economic groups. Additionally, young working-class men are difficult to reach across many methodologies, including opt-in online panels. These issues can be remedied in part through quotas and weighting, if the groups being underrepresented are identified. Other methodologies, such as postal or face-to-face surveys, can be more suitable when capturing the views of those who are digitally excluded, but these methodologies typically take longer to deliver and are more expensive. 
• Topics and complexity: Online surveys, and in general surveys that are self-administered, are useful for an overall read on issues or for exploring topics that are well understood by respondents. However, it becomes more difficult to explore in depth, complex issues where respondents do not have a good understanding of the topic. Such topics can benefit from an interviewer who can provide explanations or are better explored through qualitative research.
• Detailed demographic analysis: Due to sample size, it is not always possible to review detailed demographic cross-breaks, such as region or granular details of income with the Consumer Insights Tracker which has a sample of approximately 2,000 per month. This only allows for analysis by broad demographic characteristics such as gender, age group, income group, or the presence of children in the household. However, by combining six months of data, a sample size of 12,000 is created. This greater sample size increases the number of individuals of each demographic cross-break, thus allowing the comparison of those that do not meet the required number in the monthly figures.