Consumer Insights Tracker Technical Report: Quality Assurance

Participant verification and engagement

Several checks are applied to ensure the quality of survey data. Checks for participant identification and authentication are conducted during panel registration, and through an initial welcome survey. Some of these checks include:

• IP checks, including IP blacklists – checks on respondent’s IP addresses to ensure they have not been blacklisted from completing surveys.
• Digital fingerprinting – information collected about the software and hardware of a device for the purpose of identification.
• Double keying on panel registration and login – used to check for respondent errors in data entry.
• Email address verification, and testing for uniqueness – prevents respondents from signing up to the YouGov panel using multiple accounts.
• Contact detail de-duplication – prevents respondents from signing up to the panel using the same contact details multiple times.
• Country validation – checks to ensure that respondents on the UK panel live in the UK.
• Location verification (e.g. post code) – crosschecks respondents against other location metrics to ensure that respondents are not signing up to the panel using erroneous postcodes.
• Machine Learning based cookie checks, and checks against Cookie blacklists – uses browser-based cookies (e.g. to prevent respondents from completing the same survey multiple times).
• Email confirmation and activation code – checks to ensure that respondents are using a valid email address.

Data cleaning and quality checks

Once the survey has closed, respondents that ‘straight line’, fail the attention check questions or have answered under a certain time period  ^{(footnote 1)} are removed from the data. Additionally, inconsistent, incoherent or incomplete responses are removed from the dataset in a process of data cleaning.

The cleaned data is used to produce data tables. The YouGov team carry out a thorough review of the data tables to check for accuracy. This includes:

• Checking that all questions from the survey have been included.
• Checking that the total sample size matches the target sample.
• Checking that base sizes and text for each question are correct.
• Checking that percentages for each question sum to 100% (if single code), and that ‘net’ figures have been calculated correctly.
• Checking that crossbreaks have been calculated correctly.
• Checking that the data has been weighted correctly.
• Comparing data with previous waves and investigating the reasons behind any large shifts in results to ensure they are not the result of data errors.

When outputs are provided to the FSA, the project team also conduct quality assurance checks before data is published; including spot checks against multiple data sources (for example, previous publications and data tables) and significance testing to ensure the same results can be found using alternative methods. Any discrepancies are highlighted and discussed with YouGov. Prior to publication, reports are reviewed and signed-off by at least two researchers at the FSA.