Food and You 2: Technical Report: Questionnaire development and cognitive testing

Questionnaire design

Food and You 2 uses a sequential mixed-mode approach involving an initial online stage, with non-respondents then followed up using a postal questionnaire. Therefore, the questionnaire is designed in such a way that it can be presented online and on paper. Like many other push-to-web surveys, the online version of the questionnaire is too long and complex to translate into an equivalent self-completion questionnaire suitable for postal administration. This means there are some differences between the online and postal questionnaires. To help address this limitation, in most waves up to two versions of the postal questionnaire are developed for each country, thereby enabling more questions to be asked across the sample as a whole. However, even with two versions of the postal questionnaire, there is insufficient space to include some of the online questions.

Given the wide range of topic areas that the FSA and external stakeholders are interested in investigating, the issue of questionnaire length is considered throughout the questionnaire development period. Ipsos recommends that, in the interest of reducing drop-out rates, the online questionnaire should not take longer than 30 minutes for the average participant to complete and the postal questionnaires should not be more than 20 pages in length. This time limit for the online survey and page limit for the postal survey were recommended to minimise the risk of participants not completing the survey, and to minimise the risk of straight-lining (i.e. selecting the same answer consistently) when going through the survey.

A modular approach is required for Food and You 2 to keep the length of the survey to a maximum of 30 minutes, and to minimise the likelihood of participants starting but not completing the survey. It also maximises coverage of topics and allows for new modules or questions to be added on emerging topic areas. When developing the Food and You 2 Wave 1 questionnaire, the topic areas the FSA were interested in were grouped into broad ‘modules’ (such as food shopping, food concerns or food we can trust). These modules were then assessed for frequency of fielding (6 months, 12 months or 24 months). For instance, attitudinal questions that are used to measure the FSA’s performance (e.g. trust in the FSA) or where fluctuations over time are more likely (e.g. concerns with food) were considered to be ‘core’ and therefore collected every 6 months. Whereas behavioural questions (e.g. on food practices in the home) that were relatively stable over time in previous studies were deemed to be appropriate for fielding less frequently.

Questionnaire development draws upon the work done for previous waves. The development for Wave 1 involved questionnaire development workshops, cognitive testing, usability testing and a pilot (covered in more detail in the Wave 1 Technical Report). The questionnaire development for Wave 2 onwards was shorter as core questions and materials had been developed in Wave 1. When newly developed questions are added to the survey, a phase of cognitive testing is held to test consumer understanding. For waves with few new questions, cognitive testing is not conducted.

Design of questions

The content and nature of the questions is informed by previous research conducted by the FSA, the FSA and stakeholders’ research priorities, and by Ipsos’ prior experience in survey research.

In Wave 1 a prolonged period of questionnaire development took place which involved an extensive review of questions from previous FSA surveys (Food and You and Public Attitudes Tracker). After all relevant questions were compiled, a workshop with the Food and You 2 advisory group was held to discuss key priorities for the questionnaire. This was followed by a second workshop with key internal stakeholders to discuss their priorities for the questionnaire and provide Ipsos with direction regarding questionnaire content.

Following this, draft questionnaire modules were compiled based on questions from previous FSA surveys. Numerous alterations to the wording, ordering, format and content of the questions were made in the process based on survey design best practice, with additional questions designed based on stakeholder needs.

To determine content for the questionnaire for each wave, meetings were held between Ipsos, the FSA and key stakeholders to discuss research priorities and to decide which questions from the online questionnaire should be included in the postal questionnaires.  

To enable comparability of the data between waves, questions carried over from earlier waves are largely kept consistent in wording and format. For the exceptions, see the ‘Differences between Waves’ sheets in the Tables User Guide for each wave.

Cognitive testing

In social and market research, cognitive testing refers to a form of qualitative data collection in which participants are asked by an interviewer to examine a set of materials and explain their understanding of them. In questionnaire development, cognitive testing interviews are used to evaluate how participants approach a questionnaire so that any issues regarding participant comprehension may be highlighted.

Following the completion of the first questionnaire draft, a series of cognitive testing interviews are arranged to test a sub-set of questions from the questionnaire, specifically those new or modified for that wave. Cognitive interviews are conducted with members of the public, including some conducted in the Welsh language. During recruitment participants are screened on age, gender, ethnicity, geographical region, employment status, income and, depending on the focus of the topics, whether or not they have a food allergy, intolerance or Coeliac disease. This ensures people with relevant food behaviours and habits are spoken to, which is important for assessing the questions.

Key aims of the cognitive testing are:

• to gauge the simplicity of questions and participant comprehension of key terms;
• to note any ambiguity in the interpretation of the questions; and
• to identify any questions that may not produce meaningful data.

The Welsh language interviews also help to evaluate the accuracy and clarity of the translations. 

Each cognitive interview is undertaken with a single participant, lasts approximately one hour, and is conducted by a moderator using online video conferencing software. During each interview, the moderator records the participant’s answers and notes further observations regarding how the participant interprets the questionnaire, with attention paid to any problems encountered. The English language interviews are conducted by moderators from Ipsos, while the Welsh language interviews are conducted by a trusted external qualitative researcher. Some of the interviews are conducted in the (virtual) presence of an observer from the FSA.

Following completion of the interviews, Ipsos submits a written report to the FSA detailing the findings with recommendations. An extended meeting is subsequently held to discuss the findings and agree on further edits to the questionnaire.

The specific number of cognitive interviews conducted each wave are shown in the accompanying technical report spreadsheet.

Postal questionnaire design and modular approach

The postal questionnaires consist of a selection of questions from the online survey. The full questionnaire is not included in the postal versions due to concerns regarding questionnaire length.

Questions are selected for inclusion in the postal questionnaires based on a number of factors. For instance, questions that are a key strategic measure for the FSA (e.g. trust in the FSA) are included to provide the FSA with robust data. Questions are also included to maximise the base sizes for specific groups of interest (e.g. participants with food allergies). Finally, questions where the mode of delivery and sample profile may impact the data collected, for example questions on food security. It is important to include the majority of the demographic questions in the postal survey to enable subgroup analysis.

As with the online questionnaire, there are country-based differences in the wording of a small number of questions. Participants in Wales are sent copies of the questionnaires in English and in Welsh.

As noted, the survey is conducted using a modular approach. Certain ‘core’ modules are included in each biannual survey wave, while others are rotated every 12 or 24 months. The content of the survey for each wave is noted in the Appendices.