Questionnaire development and cognitive testing

Questionnaire design

Food and You 2 uses a sequential mixed-mode approach involving an initial online stage, with non-respondents then followed up using a postal questionnaire. Therefore, the questionnaire was designed in such a way that it could be presented online and on paper. Like many other push-to-web surveys, the online version of the questionnaire is too long and complex to translate into an equivalent self-completion questionnaire suitable for postal administration. This meant there were some differences between the online and postal questionnaires. To help address this limitation, two versions of the postal questionnaire were developed, thereby enabling more questions to be asked across the sample as a whole. However, even with two versions of the postal questionnaire, there was insufficient space to include some of the online questions.

Given the wide range of topic areas that the FSA and external stakeholders were interested in investigating, the issue of questionnaire length was considered throughout the questionnaire development period. Ipsos recommended that, in the interest of reducing drop-out rates, the online questionnaire should not take longer than 30 minutes for the average participant to complete and the postal questionnaires should not be more than 20 pages in length. This time limit for the online survey and page limit for the postal survey were recommended to minimise the risk of participants not completing the survey, and to minimise the risk of straight-lining (for example, selecting the same answer consistently) when going through the survey. 

A modular approach was required for Food and You 2 to keep the length of the survey to a maximum of 30 minutes, and to minimise the likelihood of participants starting but not completing the survey. It also maximised coverage of topics and allows for new modules or questions to be added on emerging topic areas. When developing the Food and You 2 Wave 1 questionnaire, the topic areas the FSA were interested in were grouped into broad ‘modules’ (such as food shopping, food concerns or food we can trust). These modules were then assessed for frequency of fielding (6 months, 12 months or 24 months). For instance, attitudinal questions that are used to measure the FSA’s performance (e.g. trust in the FSA) or where fluctuations over time are more likely (for example, concerns with food) were considered to be ‘core’ and therefore collected every 6 months. Whereas behavioural questions (for example, on food practices in the home) that were relatively stable over time in previous studies were deemed to be appropriate for fielding less frequently. The modules selected for inclusion in the Wave 4 questionnaire reflected this approach. 

Questionnaire development for Wave 4 drew upon the work done for Waves 1, 2 and 3. The development for Wave 1 involved questionnaire development workshops, cognitive testing, usability testing and a pilot (covered in more detail in the Wave 1 Technical Report). For Waves 2, 3 and 4 a shorter period of development was conducted, during which a phase of cognitive testing was held to test newly developed questions.

Design of questions

The content and nature of the questions was informed by previous research conducted by the FSA, the FSA and stakeholders’ research priorities, and by Ipsos’ prior experience in survey research.

Much of the content for the questionnaires had already been completed during Waves 1, 2 and 3 questionnaire development periods. To determine content for the Wave 4 questionnaire, meetings were held between Ipsos, the FSA and key stakeholders to discuss research priorities and to decide which questions from the online questionnaire should be included in the postal questionnaires. Two modules were repeated in Wave 4, from Wave 2: Eating Out and Eating at Home; and a new Emerging Issues module was introduced.
To enable comparability of the data between waves, questions carried over from Waves 1, 2 and 3 were kept consistent in wording and format.

Cognitive testing

In social and market research, cognitive testing refers to a form of qualitative data collection in which participants are asked by an interviewer to examine a set of materials and explain their understanding of them. In questionnaire development, cognitive testing interviews are used to evaluate how participants approach a questionnaire so that any issues regarding participant comprehension may be highlighted. 

Following the completion of the first questionnaire draft, a series of cognitive testing interviews were arranged to test a sub-set of questions from the questionnaire, specifically those new to Wave 4. The cognitive testing was intended to aid the development of the questionnaire by allowing Ipsos and the FSA to identify questions requiring further development. A total of 20 interviews were conducted with members of the public. Five of the interviews were conducted in the Welsh language. During recruitment participants were screened on age, gender, ethnicity, geographical region, employment status, whether or not they receive any benefits, diet type, and whether or not they have ever eaten meat alternatives. This ensured we spoke to people with relevant food behaviours and habits, which was important for assessing the questions.

Key aims of the cognitive testing included: 

• to gauge the simplicity of questions and participant comprehension of key terms
• to note any room for ambiguity in the interpretation of the questions
• to identify any questions that may not produce meaningful data.

The Welsh language interviews also aimed to evaluate the accuracy and clarity of the translations. 

Each cognitive interview was undertaken with a single participant, lasted approximately one hour, and was conducted by a moderator using online video conferencing software. During each interview, the moderator recorded the participant’s answers and noted further observations regarding how the participant interpreted the questionnaire, with attention paid to any problems encountered. The English language interviews were conducted by moderators from Ipsos, while the Welsh language interviews were conducted by a trusted external qualitative researcher. Some of the interviews were conducted in the (virtual) presence of an observer from the FSA.
Following completion of the interviews, Ipsos submitted a written report to the FSA detailing the findings. An extended meeting was subsequently held to discuss the findings and agree on further edits to the questionnaire.

Survey mailings

The survey was conducted in England, Wales and Northern Ireland using a push-to-web methodology, continuing the approach used in Waves 1, 2 and 3. As noted, push-to-web is a quantitative data collection method in which offline contact modes are used to encourage sample members to go online and complete an online questionnaire. 

The push-to-web methodology used in this survey mirrored a tried-and-tested methodology used by Ipsos in previous studies; a sequential mixed-mode approach in which participants are at first asked to complete an online survey, with non-respondents then followed up using a postal questionnaire at the third mailing. The rationale behind this methodology is that it brings the benefits of encouraging online survey completion while avoiding the exclusion of those who do not have access to the internet and/or have low levels of digital literacy.

In this study, the methodology consisted of a series of four mailings sent to selected households. The second, third and fourth mailings were only sent to households who had not responded to the survey since the previous mailing. Further details regarding the sampling approach are provided in the ‘Sampling’ section. The schedule of mailings is outlined below:

• Mailing 1: Initial invitation letter
• Mailing 2: First reminder letter
• Mailing 3: Postal questionnaire and second reminder letter
• Mailing 4: Final reminder

The first mailing invited recipients to complete the survey online. The letter invited two adults from each household to participate. Each participant was provided with a unique passcode allowing them to complete the questionnaire on the survey website. Those who did not complete the survey following receipt of the initial invitation letter were sent a reminder letter a few weeks following the mailout of the invitation.

The second mailing took the form of a reminder letter, again inviting participants to complete the online survey. In the third mailing, copies of the postal version of the questionnaire were sent alongside a letter instructing recipients how to complete and send back the postal questionnaire. Lastly, a final reminder letter was sent. Each mailing was separated by an interval of a few weeks. 

Postal questionnaire design and modular approach

The postal questionnaires consisted of a selection of questions from the online survey. The full questionnaire was not included in the postal versions due to concerns regarding questionnaire length. 

Questions were selected for inclusion in the postal questionnaires based on a number of factors. For instance questions that were a key strategic measure for the FSA (e.g. trust in the FSA) were included to provide the FSA with robust data. Questions were also included to maximise the base sizes for specific groups of interest (e.g participants with food allergies). Finally, questions where the mode of delivery and sample profile may have impacted on the data collected, for example questions on food security. It was important to include the majority of the demographic questions in the postal survey to enable subgroup analysis.

As with the online questionnaire, there were minor differences between England, Wales and Northern Ireland in the wording of a small number of questions. Participants in Wales were sent copies of the questionnaires in English and in Welsh. 

As noted, the survey was conducted using a modular approach. Certain ‘core’ modules were included in each biannual survey wave, while others were rotated every 12 or 24 months. The content of the survey for this wave is detailed in the section below. 

Overview of survey content

Introductory questions (core module)

In the online survey, this module began with a question asking for confirmation of age (as those under 16 years were not eligible to participate). This was followed by a small number of questions asking participants for some basic information about themselves and their household, such as their gender identity, and the number and age of any other household members. The module also asked participants whether they had a food allergy, food intolerance or Coeliac disease so that the questionnaire could be tailored to individuals. This module was included in Wave 1 and is kept unchanged between waves to enable comparability of subgroup trend data. 

Food hypersensitivities

This module began with a question asking participants whether there were any foods which caused them unpleasant physical reactions or which they avoided because of unpleasant physical reactions which the foods might cause. Those participants who answered ‘yes’ to this question were then asked a series of questions regarding the nature of the hypersensitivity and how they found out that they had the hypersensitivity. They were then asked details regarding any recent experiences of consuming the foods in question. Certain questions in subsequent modules were routed to those who stated in the Food Hypersensitivities module that they had a hypersensitivity. 

Eating out

In this module, participants were asked how often they eat out or buy food to take away and the factors they consider when choosing where to eat. Participants were also asked about their awareness and use of the Food Hygiene Rating Scheme (FHRS). 

Eating at home (core questions)

This module included a sub set of questions asked in the ‘full’ Eating at Home module included in Wave 1. It was intended to gauge participant knowledge of and adherence to the key FSA food safety and hygiene guidelines. Participants were asked about the ways in which they store, prepare, and consume food in the home. 

Emerging issues

A new module was introduced in Wave 4 focusing on emerging issues of interest to the FSA and its stakeholders. More specifically, the questions focused on sustainable shopping and dietary behaviours, meat alternatives and awareness of Genetically Modified (GM) and Genetically Edited (GE) food.

Food concerns (core module)

In this core module, participants were asked whether they had any concerns with the food they ate, followed by a spontaneous question asking them to give details on these. This was followed by questions which listed specific food concerns, prompting participants on the food concerns they may have.

Food You Can Trust (core module)

This core module gauged participant confidence in the food supply chain (including in farmers, food manufacturers, and shops) and asked participants questions relating to the FSA, and trust in its ability to fulfil its key responsibilities. 

Household food security (core module)

This module incorporated the USDA 10-item US Adult Food Security module, a standardised measure that uses indicator questions to assess different levels of food security experienced by participants and their households. It asked a series of questions regarding participants’ ability to afford food over the previous 12 months. It also asked about changes participants had made to their eating habits in the last 12 months, and the reason for these changes (for example, financial reasons, health reasons). The USDA has published the most up to date guidance, including how to calculate food security scores. For more detailed information please visit the guidebook.

Due to the sensitive nature of the topic area, all questions in this section were optional and included a ‘Prefer not to say’ option, in addition to ‘Don’t know’ or ‘Not stated’ options. Any questions that had any of these three responses, or that were left blank, were treated as ‘missing’, with no data imputed.  In total 212 respondents had missing responses to the first three questions and so their overall food security status was set to missing (at Waves 1, 2 and 3 there were 313, 187, and 191 such respondents respectively).

This module was also included in the Wave 1 survey. It was decided to repeat this module in Waves 2, 3 and 4 so that food security could be monitored during the Covid-19 pandemic. 
 

Defra questions (repeated from Wave 1)

This module included questions requested by the Department for Environment, Food and Rural Affairs (Defra) (who made a small financial contribution to the study). 

It asked questions about participants’ food-buying activities. This included questions relating to environmental concerns, provenance and what influences purchasing choices. 

About you and your household (core module)

This final module asked participants various questions about their personal circumstances and those of their household, including age, marital status and working status. The inclusion of these questions was primarily intended to enable demographic subgroup analysis of the data.