Effect of different questionnaire formats on item nonresponse in older patients with cancer

Abstract

Background

Self-administered questionnaire surveys are instrumental in revealing patient experiences. However, concerns about the quality of data and validity of results arise from nonresponses, either to the entire survey or to specific items therein. The format of the survey can affect the ease of answering and subsequent responses. Older patients who are more likely to have some cognitive decline may be especially sensitive to questionnaire comprehensibility and response methods.

Methods

We surveyed the experiences of patients aged 65 years and older with cancer in four hospitals. Randomly selected patients from the hospital-based cancer registries received questionnaires with either the grid or text-choice response formats. We compared item nonresponse rates for each question between the two types of questionnaires.

Results

Out of invited 1500 patients, 51.3% responded. We included data from 675 patients (331 and 326 in the grid and text-choice formats, respectively) in the final analysis. The non-response rate of items considering participant demographics, including age, gender, and cancer sites, did not differ between groups (mean age: 76.3 and 75.8 years in the grid and text-choice formats, respectively). Most items with the grid format had higher proportions of item nonresponse than those with the text-choice format. The difference was more pronounced in patients aged 75 years and above, compared to those aged 65–74 years.

Conclusions

In older patients, text-choice format survey questions elicited fewer item nonresponses compared to grid format questions. The text-choice format should preferably be used consistently throughout surveys targeting older persons.

Background

Patient experience surveys are important tools for capturing patients’ experiences during and after receiving care. Healthcare policy decision-makers require scientific evidence to develop the healthcare system, and patient experience surveys are often used to evaluate healthcare delivery to patients. Globally, numerous patient surveys have been developed, such as the Hospital Consumer Assessment of Healthcare Providers and Systems by the Agency for Healthcare Research and Quality in the United States and the patient experience surveys provided by the National Health Service in the United Kingdom [1, 2]. In particular, some surveys are designed for patients with cancer, such as the Consumer Assessment of Healthcare Providers and Systems Cancer Care Survey and the National Cancer Patient Experience Survey [3]. In Japan, for example, the Cancer Patient Experience Survey has been conducted to evaluate the experiences of patients with cancer to inform the cancer control policy by monitoring the progress of the National Cancer Control Plan [4].

While a questionnaire survey is an essential tool to capture the voices of respondents, item nonresponse is a common problem that affects survey data quality. This occurs when a survey participant skips specific questions, which thus have to be excluded from data analysis [5]. Item nonresponse may pose a serious threat to the validity of inferences due to selection bias. Statistical imputation methods can help fill in the missing data but they need statistical assumptions; hence, efforts to minimize item nonresponse in surveys are desirable [6]. Past research has explored ways to address this issue. For instance, a study by Flüß et al. investigated whether offering an internet response option or using single-sided printed questionnaires could improve survey response rates; however, their findings indicated that these strategies did not significantly boost participation, suggesting that nonresponse in surveys may not be easily reduced through format changes alone [7]. In the context of surveying patients with cancer, our study—focusing on the experiences of patients aged 65 years and older with cancer—is important because the majority of patients with cancer are older individuals. According to the Surveillance, Epidemiology, and End Results Program in the US, the median age of first cancer incidence is 66 years, and more than half of all patients with cancer are 65 years and older [8]. Similarly, the Ministry of Health, Labour, and Welfare in Japan reported that 73.8% of patients with cancer are 65 years and older [9]. Hence, cancer patient surveys usually include a substantial number of older adults. Further, although earlier studies have revealed that older participants have more item nonresponse compared to younger participants, no study, to our knowledge, has examined how the format of answer options could affect item nonresponse among older patients [10].

Coming back to the Cancer Patient Experience Survey mentioned above, which is a nationwide mail-based survey conducted in Japan, we have periodically conducted two rounds of this survey thus far to evaluate the cancer care experiences of patients with cancer and their families [4]. In these two questionnaires, we changed the format of response options, from a text-choice list of response options for each question in the first survey, to grid formats in the second survey. While this modification was useful in presenting the questionnaire in a limited space, it appeared to have resulted in higher rates of item nonresponse, particularly among older patients. Thus, we hypothesise that the grid format leads to an increased likelihood of item nonresponse.

To examine this hypothesis, we conducted a parallel cross-sectional survey, which compared the response options of the text-choice and grid formats. Our study measured the effect of question format type on the responses of older patients with cancer.

Methods

Design

We conducted a parallel, cross-sectional study using a self-administered mailed survey completed by two groups of older patients with cancer. The survey period was between January and March 2023. Each patient received one of the two types of questionnaires, both of which contained the same set of questions but differed in their formats. Subsequently, response rates, item nonresponse rates, and answers were compared between the groups. We conducted the survey in Japanese.

Ethical considerations

This study was conducted according to the guidelines of the 1964 Declaration of Helsinki and its amendments [11]. Informed consent to participate in the study was obtained from all the participants before they completed the questionnaire. Further, the study was approved by the National Cancer Center Institutional Review Board (approval number: 2022 − 224).

Setting and participants

Four hospitals participated in this study. Within these hospitals, we randomly selected 1500 patients aged 65 years and older who were diagnosed with cancer in 2020 and started with initial cancer treatment in the four selected hospitals based on hospital-based cancer registries. We excluded patients who were not able to answer survey questions by themselves, such as those who were hospitalized or those deceased at the time of the survey. We created two non-overlapping groups, selected from the same patient sample, and sent them the same questionnaire in one of the two survey formats described in the following subsection. These patients were randomly assigned to two groups through random number generation with a 1:1 ratio at each hospital.

Questionnaire formats and analysis variables

The patient survey questionnaire we analyzed had 50 questions that asked about patient experiences of care. A questionnaire was developed based on focus group discussions with patients with cancer. The focus group discussions provided information on specific aspects of patient experience, which included the following dimensions: optimal care provision, access to correct information and consultation, the financial impact of a cancer diagnosis, social isolation, and caregiver burden. The questionnaire was developed to cover these dimensions in chronological order: cancer diagnosis, choosing treatment, receiving treatment, and social life during and after treatment. Details regarding questionnaire development are described elsewhere [4]. Fifty questions had multi-item scales. Of these, 24 questions had scale answers in either of the following two formats: a grid format (the grid group) or a text-choice format (the text-choice group) (see Fig. 1). The remaining 26 questions were formatted in the same style as all the questionnaires. The complete version of the questionnaire is presented in Supplementary Appendices 1 and 2 and details regarding question selection are described in Supplementary Appendix 3.

Fig. 1

Example of a question where the format of the answer option was revised. (A) Grid format; please select one response for the statement. (B) Text-choice format; did you receive sufficient information from medical staff before making your decision regarding your cancer treatment?

We defined outcome variables for each question based on the following rule. For 24 questions with scale answers, we calculated the percentage of positive responses combining “Agree strongly” and “Agree.” For Q11 and Q12, we calculated the percentage of less than 1 month by diagnosis or starting treatment by combining the response “Less than 2 weeks” and “More than 2 weeks and less than 1 month.” For Q48 and Q49, the responses “I am somewhat familiar with them” and “Yes, I am very familiar with them” were treated as outcome responses. For questions allowing multiple responses, we calculated the percentage excluding those who chose the specific response (“None of above” for Q22). For other questions permitting a single response, we determined the proportion of specific replies as outcome responses. Finally, we calculated the mean and standard deviation for these questions using the global scale, presented in Q29.

Statistical analysis

We made several comparisons to examine the effect of format differences. First, we analyzed the item nonresponse rates for the question items with different response formats in the two questionnaire types. Second, we compared the content of answers between the two groups by calculating the proportion of positive responses. In each comparison, we stratified the samples by gender and age group (65–74 years vs. ≥75 years) to examine factors that could influence item nonresponse besides the response formats. We used a t-test and the Wilcoxon rank-sum test to assess the difference in continuous variables, and the chi-squared test to compare proportions and rates. All the analyses were performed using Stata 17 (StataCorp LLC, College Station, Texas) and R 4.1.1 (R Core Team, 2021) [12, 13].

A sample size calculation suggested a minimum of 382 participants per group (i.e., a total of 764 participants in two groups) for a multicenter study at 80% power with a 10% effect size and level of significance of 0.05 (5%). Assuming the return rate of the survey to be 50%, we decided to send the questionnaire to 750 participants per group (i.e., a total of 1500 participants).

Results

Response rates and final sample

Of the 1500 patients invited to participate in the study, 51.3% responded, and the response rates of the two groups were not statistically different (389 [51.9%] and 380 [50.7%] in the grid and text-choice groups, respectively; p = 0.64). After excluding the respondents who did not explicitly agree to participate in the study, those who were reportedly never diagnosed with cancer, and cases where the questionnaire was answered by someone else (e.g., a family member), we included data from 675 patients (331 and 326 in the grid and text-choice groups, respectively) in the final analysis.

Participant demographics

Table 1 describes the baseline characteristics of the participants. The participants’ mean age was 76.0 years (76.3 years in the grid group vs. 75.8 years in the text-choice group; p = 0.21). Among the participants, 64.0% (61.0% in the grid group vs. 66.9% in the text-choice group; p = 0.12) were men. Moreover, the proportion of answers, that is, 36 of the 43 questions, was not statistically different between the two groups. The mean overall scores of patients’ experience from diagnosis to treatment were 8.52 in the grid group and 8.65 in the text-choice group (p = 0.30) (Supplementary Table 1).

Table 1 Demographic details of the participants, including nonresponse items

Item nonresponse rate

Tables 2 and 3 present the item nonresponse results. The questions with the grid format had higher item nonresponse rates than those with the text-choice format. The nonresponse rates of items related to demographics, including cancer sites, were not statistically different between the two groups. The items related to fertility preservation (Q14, Q16, and Q17) had higher nonresponse rates compared with other items. Further, we conducted a missing pattern analysis, and the results are depicted in Fig. 2.

Table 2 Item nonresponse rates for questions where the answer format changed between the two participant groups

Table 3 Comparing the item nonresponse rates of questions between the two groups

Fig. 2

Nonresponse item patterns between the two groups. (A) Grid group; (B) Text-choice group

Stratified analysis by age (< 75 and ≥ 75 years) and gender (Supplementary Tables 2 and 3) revealed that item nonresponse rates did not differ between male and female patients in either group (18 items with a p-value of < 0.05 for men and 15 items for women). Nevertheless, patients aged 75 years and above had more item nonresponse rates compared to those below 75 years for most of the questions (5 items with a p-value of < 0.05 for patients aged 65 − 74 years, and 21 items for patients aged ≥ 75 years). The differences were larger in the grid group than in the text-choice group.

Discussion

We conducted a survey to assess the item nonresponse rate for questionnaires using two different response formats among patients with cancer aged 65 years and older. Our results showed that the text-choice format questionnaire resulted in lesser item nonresponses compared to the grid format questionnaire. Since patients were randomly assigned to the two groups (i.e., grid group or text-choice group) and had similar characteristics, we attribute the group-level differences to the questionnaire formats.

This is the first study to examine the effect of question format type on the responses of older patients with cancer. Item nonresponses not only reduce the sample size but also lead to bias when the nonresponses occur systematically. Indeed, the responses of the two groups revealed different answer distributions in 15.9% (7 out of 44) of the questions (Supplementary Table 1), indicating the possibility of differential responses; this is especially relevant because the two groups were randomly assigned either of the questionnaire formats and had similar characteristics. Efforts should be made to minimize these types of item nonresponses as much as possible. Because respondents of these types of surveys are usually older, researchers must clearly understand the characteristics of older patients’ responses and nonresponses. Earlier studies have indicated that the item nonresponse rate has a strong association with age and that it is higher in older than younger patients, aligning with our results [5, 14]. Accordingly, it is recommended that surveys targeting older adults use the text-choice format rather than the grid format.

To our knowledge, our study is the first to compare the item nonresponse rate between the grid and text-choice formats used in mail surveys. In a prior study, Voutilainen et al. compared the visual analogue scale and the Likert scale among a non-randomized sample of 150 surgical patients [15]. They found that the visual analogue scale was associated with fewer nonresponses than the Likert scale. However, their study compared the visual analogue scale of plane lines without any numbers or anchors other than the end of the line versus the Likert scale accompanied with numbered 5-level text. We have obtained consistent results suggesting that the response format can affect the frequency of item nonresponses. However, our comparison was different as both of the response formats we compared had 5 levels of numbers with a different alignment of response options; one had the grid format while the other included the list of texts that described the response levels. In addition, DeBell et al. assessed data quality and survey time between grid and single-item answer options in an internet survey conducted with 1209 participants who were US citizens aged 18 years or older [16]. Their findings are consistent with those of our study as we found that the single-item format had less item nonresponses compared to the grid format, except that their study was a web-based survey. Further, we specifically targeted older patients because they appeared more likely to skip items when they were difficult to answer. Earlier research has indicated that clear instructions and a simple format are essential for survey questions presented to older patients since they are more likely to be suffering from physical disabilities or cognitive impairment [17, 18].

While we cannot pinpoint the exact reason why the grid format was associated with more nonresponses, we make a conjecture: it may be attributable to the fact that the questions, other than the level-type questions, were in the text-choice format. Therefore, the grid format may have disrupted the flow of the response process among older respondents with a change in the format in the middle of the survey, which may have increased the item nonresponse rate. Further, the grid format requires more eye movements to capture the contents of questions compared to the text-choice format. Since respondents may need to move their eyes up and down to follow the instructions in the grid format, they may feel burdened to answer questions or may misunderstand the questions, leading to an increased item nonresponse rate [19].

We did not observe any gender difference in the item nonresponse trends. Our result is consistent with the findings of earlier studies [10, 20]. Both male and female patients had higher item nonresponse rates in the grid format compared to the text-choice format. In both formats, female participants had higher item nonresponse rates than male participants; however, the differences were not statistically significant.

Our findings present important insights; yet, a few study limitations must be noted. First, a majority of the patients we recruited were from urban areas, and they may answer differently than those residing in rural areas. Although we cannot make a definitive conclusion about related response trends because the majority of our respondents were from urban areas (three out of the four hospitals are located in urban areas), we observed different item nonresponses in surveys conducted in all four hospitals. Moreover, our response rate was similar to that of an earlier study, which considered patients with cancer across Japan [4]. Second, the responses may be influenced not only by the format of the questionnaire but also by other factors such as the content of the questions or how the options are presented. Our results may not be generalizable to all types of questions. In cases where researchers may want the respondents to report relative degrees or scales comparing the items, the visual grid format may be a better choice. Finally, the mechanisms underlying the different item nonresponse rates have not been clarified in this study. Further research is needed, such as studies examining the effect of text direction in grid formats on nonresponses or those involving in-depth interviews to explore the reasons for nonresponses. Such future research should include various questionnaire formats to test the theory behind the mechanisms.

Conclusions

Our results revealed that older patients with cancer who responded to questionnaires on their experiences of care had fewer item nonresponses when using the list-based questionnaire format compared to the grid format. When planning to survey older adults, a recommendation is to consider avoiding the use of the grid format in presenting response options.