Validation and testing of the Acceptability E-scale for Web-based patient-reported outcomes in cancer care

Article Outline

• Abstract
• 1. Introduction
• 2. Methods
  • 2.1. Design, setting, and sample
  • 2.2. Instrument
  • 2.3. Clinical procedures
  • 2.4. Sample
• 3. Results
  • 3.1. Results of scale performance testing
    • 3.1.1. Reliability
    • 3.1.2. Validity
  • 3.2. Older adults
• 4. Discussion
• 5. Conclusion
• Appendix A. Acceptability Scale
• References
• Copyright

Abstract 

The performance of the Acceptability E-scale was tested in a sample of 627 adult and older adult patients from various oncology clinics who completed an electronic symptoms survey. The revised Acceptability E-scale has strong psychometric properties and can be useful in assessing the acceptability and usability of computerized health-related programs in oncology and other health population.

Back to Article Outline

1. Introduction 

Recipients of a service or a product, in both commercial and nonprofit venues, are frequently asked to evaluate whatever service has been provided. This practice is widespread in higher and continuing adult education and in health care institutions (Chernecky et al., 2006, van den Brink et al., 2005). Virtually every accredited hospital and clinic system in the United States conducts some sort of satisfaction survey with patients (Cunningham et al., 2007, Elder, 2007). Participants in health care research often have been asked to provide feedback, particularly when the intervention requires the participant to actively engage with the intervention. For example, clinical researchers evaluated the use of interactive technologies and videotapes for patient education in cancer care and found that patient satisfaction is affected by the ways in which the interventions are delivered (Gysels & Higginson, 2007). The introduction and use of electronic patient surveys and multimedia patient education material require skills and interaction on the part of patients if the intervention is to have the desired effect. These mechanisms are increasingly being used for symptom and quality-of-life (QOL) assessments, which are integral actions in oncology clinical care. Understanding whether these new methods of assessments are feasible and acceptable to patients with cancer is a prerequisite for use in clinical practice.

The Electronic Self-Report Assessment-Cancer (ESRA-C) was developed and tested as a symptom and QOL screening program in a group of patients with cancer with varying levels of education and prior computer use (Berry et al., 2004). The investigators and clinicians involved in the project were committed to assuring the program's acceptability by the diverse population receiving care at the University of Washington Cancer Center and Seattle Cancer Care Alliance. With permission, the interdisciplinary team adapted an acceptability questionnaire developed first by the research group of Taenzer et al. (1997) and then applied by Carlson, Speca, Hagen, and Taenzer (2001) in a Canadian cancer pain clinic. Mullen, Berry, and Zierler (2004) first reported the findings from the adapted Post-Survey Patient Impression Form, demonstrating a high level of acceptability of the ESRA-C program by the earliest users of the program, 101 patients in a radiation oncology clinic. Recently, the ESRA-C team analyzed the acceptability of the ESRA-C program with 342 participants in a multisite, randomized controlled clinical trial, concluding that the program was easy and satisfying to use by the great majority of patient respondents (Wolpin et al., 2008).

The reliability of this new Acceptability E-scale had not been formally evaluated. Given the availability of results from an additional large group of patients with cancer, further instrument development has been possible. These new reliability data have not been published elsewhere. On this article, we describe the performance testing of the Acceptability E-scale in a sample of adult and older adult patients with various cancers from medical oncology, radiation oncology, and stem cell transplant services of a comprehensive cancer center in the Pacific Northwest.

Back to Article Outline

2. Methods 

2.1. Design, setting, and sample 

Data for this analysis were collected in a longitudinal randomized clinical trial designed to assess the impact of the ESRA-C program at the Seattle Cancer Care Alliance. Eligible study participants were adults and older adults with various types of cancer receiving radiation and/or chemotherapy who were able to speak and read English at a sixth-grade level. All research procedures were reviewed and approved by the Fred Hutchinson Cancer Center Institutional Review Board. Participants were enrolled during clinic appointments from May 2005 to June 2007.

2.2. Instrument 

The Acceptability E-scale is an adaptation of a 10-item tool, the Post-Survey Patient Impression Form developed by Canadian researchers in the 1990s (Carlson et al., 2001, Taenzer et al., 1997), to assess patients' impressions of a computerized QOL screening program. No reliability or validity data for the Post-Survey Patient Impression Form were reported by the original developers. Mullen et al. (2004) reported the use of a 7-item version of the acceptability scale, resulting in a reliability coefficient of .91.

The Acceptability E-scale reported here includes 6 items regarding how easy and enjoyable the program was to use, how understandable were the questions, how helpful was completing the program, whether the participant liked the program, whether the amount of time to complete the program was acceptable, and overall satisfaction with the program. This latest version excluded an original item on how much the participant “liked” the program due to overlap with other items. Please refer to the Appendix for a full depiction of scale.

The ordinal scale has a simple 5-point numerical response scale. The 6 items each offer five response options: A response of 1 indicates a negative and 5 indicates a positive evaluation of each aspect of the ESRA-C program. A score of 3 indicates a neutral response.

The readability level of this scale is approximately fifth-grade level as no single item is made up of more than 14 words or more than 18 syllables (DeVellis, 2003). The scale can be completed in less than 2 minutes by most participants. To obtain a summary score, the score is applied on each item as it appears on the scale and all response scores are summed. Respondents with missing items have not been included in the analysis, and no reverse scoring is required. Scale total scores range from 6 to 30.

2.3. Clinical procedures 

As components of the ESRA-C program, computerized symptom and QOL reports were collected at the beginning of patients' treatment (T1) and at about 6 weeks from the initiation of treatment or at the completion of at least one cycle of their treatment, whichever came first (T2). A 6-item Acceptability E-scale was administered to patients immediately after they used the electronic program at T2. The ESRA-C was presented by research coordinators to participants using notebook computers equipped with touch screens and wireless network cards in the clinic reception areas or examination rooms just prior to a provider visit. When completing the survey in reception areas, participants were seated in areas reserved for research studies containing privacy partitions. If this was not possible, care was taken to locate a suitable distance from other patients. Details of the technical aspects are described elsewhere (Karras et al., 2006).

2.4. Sample 

Six hundred twenty-seven (N = 627) patient participants from medical oncology, radiation oncology, and stem cell transplant clinics reported a mean age of 54.3 ± 13.9 years (age range = 18–90 years), 54% were men, and 8% were persons of minority race. Out of the 627 patients, 25% (129 patients) were older adults (age range = 65–90 years). Eighty percent reported at least some college education.

Back to Article Outline

3. Results 

3.1. Results of scale performance testing 

Reliability testing of the 6 items revealed an alpha coefficient of .757 (N = 627), which is a very good internal consistency for a new, immature scale (Netemeyer, Bearden, & Sharma, 2003). Item analysis showed very good item-to-item correlations among the six items, with most of the items having a coefficient between .30 and .70 (Table 1), and good item-to-total scale correlation, with most of the items having a corrected item–total coefficient of >.40 (Table 2).

Table 1. Interitem correlation matrix (N = 627)

Table 2. Item–Total statistics (N = 627)

Exploratory factor analysis utilizing principal axis factoring revealed one factor with 6 items having an item load value of equal to or greater than .40 on a factor and with equal to or greater than .20 difference between loading on other factors except for Item 6. Exploratory factor analysis using principal component factoring revealed one factor with all 6 items having a value of item load at equal to or greater than .40. This factor analysis revealed the unidimensionality of the scale.

3.2. Older adults 

Subset analyses of the reliability and validity of the Acceptability E-scale in 129 older adults with cancer were also conducted (age range = 65–90 years). Reliability testing of the six items revealed an alpha coefficient of .721. Item analysis showed very good item-to-item correlations among the 6 items, with most of the items having a coefficient >.30 (Table 3). Corrected item–total scale correlation also showed strong coefficients, with most of the items having a coefficient of >.40 (Table 4).

Table 3. Interitem correlation matrix (n = 129)

Table 4. Item–Total statistics (n = 129)

Back to Article Outline

4. Discussion 

We presented a new Acceptability E-scale for assessing acceptability and usability of ESRA-C, an Internet-based symptom and QOL screening program. This scale was tested and validated in a large sample of adults and older adults with cancer. The major strengths of the scale are excellent reliability and construct validity and a less-than-1-minute completion time. The generalizability of this study, however, is limited by the lack of diversity in the fairly educated sample. The reliability and validity may differ in a group of respondents with varying cultural backgrounds. However, in an earlier analysis of acceptability, education level and race were not significantly associated with acceptability of the ESRA-C program, although gender and age were (Wolpin et al., 2008).

Computerized assessment has been introduced into a number of oncology settings as an efficient, error-free, and real-time approach to communicate symptoms and QOL reports to clinicians (Basch et al., 2005, Basch et al., 2007, Dockrey et al., 2005, Karras et al., 2006, Middeke et al., 2004, Taenzer et al., 1997). Technology-enhanced assessment has gained acceptance as an alternative method of self-reporting symptoms and QOL issues in lieu of paper forms because of its comprehensiveness, time savings, and accessibility. There is a concern, however, about how people with cancer, particularly among older adults, would view such technology. Studies assessing the acceptability of computer-based programs in oncology settings have reported moderate to high level of acceptance among patients (Basch et al., 2007, Mullen et al., 2004). Most investigators described acceptability and usability of computer programs using various types of exit interview questionnaires, with no report on the reliability of the instruments used (Basch et al., 2005, Basch et al., 2007, Bush et al., 2005, Middeke et al., 2004, Taenzer et al., 1997).

The Acceptability E-scale can be used in research or clinical practice to assess acceptability and usability of computer-based assessments or interventions. The inclusion of this scale in an exit questionnaire after use of a computer-based program can provide researchers with critical information regarding patients' impressions of or attitudes toward the program.

Clinical researchers may want to know what score on the Acceptability E-scale would indicate adequate “acceptability.” We propose that 80% of the highest possible summary score be used as the threshold score, indicating that the program or intervention being evaluated is acceptable to users. There are several permutations based on individual scores from each item that can have a summary score of 24 or above. Therefore, one must exercise caution by examining the sample scores on each item to detect any specific weaknesses of the program or intervention. For example, an intervention could be easy, understandable, and enjoyable but not particularly “useful.” Additional rigorous reliability and validity testing will further elucidate the utility of the Acceptability E-scale in different clinical settings and ethnic patient populations.

Back to Article Outline

5. Conclusion 

Overall, the Acceptability E-scale has strong performance properties as tested in adults and older adults with cancer. The scale has promise for application in assessing the acceptability and usability of computerized screening or health-related intervention programs in oncology and perhaps with other health consumer groups. The authors recommend that clinicians and clinical researchers further evaluate and test this revised Acceptability E-scale in other patient population.

Back to Article Outline

Appendix A. Acceptability Scale 

Thank you for your answers on how you are feeling—now we would like to ask you about your thoughts on using this computer program (ESRA-C).

Back to Article Outline

References 

1. Basch E, Artz D, Dulko D, Scher K, Sabbatini P, Hensley M, et al. Patient online self-reporting of toxicity symptoms during chemotherapy. Journal of Clinical Oncology. 2005;23(15):3552–3561
   • View In Article
2. Basch E, Artz D, Iasonos A, Speakman J, Shannon K, Lin K, et al. Evaluation of an online platform for cancer patient self-reporting of chemotherapy toxicities. Journal of the American Medical Informatics Association. 2007;14(3):264–268
   • View In Article
3. Berry DL, Trigg LJ, Lober WB, Karras BT, Galligan ML, Austin-Seymour M, et al. Computerized symptom and quality-of-life assessment for patients with cancer part i: Development and pilot testing. Oncology Nursing Forum. 2004;31(5):E75–E83
   • View In Article
4. Bush N, Donaldson G, Moinpour C, Haberman M, Milliken D, Markle V, et al. Development, feasibility and compliance of a Web-based system for very frequent qol and symptom home self-assessment after hematopoietic stem cell transplantation. Quality of Life Research. 2005;14(1):77–93
   • View In Article
5. Carlson LE, Speca M, Hagen N, Taenzer P. Computerized quality-of-life screening in a cancer pain clinic. Journal of Palliative Care. 2001;17(1):46–52
   • View In Article
   • MEDLINE
6. Chernecky C, Macklin D, Walter J. Internet design preferences of patients with cancer. Oncology Nursing Forum. 2006;33(4):787–792
   • View In Article
7. Cunningham L, Weber Z, Cook K. Infinite excellence. Live the brand promise by emphasizing “always”. Marketing Health Services. 2007;27(2):12–19
   • View In Article
8. DeVellis RF. In: Scale development: Theory and applications. 2nd ed.. Thousand Oaks, CA: Sage Publications, Inc; 2003;p. 67
   • View In Article
9. Dockrey MR, Lober WB, Berry D, Wolpin S, Rae LJ. Distributed health assessment and intervention research software framework. AMIA Annual Symposium Proceedings. 2005;940:
   • View In Article
10. Elder JS. Assessing patient satisfaction—the government has a solution. Journal of Urology. 2007;177(4):1223
    • View In Article
11. Gysels M, Higginson IJ. Interactive technologies and videotapes for patient education in cancer care: Systematic review and meta-analysis of randomised trials. Supportive Care in Cancer. 2007;15(1):7–20
    • View In Article
12. Karras BT, Wolpin S, Lober WB, Bush N, Fann JR, Berry DL. Electronic self-report assessment–cancer (esra-c): Working towards an integrated survey system. Studies in Health Technology and Informatics. 2006;122:514–518
    • View In Article
    • MEDLINE
13. Middeke M, Bauhofer A, Kopp I, Koller M. Computerized visualization of quality of life data of individual cancer patients–the qol-profiler. Inflammation Research, 53 Suppl 2. 2004;S175–S178
    • View In Article
14. Mullen KH, Berry DL, Zierler BK. Computerized symptom and quality-of-life assessment for patients with cancer part ii: Acceptability and usability. Oncology Nursing Forum. 2004;31(5):E84–E89
    • View In Article
15. Netemeyer RG, Bearden WO, Sharma S. Reliability. In: Scaling procedures: Issues and applications. 1st ed.. Thousand Oaks, CA: Sage Publications, Inc; 2003;p. 41–70
    • View In Article
16. Taenzer PA, Speca M, Atkinson MJ, Bultz BD, Page S, Harasym P, et al. Computerized quality-of-life screening in an oncology clinic. Cancer Practice. 1997;5(3):168–175
    • View In Article
    • MEDLINE
17. van den Brink JL, Moorman PW, de Boer MF, Pruyn JF, Verwoerd CD, van Bemmel JH. Involving the patient: A prospective study on use, appreciation and effectiveness of an information system in head and neck cancer care. International Journal of Medical Informatics. 2005;74(10):839–849
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (382 KB)
    • CrossRef
18. Wolpin S, Berry D, Austin-Seymour M, Bush N, Fann JR, Halpenny B, et al. Acceptability of an electronic self report assessment program for patients with cancer. Computers, Informatics, Nursing. 2008;26(6):332–338
    • View In Article