Project Summary
Developing a Social Validation Model for Effective Utilization of Disability and Rehabilitation Research
By
Pimjai Sudsawad, Sc.D., OTR
Department of Occupational Therapy
University of Wisconsin-Milwaukee
P.O. Box 413
Milwaukee, WI 53201
pimjais@uwm.edu
The Mary E. Switzer Tesearch Fellowship (Merit)
National Institute on Disability and Rehabilitation Research
September 2002 - August 2003
Grant # H133F020023
Project Officer: Ellen Blasiotti
Acknowledgments
The author would like to thank the following parties who were instrumental for the successful completion of this research project.
- The U.S. Department of Education, Office of Special Education and Rehabilitative Services, National Institute on Disability and Rehabilitation Research, Mary E. Switzer Fellowship Program, who funded this research project. Special thanks to Ellen Blasiotti for her guidance through this project.
- Catherine A. Trombly, Sc.D., OTR, who provided invaluable mentorship, guidance, advices, and support throughout the project period.
- Randall Lambrecht, Ph.D., Dean, College of Health Sciences, University of Wisconsin-Milwaukee, whose administrative and additional fiscal support was instrumental for this project’s success.
Background and Significance
Evidence-based practice is defined as the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients (Sackett, Richardson, Rosenberg, & Haynes, 1997). Evidence-based practice is the current approach recommended for use in occupational therapy (Holm, 2000), physical therapy (Turner & Whitfield, 1997), and speech-language pathology (Logemann, 2000).
Using ‘current best evidence’ often implies using research findings. In order to execute the evidence-based practice approach effectively, information from research reports must be adopted and utilized by the intended consumer group—therapists who would like to use research evidence to support their practice. For the research information to be utilized, it must relate to a perceived need and must be understandable to the potential users of the information.
Research articles in professional and interdisciplinary journals have been the main sources of the most current evidence for practice. Practitioners, however, often reported barriers when using information from research articles. Several issues were identified both in nursing and in rehabilitation such as lack of someone to help translate findings into practice (Champion and Leach, 1989); the implications for practice are not made clear in the research presentation (Funk, Champagne, Tornquist, & Wiese,1995); difficulties with communication of research (Kajermo, Nordström, Krusebrant, & Björvell, 2000); lack of skills in interpreting research evidence (Law & Baum, 1998); and knowledge from the literature was not always viewed as relevant, easily applied, or oriented to professional practice (Dubouloz, Egan, Vallerand, & von Zweck, 1999).
With those difficulties, it is foreseeable that practitioners would be reluctant in using research evidence for evidence-based practice. Di Fabio (1999) expressed concerns about the unrealistic expectation on the physical therapy practitioners in using research evidence for evidence-based practice due to discrepancies between real clinical situations and the situations presented in outcome studies. The results from the Turner and Whitfield study (1997) supported Di Fabio concerns, as the researchers found that research literature was ranked the lowest in importance as a basis for choosing treatment techniques by physical therapists in Australia and the United Kingdom. The slow diffusion of research findings into practice and the scarcity of practice-based research were identified as two of the factors that had negative impact on the ability of health care professionals to deliver high-quality patient care and to continually improve the quality of those services (Schiller, 1998). Despite findings that pointed to a problem of research information not optimally usable for practice, there had been no systematic approach to create research information that is more usable for practice. This project, then, proposed a systematic and theoretically-based method to examine the issue of research utilization related to characteristics of research information that would facilitate its adoption and use by rehabilitation practitioners for evidence-based practice.
The desirable characteristics of research information that might help to facilitate adoption and use were identified through the framework of the Diffusion of Innovations theory (Rogers, 1995). Those characteristics include relative advantage—the degree to which an innovation is perceived as better than the idea it supersedes; compatibility—the degree to which an innovation is perceived as being consistent with the existing values, past experiences, and the needs of the potential adopter; complexity—the degree to which an innovation is perceived as difficult to understand and use; trialability—the degree to which an innovation may be experimented with on a limited basis; and observability—the degree to which the results of an innovation are visible to others. Although it is commonly accepted that there is an advantage of using research evidence (relative advantage) when compared to other kinds of evidence, other characteristics seemed to be lacking according to the barriers reported by practitioners as discussed earlier in this report.
Using the concepts of social validity (the relevance of research questions, treatments investigated, and research outcomes to society at large) (Wolf, 1978); ecological validity (the transferability of research outcomes to real-life settings) (Sbordone, 1996); and clinical significance (a magnitude of change in research outcomes that is large enough to be meaningful to clients’ everyday lives) (Ogles, Lunnen, & Bonesteel, 2001) to design and conduct outcome studies were identified as tools that would help to create those desirable characteristics as identified through the theory. It was proposed that involving clients, practitioners, or the community to determine research goal, acceptability of treatment techniques being investigated, and the meaningfulness of performance change should increase compatibility; conducting research using “common sense” outcomes (concrete, of observable magnitude, and determined by consumers) should help to reduce complexity; investigating treatment methods that could be implemented in clinical settings should increase trialability; and measuring treatment outcomes in a way that demonstrates changes in real-life activities rather than trivial changes or merely statistically significant changes should help to increase observability.
The overall goal of this research project, therefore, was to investigate whether these proposals were perceived to be useful by rehabilitation practitioners and, if the usefulness was indicated, to develop a preliminary model of creating research information based on the concepts of social validity, ecological validity, and clinical significance to be used when conducting outcome studies in order to increase research utilization among rehabilitation practitioners for evidence-based practice.
Aims and Methods
The specific aims of this project were to investigate 1) whether the concepts of social validity, ecological validity, and clinical significance had been incorporated into outcome studies of OT, PT, and SLP interventions for children with disabilities; 2) whether incorporation of these concepts was perceived to be useful by OT, PT, and SLP practitioners who worked with children with disabilities in school settings; and 3) whether incorporation of these concepts was perceived to have a potential for a positive impact on research utilization among those practitioners.
The aims of this project were met by 1) a systematic literature review of published outcome studies of the OT, PT, and SLP interventions for children with disabilities over the past 10 years (1992-2001) to investigate the trends, frequencies, and methods used to apply the concepts of social validity, ecological validity, and clinical significance; 2) a focus group study of OT, PT, and SLP practitioners who worked with children with disabilities in school settings to gain a preliminary understanding of the perceived usefulness of the application of social validity, ecological validity, and clinical significance concepts; and 3) a national mail survey study of OT, PT, and SLP practitioners who served children with disabilities in school settings, to gain, on a larger scale, the perceived usefulness and impact of applying the concepts of social validity, ecological validity, and clinical significance (in outcome studies) on research utilization of practitioners for evidence-based practice.
Systematic review of published outcome studies
A systematic review was conducted for outcome studies of OT, PT, and SLP interventions for children with disabilities published in peer-reviewed journals in the time span of 10 years (1992-2001). An outcome study was defined as a study that investigated the effectiveness of a treatment intervention that were authored by OT, PT, or SLP researchers; that investigated OT, PT, or SLP treatment interventions as indicated by the researchers; or that treatment interventions investigated were implemented by OT, PT, or SLP practitioners (therapists who provided treatment in that study). Children was defined as birth to 21 years of age. Outcome studies were searched through electronic databases including Medline, CINAHL, ERIC, and PsychInfo using broad keywords occupational therapy and children, physical therapy and children, speech and children, and language and children, to identify outcome studies in OT, PT, and SLP for children with disabilities. Case studies, and retrospective studies were not included in this review because of their limited contribution as evidence for practice. Studies that included both children and adults as participants were also excluded, unless the results were reported separately for children and adults.
Each study was reviewed and coded to indicate whether the researchers had implemented the concepts of social validity, ecological validity, and social significance in their studies. The coding categories included whether the researchers 1) verified with any consumer groups the importance/relevance of the research questions/hypothesis/treatment goal being investigated, 2) verified with any consumer groups the acceptability of the treatment being investigated, 3) verified with any consumer groups the importance/relevance of the study's outcomes, 4) measured outcomes related to research participants’ performances in natural settings, and 5) reported magnitude of change ( as a result of the convention) in ways that demonstrated meaningful changes in the participants’ real-life performances.
Focus group study
Two focus groups were conducted with OT, PT, and SLP practitioners who served children with disabilities in school settings in the State of Wisconsin, and who were members of the Wisconsin Occupational Therapy Association, Wisconsin Physical Therapy Association, and Wisconsin Speech-Language Pathology & Audiology Association. The participants were recruited by mail through the mailing lists purchased from those organizations. Each focus group discussion lasted two hours. During the focus group, practitioners participated in the discussion related to the difficulties they perceived in using research information for their practice, the solutions they perceived would make it easier for them to use research information, and their perceptions about the usefulness of the social validity, ecological validity, and social significance concepts related to usability of research information. Both the nominal group technique (Delbecq, Van de Ven, & Gustafson, 1975) and traditional focus group technique were used during the focus group discussion. The information obtained from the focus group study was used to develop the survey questionnaire items used in the mail survey study.
Mail survey study
A mail survey study was conducted with OT, PT, and SLP practitioners who practiced in school settings and who were members of the American Occupational Therapy Association (ADTA), the American Physical Therapy Association (APTA), or the American Speech and Hearing Association (ASHA) at the time of the study. The prospective participants were randomly selected from a pool of eligible prospective participants from mailing lists made commercially available by the AOTA, APTA, and ASHA. A paper-and-pencil survey questionnaire with close-ended questions was used as an instrument. The content of the survey questionnaire included frequency and patterns of evidence used for practice and perceived usability of research information; perceived importance of research information characteristics as indicated by the Diffusion of Innovation theory; current problems in using research information and suggested solutions; perceived benefits and likelihood to increase use of research information by applying the social validity, ecological validity, and clinical significance concepts to outcome studies; and other relevant demographic information.
First, invitation letters were mailed to all prospective participants explaining the needs for information sought and the possible contribution of the findings for practice. Approximately 5 days after the invitation letters were sent, the survey questionnaire (with a cover letter and a self-addressed stamped envelop) was sent to each prospective participant. At approximately 3 weeks after the first mailing, a postcard reminder was sent to each prospective participant who had not yet returned the questionnaire. At approximately 4 weeks after the first mailing, a second copy of the questionnaire (with a reminder letter and a self-addressed stamped envelop) was sent to each of the prospective participants who had not returned the questionnaire at that point.
Preliminary Findings
Systematic review of published outcome studies
Two hundred forty-six pediatric outcome studies in OT, PT, and SLP that met the inclusion criteria were reviewed. The number of outcome studies for each year showed an upward trend from earlier (1992) to later (2001) years. The research design used most often was the pretest-posttest research design (with or without control groups), followed by single subject research design. There were equal and small numbers of the randomized pretest-posttest research design and randomized controlled trial design used in studies reviewed.
Overall, it was found that less than 15% of the outcome studies contained the aspects of verifying (with any consumer groups) the importance/relevance of the research questions/hypothesis/treatment goal being investigated, the acceptability of the treatment being investigated, or the importance/relevance of the study's outcomes. Higher number of studies, (although still less than 40%) included outcomes that related to the children’s performances in natural settings and reported magnitude of change ( as a result of the intervention) in ways that demonstrated meaningful change in the participants’ real-life performances. Examples of methods of verifying the importance/relevance of the research questions/hypothesis/treatment goal were seeking parents’ (of the children who participated in the study) agreement on the relevance of research question being investigated; developing treatment goals (for the study’s intervention) in conjunction with parents, teachers, staff therapists, and the children; or using the children’s existing treatment goals according to their Individual Educational Plan. Examples for the methods researchers used to obtain outcomes that related to children's performance in natural settings included researchers’ observation of the children in natural settings either in person or via videotape; and parents’ report of performance in natural settings. The meaningful change in the children's real-life performances were reported in ways such as descriptive explanation of how the children improved in their performances in daily activities; perceived change or expressed satisfaction with change by the children themselves, parents, teachers, or therapists; using normative comparison either by direct observation or rating scale or comparing with normative distribution; using other statistical methods such as effect size, and indicating the disappearance of symptoms in children who received the intervention.
A linear regression line was imposed on data plots of each coded aspect to examine the trend from 1992 – 2001. The aspect that showed the clearest upward trend was that of reporting meaningful change in the children's real-life performance as a result of the intervention. The trends were either relatively unchanged or slightly downwards for others. It was noted that the number of studies that included each aspect (as being coded in the review) vary considerably from year-to-year with mostly no noticeable pattern, except that of reporting meaningful change as indicated above.
Focus group study
Five occupational therapists, one physical therapist, and seven speech language pathologists who practiced in the Greater Milwaukee area participated in the focus group study. All participants were females with age groups ranged from 21-25 to 51-55. Two participants had a Bachelor’s degree while others had Masters degrees. The participants’ practice experience ranged from 3 to 31 years. The majority of participants indicated that they read 1-2 research articles per month.
Participants identified the difficulties they experienced when trying to use research information for practice and the solutions they felt would make research information more usable. From the discussion, there seemed to be different aspects of difficulties and solutions related to research design, research article content, research article format, researchers, and practitioners’ situation. Examples of top barriers identified by participants included 1) research topics not relevant to practice; 2) study samples not relevant to practice; 3) lack of explanation for practical applications; 4) researchers’ (perceived) lack of real-life knowledge about clinical situations; and 5) presentation of research not user-friendly. Example of top solutions included 1) summarize and highlight results and how they can be applied real-life setting in research article, 2) clearly state the purpose of the research at the very beginning, 3) using reader a friendly language and presenting research information including using concise language, short sentences, and bullet points, 4) articles rated by a panel of practitioners for their usability for practice, and 5) more collaboration between researchers and clinicians in specific setting to which the research pertains.
Practitioners expressed favorable responses to the introduction of the social validity, ecological validity, and clinical significance concepts. They also indicated that most of the techniques researchers used in the past (as drawn from the earlier systematic literature review) were appealing to them and were perceived as likely to increase their use of outcome studies for practice if those concepts and techniques were to be implemented by researchers on a regular basis.
Mail survey study
Nine hundred sixty-two returned questionnaires were determined to be usable for analysis (some were eliminated due to excessive missing data), constituting a return rate of approximately 64%. The responses were received from 349 occupational therapists, 320 physical therapists, and 288 speech language pathologists in 50 states with ages ranged from 24 - 80 years, and an average age of 43 years. The majority of the respondents reported having a Master's degree, and indicated that they read between 1-2 articles per month and 1-2 research articles per six months. The respondents' practice experience ranged from less than 1 year to 43 years with the average experience of 17 years. Less than 5% of the respondents were male.
Even though almost 80% of the respondents indicated that it was either important or very important to use research evidence for treatment efficacy/effectiveness as a basis for practice, and more than 50% of respondents either agreed or strongly agreed that research was a stronger evidence than other evidence, the respondents indicated they often used colleagues and their own experiences as source of evidence, and only sometimes used research evidence as a basis for practice. The majority indicated that it was very important for research information to be easy to understand and use, and the treatment investigated to be easy to implement in their practice settings.
The three situations that the majority of respondents indicated to be serious problems when trying to use research information for practice were the lack of clear conclusion as to whether the treatment investigated was effective, insufficient practical information within research articles, and the lack of explanation for application of treatment in practice settings. Examples of situations that were indicated to be moderate problems included 1) study topics were not relevant for practice, 2) no clear reason why the study was done related to the need in practice setting, 3) treatment investigated could not be implemented in practice settings, and 4) not enough information about the meaning of test scores used to measure treatment outcomes.
The solutions suggested by the majority of respondents as very likely increase their use of research information for practice were for the researchers to summarize and highlight the study results and how they could be applied in real-life setting, and for the researchers to provide a clinical meaning of the outcome rather than presenting only numbers (test scores). Examples of other solutions perceived as likely to increase the use of research information were for the researchers to conduct research in real clinical environments, use simpler methodology in the study to enable application, use reader-friendly language when writing research articles, and use more charts and graphs when reporting results.
The majority of the respondents indicated that the use of the social validity, ecological validity, and clinical significant concepts would be either helpful or very helpful to make research information on treatment efficacy/effectiveness more usable for practice. All examples (given in the survey based on the systematic literature review) of the implementation techniques for the social validity, ecological validity, and clinical significance concepts from a number of past outcome studies were rated by the majority of practitioners as either likely or very likely to increase their use of research information for practice.
Conclusions and Implications
The preliminary findings of this project seem to indicate that using the social validity, ecological validity, and clinical significance concepts in designing and conducting outcome studies would be beneficial and would likely increase the use of research information by rehabilitation practitioners for evidence-based practice. The use of social validity ecological validity and clinical significant concepts was then termed the social validation model for creating outcome research. The model includes the following aspects that should be explicitly verified or included in an outcome study: 1) the topic of study is what the research consumers are interested in or need to know about, 2) the treatment procedures used in the study are acceptable to the research consumers, 3) the treatment outcome chosen to be measured in the study is what the research consumers feel to be an important outcome of interest, 4) the outcomes chosen are linked to performance of daily activities in natural settings, and 5) the improvement as a result of treatment is reported in a clinically meaningful way in addition to statistical significance.
Even though those concepts had been implemented somewhat in the past by researchers, they were applied sparingly and, in many cases, only one concept was implemented but not the others. Creating research information using the social validation model could have a significantly positive impact on the research utilization of rehabilitation practitioners for practice. Although there are certainly many factors that can have an impact on research utilization aside from the characteristics of research information itself, as indicated in this study, improving this one aspect in conjunction with other efforts (that others are investigating) would certainly help to contribute to a larger and timely gain in moving evidence-based practice forward in rehabilitation. Increased use of research information by rehabilitation practitioners will ultimately benefit rehabilitation consumers who will receive rehabilitation services that are based on current and best evidence.
Passive distribution of this information will neither be sufficient nor timely for the social validation model to be used by its potential users. In the next step, a structured utilization plan that caters to the needs of each potential user group, and that is developed through the potential users’ active engagement, is necessary for the model to be successfully adopted and implemented by its potential users. Presently, a grant proposal has been submitted to NIDRR's Field-Initiated Program to develop such a utilization plan, and it is currently in the review process. In the meantime, the findings from this project have been, and will be disseminated through speaking engagements, presentations at conferences, publications in peer-reviewed journals, and Web sites.
References
Champion, V. L., & Leach, A. (1989). Variables related to research utilization in nursing: An empirical investigation. Journal of Advanced Nursing, 14, 705-710.
Delbecq, A. L., Van de Ven, A. H., and Gustafson, D. H. (1975). Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Dallas, Texas: Scott, Foresman & Co.
Di Fabio, R. D. (1999). Myth of evidence-based practice. Journal of Orthopeadics & Sports Physical Therapy, 29, 632-634.
Dubouloz, C.-J., Egan, M., Vallerand, J., & von Zweck, C. (1999). Occupational therapists’ perceptions of evidence-based practice. The American Journal of Occupational Therapy, 53, 445-453.
Funk, S. G., Champagne, M. T., Tornquist, E. M., & Wiese, R. A. (1995). Administrators views on barriers to research utilization. Applied Nursing Research, 8, 44-49.
Holm, M. B. (2000). The 2000 Eleanor Clarke Slagle Lecture: Our mandate for the new millennium: Evidence-based practice. The American Journal of Occupational Therapy, 54, 575-585.
Kajermo, K. N., Nordström, G., Krusebrant, å. & Björvell, H. (2000). Perceptions of research utilization: Comparisons between health care professionals, nursing students and a reference group of nurse clinicians. Journal of Advanced Nursing, 31, 99-109.
Law, M., & Baum, C. (1998). Evidence-based practice occupational therapy. Canadian Journal of Occupational Therapy, 65, 131-135.
Logemann, J. A. (2000). From the President: Are clinicians and researchers different? ASHA Leader, 5(8), 3.
Ogles, B. M., Lunnen, K. M., & Bonesteel, K. (2001). Clinical significance: History, application, and current practice. Clinical Psychology Review, 21, 421-446.
Rogers, E.M. (1995). Diffusion Innovations New York: The Free Press.
Sackett, D. L., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (1997). Evidence-based medicine: How to Practice and Teach EBM. New York: Churchill Livingstone.
Sbordone, R. J. (1996). Ecological validity: Some critical issues for the neuropsychologist. In R. J. Sbordone, & C. J. Long (Eds.), Ecological Validity of Neuropsychological Testing (pp.15-41). Delray Beach, FL: GR Press/St. Lucie Press.
Schiller, M. R. (1998). Linking outcomes research, education, and practice. Journal of Rehabilitation Outcomes Measurement, 2(6), 1-8.
Turner, P., & Whitfield, T. W. A. (1997). Physiotherapists’ use of evidence based practice: A cross-national study. Physiotherapy Research International, 2, 17-29.
Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavioral Analysis, 11, (2):203-214.
