ADA Systematic Review—Technical Report of Scoping Review
ADA Systematic Review—Technical Report of Scoping Review
Americans with Disabilities National Network
Knowledge Translation Center
Systematic Review Project
Year 1: Scoping Review of the ADA
TECHNICAL REPORT
Authors:
Department of Disability & Human Development
University of Illinois at Chicago
Sarah Parker Harris, Principal Investigator
Robert Gould, Project Director
Glenn Fujiura, Co-Investigator
Robin Jones, Co-Investigator
Patrick Ojok, Research Assistant
Avery Olmstead, Academic Librarian
February 15, 2013
Funded by NIDILRR Grant # H133A11014
Table of contents
- EXECUTIVE SUMMARY
- Overview
- Highlights of Results
- Dissemination
- 2. BACKGROUND
- 3. METHODOLOGY
- 3.1 Overview
- 3.2 Identification of databases and sources for literature
- 3.3 Locating, Screening & Selection of Records
- 3.4 Extraction, Analysis & Synthesis of Data
- Initial extraction
- Mixed categorical coding
- Reliability Checking
- Analysis and Synthesis
- 3.5 Limitations of the Study
- 4. RESULTS
- 5. SUMMARY & NEXT STEPS
- REFERENCES
- APPENDICIES
EXECUTIVE SUMMARY
Overview
More than 20 years after its implementation, there are still conflicting arguments and fragmented research findings about the ADA’s effectiveness across the various stakeholder groups it impacts. To address this fragmentation in research the University of Illinois at Chicago is conducting a five-year multi-stage systematic review of the ADA as part of the NIDRR funded National ADA Knowledge Translation Center based at the University of Washington. The project includes a scoping review, a rapid evidence review, and systematic reviews. This report provides a summary of the progress of the first year of the project and the findings of the scoping review.
The primary purpose of the scoping review has been to map the landscape of ADA research by examining the extent, range and nature of research evidence while consulting with key ADA stakeholders in the implementation of the policy. The first stage has entailed:
- Searching for “ADA” and the appropriate truncation in academic databases.
- Locating the academic research and grey literature on the ADA and extracting the key information from only the abstracts of research materials.
- Inputting key data points from the abstracts of research materials to be used to formulate research questions that will guide the future rapid evidence and systematic reviews.
- Generating a descriptive overview of the subgroups studied, topics covered, research questions asked in the research evidence, and direction of findings.
Highlights of results
The initial review of the scoping review yielded some surprising results and also confirmed some previous notions from scholars of ADA research. The research evidence that was tracked across research records included: the different stakeholder groups and specific stakeholders, topic and subtopics, research methods, and primary factors that were studied in each study. Some of the key findings include:
- The majority of research (67%) comes from academic journals and the remainder comes from conference proceedings, books, dissertations and theses, and organizational reports.
- A range of different research methods were used in ADA research include quantitative, qualitative, mixed-methods, and theoretical/policy analysis techniques. About half of all the research on the ADA is quantitative, and a substantial portion of mixed-methods research also includes quantitative analysis.
- The most common issues studied across research included attitudes and knowledge about people with disabilities and the ADA; barriers and facilitators to implementing the policy; compliance rates; and costs associated with the Act.
- There is very little research across different stakeholder groups and the majority of research is either on people with disabilities or businesses/employers.
- The most common topic of the ADA research gathered is employment (48% of the research records) and the second most common is education (16%).
- Critical access and accommodation issues (i.e. emergency preparedness (?) and voting) have virtually no research.
Dissemination
One of the primary goals of the systematic review project is to disseminate research findings nationally to a broad range of stakeholders to ensure its application in improving the implementation of the ADA. The two primary methods of achieving this goal are by creating plain language reports and publications, in addition to national presentations. To date we have:
- Drafted two plain language summaries (defining what constitutes ADA evidence and an overview of the ADA systematic review project and methods).
- Provided four presentations to different stakeholder groups (an expert panel of ADA stakeholders; local ADA Center’s advisory center; ADA Network’s Directors meeting; and the ADA Network National meeting).
Additional publications and presentations are planned based on the research conducted in year one of the project, which include: one plain language summary and an academic paper on the scoping review findings; academic paper on systematic review methodology; and presentations at two national academic conferences (Society for Disability Studies, the American Sociological Association), and a symposium on disability rights at the University of California-Berkeley.
1. INTRODUCTION
The Americans with Disabilities Act (ADA) is the cornerstone of US civil rights policy for people with disabilities, and the primary legislative tool for ensuring their full and equal treatment as citizens. The legislative intent of the ADA is to protect against discrimination while simultaneously fostering social inclusion across all domains of public life. While the law is the primary means of ensuring protection of rights, there has been much debate on its impact and effectiveness as a social policy. When the drafting of the Americans with Disabilities’ Act began in 1988, a wide body of research documenting the discrimination and inaccessibility across the country was developed to convince the government about the importance of a new federal policy. Since its inception into law in 1990, an even broader body of research has been developed to track the progress in addressing such inequalities. However, the research to date on its impact has often been duplicative, inconclusive, or conflicting. The state of ADA-related research is highly fragmented, comprising of a multitude of definitions, theories, topics, disability subgroups, stakeholder perspectives, multiple methods and outcomes. This represents a rich resource, but the volume of research on the ADA makes it hard for researchers, practitioners and policymakers to distill what is most useful for their information needs. More than 20 years after its implementation, there are still conflicting arguments and fragmented research findings about the ADA’s effectiveness across the various stakeholder groups it impacts. The National Council on Disability (2007) explains that the lack of ongoing systematic study and disagreements within existing research evidence has resulted in “significant knowledge gaps about the impact of the ADA.” The discordant state of research on the ADA continues to be a significant barrier to improving its implementation.
To address this fragmentation in research the University of Illinois at Chicago (UIC) is conducting a five-year systematic review of the ADA as part of the National Institute on Disability and Rehabilitation Research (NIDRR) funded National ADA Knowledge Translation Center based at the University of Washington. The project was developed in response to a call by NIDRR to “increase the use of available ADA-related research findings to inform behavior, practices, or policies that improve equal access in society for individuals with disabilities.” UIC addresses this call through a multi-stage systematic review that includes: (1) a scoping review of how the ADA has been studied to map the literature landscape; (2) a rapid evidence review to refine priorities and analyze selected topics for preliminary assessment of the research; and (3) systematic reviews to synthesize research and answer specific key questions in the area. We will use these reviews and syntheses to create a foundation of knowledge, inform the subsequent policy, research and information dissemination, and contribute to the overall capacity building efforts of the ADA Regional Centers. Together, these reviews will identify a future plan for research that spans core ADA research topics, methodological approaches to ADA-related research and potential outcomes to inform policy and practice, and positively impact on ADA stakeholders. To date, no systematic reviews have been conducted in this area.
In order to increase the use of available ADA-related research findings to inform policies and practices that maximize the full inclusion in society for individuals with disabilities, and taking into account the heterogeneous nature of ADA research, a three stage approach to a systematic review is necessary. This report details the activities and findings from stage one (scoping review).
2. BACKGROUND
A scoping review is a review that is often carried out in advance of a systematic review to develop an understanding of the research landscape by examining the extent, range and nature of research evidence (Grant & Booth, 2009). This may include subgroups that have been studied, research questions asked, methods used, and direction of findings. Scoping studies tend to address broader topics than systematic reviews. Scoping reviews are particularly useful to investigate key research priorities and point the way to future research (Levac, Colquhoun, & O’Brien, 2010). Jesson, Matheson, and Lacey (2011) define the purpose of the scoping review:
A scoping review sets the scene for a future research agenda…The review documents what is already known, and then, using a critical analysis of the gaps in knowledge, it helps to refine the research questions, concepts and theories to point the way to future research” (p. 15).
In short the primary purpose of a scoping review is to provide a broad overview of the current research on a topic and to document key components of the research in order to identify specific gaps and key research needs based on the existing research evidence. While the primary purpose of a scoping review is to map the state of evidence in a given area, there is much debate about what constitutes evidence when it comes to research on the ADA. Additionally, as noted above, the fragmentation and volume of research on the ADA makes it hard to know what is currently useful. Sourcing records is more complex than in traditional systematic reviews. A substantial source of evidence for key ADA decisions to date has been academic and technical research. Given the non-availability of many qualitative accounts, search strategies must augment electronic searchers with more traditional methods of reviewing including back-tracking of references and engaging with key experts in the field (Petticrew & Roberts, 2006)). ADA-related evidence can come from published and/or unpublished research from a variety of sources:
- Published studies are found in academic databases, journal articles, books and book chapters, dissertations, background material used in court case decisions; as well as research conducted or commissioned by government departments, disability, technical, and other stakeholder agencies, ADA Regional Centers, research institutes, think tanks, and employer and business-related organizations.
- Unpublished studies (‘grey literature’) are found in many of the above same sources, but especially in government agencies, non-government and technical organizations, dissertations, and research institutes. Because grey literature is unpublished, obtaining the research and reports requires the expertise and knowledge of key contacts and stakeholders working in the field.
Evidence-based practice in other domains (i.e. medicine) often relies on obtaining quantitative research and intervention studies conducted on narrow topics. However, the state of ADA research provides a much wider array of topics and fields to draw from. For example, studies may include qualitative, quantitative and mixed methods research that address a myriad of areas, including: social movements, rights and participation; legal and economic issues; socio-political discourse and cultural values; stakeholder perspectives and experiences; and issue/title-based topics such as employment, education, health and technology, etc. The heterogeneity in both method and content of ADA-related research provides a wide basis of what constitutes evidence.
With this in mind, the research team formed an ADA Expert Panel. Systematic review projects often rely on consultations from content experts to exhaustively locate materials that are seldom disseminated, confirm findings and assumptions about the overarching themes from categories of research, and validate key research findings (Petticrew & Roberts, 2005). The ADA Expert Panel consists of key ADA stakeholders, including representatives from the National Council on Disability, the Disability Rights Education and Defense Fund (DREDF), the National ADA Network, Mathematica Policy Research Group, the US Business Leadership Network, various Universities, and other pertinent organizations (for a complete list of members refer to Appendix I) The research team has worked in collaboration with the ADA Expert Panel throughout the scoping review to decide what is evidence, what should be included as evidence, and what is important in the evidence. In subsequent reviews, the research team will continue to work with the Expert Panel to identify ADA knowledge gaps, shape future ADA research topics, and, through dissemination and knowledge translation, contribute to an improved understanding by ADA stakeholders of their rights and responsibilities under the ADA.
3. METHODOLOGY
3.1 Overview
From January to September 2012, the UIC research team worked on forming the Expert Panel committee; developed the research protocol, research questions, and inclusion criteria; and identified, screened and gathered relevant research on the ADA. Data extraction, analysis and synthesis were conducted during October to December 2012. The scoping review entails locating both published research materials from various online databases and also unpublished and hard-to-locate (grey) research literature through specific data base searching, interlibrary loan requests, and contacting various agencies and organizations. In addition to retrieving materials, the scoping review also requires cataloguing the literature in a database system to identify key components of each study.
The research team conducted an extensive scoping review of the literature that involved:
- Searching for “ADA” and the appropriate truncation in academic and other databases. Search terms of general scoping reviews are broad, and do not typically go beyond full text searches for the primary topic under investigation.
- Locating the relevant research and grey literature on the ADA (referred herein as ‘records’), and extracting the key information (referred herein to as ‘data’) from only the abstracts of research materials.
- Generating a descriptive analysis and synthesis of the data to map the landscape of how the ADA has been researched and studied.
The research questions guiding the scoping review were developed by the research team in conjunction with the ADA Expert Panel, with member-checking with disability consultants serving as key population informants (refer to Table 1). The questions were kept intentionally broad at this stage of the project in order to understand and assess what has been studied. Subsequent reviews will seek to answer more specific research questions.
Table 1: Scoping Review Research Questions
The inclusion criteria consists of citations to all records identified as empirically[1] examining the ADA by a literature search using the following parameters: (a) published or dated from 1990; (b) be written in English; (c) have been carried out in the United States; (d) relate to the ADA; and (e) be based on published studies reporting the gathering of primary or secondary data or the collating and synthesis of existing information to answer ADA-related research questions. Items that are not included in the initial scoping review are established facts about the ADA (i.e. court-case decisions, technical materials on compliance, general fact sheets), opinion pieces (i.e. by various stakeholders, lawyers or academics), and anecdotal evidence. The inclusion criteria is broad, as the topic of study is broad.
The following sub-sections contain a detailed description of the scoping review record and data collection process, involving: identification of records; location, screening and selection of records (limited to abstract); extraction, analysis and synthesis of data (limited to abstract/descriptive).
3.2 Identification of databases and sources for literature
The first step of the scoping review process was to identify key databases to access pertinent research records. The research team took multiple steps to ensure a broad expanse of research was investigated. We first consulted a checklist of key sources based on the suggestions of the Centre for Reviews and Dissemination,[2] a UK based research center that provides guidance to ensure academic rigor in the systematic review process. This checklist was used to compile an initial spreadsheet of databases. Further, an academic librarian consultant reviewed the list and added in additional databases that were believed to contain return relevant academic literature. A full listing of the databases searched is included in Appendix II. These databases include a wide range of academic journals that were potentially relevant to the research project. 321 different journals revealed relevant literature. Appendix III contains a listing of all of the journals that contained included research on the ADA.
In addition to a comprehensive list of academic database, multiple steps were necessary to locate the broad expanse of ADA research evidence. A comprehensive listing of relevant organizations that have potentially researched the ADA was compiled by the research team. This listing was expanded through general searching on Google for disability organizations with assistance from the academic librarian. The research team retrieved all relevant research records from the organizational websites. The research team conducted a thorough investigation of the organizations’ websites using appropriate search teams (i.e. “Americans with Disabilities Act;” “ADA”; or truncated search terms for websites that had database-like search capabilities and not general site searches such as American* Disab* Act). If no resources were found via the websites, the organizations were called or emailed to verify the availability/unavailability of research regarding the ADA. This search method, however, did not yield any additional pertinent research evidence that had not already been retrieved from either academic databases or the previous grey literature searches. A full listing of the organizations that were searched is included in Appendix IV The listing will be reviewed by the Expert Panel at the next meeting (February) to ensure that all relevant organizations are included. This finalized list of organizations with possible ADA resources will be utilized in future systematic reviews to exhaustively search for research on specific top areas. Due to the nature of grey literature (not published by mainstream publishing clearinghouses, independently published records/reports from agencies, or fully unpublished), alternative search methods were also employed drawing on several different search strategies. These methods include utilizing the primary researchers’ knowledge and familiarity with the research evidence; communicating with the network of researchers involved with ADA research; cross-checking bibliographies; and collaborating with the Expert Panel to locate hard-to-reach-studies and confirm saturation. Additionally, we used an expansive database system of grey literature, OCLC Worldcat,[3] to examine different library holdings of unpublished and hard to reach materials. One advantage of searching through this system database is that searching the database permitted the use of more traditional systematic review search methods (i.e. tracking search results, records, and search terms). However, there is greater difficulty in identifying and confirming the relevance of the studies retrieved from the database as there is limited information available. Records often include a title and/or brief description rather than a full abstract (necessary for a scoping review), and may need to be located/examined in print before they can be confirmed as research. Furthermore, references to research studies are sometimes incomplete or referred to in multiple data repositories (i.e. organizational websites and/or grey literature search databases).
At this time, other grey literature databases besides Worldcat have been excluded from the initial scoping review per the advice of the research librarian, Most (if not all) pertinent research will be found in either published articles, reports, or unpublished dissertations that are included in this database. For example, Worldcat includes dissertations and theses from the same institutions as other grey literature clearinghouses, such as the OpenDoar system and others which are often based on Google search platforms. However, once specific systematic review questions are identified and we can introduce additional and more specific search terms, it is possible that these grey literature database may return previously unfound research records regarding the ADA. For the scoping review process, it would not be viable to go through the full records of the various Google based platforms.
3.3 Locating, Screening & Selection of Records
The next step of the scoping review process involved locating, screening and selecting records based on the inclusion criteria. The initial search criteria yielded 34,599 records. This included:
- 26,371 records from Academic databases
- 6,975 records from the Worldcat library of grey literature items
- 1,253 records provided from the National Rehabilitation Information Center (NARIC[4])
Records were scanned for initial eligibility and key article information (i.e. journal, title, date, authors, abstract) and saved to an online bibliographic citation management system called RefWorks.[5]
- For academic database and NARIC records: the initial scanning process consisted of reviewing the full title and abstract for eligibility. Articles excluded were those that were not research on the Americans with Disabilities Act; conducted before 1990; not conducted in the US or in English; and/or were duplicate records or separate publications of the same study (i.e. research briefs that provided a short overview of complete doctoral dissertations were excluded in favor of a record of the full dissertation).
- For grey literature: the initial scanning process consisting of a general search using the terms “Americans with Disabilities Act” in the Worldcat system in addition to specific searches in the WorldCat dissertations, conference proceedings, and papers databases. Preliminary screening found the majority of records did not include the analysis or collection of primary or secondary data (e.g. not research-based but rather a variety of congressional records, governmental hearings, training manuals, and compliance guides about the ADA, etc). Items that could not be verified as relevant through preliminary screening were ordered as print materials, microfilm, and/or microfiche and screened. Relevant records were subsequently scanned and saved electronically.
Further screening and selection by the research team reduced the number of records from the initial 34,599 to 3,351 records that were identified as potentially relevant and required data extraction (see next section). This number included both retrieved academic literature and grey literature that had been preliminary identified as meeting the initial inclusion criteria (refer to p. 8) or possibly meeting the search criteria (but had not yet been received). Also included in this group were legal documents that had to be vetted for inclusion (i.e. legal or policy analysis that clearly indicated an ADA-related research question and the collation and synthesis of existing evidence in the research abstract), and books/other grey literature that had been scanned but not yet vetted for inclusion/exclusion.
A final screening was conducted by a secondary reviewer who examined the full 3,351 records for meeting the inclusion criteria. After this screening, 960 studies were dropped after the secondary reviewer noted that the studies did not meet the inclusion criteria. Additionally, 1,417 research records were saved for categorized as legal research that fit outside the scope of this project. Lastly, 37 organizational records were saved for future reviews because they could not easily be coded due to their expansive coverage of multiple ADA issues. Future reviews will include these 37 organizational records as well as supplementary organizational materials relevant to different topic areas. Figure 1 below provides a visual representation of the decision processes that were used to finalize the selection of 980 pertinent research records.
3.4 Extraction, Analysis & Synthesis of Data
From the selected 3,351 potentially relevant records we proceeded to the data extraction stage of the scoping review process. Key information points from the academic record abstracts were electronically extracted and entered into a formal spreadsheet document. This included basic record information as noted above, as well as information related to the research questions (e.g. topic of research, stakeholders involved, research design, etc). Extracting similar information from the grey literature and other non-academic records involved a more time-consuming and complicated process. Citations from research in whole books, book chapters, agency reports, and other grey literature reports needed to be manually entered into the spreadsheet. Further, these records were predominantly in hard copy and thus required manual scanning (i.e. from microfiche), where individual reports and/or documents took several hours to scan.
In line with conventional systematic review techniques the data extraction included intermittent reliability checks along with specific steps to ensure that the data entry and coding was methodologically sound. Data extraction consisted of three separate steps: (i) initial extraction, (ii) Mixed categorical coding, and (iii) reliability checking. These steps are described below.
Initial extraction
The first step of the data extraction process involved retrieving all of the bibliographic information from the pertinent research records. Citation information for all literature was saved to Refworks. The citation information was then exported to an excel database where additional coding and data entry was conducted.
The initial extraction process involved recording data from (i.e. direct quotes; or, when unavailable, a summary) from all (3,351) research abstracts and entering this into the Microsoft excel spreadsheet under the following broad headings:
- Organization
- Method/Research Design
- Study Goal/Aim/Purpose
- Key Findings (when provided in the abstract)
- Stakeholder category
- Topic/Sub-topics
In addition to the data entry portion of the data extraction, we also verified that the studies met the inclusion criteria during this stage of the research. 2,377 of the 3,351 records were excluded. Reasons for exclusions were that the records did not meet inclusion criteria or the material was unobtainable. Many of these records were found to be legal research, which was beyond the scope of this scoping review (see Limitations section below). Although research findings and study goals were retrieved when they were reported, these data points were not synthesized for this portion of the review. There was not sufficient data to complete an overview of these categories for the scoping review as many social science abstracts do not include descriptions of the findings or goals.
The initial extraction process involved taking direct quotations that applied to each of the headings. When no quotation was available, a descriptive or summative account was provided by the initial reviewer. In addition to the quotations, these descriptions were used to create a list of potential codes to be used for the coding process.
Mixed categorical coding
After the initial data extraction, an preliminary list of coding categories was developed to begin the coding process. A set of categories of topic areas, stakeholder groups, and research methods was collaboratively developed by the research team that had been involved with the data extraction. Additional codes were then developed using two steps. First, a list of potential categories for topic areas and stakeholder groups was generated based on input from the Expert Panel (see Appendix V). The generation of codes stemming from stakeholder input is a common systematic review strategy known as categorical coding. It is useful for studying pre-defined concepts and theories (Gough, Oliver, & Thomas, 2011). The second step for generating codes was an iterative process. Codes were assigned to each record primarily from quotations and the records were then placed into categories. The initial listing of potential codes was refined at this point based on a secondary review of the initial extraction and expert panel input. Additional codes were generated directly from the quotations and summary statements created during the first stage of the data extraction process that could be used to categorize and sort the different research records. When quotations were unavailable, the summary statements or descriptive keywords were used to generate the appropriate code for the research record. This coding process, often referred to as mixed coding, involves building from pre-defined concepts about a topic area to explore other potential topic areas that emerge from the literature (Gough, Oliver, & Thomas, 2011).
The secondary review of the initial data extraction revealed additional codes and topic areas (marked with an asterisk in Appendix V). Subgroups and subtopics were also assigned based on the initially extracted data. One new topic area (people without disabilities) was added to the final list of codes. This final coding list was used to review the 980 relevant research records that met the inclusion criteria. 37 from organizational reports were additional records were set aside at this point for future systematic review (see Limitations section below).
Reliability checking
Using the coding sheet, a second reviewer re-examined the initial data entry and cross checked it against the abstract. This process allowed for a brief check to enhance the interrater reliability of the initial data entry and to confirm the categorical codes. This secondary review was primarily useful to ensure the reliable entry and coding of data when no direct quotation was available from the abstract. At this point, the coder marked any disagreements between the assigned codes and the previously generated description into a new column in the Microsoft excel spreadsheet (without physically changing any of the work of the initial data extracter {from step i.} or the coder {from step ii}). Resolutions occurred by the initial reviewer either agreeing with the suggestions, or when agreement could not be made, a third senior reviewer then cross checked and made the final decision about the assigned codes. All codes were finalized before the synthesis stage.
Analysis and synthesis
The final stage of the scoping review process is descriptive analysis and synthesis of the data which was completed using a descriptive numerical summary (e.g. overall number of studies included, types of study design, topics and/or titles studied, characteristics of disability sub-groups and/or stakeholders, years of publication). The frequency that different research design elements occur in ADA research was calculated to provide an overview of the most common approaches to studying the ADA. The analysis provides a descriptive account of the various factors and data points coded from the different research records. The coded data was grouped in broader methodological, stakeholder and topic categories. The topic and stakeholder categories were created from the expert panel feedback and the initial research team’s input during the preliminary data extraction phase. The subtopics were created from a scanning of the research abstracts for themes and context and reflect the codes that were assigned by the reviewers during the data extraction phase. The factors indicate descriptions of the subtopics and how they were studied in the research and were assigned to each record by the Principal Investigator and Project Director jointly during the final review of the coded data.
3.5 Limitations of the Study
The primary challenge for the UIC research team has been the retrieval of unpublished and print materials. Unpublished and print materials pose several challenges regarding locating and entering appropriate data. First, the majority of the difficulties arise from obtaining material from 1990-1994 before the internet became readily available to many libraries. These materials are expensive to ship, and libraries are often reluctant to send them. Additionally, many of these materials are not permitted to leave the special collections departments of various libraries (academic and public). When possible, the research team requested photocopies and/or electronic scanned files of the abstracts, introductions, and table of contents. However, a large portion of print materials are still unobtainable at this time. Lastly, a number of unpublished dissertations and research reports are only available via microfilm or microfiche format. When lending libraries are able to deliver these items, it is a very time intensive process to review the research materials and it typically takes between two and three hours to scan one microfilm/microfiche document (such as a thesis or dissertation) in its entirety.
A second limitation in the scoping review is that the research time could not exhaustively search all clearinghouses of grey literature through online databases. Various grey literature databases (such as the OpenDOAR database system) pose additional challenges to retrieve pertinent information in that they are not connected with the electronic library cataloguing system and are only searchable using a Google platform. The Google platform does not offer a way to systematically exclude duplicate and/or irrelevant research records, (for example, a Google scholar search for “Americans with Disabilities Act” yields over 55,000 results). Furthermore, many grey literature databases, including OCLC Worldcat which was used in the scoping review, do not contain abstracts and at times it can be very difficult to identify the nature of the item to indicate if it is a research document. More often than not there is little information available about the records other than the title, author, and holding library. A portion of the records from OCLC Worldcat system were found on the internet to help deduce the nature of the material and whether or not it was research evidence. When there is no information available, it is often necessary to have the materials shipped to the research team although an extensive amount of materials end up being irrelevant to the study. Therefore only the OCLC Worldcat system was used to search for grey literature due to the extensive time and resource commitment for this type of search and the likelihood of retrieving impertinent materials.
Additional limitations involve the categorization of research records. There are a number of difficulties in separating out records and categorizing them appropriately. When closely examining the analysis, some of these frequencies of different topic areas and stakeholder groups might seem surprisingly low. One of the reasons for this low frequency is in the overlap of topic areas. For example, although only one study pertains to the topic area of nursing (coded as healthcare services), nurses have been stakeholders in many other research studies on the ADA. For example, multiple studies pertain to the education of nurses and a number of studies also pertain to workplace factors such as discrimination or employment of nurses. While there would be valid arguments that these other areas ultimately impact healthcare, this category specifically deals with the actual implementation and delivery of healthcare services as impacted by the ADA.
The difficulty in separating and categorizing research records was further exemplified in the search for research evidence from major disability organizations and possible stakeholders in ADA research. When research records were gathered from organizations, it was often not in a form that could easily be reviewed and included in its current form. For example, the exhaustive and thorough research of the National Council of Disability often includes multiple stages of research, covers various topic areas, and includes a multiplicity of stakeholder groups. Thirty-seven reports containing the collection and/or analysis of existing research were excluded from the scoping review because the wide variety of topics covered in the varying research reports could easily cover each of the broad categories created for the scoping review. It was thus decided that the evidence retrieved from organizational searching would be saved for future more review where key study elements would be analyzed more closely.
Another topic that requires attention in future review is the research of legal scholarship. One priority topic identified by multiple members of the Expert Panel was the legal analysis of the ADA. The Panel particularly emphasized that analysis of early legal research from national organizations, such as the National Council on Disability, is important to explore as a baseline point in the progress and development of research on the ADA. In reviewing legal research, however, there are not clear criteria used to distinguish empirical legal research (legal research that contains the collection and analysis of non-legal data) from other work that appears in law reviews and other typical sources of legal scholarship. In relation to the literature on the ADA, few research studies have a clear data collection and methodology for review. Often times it is difficult to assess if law review articles are driven from specific research questions or strategies and many journals do not require an overview of methodology or theoretical framework. A common and longstanding critique when evaluating legal scholarship is that it often can blend policy and opinion without research-based evidence (Kissam, 1988). The empiricism of legal scholarship thus is a topic that is often in critical debate and falls beyond the scope of this study.
4. RESULTS
The final search yielded 980 separate research records on the ADA. The results have been descriptively analyzed and synthesized into the following categories: record type, stakeholder groups, topics, and research methods.
4.1 Record Type
Record type refers to the source of the literature. The main source of literature was academic journal articles, specifically the Journal of Vocational Rehabilitation (31 articles); Work: A Journal of Prevention, Assessment, and Rehabilitation (26 articles); the Journal of Disability Policy Studies (26 articles); the Journal of Rehabilitation (17 articles); and the American Journal of Occupational Therapy (20 articles) where the majority of ADA research has been published to date. Refer to Figure 2 for the overview of the different types of records that were gathered.
4.2 Stakeholder Group
The stakeholder group pertains to the research participants and/or subject of the research study. We identified eight main stakeholders and 54 subgroups (refer to Table 2). 868 studies involved key stakeholder groups, and 107 studies involved two primary groups. The majority of research focused on people with disabilities, followed by business/employers. Families/Advocates and people without disabilities (i.e. coworkers of people with disabilities) were the least researched groups.
Table 2: ADA Stakeholders and Subgroups
TOTAL*: 975 100% *107 studies included two or more stakeholders. 112 records did not have a primary stakeholder group and were either theoretical or architectural design/compliance related studies.
4.3 Topics
We identified 16 main topics and a number of different sub-topics. The description of each topic can be found in Appendix VI. The clear majority of research (about half of all studies identified in the scoping review) on the ADA relates to employment, specifically as it relates to discrimination and harassment. While most of the topics have 17 or more records, there were a group of topic areas that have been less studied in comparison to the other major topics. These include: child care, emergency preparedness and response, and voting. Refer to Table 3.
Table 3: Topic and subtopics of ADA Research
TOTAL: 980 100%
Sub-Topic Issues
The top five topics that were researched (employment, education, accessibility and compliance, transportation, health) were additionally analyzed into the key ways in which the sub-topics were studied (refer to Table 4). There a number of different issues that cross-cut topic areas, the most common being compliance rates (263 records); attitudes and knowledge about people with disabilities and the ADA (131); barriers/facilitators to implementation (77); and costs (37).
Table 4: Most common sub-topic issues (by number of records)
4.4 Research Methods
The research evidence on the ADA includes a wide range of methodologies and research design to study the policy’s effects and implementation. The research was categorized into four primary categories: qualitative, quantitative, theoretical/policy analysis, and mixed methods (refer to Figure 3). Almost half of all the research on the ADA is quantitative.
5. SUMMARY & NEXT STEPS
5.1 Key Findings
A scoping review has been used in this project to understand what has been researched and studied in relation to the ADA. It is the first step in a full systematic review. This stage of the research is mostly useful for providing a broad descriptive overview of the state of ADA research. Future analysis over the course of this project will generate more specific summative evidence regarding the effects of the ADA on implementation in key policy areas. Also, moving forward, the initial results have many broader implications for the future study of the ADA. The results from studying each of the key areas in the scoping review (method, topics, subtopics, and stakeholders) suggest the need for future inquiry in seldom-researched areas as well as expansive analysis in the areas for which we have more thorough and substantive evidence.
Future analysis in the systematic review will closely examine the variation in approach and findings that are arrived at using different methods and research designs. We found that the majority of existing research on the ADA is quantitative, followed by theoretical/policy analysis. Further qualitative studies are needed to understand the lived experienced of disability since the passing of the ADA. As we continue to see disagreement and fragmented results amongst the quantitative studies, qualitative research will provide a more analytical and exploratory account of ADA implementation.
In terms of stakeholders (i.e. either as participants in the research studies, or the subject of the research), the main focus of ADA research has been on people with disabilities and business/employers. While neither of these findings are particularly surprising, a more detailed account of how these groups have been approached within the research will tell us more about research directions and gaps in findings. Although people with disabilities are frequently the target of ADA research, it is still not clear how and which communities with disabilities have been approached. In the study of the ADA, future inquiry, including the future systematic review, need to more closely investigate how multi-layered categories are impacting research findings to fully comprehend how the policy is impacting rights and social justice for people with disabilities. Future inquiry of the ADA evidence requires a thoughtful analysis of how the stakeholders have been included in the research, and their role in their research in relation to social justice. In addition to defending the rights of the people with disabilities, the ADA also impacts numerous other stakeholder groups. Besides employers, other groups such as disability service providers, families and associates of people with disabilities, health care workers and many industry professionals are impacted by and are a part of the ADA’s implementation. However, many of these groups are inadequately represented in the literature. One of the most noticeable gaps is the very small pool of research specifically about family and advocates of people with disabilities. This group represents an additional gap that will require additional attention in relation to the ADA, especially as housing and support policies continue to shift away from the state to families and the community. Research concerning this stakeholder group spans many topic areas, including the developing discussions of care and aging. There has been minimal research that discusses the intersection of family policy and the ADA even with emerging trends and concerns for the well being of this stakeholder group.
While critical stakeholders and subtopic areas that are vastly important to the changing landscape of national disability policy remain understudied, some topic areas have been at the forefront of ADA inquiry since the policy’s inception. The main ADA topic studied has been employment, particularly as it relates to discrimination and harassment. These findings further support previous academic claims about the existing research evidence about the ADA. For example, that the ADA is overanalyzed in relation to the impact of Title I on the employment rate of people with disabilities. Reasons for the ADA’s failure are subsequently discussed in relation to the court’s actions and not other social institutions. These findings posit multiple theoretical questions and directions for the future of the field. First, with this large body of research can we make empirical claims on the ADA’s effectiveness in this area? Secondly, although the ADA covers many domains of social living, why is it so overwhelmingly analyzed in relation to employment? These questions require exploration in relation to the other key areas that were tracked in the scoping review, such as the methodologies used to study the ADA’s implementation. The need to investigate how research varies across different methodological pursuits can provide additional insight into key topic areas and pragmatic debates on the ADA. For example, although this review revealed that the topic area of employment is the most prevalent ADA research area, key facts and figures for which we have empirical and longitudinal data have still not provided us with an adequate depiction of how or why changes to labor market participation occur. Foregone conclusions on seemingly straightforward concepts such as employment and hiring rates have been challenged in research. Key questions, such as how the ADA has impacted the open labor market participation and experience of people with disabilities, remain unsettled and underdeveloped in research. Different quantitative research studies most often explore the issue of open labor market participation by analyzing hiring rates, labor participation rates, and frequency of discrimination charges. More than 150 research records pertain to the study of “rates” in relation to different areas of employment. In addition to rates, however, there has been a plethora of research that looks more into labor market participation from other angles such as attitudes or perceptions about people with disabilities. These issues have also been commonly explored using a quantitative methodology, looking at the frequency and prevalence of discrimination. Further qualitative inquiry can better piece together the relational aspect of many of these factors. The benefits of a full mixed-methods review is that future study will allow for the comparison of the many outcomes in key topic areas, and to simultaneously analyze how these conclusions have been drawn. Additional research needs to provide a more nuanced overview of the employment experience of people with disabilities to better settle some of the debates on the impact of the ADA’s implementation.
In addition to a broader analysis of the often-studied topic areas, future ADA research needs to explore often discussed areas that have little research evidence. Critical topic areas that have clear policy implications for people with disabilities remain under synthesized. For example, emergency preparedness has virtually no empirical research on its relation to the ADA. Federal government and other policy/agency organizations have historically neglected people with disabilities from emergency preparedness initiatives and policies. However, more recently there has been some push to develop policy that better includes people with disabilities in emergency planning. Yet, these policy suggestions and large-scale organizational reports have not resulted in subsequent research investigating the impact of the ADA in this area. Although emergency preparedness and disaster management are pressing policy issues, the intersectional analysis of emerging policies with the ADA is seldom explored. This area will require further attention given the increase in natural and other emergencies that require better policy responses.
Additional findings related to underexplored research areas are puzzling both due to their importance in relation to the ADA and the large pool of unutilized data in their areas. For example, the limited research on the ADA and housing is also surprisingly low, particularly when there is access to longitudinal data in this area (e.g. State of the States on Developmentally Disabilities and other related projects). However, only a small amount of research asks how the ADA specifically has impacted housing. Additionally, there is a dearth of information of how the ADA has impacted the civic engagement of people with disabilities. Similar to the area of housing, large pools of longitudinal data exist that track progress in relation to issues such as voter turnout (e.g. the Harris Poll and various institutional reports that make use of their data sets). The dearth of literature on these two topic areas is partly indicative of a limitation of this scoping review because large scale organizational reports spanning multiple topic areas, including voter turnout, were excluded from the scoping review. These reports could not be easily categorized into the various subtopic grouping and would skew the findings. Instead, such reports will be further utilized in future systematic reviews addressing more specific research questions. Future exploration of the compounding factors impacting the ongoing discrimination in the civic engagement of people with disabilities is necessary and requires thorough analysis of the available resources and longitudinal data. Additionally, next steps of the systematic review will further take into account how and what research evidence organizational reports contribute to the body of knowledge on the ADA.
5.2 Stakeholder Feedback
To help direct future research and to ensure the usefulness of this research, the research team collaborated with a variety of ADA stakeholders. In February 2013, the expert panel met with the research team to help confirm findings, identify gaps in the research, and direct the upcoming rapid evidence and systematic reviews. Prior to our February meeting, expert panel members were asked to review the technical brief and provide some initial feedback to guide our discussion. Additionally, directors of the ADA National network centers were also individually consulted for feedback, as well as members of the national Knowledge Translation committee, who provided feedback during a phone presentation. Both the expert panel and center directors completed a short worksheet to provide commentary on the findings. The worksheet asked a series of questions to identify the important areas of research most relevant to the various stakeholders. This information has been incorporated below.
The first task that we asked of the stakeholders was to identify the most commonly debated conclusions about the ADA that lack decisive answers. The panel and directors identified three commonly debated areas in the ADA implementation research. Of those that responded to the question, all agreed that the area of employment is most commonly studied and debated but has drawn the least amount of decisive conclusion. There are still frequent discrepancies on the study of ADA and employment in regards to access to the open labor market, how perceptions of people with disabilities have changed in the employment sector, and more generally, how the ADA has enhanced employment outcomes and opportunities. An additional often-debated area that was identified was the area of social participation of people with disabilities in regards to how the ADA has impacted, social capital and the quality of life of people with disabilities across all domains of life. One respondent noticed that the ADA has had a substantial impact on improving architectural accessibility in many aspects of everyday life, but noted that there is little conclusive research evidence to document this change. Lastly, the group agreed that here has not been a summative conclusion on the ADA’s impact in regard to compliance and accessibility. Stakeholders noted how there has been much debate about the success and failures of implementing the ADA by service providers. Question remain unanswered: has the ADA improved the quality of and access to goods and services? Has the ADA really improved access to public accommodations?
Addressing these types of questions are difficult and requires carefully assessing literature to gauge its application to the ADA. One panel member asked us to consider in future reviews what the different is between research relevant to the ADA versus “ADA research?” As noted by another panel member, ADA research refers to the evidence specific to the implementation goals of the ADA in regards to ensuring the full and equal participation of people with disabilities. It was reiterated that participation is a vital issue for any systematic study of the ADA.
As the stakeholders indicated, research questions pertinent to employment and access are frequently studied, often debated, and remain unanswered. Few people were thus surprised at the prevalence of research in these topic areas. One respondent noted that with the wide breadth of studies, future more comprehensive analysis using qualitative and mixed-methods can bring greater clarity to the issues impacting the ADA in the area of employment. We specifically asked if any of the results from the scoping review were surprising. This question was asked partly to generate discussion and also partly to add in a member check to ensure that sufficient research in the various topic areas had been retrieved. In response, we received some additional published and unpublished reports that will be useful for future topic-area specific reviews, but did not meet the initial scoping inclusion because they covered multiple stakeholders and topics. In general, stakeholders found that few results were surprising. The lack of publications in trade- specific areas, not including those found in disability related publications and journals, was identified as an area that needs further attention in ADA research. One of the only surprises was the high prevalence of studies that try to investigate employment rates, even though much of this research has not produced a shared conclusion. Respondents noted that there has been less evidence regarding other issues such as attitudes, barriers, and retention rates. Other surprisingly under-researched topic areas were deinstitutionalization, self-advocacy, and criminal justice, although the stakeholders agreed that these areas have been studied more commonly outside of their relation to the ADA. There was additional surprise at the lack of stakeholder involvement with substance abuse providers as the topic of substance abuse was highly contested in the drafting of the ADA and continues to be a contentious issue debated in the legal literature today.
Of the research areas we identified as having a sufficient amount of evidence to systematically review (employment, healthcare, education, compliance/accessibility), the panel members and directors indicated that employment and healthcare were priority areas to address in subsequent reviews. We left one research area open-ended to seek opinions on additional priority areas. Some stakeholders expressed interest in examining the area of transportation, but also noted that a lot of essential research still needs to be conducted in this area. Additionally, the area of deinstitutionalization is considered an important area, particularly as the Olmstead decision applies to both community living and employment. Although almost all participants selected employment as the key area to review, a few respondents thought this was the least important area because of the wide range of research already prevalent in this area. These respondents noted that the core issue of physical accessibility and compliance in both the public and private sector were more essential to summative understand the ADA’s impact.
During the expert panel meeting, the selection choices were discussed and complicated. For example, a number of panel members insisted that it would be important to look at what was referred to as the “covariance” between topic areas: carefully addressing how the topics relate to and impact each other. As one panel member noted, although the topic of employment is the most ubiquitous of ADA research, it is also one that cuts across all of the other topic areas identified in the scoping review and is not easily separated as its own stand alone category. The stakeholder feedback to examine the core area of accessibility and compliance also supports the notions that this area should be closely examined as it relates to the employment sector as well. In terms of specific research questions for the systematic reviews, the feedback provided on the worksheet initially indicated that the two most important research questions to ask in relation to the topics were: “how has the ADA impacted the rate and experience of people with disabilities in the open labor market?” and, “how was the ADA improved access to healthcare equipment and services?” One person noted that an additional healthcare subtopic should include the intersection of how other factors can impact access such as accreditation or Medicaid eligibility. The topic area of Internet access, assistive technology, and information technology also emerged as an additional priority area during the group discussion, and was supported by ADA director feedback. It was noted that this interest was partly driven be recent testimony in Congress by the Department of Justice that cited the Internet as the most vital area to improve policy and access for people with disabilities. This is a topic area that can stand alone as a research area, and also importantly cross cuts across the aforementioned areas of health and employment.
The stakeholder feedback was also useful to narrow down subtopic areas for our future research questions. Two policy experts reiterated the notion that the study of employment rates are of little use, and can lead the research down an unnecessary “worm hole” investigating the nuances of studies that are not useful to ADA stakeholders. This advice further supported our previous inclination to look at more nuanced issues of the employment experience of people with disabilities including retention, access, and attitudinal studies. Additionally, studies pertaining to physical and web accessibility were commonly noted by the directors as essential subtopics for any systematic review question. There was some concern about the relevance of a systematic review of the ADA and healthcare due to the broad changes imminent to our health care system in the next year, in particular with the Affordable Care Act. However, many stakeholders reiterated the importance of health care research, and noted that a review of the ADA’s impact on healthcare could provide a substantive and useful baseline to understand future policy directions. An additional issue identified as particularly relevant for future reviews was to note differences in findings between disability types.
All of these suggestions will be taken under further consideration for the development of research questions in the rapid evidence and systematic reviews. The Expert Panel will continue to meet bi-annually with the core research team to provide feedback as the project evolves. The collaboration efforts with the Expert Panel on developing and refining review questions at each stage of the systematic reviews proposed in this project will ensure that the results have policy and practice implications.
5.3 Dissemination Activities
To date, the research team has presented in the following forums:
- ADA Knowledge Translation Systematic Review Expert Panel. July 13, 2012. (Online webinar).
- National ADA Network Director’s Meeting. August 1, 2012. (Monthly phone meeting).
- Great Lakes ADA Center State Affiliate Advisory Council meeting. November 15, 2012. Chicago, IL.
- National ADA Network annual meeting. December 3, 2012. Washington, DC.
Planned presentations are as follows:
- UC-Berkley Symposium on Implementing Disability Rights, March 2013
- NARRTC Annual Conference, April 2013
- Society for Disability Studies Annual Conference, June 2013.
- American Sociological Association Annual Conference, August 2013
- Society for the Study of Social Problems Annual Conference, August 2013
Publications, current and planned, include:
- Plain language summaries:
- Defining evidence in the systematic review process (2012)
- The ADA systematic review project: overview and methods (2012)
- Scoping review of the ADA: a year 1 summary (2013)
- Systematic Review of the ADA Research: what exists, and where do we go from here? (journal article in development)
- Epistemological Considerations of a Mixed-Method Systematic Review of the ADA (journal article in development)
REFERENCES
Crotty, M. (1998). The foundations of social research. Sage: Thousand Oaks, California.
Gough, D. Oliver, S. & Thomas, J. (2011) An Introduction to Systematic Reviews. Sage: Thousand Oaks California.
Grant, M. & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91-108
Guba, E. & Lincoln, Y. (1994). Competing paradigms in qualitative research. In N.K. Denzin &
Y.S. Lincoln (Eds.) Handbook of qualitative research (pp. 105-117) Sage: Thousand Oaks, California.
Jesson, J., Matheson , L., & Lacey, F. (2011). Doing your literature review: Traditional and Systematic Techniques. Sage: Thousand Oaks California.
Kissam, P (1988). The evaluation of Legal Scholarship. Washington Law Review. 63. 221.
Levac, D., Colquhoun, H., O’Brien, K. (2010). Scoping studies: advancing the methodology. Implementation Science. 1(5) 69. Retrieved from https://implementationscience.biomedcentral.com/track/pdf/10.1186/1748-5908-5-69.pdf
Mertens, D. (2010). Research and evaluation in education and psychology: Integrating diversity
with quantitative, qualitative, and mixed methods. Sage: Thousand Oaks California
National Council on Disability (U.S.). (2007). The Impact of the Americans with Disabilities Act: Assessing the Progress Toward Achieving the Goals of the Americans with Disabilities Act. Washington, DC : National Council on Disability. Retrieved from https://www.ncd.gov/publications/2007/07262007
Petticrew, M., & Roberts, H. (2005). Systematic reviews in the social sciences. A practical guide. London: Blackwell.