Remote Data Entry (RDE) as the Predecessor to Web-based EDC

The earliest use of computers in research at clinical sites likely occurred in 1963.⁵ The earliest reports of data collection via computers at clinical sites described locally developed systems.⁶ An example early implementation is described in “Development of a Computerized Cancer Data Management System at the Mayo Clinic.⁷

The foundations of distributed data entry in multicenter clinical studies originated before the advent of the internet. Starting in the early 1970s, sporadic reports of development and implementation of computer systems for Remote Data Entry (RDE) at clinical sites appeared in the literature. The earliest reported use of remote data entry in a multicenter clinical study was in the National Institutes of Health (NIH)-sponsored Kidney Transplant Histocompatibility Study in 1973, in which two of the twenty-five sites piloted data entry via remote terminals connected by telephone lines to a mainframe at the data center.^8,9

In the mid-1980s RDE use expanded with availability of microprocessors.^{10,11,12,13,14,15} RDE use in multicenter studies was a sponsor or coordinating center driven process in which sites were provisioned with computers, data entry software, and a data transmission device such as floppy discs or a modem. Early RDE implementations distributed database management to the clinical sites with varying levels of responsibilities.¹⁶ Some implementations required sites to first complete paper CRFs and afterward perform first data entry with limited edit checks. Other implementations included centrally performed second entry or additional centrally run edit checks. While others relied upon single entry, with sites having varying amounts of local edit checking during or shortly after entry, and differing extents of batch or central batch edit checks afterwards.^8,9 Early registries from the 1970s through the 1990s similarly employed systems where data were entered at clinical sites on thick- or full clients, i.e., software applications where most or part of the program was stored on the user’s computer with subsequent transfer of data to a central database.

Early commercial systems based on a distributed model (transferring data by media or modem) started appearing in the 1990s.⁹ The earliest of these commercial systems required distribution of laptop computers to investigational sites. These off-line systems were vulnerable to local data loss and, due to the distributed nature, were more difficult to secure. The expense of hardware distribution to sites, especially in multiple countries, was cost prohibitive and logistically challenging. With few exceptions, RDE largely failed to show significant improvement over traditional paper-based data collection and central processing.¹⁷ Clinical data capture was not listed among major improvements in clinical trial procedures and implementation realized during the 1980s.¹⁸

Collen (1990),⁶ Lampe and Weiler (1998),¹⁹ Hyde (1998),²⁰ Kubick (1998),²¹ and Prove (2000)²² provide more complete historical descriptions of RDE. Reports of RDE were published up to the turn of the century^{11,23,24,25,26,27,28} when RDE ultimately gave-way to web-based approaches.

Web-based EDC

Three main advances in Internet technology made web-based EDC as we know it today possible: (1) internet enabled connectivity of geographically distributed sites, (2) executable code embedded within web pages, and (3) central database storage with decentralized retrieval of information via the internet.²⁹ Early use of web-based forms to collect data over the internet first started appearing in 1995 and continued increasing in frequency to the turn of the century.^{29,30,31,32,33,34,35,36,37,38} Several provided descriptions of early system architecture and design.^{29,34,36,39,29,41}

The first large web-based trial was initiated in 1997.³⁸ A fifteen-country rare disease trial reported in 1999²⁹ was likely initiated in the same time frame. The first surgical web-based trial was reported in 2002.⁴² These early web-based EDC attempts used locally developed web-based systems. During this time period, early speculation and sporadic reports of cost and time savings associated with web-based EDC piqued the interest of the therapeutic development industry and the number of commercially available systems grew. Industry-sponsored studies primarily used commercially available systems. However, reports of locally developed web-based systems with varying combinations of functionality continued to be reported at academic centers through 2010.^42,43

Even though 86% of respondents in a 2002 survey of large pharmaceutical companies agreed that “new technologies would have great impact in the near future,” and 56% agreed that there was “already a high level of urgency within their organizations to adopt new technologies,” adoption of web-based EDC remained low at the start of the new millenium.⁴⁴ At the time, it was widely recognized that “reliance on traditional paper-based processes” in clinical trial data collection commonly incurred a three to four month delay before information became available.⁴⁵ Others reported longer lag times in data availability when comparing use of EDC electronic Case Report Forms (eCRFs) to using traditional paper-based data collection.^46,47,48 In the same year, an Association of Clinical Research Professionals (ACRP) survey of 2,300 respondents found that 66%, 60%, and 50% of sites, CROs, and Sponsors, respectively, expected to adopt eCRFs within two years.⁴⁴

EDC adoption progressed more slowly than the early surveys predicted. Reported reasons included organizational lack of strategic planning, the customized requirements of each trial, the immaturity and fragmentation of commercially available EDC software, lack of scalability from pilots to enterprise adoption, and lack of addressing both process and organizational change.^44,45,49 The lack of process change needed to implement EDC has been attributed to lack of leveraging of the technology to re-engineer clinical study processes, lack of organizational strategy and planning to move from pilots to enterprise adoption, and inability to measure success.⁴⁵ Further, failure to change historic delineations of job roles and responsibilities has been cited as a significant factor inhibiting process re-engineering.⁴⁹ Additionally inhibiting the introduction of new technologies was the potential of new technology to create unanticipated bottlenecks in processes. For example, new systems may be larger and more complex, increasing the need for training or the need for specialists.⁵⁰ More complex systems result in fewer team members understanding the workflow, data flow, or other operations of the system, increasing the potential for mistakes and longer times to troubleshoot problems. These situations would, in turn, require more specialists or increase the number of individuals needed to configure, test, implement, troubleshoot, and maintain systems.⁵⁰ EDC study start-up required more up-front work and data management staff.⁵¹ Lack of leadership mandate or support and lack of alignment throughout the organization have also been faulted for slower-than-projected adoption, as well as lack of delighting stakeholders (sites) with an improved trial experience.⁴⁹

By year 2001, only 5% of new clinical trials reported using EDC or Remote Data Entry.⁴⁹ In the same time frame, CenterWatch reported that only 16% of sites were required to use EDC (probably inclusive of RDE).⁵² In the concurrent literature others also lumped both RDE and web-based EDC under the label “EDC”.^9,17,45 By year 2004, however, 70% of sites reported using an EDC system.⁵² El Emam et al. (2009) estimated that by the 2006–2007 time frame, 41% of Canadian trials were using EDC.⁵³ Varied EDC system designs and architectures were reported during this time. Examples of these include both client-server-based systems on laptops,⁵⁴ and wholly internet-based approaches developed for single studies,^55,56,57 as well as use of commercial web-based EDC platforms.^41,58,59,60

Commercial EDC platforms were more heavily used by industry than academia. Disproportionately fewer studies were published by industry authors, however, because information about the methods and technology industry utilized were often considered proprietary. Thus, the earliest reports were predominantly from academic institutions and not reflective of industry practices. Broad adoption of web-based EDC at academic centers lagged in favor of basic general-purpose or office applications such as spreadsheets and inexpensive pseudo-relational databases.⁶¹ Delineated reasons for slower adoption in academia primarily stemmed from lack of motivation or resources to scale tools built for single studies, high cost of early commercial solutions compared to low-budget small investigator-initiated studies, and lack of institutional support to defray costs to academic investigators.^62,63 Later, free solutions such as REDCap® filled the gap in academia by providing free publicly available software meeting the needs of academic researchers.⁶⁴ Multiple such in-house solutions were developed and used at academic institutions. Today, the REDCap® system is nearly ubiquitously adopted at clinically-oriented academic institutions in the United States (and likely abroad) for EDC. An important difference between commercial and academic EDC platform usage is that although the technical controls requisite in Title 21 CFR Part 11 are often present in software developed and implemented at academic institutions, in our experience, they are not routinely locally validated to Part 11 standards. In contrast, commercial EDC platforms serving the therapeutic development industry meet these regulatory requirements for validation.

Perceived Benefits of EDC

Benefits of web-based EDC Implementation such as cost savings, more timely availability of data, increased accuracy, and fewer queries (Tables 1 and 2) have been widely touted.^29,65,66,67 These and other benefits stem from the internet providing both (1) rapid and continuous central data acquisition from geographically distributed sites and (2) immediate and decentralized data use. Internet provided connectivity allowed common information system benefits such as shared data use, decision support, automation, and knowledge generation across geographically distributed sites and teams in multicenter studies. Thus, the value of EDC to an organization increases as more data, especially data traditionally managed in separate systems, become available via EDC systems.^65,68 Echoing what process engineers have been saying for the past several decades, although web-based EDC implementations will improve most data collection scenarios, their real benefit arises when processes are re-designed to take advantage of having real-time information simultaneously available to all research team members.³⁷

Reported benefits of EDC are classified here according to mechanisms through which health information technology creates value.⁶⁹ In addition to the categorization from this review (Tables 1 and 2), early advantages and disadvantages of EDC have been particularly well articulated by Marks et al.³⁸

Barriers to and Unmet Potential of EDC

Barriers to use of web-based EDC that surfaced during the early phases of EDC adoption have largely been overcome. These included problems with hardware and internet provision, internet connectivity, as well as new roles and large learning curves for data management, monitoring, and site personnel. Other barriers to early EDC adoption that have now been largely overcome include management of system access and privileges, provision of technical support for site personnel, and privacy and confidentiality concerns due to transmission of patient data over the internet.^37,41,49,50 The latter necessitated new technology and methods to secure systems open to or operating over the internet^3,73; vulnerability to local bandwidth and network traffic being outside the control of the study team; increased dependence on informatics and IT expertise and institutional processes, for approval to install software that abided by local-client requirements or allowed institutional data to be housed in Sponsor’s web-based systems.^46,73 Early adopters identified inconsistency between institutional policies at sites for retention of paper copies and EDC-based electronic research documentation retention guidelines.⁴⁶ Early lack of enthusiasm of site personnel for performing data entry, perceived as “clerical”,⁹ was largely overcome as site personnel perceived reduction in query-related effort and other benefits related to EDC implementation.

Most early barriers to EDC have been reduced or overcome through improvements in technology, process, and implementation or through evolution of trial personnel perceptions and acceptance over time. For example, consensus has been reached on interpretation and implementation of Title 21 CFR Part 11.⁴⁹ Today the regulation is understood as a requirement and organizations have procedures in place to ensure compliance. Though largely overcome today, the cumulative impact of these challenges has hampered, obstructed, and slowed implementation and adoption of EDC.⁶⁵

Today, despite all the progress made, EDC has still not reached full potential. For example, EDC was supposed to eliminate recording data on paper study forms at clinical sites. Early reports documented that using a paper form as an intermediate step between the source and the EDC system “doubled the data entry workload at the sites and also increased the monitor’s workload”.³⁷ However, use of paper forms at sites in this way has perpetuated at some sites^{59,67,78,81,82} indicating that further process improvements or better integration into site-specific workflow and data flow are needed. Early EDC adoption was hindered by transiency of internet resources (e.g., controlled terminologies), web based knowledge sources, and interfaces used for web-based information exchange.⁷³ Transiency was a particularly difficult problem given increased desire to receive and use data from one system to another, which subsequently intensified need for system interoperability, technology reliability, and standards for data exchange. Though in some cases the transiency has been overcome, significant interoperability challenges remain.

Early Site-user Evaluations of EDC

Early EDC adopters were concerned about site acceptance of new technology and process changes, in particular the aforementioned relocation of data entry to clinical sites.⁹ Three site investigator surveys were reported following EDC pilots. Dimenas (2001) reported that 77% of investigators and 74% of monitors “found the workload to be reasonable”.³⁷ Similarly, 71% of 107 participants at the final investigators’ meeting indicated that the web-based EDC system offered definite advantage over other alternatives.³⁷ Litchfield et al. (2005) reported equivocal results with 57% of investigators indicating that setting up the EDC study sites took either a little more or much more time than past, non-EDC paper studies.⁴⁷ Half of the respondents reported that monitoring visits took more time, with only 28% reporting monitoring visits taking less time than in paper studies.⁴⁷ Similarly, Litchfield et al. go on to report that 42% perceived CRF completion in the EDC study as easier than their experience on past paper studies, while 28% perceived it as more difficult.⁴⁷ Half of the responding site investigators in this survey reported the perception that the number of queries was slightly fewer and their handling easier with EDC, while 7% perceived the number of queries to be much fewer.⁴⁷ Thirty-six percent of the responding site investigators in the Litchfield survey perceived the handling of queries to be more difficult with EDC than their experience on past paper studies.⁴⁷ Investigators perceived a worsening in increased time (1) to set up the study (58%) and (2) to complete the CRF (50%.)⁴⁷ However, in the same survey, 36% of the internet sites thought use of EDC reduced the time and costs associated with the trial overall.⁴⁷ Eight of the 14 centers (57%) perceived that use of EDC for clinical trial data recording was better or much better than conventional systems, though 28% still thought it was worse or much worse.⁴⁷ A majority (71%) of the responding site investigators indicated they would prefer to use EDC over paper CRFs for future studies.⁴⁷ In a separate, brief site investigator survey conducted in 2005 following a pilot EDC study, respondents reported data quality (67%), data entry (78%), and workload (59%) to be an improvement over paper.⁶⁶ Overall, site investigators in the reported survey studies responded favorably to EDC.

Quantitative Evaluations of EDC

Today EDC is largely accepted as having cost, time, and quality advantages over data collection on paper.^{33,38,46,48,57,58,65,66,72,73,76,78,83,84} While early direct comparisons between EDC and paper were quite favorable toward EDC in terms of reduction of query volume and data collection time, only a few were published. Mitchel (2006) lamented that while it was clear that there were theoretical advantages of internet-based trials over traditional paper-based clinical trials, there was a paucity of evidence and that data were needed regarding the efficiencies of data entry, trial monitoring, and data review.⁷² Twelve studies reporting quantitative evaluation of EDC were identified in the literature search.

Of the twelve identified EDC evaluations (Table 3), only Litchfield et al. (2005) employed a randomized, controlled design.⁴⁷ Ten of the remaining eleven studies were observational. Six of the observational studies employed a comparator such as data double or single entered centrally from paper forms on the same study or on a comparable but different study.^{48,58,70,85,86,87} Four other of the observational EDC evaluations reported operational metrics including cycle times, number of data changes, and discrepancy rates from source-to-EDC audits from actual studies but with no comparator.^37,59,72,81 The one remaining report made comparisons between EDC and paper data collection, but did not describe the methods in sufficient detail to classify the research design.⁴⁵ Surprisingly, given the desire to retain the best clinical sites and sensitivity to site burden in clinical research,⁸⁸ no formal workflow or usability studies of EDC technology were found among published EDC evaluations.

Due to the small number of evaluations documented in the published literature, we do not know whether the results from the evaluations are representative of most EDC evaluations. It is likely, based on the identified studies, that most were observational in design and used existing organizational, operational metrics similar to those listed in Table 4 as the basis of their appraisal. Cost, quality, and time metrics were variously used in the evaluations (Table 4). Only one evaluation reported metrics in all three categories. The cost, quality, and time outcome measures are summarized to inform design of future evaluation of data collection and management technology. Since the observed operational metrics varied between studies and were not well specified, further synthesis is dubious.

Data Accuracy in EDC Evaluations

The fundamental belief that EDC improves data accuracy is widely held and reflected both in the literature and practice. Yet, in preparing this review, only four comparisons of clinical trial data accuracy between EDC and traditional paper-based data collection in clinical studies were found.^59,70,81,86 All four identified studies compared data initially recorded on paper forms to the same data subsequently entered in the EDC system. Thus, all four evaluation studies measured transcription fidelity from paper forms to the EDC system.

These studies all suffer from methodological weaknesses that limit their usefulness in assessing the accuracy of data entered at clinical sites via EDC. Only two of the studies^59,70 reported a discrepancy or error rate. In a third evaluation study, Staziaki et al. (2016) employed a crossover experimental design to compare EDC with data collection via key-entry into a spreadsheet.⁶¹ However, the sample size was too small to assess data accuracy. Thus, the Staziaki does not appear in Table 4. In a fourth evaluation study, Takasaki et al. (2018) reported data quality assessment for a rehabilitation nutrition study.⁸⁹ They enumerated missing, out of range, and inconsistent data as detected by rule-based data quality assessment, i.e., query rules. Four errors were detected in the data entered for 797 patients with data type errors, out of range errors, input errors, and inconsistency all assessed. The objectives of the Staziaki et al. and Takasaki et al. studies were not EDC evaluation.

The paucity of evaluative results regarding data quality is striking. Multiple authors (Table 4) have reported counts and rates of data discrepancies identified through rule-based data quality assessment, also called edit checks or query rules. Data discrepancy reports such as these are often and unfortunately confused with data accuracy. Discrepancies identified by rule-based methods may serve as an indicator of data accuracy since it is reasonable to infer that data discrepancies (detected by rules) are a by-product of actual errors in the data – and vice versa. However, (1) rules, real-time or otherwise, miss conformant but inaccurate data and (2) data discrepancies identified through rules are dependent on the number of rules, the logic of the rules, and the data elements to which the rules are applied. These aspects vary from study to study and, as such, so do the number of identified discrepancies and the percentage of fields, forms, patients, etc. with data discrepancies. In other words, the number of discrepancies for two data sets of equal accuracy may vary based on the rules used. Thus, neither the number nor rate of rule-identified data discrepancies provide a reliable estimate or basis for comparison of data accuracy across studies. Rule-identified data discrepancies, while easy to measure, are only a broad indicator- but not a measure of data accuracy.

Accuracy of a data value can only be determined by comparison with the true value, which in most cases is not known. Redundancy-based methods of identifying data discrepancies, i.e., comparing the data to an independent source of the same information, are more comprehensive than rule-based methods in that they detect all divergences from the comparison values. For example, in-range but wrong data will be detected by redundancy-based methods to the extent that the redundant data are themselves accurate. As such, where good sources of comparison can be found, redundancy-based methods as proposed by Helms (2001) are better indicators of data accuracy than rule-based methods.⁹ Sending two samples from the same time-point, patient, and instance of sample collection (a split sample) to two independent labs is an example of a redundancy-based method; as is comparing data collected for a study back to the original recording of the information, i.e., the source, as is done in clinical trial source data verification processes.⁹⁰ To the extent that redundancy-based methods cover all data elements of interest and employ an independent and more “truthy” comparator, they get closer to actually assessing accuracy than rule-based methods. Comparison to the source is still not a complete measure of accuracy because errors in the source are usually not known and not measured in the comparison. In the reports of EDC data quality assessment, only one, Nahm (2008), compared EDC data to the source; however, the source was a structured form completed during or after the patient visit and may have still contained errors itself.⁵⁹

Claims that EDC increases data accuracy (Table 1) are likely based in first principles, mainly that (1) the “forced rigor” of structured fields, limited response options, and real-time checks for or prohibition of missing or inconsistent data during entry decrease errors and (2) entering data closer in space and time to the original capture allows correction where there is recollection or an original recording. While the forced rigor has been associated with lower error rates,⁹⁰ these likely true claims have not been proven in a generalizable way. Further, error rates measured from data entry were an order of magnitude lower than those measured between collected data and medical record source documents from which they were abstracted. Unfortunately, the only appropriate conclusion from this review regarding evaluation of data quality in clinical studies using EDC is that little is reported in the literature about the accuracy of data captured through EDC.

EDC Adoption

In the last decade, the therapeutic development industry has made extensive use of EDC. In 2008, the CenterWatch EDC adoption survey of investigative sites reported that 99% of sites were using EDC in at least one of their trials, with 73% of trial sites utilizing EDC for at least one-fourth of their studies.⁵² In the same survey, 36% of sites reported using some type of EDC including interactive voice randomization, web-based data entry, fax-based data entry, or electronic patient diaries for at least one-half of trials they conducted.⁵² Only 2% of responding sites, however, indicated they no longer collected any case report form data on paper.⁵² A decade later in the 2018 eClinical Landscape Survey, 77.5% of eligible respondents reported managing CRF data in the primary EDC system.⁴ Today with most traditional CRFs entered and cleaned over the internet, the primary mechanism for collection and management of CRF data is web-based EDC.

The time period from realization that clinical study data could be collected electronically at sites and immediately available to the study team until full adoption seems long. The time period from the earliest report of RDE (in 1973), until the most recent adoption reports, spans four decades. However, as seen in Figure 1, the EDC adoption curve (bold blue line) is comparable with most contemporary new technology adoption rates.⁹¹ The entry tail of the adoption curve may have been more protracted for EDC than those for most other reported technologies. However, early data before the inflection point are not provided for most other innovations on the graph and such a comparison is not possible. Given complexities involving the regulated international nature of therapeutic development, the primacy of human subject protection concerns (including protection of individual health information), the challenges of crossing organizational boundaries for EDC implementation, and the sociotechnical aspects of EDC changes wrought in workflow and information flow (for sites and sponsors alike), it would not be surprising if the approach to EDC adoption were slower than for other innovations.

Figure 1

Estimated EDC Adoption Curve Superimposed on US Technology Adoption Data.

Source: US Technology adoption in US Households data and image from Ritchie and Roser.⁹² The data sources from which the image was compiled are listed at https://ourworldindata.org/technology-adoption#licence. The image was reproduced and adapted here under the creative commons license. EDC adoption data were obtained from the studies referenced here^4,52,53 and superimposed in bold blue “X’s” on the image. Point estimates were made by the authors averaging percent adoption of sites and trials where both existed at the same timepoint. Thus, the actual EDC adoption curve may lie several years or percentage points in either direction. Some regions of the world likely experienced significantly shifted curves from others.

Lessons Learned from EDC Adoption

We have learned from the history of EDC adoption that non-technical factors significantly impeded adoption in the therapeutic development industry.^48,77,80 For example, companies demurred from significant role re-definition and therefore missed significant re-engineering opportunity.^80,101 In clinical informatics, it is commonly held that successful adoption of new information systems is “about 80% sociology, 10% medicine, and 10% technology.”¹¹² Thus, to the extent that individuals, groups, and organizations will interact with new technology, needs assessment, design, development, dissemination, and implementation should equally account for human and sociotechnical factors and potential barriers.

The EDC adoption story demonstrates that “beneficial effects are obtained when the ways of thinking and working are changed to take advantage of the opportunities arising from having information online.”³⁷ Web-based EDC offers the ability to centralize information while decentralizing its use, making information and information products (such as decision support and automation) available to everyone on the study team simultaneously and in real-time. This enables study teams to do things not possible, or at least unwieldy, prior to web-based EDC technology. As evidenced by the aforementioned limitations 1 and 2, implementing new technology without transformative process change to leverage these opportunities has not and will not yield the expected improvement.^{48,60,67,77,80,94,113}

In addition to the process re-engineering needed to realize the benefits of new technology, infrastructure that comprise Quality Management Systems such as technical, managerial, and procedural controls need to be adjusted to the new technology. Roles and responsibilities have to be adjusted to the new working processes.^{60,67,77,80,94} Individuals in affected roles need training and time to adjust and gain experience with the new technology.^60,67,77,94 The implemented processes and software need to be monitored in order to ensure expected performance in local contexts.^77,80,74,75 This capacity building and infrastructure development is a project unto itself and should be managed as such, separately from evaluation pilots and stabilized before using new technology in routine operations.⁸⁰ Even then, new technology faces Solow’s Paradox; i.e., that IT investments aren’t often or immediately evident as increased productivity.¹¹⁴ Hypothesized reasons why immediate benefit is not perceived with new technology include lack of accounting for work redistribution, lack of accounting for work to meet needs for increased explicitness needed for automation and decision support, and failure to leverage and integrate the new technology with pre-existing technology, infrastructure, and processes. Looking back on EDC adoption, we can see their mark. The difficulty prospectively conceptualizing and valuating opportunities made possible by new technology adds another reason why increased productivity and return on investment (ROI) often isn’t immediately evident. Such ROI projections should account for increasing value of information technology to an organization as more data, especially data traditionally managed in separate systems, become available for use.^65,68 Data sharing from early pilots of new technology may help overcome the paradox through more accurate prediction as will methods that take into account likely causes of the IT productivity paradox.

In major re-engineering endeavors, large companies with existing and stable infrastructure (including roles, responsibilities, procedures, and technology in place) tend to have more inertia and are slower to change.¹⁷ On the other hand, companies without legacy infrastructure are usually more nimble in implementing beneficial change.¹⁷ Thus, large organizations need clear strategy, strong leadership, meticulous goal alignment, and thorough understanding of how new technology will mesh with pre-existing infrastructure in order to re-engineer at a pace similar to their smaller and newer competitors.²⁵

In the authors’ experience many organizations piloted one or more EDC systems. As evidenced by this literature review, very few of these pilot projects published results, and even fewer published in peer-reviewed scientific literature. Practices regarding information sharing in therapeutic development have evolved over the last three decades and pre-competitive information sharing now occurs much more frequently than it did in the past. Examples include initiatives such as TransCelerate Biopharma (https://transceleratebiopharmainc.com/), the Clinical Trials Transformation Initiative (CTTI, https://www.ctti-clinicaltrials.org/), the Society for Clinical Data Management (SCDM) eSource Consortium (https://scdm.org/esource-implementation-consortium/), projects undertaken by members of European Federation of Pharmaceutical Industry Associations (EFPIA), and the VULCAN Accelerator in HL7 (http://www.hl7.org/vulcan). Through initiatives such as these, future technology evaluations are expected to be more collaboratively undertaken and published.

The history of EDC as told through the literature is that, with one exception,⁴⁷ the EDC evaluations that were published leveraged observational methods and empirical data without the support of experimental controls. In many cases the research was done without contemporaneous comparators. This review found no record of ongoing productivity monitoring past initial evaluations. Further, variability in the outcome measures used in the reported studies precluded formal meta-analysis to synthesize evaluation results (Table 4). A general lack of sharing evaluation results and the heterogeneity of evaluation outcomes, along with the aforementioned barriers all likely contributed to the “perpetual piloting” phenomenon mentioned in the literature, and lengthened the EDC’s adoption curve. More rigorous evaluation on consistent outcome measures, along with earlier information sharing will likely benefit the industry in future new technology adoptions. Collaborative evaluation could further decrease uncertainty earlier in and shorten the adoption curve.

There have been three major paradigm shifts in clinical research data management: (1) the use of structured forms for data collection; (2) the advent of clinical data management systems in the late 1990s, in which data were entered, imported, integrated, stored, cleaned, coded, and otherwise processed; and (3) web-based EDC that decentralized data entry, cleaning, and use. While we argue that, though the potential of the latter is not yet fully realized, all three innovations have positively increased the ability to plan, conduct, and manage clinical studies, increased the types and number of studies that we can conduct, and improved the documentation, if not the quality of the study data. Three major limitations remain with EDC technology: (1) interoperability deficits, (2) unexploited process re-engineering potential, and (3) lack of error rate estimation. Significant untapped potential exists for the use of automation and immediate information availability to support, make, and act on study management decisions in real-time such as following-up on data and resolving discrepancies within a day of their commission and detecting and intervening in operational anomalies like protocol violations and non-compliance immediately. The potential gains are magnified when all sources of data on a clinical study are centrally available through interoperability to signal detection algorithms and for decision support.

Due to these limitations, in terms of diffusion of innovation,^115,116 our current plateau on the innovation “S-curve” falls short of what could be achieved with current EDC technology (Figure 2).

Figure 2

Major Events on the Trajectory of Innovation and Performance Improvement in Data Collection, Management, and Use in Clinical Research.

Diagram adapted from: Clayton M. Christensen, “Exploring the limits of the Technology S-Curve. Part 1: Component Technologies”. Production and Operations Management 1, no. 4, (Fall 1992) 340.

Advances in processing and use of different types of data (beyond those obtained from CRFs) when integrated into EDC have the potential to further, and sustain, an upward performance trajectory in EDC effectiveness, adoption, and function. There are many technological advances on the horizon including new data sources, advanced ways to extract information from data such as image processing and natural language processing, more advanced ways to generate knowledge from information (such as data mining and machine learning), and advanced ways to apply new knowledge to augment human performance such as use of artificial intelligence for signal detection and decision support. Likely similar advances will positively impact clinical study design, conduct, oversight, and reporting. Parallel advances in data standards such as those currently pursued through the HL7 VULCAN accelerator for clinical research could exponentiate performance gains through agreement on, and availability of more precise data definition and mechanisms for data exchange. If fully pursued to support clinical research use cases, FHIR® standards will unlock data not previously available (or not previously computationally accessible) and will enable new uses of operational data such as computationally aligning EHR data to a study schedule of events, or enabling new opportunities to shorten therapeutic development such as seamless conversion of open-label extension studies to EHR- or claims-based post-market registries.

Many organizations will pilot and eventually adopt new data sources and novel ways of collecting, processing, and using data in clinical studies. To gain value, organizations will need to integrate them into existing processes or re-engineer existing processes to optimally exploit them. For example, use of artificial intelligence to detect operational anomalies is easy to implement separate from existing data collection and processing pipelines. However, with a recent and notable exception, none as of yet have integrated such data into study data processes or at-scale, routine use by clinical trial teams. The EDC adoption story indicates that each advance may experience a protracted entry to the adoption curve for similar reasons experienced by EDC and will likely have the characteristic 15–20 years¹¹⁷ to reach maturity and widespread adoption as experienced by many other recent technological advances (Figure 1). Each of these advances has the potential to sustain performance increases in clinical research moving us beyond today’s EDC (Figure 2). While only wild speculation, the next major paradigm change in data collection, management, and use in clinical studies may be comprehensive adaptation and adoption of the FHIR® standards in clinical research. Application of the FHIR® approach to enable seamless exchange of data within clinical site facilities, between clinical sites and study teams, and among organizations working together to conduct clinical studies would end the current siloed state.

Limitations

Though extensive attempts were made to identify all seminal events and relevant evaluations in the history of EDC, few were published in peer-reviewed literature. Many pertinent and important articles and papers may have been missed because they were published in outlets not indexed or preserved for academic retrieval. This review completely misses the likely substantial work undertaken and communicated only within organizations. Additionally, one individual extracted information for this historical review. Although the articulation was independently reviewed by the two co-authors, the initial extraction is subject to human error and bias associated with first author’s single reviewer synthesis. Commentary on the article is encouraged to counter any bias or missed information relevant to this review. In this vein, materials used in this review will be made available for re-interpretation and analysis.

Evaluation 1

Banik and Mochow, presented in 1998.

During the 1998 8th Annual European Workshop on Clinical Data Management,⁵⁸ Banik and Mochow reported on their comparison of EDC to traditional paper data collection in a study conducted at Bayer Vital GmbH & Co. The EDC evaluation compared two similar studies from the same drug development program sponsored by Bayer Vital GmbH & Co. One study employed traditional paper data collection, and the other implemented web-based EDC.⁵⁸ The results were subsequently published by Green (2003).⁴⁸ Banik and Mochow measured a 30% reduction in trial duration, an 82% reduction in queries, an 86% reduction in query resolution time (attributed to “use of immediate edit checks that are not possible with paper”), a 43% reduction in time to database lock and a 9% increase in the number of evaluable patients.^48,58 This was the earliest identified quantitative evaluation of EDC.

Evaluation 2

Green, published in 2003

Green reported results from a Gilead Sciences study using the same EDC system deployed by Banik and Mochow (1998). Green reported a 75% reduction in query rate and a 45% reduction in time to database lock.⁴⁸ However, details of the comparator, traditional paper data collection with central data entry, were not provided. It is not clear whether the metrics from the comparator used by Banik and Mochow (1998) were used, whether a separate and comparable study to the Gilead trial was used, or whether the data for the comparison were from data processed via traditional paper data collection on the same study in different sites, different patients, or in the same patients but in parallel to the EDC processes.

Evaluation 3

Mitchel et al., published in 2001

A third early evaluation comparing EDC versus traditional paper data collection was reported by Mitchel et al. (2001). In the evaluation, three similar studies using an observational cohort design were examined.⁸⁵ The first study (study 1) was performed with traditional paper CRFs and a rule-based data cleaning system. The second study (study 2) used a CRO-developed, Web-based data collection system with no rule-based edit or logic check functions. The third study (study 3) employed the same web-based data collection system with full rule-based edit and logic check functions. In this comparison, EDC with full rule-based edit and logic check functions achieved a 63% decrease in queries at the time of data entry and a 65.5% decrease in the queries generated by the monitoring group compared with the traditional paper data collection.⁸⁵

Evaluation 4

Dimenas et al., published in 2001

In the fourth EDC evaluation identified by this review, Dimenas et al. (2001) observed operational metrics from two EDC pilot studies.³⁷ In the two pilots, 69% and 54% of visits were entered the same or next day, and 23% and 24% of queries were resolved the same or the next day with the average time from query generation to resolution of 18 and 17 days for the two pilot studies.³⁷ Last Patient Last Visit (LPLV) to clean file on average was 14 and 20 days, respectively.³⁷

Evaluation 5

Spink, published as an industry white paper in 2002

A fifth report of EDC evaluation provided metrics from ten phase three studies conducted over a three-and-a-half-year period and involving 6,700 subjects. The report was made via an industry white paper by Spink (2002).⁴⁵ Neither therapeutic area nor the evaluation design were described. Spink reported a 50% decrease between EDC and traditional paper data collection in the percentage of invalid enrolled subjects, an 80% decrease in the cost of raising and resolving a query, a 95% decrease in the number of queries per subject, a 95% reduction in the percentage of data requiring correction, a 100% decrease (from 48% paper to 0% with EDC) in the percentage of queries caused by missing data, an 86% reduction in the percentage of queries caused by inconsistent data, an almost complete reduction in the percentage of queries caused by out-of-range data (from 8% with paper to 0.1% with EDC), a 100% reduction (from 6% with paper to 0% with EDC) in the percentage of queries requesting clarification, and a 50% decrease in the percentage of queries caused by invalid data.⁴⁵

Evaluation 6

Mitchel et al., published in 2003

In a sixth relevant evaluation identified by the review, Mitchel et al. (2003) reported an overall error rate of 0.41% in EDC generated data, after detecting 950 errors in 229,152 fields. This observational evaluation compared single entry from forms into an EDC system with double data entry of the same data in a 124-subject trial conducted at 15 clinical sites.⁷⁰ Though the aforementioned evaluations reported substantial reductions in queries, Mitchel et al. (2003) was the first EDC evaluation reporting measures of data accuracy.

Evaluation 7

Litchfield et al., published in 2005

In the seventh and most rigorous of the aforementioned evaluations identified for this review, Litchfield et al. (2005) directly compared EDC and paper data collection in a cluster-randomized experiment in which investigational sites were randomized to EDC or paper data collection. They report that the time from the last patient completing the study to the release of the database was shorter in the EDC sites (33 rather than 48 days, a 31% decrease) in spite of the much larger number of patients in the internet group.⁴⁷ The study reported no appreciable difference in time from the first patient first visit (FPFV) to the date of the last data change between the two arms of the study.⁴⁷ However, site differences (site at which the data were entered) in lag time to database release was highly significant, indicating that the differences in times were due to site effects rather than group effects.⁴⁷ The majority of EDC data were entered within a few days after the visit, with 90% of the data entered within three weeks after a study visit. This was a stark contrast to the paper group where data entry took up to six months.⁴⁷ Though there were more queries in the EDC group (11.4 queries per patient with EDC versus 1.4 in the paper group) on average, the query volume was not perceived as significantly higher. The study authors attributed this to the fact that queries in the EDC group were posed at the time of entry when correction could be made immediately, whereas paper queries came after-the-fact.⁴⁷ In the same study, queries resulting from programmed validation checks were resolved on average 13.2 days faster while clinical, and presumably manual, queries took on average 3.7 days longer.⁴⁷ Similar data entry time site effects were also reported in an observational analysis of studies using Direct Data Entry, i.e., using the EDC system as the source, reported almost a decade later.^93,107,113

Evaluation 8

Meadows, published as a Doctoral Dissertation in 2006

The eighth evaluation identified by the review and reported by Meadows (2006) used secondary analysis of existing observational data to assess EDC. To conduct the evaluation, Meadows compared data entered at clinical sites, from paper CRFs, into an EDC system to data from sites submitting paper forms for data entry at a central data center.⁸⁶ The study found a significantly higher proportion of forms with rule-detected discrepancies (queries) for paper-based forms as compared to EDC (46.5% vs. 31.7%), with the odds of having an error from one to two times higher for the paper process.⁸⁶ The average rate of both univariate and multivariate discrepancies was greater for paper-based forms than EDC forms.⁸⁶ Meadows found a statistically significant higher proportion of resolved discrepancies for EDC, as compared to paper-based forms (62% vs. 48%); i.e., discrepancies on EDC forms were 2.2 times more likely to be resolved compared to discrepancies on paper forms.⁸⁶ This varied across the analyzed form types from 1.32 to 3.76 times greater likelihood for discrepancy resolution for EDC forms as compared to paper.⁸⁶

Evaluation 9

Mitchel et al., published in 2006

A contemporaneous observational case study conducted by Mitchel et al. (2006) reported metrics obtained from the first year of data collection on a 170 patient prostate cancer trial conducted using web-based EDC.⁷² They reported that 85.9% of all forms did not require any data modification. Of the forms evaluated, concomitant medication and adverse event forms required more data corrections (29% and 29.8%, respectively, required modification). Visit date and demographics forms required little correction with EDC (99.3% and 96%, respectively, requiring no modification).⁷² When forms required modification, 76%–99.5% across form types from concomitant medications (76%) to the visit date form (99.5%) had the final modification within 30 days of data entry.⁷² The high percentage of clean data early in the study was noted as a clear advantage of EDC.⁷² Regarding monitoring, 69%–83% of all forms were reviewed by the CRA in the field within 60 days of data entry or final form modification.⁷² Further, 9.8% of all forms were reviewed by the in-house data reviewers on the same day the monitor reviewed the form, i.e., immediately after the monitor completed SDV.⁷² In general, 88%–97% of all forms were reviewed by the clinical data manager within 60 days of form review by the clinical trial monitor, facilitating early and ongoing “by patient” locking of the database.⁷²

Evaluation 10

Nahm et al., published in 2008

In the tenth identified study quantitatively evaluating data quality, Nahm, et al. (2008) reported observational metrics from completed source-to-database audits of 24 sites participating in four EDC trials.⁵⁹ All trials and sites audited used study-standardized, paper worksheets as source documents for capturing trial data. Data from these CRF-like worksheets were single-entered by site staff into an EDC system with extensive on-screen checks.⁵⁹ The average error rate across all four trials assessed was 14.3 errors per 10,000 fields, with a 95% Confidence Interval (averaged across audit Confidence Intervals) of 12–39 errors per 10,000 fields.⁵⁹ This compared favorably with a contemporaneous pooled analysis of data discrepancy and error rates measured for single and double entered data in other studies,¹¹⁹ indicating that single-entry at sites with intensive on-screen edit checks can produce data quality comparable to centrally, double entered data.

Evaluation 11

Mitchel et al., published in 2006

In the eleventh evaluation of EDC identified for this review, Mitchel et al. (2011) reported operational metrics from a multicenter clinical trial investigating the efficacy and safety of a new treatment in 492 randomized men.⁸¹ The trial was conducted using EDC and data were transcribed from paper source documents to the EDC system.⁸¹ Of the 2,584 data changes, 71.1% were designated due to data entry errors, 18.8% due to additional information, and 10.1% due to other reasons. The data were also analyzed by form. While the means of trial variables did not change appreciably from before to after data cleaning, in all cases the estimate of the standard deviation was smaller after cleaning than before cleaning.⁸¹

Evaluation 12

Pawellek et al., published in 2012

The last evaluation identified was reported by Pawellek et al. in 2012.⁸⁷ Use of a commercial EDC system in a multi-center double-blind randomized clinical trial conducted in eleven centers across five European countries was evaluated.⁸⁷ The EDC software was pre-loaded on laptop computers provided to sites. Early visits were conducted using paper forms, presenting the opportunity for comparing observational pretest and posttest metrics to identify differences associated with the use of EDC at one site. Following EDC data collection, plausibility of single-anthropometric values, as well as changes in the values between two study time points, were checked independently of the EDC process. These checks identified data anomalies in 14.6% of visits documented by EDC compared to 35.6% of visits documented with paper-based CRFs (Chi-squared test, P < 0.001).⁸⁷ Overall, 44.0% of all data anomalies detected by the independent data management checks were detectable by the automatic checks implemented in the eCRF.⁸⁷ Pawellek et al. concluded that the need for after-trial plausibility checks of anthropometric data was significantly reduced for eCRF-collected data compared to data collected on paper and that, “the planning and implementation process before starting the trial is more time-consuming” for studies collecting data via EDC than via paper forms.⁸⁷