1) Learning Objectives
To define a Clinical Outcome Assessment (COA) and the different types of COA(s).
To assess the pros and cons of a traditional paper COA versus an electronic COA (eCOA).
To detail different considerations/minimum standards and best practices for data managers during an eCOA build, i.e., determining if it is Fit-for-Purpose (FFP).
To establish who provides the correct source version of the COA that will be used for the build (e.g., an electronic version or a paper version).
To outline the fields that can/should be adjusted when moving from a paper source COA to an electronic COA during the build. For example ‘please tick’ may need to be adjusted to ‘please click’.
To give an idea as to the instructions that may need to be added (outside the assessment questions) to facilitate the flow of eCOA design and best practice (e.g., factors such as age and experience with smartphones).
To advise on the decisions regarding the appointment of relevant team members and the definition and allocation of their responsibilities.
To inform regarding the regulators’ guidance towards changing or correcting data collected through eCOA and how this has changed in the past decade.
To list the key documents expected to support eCOA.
To detail the way that key reports and/or patient profiles for eCOA data are generated, i.e., within eCOA system capability vs. outside of it – generated by eCOA vendor vs. a separate vendor.
To describe the eCOA lock and post-lock process.
To compare the eCOA software modification process with Electronic Data Capture (EDC) database modification.
2) Introduction: Clinical Outcome Assessment
Clinical outcome assessments (COAs) are often used to define efficacy and safety endpoints in clinical research when developing a therapy. “COAs include any assessment that may be influenced by human choices, judgment, or motivation” (Walton et al., 2015).2 COAs must be well-defined and possess adequate measurement properties to demonstrate directly or indirectly the benefits of a treatment. COAs can be categorized into four types (FDA, Oct 2023):
A) Patient-reported Outcome (PRO)
B) Observer-reported Outcome (ObsRO)
C) Clinician-reported Outcome (ClinRO)
D) Performance Outcome (PerfO)1
A. Patient-Reported Outcome
A PRO comes directly from the patient about the status of the patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else. A PRO can be measured by self-report or by interview provided that the interviewer records the patient’s exact verbal response. Symptoms or other unobservable concepts known only to the patient can only be measured by PRO measures. PROs can also assess the patient’s perspective on functioning or activities that may also be observable by others. Examples of PRO assessments include:
Visual Analog Scale (VAS).
Health related Quality of Life questionnaire (QOL).
Hospital Anxiety and Depression scale (HADS).
Treatment Satisfaction Questionnaire for Medication (TSQM).
Counts of events (e.g., patient-completed log of emesis episodes or micturition episodes).
Daily Diary* (whilst this can include a PRO, e.g., seizure episodes, it can also include non-PRO data, e.g., time of medication dose)
*While not necessarily a clinical outcome assessment, this paper includes electronic daily diaries as an electronic patient-reported outcome (ePRO) as it is a patient- or caregiver-facing collection tool.
B. Observer-Reported Outcome
An ObsRO is based on a report of observable signs, events, or behaviors related to a patient’s health condition by someone other than the patient or a health professional. Generally, ObsROs are reported by a parent, caregiver, or someone who observes the patient in daily life and are particularly useful for patients who cannot report for themselves (eg, infants or individuals who are cognitively impaired). An ObsRO measure does not include medical judgment. Examples of ObsRO assessments include:
Acute Otitis Media Severity of Symptoms scale (AOM-SOS), a measure used to assess signs and behaviors related to acute otitis media in infants.
Face, Legs, Activity, Cry, Consolability scale (FLACC), a measure used to assess signs and behaviors related to pain.
Counts of events (e.g., observer-completed log of seizure episodes, snore episodes).
C. Clinician-Reported Outcome
A ClinRO is based on a report that comes from a trained health care professional after observation of a patient’s health condition. Most ClinRO measures involve a clinical judgment or interpretation of the observable signs, behaviors, or other manifestations related to a disease or condition. ClinRO measures cannot directly assess symptoms that are known only to the patient. Examples of ClinRO assessments include:
Reports of particular clinical findings (e.g., presence of a skin lesion or swollen lymph nodes) or clinical events (stroke, heart attack, death, hospitalization for a particular cause).
Psoriasis Area and Severity Index (PASI) for measurement of severity and extent of a patient’s psoriasis.
Hamilton Depression Rating Scale (HAM-D) for assessment of depression.
D. Performance Outcome
A PerfO is based on standardized task(s) actively undertaken by a patient according to a set of instructions. A PerfO assessment may be administered by an appropriately trained individual or be completed by the patient independently. Examples of PerfO assessments include:
Measures of gait speed (e.g., timed 25-foot walk test using a stopwatch or using sensors on ankles).
Measures of memory (e.g., word recall test).
The continuous evolution of electronic Clinical Outcome Assessment (eCOA) solutions in clinical trials reflect advancements in technology, regulatory changes, and a shift toward more patient-centered data collection methods. eCOA solutions have developed over time from paper-based to electronic systems, through integration with mobile and web technologies, regulatory support and validation, patient-centric and decentralized trail, and finally advanced analytics and integration Stone, A.A. et al (2002) and Coons, S.J. (2015).3,4 These developments have positioned eCOA as a critical component of modern clinical trials, offering robust, patient-friendly data collection that aligns with the increasing demand for remote and decentralized trial designs.
3) Scope
The scope of this paper is to provide a comprehensive overview of the implementation of Electronic Clinical Outcome Assessments (eCOAs) in clinical trials. Paper COA is outside the scope of this paper as typically the responses in the paper version are captured in EDC. Please refer to the Good Clinical Data Management Practices (GCDMP) paper regarding Electronic Data Capture for more information (Pestronk et. al, 2021).5
This paper also discusses the pros and cons of the different eCOA collection modes, e.g., Bring Your Own Device (BYOD), provisioned or mixed.
This paper will share data management considerations that may be utilized while building and managing a study with eCOA.
It is not within the scope of this paper to discuss eCOA vendor selection or the software development life cycle of eCOA. Please refer to the GCDMP’s Vendor Selection and Management chapter (Amatya and Edgerton, 2021).6
PRO instruments and their implementation for adolescents and children are often different compared with those for adults. While data management should be knowledgeable of the differences, it is outside the scope of this paper to cover ePRO build considerations in each section based on age group.
Choosing which COA to be used in the protocol is determined by the clinical research team. As such, this paper will not focus on choosing the eCOA but focus on implementing the eCOA.
It is not within the scope of this paper to discuss other electronic data collection devices, such as continuous monitoring devices or all digital collection methods. This paper will discuss some of the pros and cons of integrating eCOA data in EDC, and provides an overview of general documentation and process around the integration.
4) Minimum Standards
This section focuses on eCOA related regulatory considerations.
| Keyword | Section | Regulatory Document |
| Audit Trail | Section 5.5.4 concerns traceability and states that “If data are transformed during processing, it should always be possible to compare the original data and observations with the processed data.” | ICH E6 (R2).7 |
| Audit Trail | Section V.C.1 eSource Principles for EHRs, Data Originator “For the purposes of recordkeeping, audit trails, and inspection, each electronic data element should be associated with a data originator.” |
Use of Electronic Health Record Data in Clinical Investigations: Guidance for Industry (FDA, 2018).8 |
| Audit Trail | The FDA’s acceptance of data from clinical trials for decision-making purposes depends on its ability to verify the quality and integrity of the data during FDA on-site inspections and audits. (21 CFR 312, 511.1(b), and 812). It is important to keep track of all changes made to information in the electronic records that document activities related to the conduct of the trial (audit trails). We recommend that you incorporate prompts, flags, or other help features into your computerized system to encourage consistent use of clinical terminology and to alert the user to data that is out of acceptable range. Information provided to the FDA should fully describe and explain how source data were obtained and managed, and how electronic records were used to capture data. |
Electronic Source Data in Clinical Investigations: Guidance for Industry (FDA, 2013).9 |
| Source | FDA Guidance eSource data in clinical investigations, Sept 2013- III, A, 2, b “When a device or instrument is the data originator (e.g., blood pressure monitoring device or glucometer) and data are automatically transmitted directly to the eCRF, the eCRF is the source.” “When a PRO instrument is used by a subject to transmit data elements directly to the eCRF, the subject is the data originator and the eCRF is the source. If a process is used by which the subject uses the instrument to transmit data to a technology service provider database, the service provider database is the source.” |
Electronic Source Data in Clinical Investigations: Guidance for Industry (FDA, 2013).9 |
| Change Control | Section 4.7 (Software Validation After a Change): “Whenever software is changed, a validation analysis should be conducted not just for validation of the individual change, but also to determine the extent and impact of that change on the entire software system.” | General Principles of Software Validation; Final Guidance for Industry and FDA Staff (FDA, 2002).10 |
| Data Integrity | Section 5.0.1 further advocates a process-oriented quality management system approach stating that “During protocol development the Sponsor should identify processes and data that are critical to ensure human subject protection and the reliability of trial results.” | ICH E6 (R2).7 |
| Data Integrity | Section 5.1.1 further states that “The sponsor is responsible for implementing and maintaining quality assurance and quality control systems with written Standard Operating Procedures (SOPs) to ensure that trials are conducted, and data are generated, documented (recorded), and reported in compliance with the protocol, GCP, and the applicable regulatory requirement(s).” | ICH E6 (R2).7 |
| Data Integrity | Section 5.1.3 states that “Quality control should be applied to each stage of data handling to ensure that all data are reliable and have been processed correctly.” | ICH E6 (R2).7 |
| Data Integrity | Section 5.5.3 concerns validation of computerized systems and states that “When using electronic trial data handling and/or remote electronic trial data systems, the sponsor should, a) Ensure and document that the electronic data processing system(s) conforms to the sponsor’s established requirements for completeness, accuracy, reliability, and consistent intended performance (i.e., validation).” | ICH E6 (R2).7 |
| Data Integrity | Section 8.0 states that documents that “individually and collectively permit evaluation of the conduct of a trial and the quality of the data produced” are considered essential documents (ICH E6) and shall be maintained as controlled documents. Depending on the eCOA vendor, determination needs to be made on the separate Data Management Plan (DMP) needed for eCOA data. |
ICH E6 (R2).7 |
| Data Integrity | Section 6.4 states that “Data integrity is the degree to which data are complete, consistent, accurate, trustworthy, reliable and that these characteristics of the data are maintained throughout the data life cycle. The data should be collected and maintained in a secure manner, so that they are attributable, legible, contemporaneously recorded, original (or a true copy) and accurate.” | Medicines & Healthcare products Regulatory Agency (MHRA, 2018).11 |
| Data Integrity | Section III.A.1. states that “In the case of electronic, patient-reported outcome measures, the subject (e.g., unique subject identifier) should be listed as the originator. … When identification of data originators relies on identification (log-on) codes and unique passwords, controls must be employed to ensure the security and integrity of the authorized usernames and passwords. When electronic thumbprints or other biometric identifiers are used in place of an electronic log-on/password, controls should be designed to ensure that they cannot be used by anyone other than their original owner.” | Electronic Source Data in Clinical Investigations: Guidance for Industry (FDA, 2013).9 |
| Data Integrity | Section III.A. 5 FDA encourages the “use of electronic prompts, flags, and data quality checks… to minimize errors and omissions during data entry. Prompts can be designed to alert the data originator to missing data, inconsistencies, inadmissible values (e.g., date out of range), and to request additional data where appropriate.” | Electronic Source Data in Clinical Investigations: Guidance for Industry (FDA, 2013).9 |
| Data integrity | Section E likewise provides further detail regarding expectations for security, ie, sponsors should maintain a cumulative record that indicates, for any point in time, the names of authorized personnel, their titles, and a description of their access privileges and recommends that controls be implemented to prevent, detect, and mitigate effects of computer viruses, worms, or other potentially harmful software code on study data and software. | Guidance for industry: Computerized systems used in clinical trials. U.S. Department of Health and Human Services (FDA, 1999).12 |
| Data Integrity | Title 21 CFR Chapter I Subchapter A PART 11 — ELECTRONIC RECORDS; ELECTRONIC SIGNATURES section 11.300 – Controls for Identification codes/passwords: No two individuals should use the same identification/password to access the system. Passwords should be changed periodically to protect against password aging. |
Title 21CFR Part 11.13 |
| Data Integrity | Section V.C.1 eSource Principles for EHRs, Section C.2 Data Modifications states that “After data are transmitted … the clinical investigator or delegated study personnel should be the only individuals authorized to make modifications or corrections to the data. Modified and corrected data elements should have data element identifiers that reflect the date, time, data originator, and the reason for the change. Modified and corrected data should not obscure previous entries. Clinical investigators should review … for each study participant before data are archived or submitted to FDA … the changes should be reviewed and approved by the clinical investigator.” The acceptable parameters for changing the data in eCOA, with evidence (in the case of ePRO, by the patient), must be clearly discussed, agreed, and documented. Discussion with regulatory representative is recommended. |
Use of Electronic Health Record Data in Clinical Investigations: Guidance for Industry (FDA, 2018).8 |
| Data Integrity Retention | Section 5.1.2 protects access to source data and documents, stating that “The sponsor is responsible for securing agreement from all involved parties to ensure direct access (see section 1.21) to all trial-related sites, source data/documents, and reports for the purpose of monitoring and auditing by the sponsor, and inspection by domestic and foreign regulatory authorities.” | ICH E6 (R2).7 |
| Data Privacy | Section 2.11 states “The confidentiality of records that could identify subjects should be protected.” Functionality to meet this requirement becomes criteria used in software evaluation and selection. | ICH E6 (R2).7 |
| Fit-For-Purpose | Section 2.4 states that “All production and/or quality system software, even if purchased off-the-shelf, should have documented requirements that fully define its intended use, and information against which testing results and other evidence can be compared, to show that the software is validated for its intended use.” | General Principles of Software Validation; Final Guidance for Industry and FDA Staff (FDA, 2002).10 |
| Retention | Section 2.10 states that “All clinical trial information should be recorded, handled, and stored in a way that allows its accurate reporting, interpretation, and verification.” | ICH E6 (R2).7 |
| Retention | Section III.C states that “clinical investigator(s) should retain control of the records”, that “clinical investigator(s) should provide FDA inspectors with access to the records that serve as the electronic source data”. The rationale being, similar to eCRF data, that eCOA data also needs to be provided to the sites at the end of the study or site closure. | Electronic Source Data in Clinical Investigations: Guidance for Industry (FDA, 2013).9 |
| Retention | Subpart J – Records and Reports Sec. 211.180 General requirements. (c) All records required under this part, or copies of such records, shall be readily available for authorized inspection during the retention period at the establishment where the activities described in such records occurred. These records or copies thereof shall be subject to photocopying or other means of reproduction as part of such inspection. Records that can be immediately retrieved from another location by computer or other electronic means shall be considered as meeting the requirements of this paragraph. |
Title 21CFR Part 11.13 |
| Retention | Chapter I Subchapter D PART 312 — INVESTIGATIONAL NEW DRUG APPLICATION Subpart D – Responsibilities of Sponsors and Investigators Sec. 312.62 Investigator recordkeeping and record retention. (c) Record retention. An investigator shall retain records required to be maintained under this part for a period of 2 years following the date a marketing application is approved for the drug for the indication for which it is being investigated; or, if no application is to be filed or if the application is not approved for such indication, until 2 years after the investigation is discontinued and FDA is notified. |
Title 21CFR Part 11.13 |
| Risk Management | Section 5.5.3 states that validation of computer systems should be risk-based. “The sponsor should base their approach to validation of such systems on a risk assessment that takes into consideration the intended use of the system and the potential of the system to affect human subject protection and reliability of trial results.” And also “maintains SOPs for using these systems.” | ICH E6 (R2).7 |
| Risk Management | Section 2.6 states that “Users of this guidance need to understand their data processes (as a lifecycle) to identify data with the greatest GXP impact. From that, the identification of the most effective and efficient risk-based control and review of the data can be determined and implemented.” | Medicines & Healthcare products Regulatory Agency (MHRA, 2018).11 |
| Risk Management | Section 5.2.2 states that “Software requirement specifications should identify clearly the potential hazards that can result from a software failure in the system as well as any safety requirements to be implemented in software.” | General Principles of Software Validation; Final Guidance for Industry and FDA Staff (FDA, 2002).10 |
| Risk Management | Section III.D emphasizes that the FDA encourages viewing of the data early and by sponsors, CROs, data safety monitoring boards, and other authorized personnel to prompt detection of study-related problems. | Electronic Source Data in Clinical Investigations: Guidance for Industry (FDA, 2013).9 |
| This is particularly important for eCOA, because the correction process is cumbersome and lengthy. So, if there is any software modification needed or data correction needed or training needed, identifying issues sooner rather than later is healthy for the study data quality. | ||
| SOP | Section 5.5.3 The addendum introductory statement enumerates topics that should be covered in SOPs. “The SOPs should cover system setup, installation, and use. The SOPs should describe system validation and functionality testing, data collection and handling, system maintenance, system security measures, change control, data backup, recovery, contingency planning, and decommissioning.” | ICH E6 (R2).7 |
| User Training and Access Management | Section 2.8 states that “Each individual involved in conducting a trial should be qualified by education, training, and experience to perform his or her respective tasks.” Echoing similar statements elsewhere in ICH Good Clinical Practice (GCP) and in Title 21 CFR Part 11, this requirement applies to eCOA software selection in that it applies to individuals involved in eCOA system selection, installation, testing, use, and maintenance whether they are performed in-house or elsewhere. Where tasks are performed by other organizations, this requirement is met through vendor qualification assessments, usually part of software selection decision making. Functionality to record and track system privileges assigned to users over time, i.e. tasks that users are allowed to perform in the system, become criteria used in software evaluation and selection. While ICH E6 (R2) does not explicitly call out “subject training” under “user training,” it indirectly encompasses it as part of the investigator’s and sponsor’s responsibilities to ensure all trial participants, including subjects, are adequately informed and prepared to perform their roles in the study. |
ICH E6 (R2).7 |
| User Training and Access Management | Section 5.5.1 refers to qualifications of study personnel and states that “The sponsor should utilize appropriately qualified individuals to supervise the overall conduct of the trial, to handle the data, to verify the data to conduct the statistical analyses, and to prepare the trial reports.” | ICH E6 (R2).7 |
| User Training and Access Management | Section III.D also suggests aspects of access control: (1) a list of the individuals with authorized access to the eCOA should be maintained, (2) only those individuals who have documented training and authorization should have access to the eCOA, (3) Individuals with authorized access should be assigned their own identification (log-on) codes and passwords, and (4) log-on access should be disabled if the individual discontinues involvement during the study. | Electronic Source Data in Clinical Investigations: Guidance for Industry (FDA, 2013).9 |
| User Training and Access Management Audit Trail Retention |
Chapter I Subchapter A PART 11 — ELECTRONIC RECORDS; ELECTRONIC SIGNATURES Subpart B – Electronic Records Sec. 11.10 Controls for closed systems. Limited access to personnel, routine device checks ensuring the integrity of data and signatures, and written policy procedures for system security. Maintain an audit trail of revisions and change controls. Persons who use closed systems to create, modify, maintain, or transmit electronic records shall employ procedures and controls designed to ensure the authenticity, integrity, and, when appropriate, the confidentiality of electronic records, and to ensure that the signer cannot readily repudiate the signed record as not genuine. Such procedures and controls shall include the following:
|
Title 21 CFR Part 1113 |
5) Best Practices
The following list of best practices is provided for informational purposes and is based on the authors’ collective experience and expertise in eCOA studies as well as on the cited publications. Some practices included may primarily be applicable only to studies that involve provisioned devices, and their relevance may vary in different contexts, such as BYOD or Web-based scenarios. It is important to consider the specific requirements and complexity of each study when applying these recommendations. The authors have compiled this list based on their experiences with complex eCOA implementations; however, simpler studies may not require the same level of detail or resources as described. This list should be viewed as a general guide rather than a prescriptive set of rules.
Engage your eCOA system provider early in the protocol development process so that all elements of electronic data capture requirements are included in the protocol (Fleming, 2015).14 An agnostic approach is recommended to the mode of administration adopted when developing study protocols. This strategy ensures that if paper-based methods or any unplanned backup data collection becomes necessary, it does not lead to a protocol deviation. eCOA-specific instructions are intentionally placed in study manuals, with the protocols referencing these manuals. This allows for necessary updates to be made within the manuals without requiring a protocol amendment.
Identify core team members to participate in User Acceptance Testing (UAT), including sponsor representative(s) and provide appropriate access and/or devices to testers prior to UAT if applicable in a timely manner. Sponsor participation is important for oversight of design and reports. Refer to Gordon, S. (2022) for details on eCOA UAT best practices.15
Ensure the design is easy and convenient for the study population to use (Fleming, 2015).14
If it is a new COA, then psychometric validation may be required. This is the process of evaluating and confirming the reliability, validity, and overall measurement properties of a questionnaire or scale used in research, particularly in clinical and psychological studies. This validation ensures that the instrument accurately measures what it is intended to measure and produces consistent, reliable results across different populations and settings.
It is not recommended to have a free text entry but to provide a drop down or button list of choices with single selection or multiple selections applied.
Implement skip logic where possible. Example, if patient answered “No” to “Did you have pain today” then follow-on questions rating level of pain are not presented for response.
Implement only critical audible alerts/alarms and notifications to encourage subject compliance with ePRO data entry using home devices or BYOD apps, whilst not increasing the burden to the patient.
Implement practical data entry time window restrictions in the ePRO design. Stone et al. (2002) discuss the issues of patient compliance and the impact of recall bias in the context of paper diaries, which has significant implications for the design and use of electronic clinical outcome assessments (eCOAs) as well.3
Set up site and/or sponsor email alerts for subject non-compliance to follow-up and re-train as needed.
Ensure help desk information is provided to all the relevant users including how the site can help the patients as many patients will call the site before calling the help desk.
Ensure all training materials for stakeholders, including site users, patients, Clinical Research Associates (CRAs), caregivers, and others as applicable are robust to facilitate the eCOA completion in an intended manner. The timing of the training materials’ availability towards site initiation visit is critical (Matza, 2013).16
Ensure enough devices are acquired to last the length of the trial as device models change rapidly.
Ensure that translations are checked against local dialects.
Ensure all devices are checked for unsent data upon return.
Only supply a few extra devices to each site and re-supply when needed. Don’t send it all at once.
Proactively discuss how to handle missing data and any process for remediation with the core team.
Ensure all available web or electronic back up options are implemented as much as possible to ensure data is not lost due to failure of study eCOA/ePRO device if applicable.
It is important to ensure that paper-based backup measures are “only used as a last resort and not used as the result of convenience or preference on the part of the site or subject. The sponsor should provide sites with procedures for troubleshooting and an escalation plan that must be followed prior to sites implementing paper as a source back up” (Fleming, 2015).14
If utilized, all the paper backup details should be specified in the DMP, including the critical components of paper CRF including study id, site id, subject id, visit id, date data was recorded, time and actual PRO equivalent to the electronic version.
The system should be able to systematically identify data that is entered via paper back up or changed later due to the site request via audit trail or system flag.
Have a documented oversight plan and documented site staff training
Follow clearly defined and standard workflows–but also mitigate undefined change types
Allow trial documentation per ALCOA+ principles
Induce sponsors into timely reviews and reconciliations of discordant data (i.e., via sensitivity analysis) and not by arbitrarily overruling the investigators
6) Data Management Considerations
A. eCOA System Vendor Selection
Vendor selection and management is not within the scope of this paper. Please refer to SCDM’s paper on Vendor Selection and Management (Amatya & Edgerton, 2021)6 for more details. The paper outlines the process for selecting vendors including the Request of Information (RFI)/Request for Proposal (RFP) through contract award and then recommends best practices for maintaining effective vendor relationships with respect to time, cost, and quality.
Below are some suggested questions for the RFI/RFP for eCOA specific vendors:
COA design can be categorized into two types – simple designs (e.g., a standard questionnaire) and complex designs (e.g., branching conditional to other responses). Please present your experience with both.
State your experience with provisioned devices vs. BYOD vs. hybrid considering patient age group, countries, number of sites, and therapeutic areas. What therapeutic areas do they support and what languages and countries have they worked in?
Describe your user support services (e.g., IT helpdesk locations, language and hours).
Describe how your company complies with 21 CFR 11.13
Is your company Clinical Data Interchange Standards Consortium (CDISC) compliant?
Describe your company’s experience and recommendations for user-training. Differentiate between clinical studies with a few sites and those with many sites and global studies.
Describe your process for quality oversight of subcontractors (e.g., translation vendors).
What is your experience of cloud services?
What user feedback have you solicited or received from study-site personnel, patients, or clients about any limitations of your system? How was the feedback addressed?
Regulatory guidance on ePRO data collection and data management is constantly evolving. Please provide your experience and understanding of how this has evolved.
Describe your team structure, communication, and escalation processes.
Describe your process for handling system updates during the study.
What is the average time between data entry and server upload; when does the data become accessible to the team?
What reports do you provide; what is their format and timing?
How do you monitor compliance with data entry and what actions do you take to improve compliance? Describe your process for sending notifications to participants and sites regarding missing or incomplete data.
Describe what happens when data cannot be sent successfully to the server. Does the data stay locally stored on the device until it can be successfully transmitted?
What happens to the devices at the end of the trial?
What will be the format and structure of the eCOA site archive? For example, the data types, metadata, audit trails, and the medium of delivery (e.g., digital repository, encrypted media).
What is the process to replace broken/lost devices and how is that reconciled?
Describe your process for providing audit trail report(s) and the format.
Describe your safeguards to ensure personally identifiable information or participant locations are secure at rest and in transit.
Describe your process to inactivate the device or application in the event of loss or theft.
Request relevant references from previous customers.
B. COA in Planning
a) The choice of tool
There are several points to consider when choosing the mode of implementing a COA. The first is whether to implement the COA as a paper tool, web-based tool, or a smart device application (app). Additionally, if the intention is to use electronic capture, the decision to use a vendor-provided device (provisioned device) or to allow the subject to use their own BYOD or mixed module must be considered. Consideration of the demographic and location of the patient population for the study may influence the choice, as will the practicalities of shipping devices and how data will be uploaded from different environments.
b) Paper COA vs. electronic COA
Research into paper vs. electronic COA shows that “The new electronic version shows good reliability and face validity, and scores obtained from paper or electronic modes share comparable accuracy and interpretation” (Duracinsky et al., 2014).17 Regulatory preference has moved in the direction of collecting data electronically due to the limitations of paper collection, which “result in untimely, unreadable, missing, illogical or otherwise faulty data” (Coons et al., 2015).4 In contrast, this paper further states that electronic data collection systems can “lead to more accurate and complete data, improved protocol compliance, avoidance of secondary data entry errors, easier implementation of skip patterns, less administrative burden, high respondent acceptance, reduced sample size requirements, and potential cost savings”. It also states that the use of an ePRO “enhanced data integrity and accuracy of PRO data in clinical trials. The US Food and Drug Administration has made it clear that electronic capture of clinical trial source data is preferred over paper-based data collection.” Some other benefits of eCOA compared with paper COA as enumerated by the FDA include it reducing the possibility of transcription errors, enabling remote monitoring of data and real-time access of data review, and facilitating the collection of accurate and complete data (FDA, 2013).9
The following table shows the comparison of COA vs. eCOA:
| eCOA Considerations | Paper COA Consideration | |
| Missing Data | Responses are required, which reduces the risk of missing data. | High volume of missing data is possible, e.g., lost or destroyed papers. |
| Privacy Concerns | Some subjects have privacy concerns about ePRO. Confirmation from ePRO vendor that system complies with GDPR and the Health Insurance Portability and Accountability Act (HIPAA), i.e., cannot be traced to subject. | GCP/HIPAA rules for paper CRF design apply. |
| Data Integrity | Each response is timestamped, and an audit trail is available. | While site-based data entry of PROs can be timestamped, home data entry cannot be determined. Example: Patient diaries can be filled out at the expected time or retroactively in the clinic parking lot. |
| Technical Issues | Issues, such as problems with login procedures, system outages or lost devices, can result in missed data if not resolved in a timely way. | Not applicable. |
| Data Accuracy | Edit checks can be programmed for missing, inconsistent, or abnormal values so that potential errors are identified at the time of data collection. Ensure edits are logical and do not influence patient’s reporting. | Data entry errors are possible. Edit checks can be implemented in EDC during data entry but it will be after the fact and data correction may not be an option. |
| Cost | It is a significant financial investment. | Comparatively cheap. |
| Safety Alert | Safety alerts can be sent to the principal investigator (PI) and/or sponsor, at the time of data collection allowing for more immediate action for urgent matters only. | Not possible till data entry in EDC. |
| Subject Population | Success of eCOA could depend on targeted subject populations by technical capability, indication, etc. | Less restrictive to any target demographic. |
| Reminders/Notifications | Automated reminders can be programmed to have subjects complete ePRO. | Not applicable. |
| Minimize Lost Data | Secure storage in the database server at time of data collection as long as the device is synced. | Paper could be lost or damaged between collection and site visit. |
| Source Data Verification | Source Data Verification (SDV) is not required. | SDV can be required per the monitoring plan since it is in EDC. |
| eCOA Device Management | Device provision and return procedures may be difficult in some countries. | Not applicable. |
| Key Topics | Provisioned Device Considerations | BYOD Considerations |
| Technical Support | If it is a vendor-provisioned device, then the vendor’s help center supports the device and app. | If utilizing BYOD, the vendor support will vary. It should be made clear to the sponsor, the site, and all users what level of support is provided. |
| Enrollment Impact | Enrollment impact is lower. | Could limit enrollment to only those who have suitable devices. |
| Applying Restrictions on Alarms and Notifications | Participants can change settings as allowed in the device sponsor/vendor approved configuration. Examples include changing settings for volume and screen orientation. | Patients can change the setting and turn off or silence the notifications and alarms. |
| Losing a Device | If lost, the same device can be shipped with the same settings without delay. | No guarantee that the patient will buy the same device immediately, i.e., higher risk of lost data with new settings. However, if accessing a website, it could be quicker than waiting for a replacement device. |
| Cost | Expensive hardware is needed. | Reduced costs as no device needs to be provided and shipped. |
| Patient Burden | Extra training is required. Also, patients must carry this device in addition to their personal phone. | Reduces burden on the subjects as they are using the device they are most familiar with and have access to in their day-to-day lives. |
| Site Burden | Site needs to manage the devices, i.e., provide devices, charging before the first visit, collecting, etc. | Reduce burden on sites as they don’t need to manage the hardware. |
| End of Study Process | A specific plan needs to define the process on what to do with the device once the subject completes or withdraws from the study. | Needs to ensure that the app is disabled or removed from the patient’s phone or access removed from a website. A study plan should be identified to define this. |
| Device | Provision devices could be tablets for site staff or handheld devices for patients. | Usually, smart phone-based apps or web based. |
| Location | Country specific issues may also affect your choice of tool as it may be difficult to get vendor-provisioned electronic devices into some countries and also to have vendor provisioned electronic devices returned at the end of the study | Internet connection might not be available everywhere and/or smartphones might not be available everywhere. |
c) Qualified COA and licensing
The decision of which COA assessment to be used in the study is not typically made by data management, as it is often clinician-driven and indicated in the protocol. However, data management does need to confirm the selected COA is validated and the license is in place before any build activity starts. This needs collaboration with the study team. The instrument author’s license approval process is not within the scope of this paper. It is recommended to utilize your study project manager for this process. This process could vary by instrument or instrument license owner. While implementing eCOA that is in the public domain, this step can be skipped. However, there could be an assessment of specific criteria that need to be implemented. For example, the VAS scale should be horizontal vs. vertical view. based on regulatory requirements and the advice on ePROVIDE (a platform developed by the Mapi Research Trust https://www.mapi-trust.org to provide information on COAs including the digitization of paper COAs).
C. eCOA in Study Start-Up
The mode of collection, whether it will be a provisioned device, BYOD, or mix module should be decided during the design phase.
a) eCOA: Provisioned device vs. BYOD vs. mixed module
A Provisioned Device approach needs the eCOA vendor to ship the study device to the sites and/or to the patients. BYOD approach employs the patient’s personal device using a specific app or access to a web-based solution. A mixed module is when a study adopts both provisioned devices as well as BYOD to include different patient populations.
All approaches have considerable promise, but a number of questions need to be answered prior to deciding which approach to adopt.
Mixed Module Considerations
If the study is using a mixed method, then in most cases the provisioned and BYOD considerations will apply in mixed mode. Technical support could be complicated for the site as different patients may need different information. On the other hand, enrollment could have a positive impact with the possibility of including a diverse patient population. Regarding the other features, including applying restrictions on notification and alarms, the impact will be in relation to the proportion of provisioned vs. BYOD devices used towards the study.
Allowing patients or physicians to use alternative devices in parallel (PC, laptops, tablets, phones) minimizes the risks of data loss if one device is lost. This depends on the eCOA vendor’s ability to support multiple configurations.
b) eCOA design
Needless to say, eCOA design is very critical. A straightforward and clear design is more likely to improve compliance and data quality. Simple reminders in the form of alerts or alarms can help patients or caregivers to complete the eCOA correctly. Simple validation rules, i.e., logical edit checks and entry restrictions can be a guide to completion whilst not influencing the data. It is therefore important that the study team is involved in the specifications review phase to contribute and ensure the design is optimal. Catching these in the user acceptance testing (UAT) phase will cause a significant delay in the timeline.
c) Scrolling design impact
One of the eCOA build considerations, especially with BYOD and different sizes of devices, is the scrolling impact on the data collection. Scrolling is a perceived barrier in the use of BYOD to capture electronic patient reported outcomes (ePROs). Shahraz et al. (2021) “explored the impact of scrolling on the measurement equivalence of electronic patient-reported outcomes” … “in the presence and absence of scrolling.”18 This study, to our knowledge, is the first research that evaluates scrolling; it provides some positive signals to help mitigate concerns over use of a scrolling feature when it is necessary. While the need for scrolling is unlikely on larger devices and can be completely prevented when providing a provisioned smartphone to study participants, the need to scroll cannot be completely eliminated in a BYOD setting in which a pre-defined criteria to exclude small BYOD devices is not set up. Below are four recommendations relevant to future ePRO design:
Continue to design ePRO to avoid scrolling when using a provisioned device or in case of BYOD, a smaller than average screen.
Mitigate scrolling by using one of the approaches described (smart-scrolling, scrolling indicator/pop-up, or navigation buttons at the foot of the screen requiring scrolling to progress),
Override certain user-adjusted screen display settings within the app display where possible; and
Always provide partial provisioning as an option to allow for patients with unsuitable smartphones, which can be facilitated by defining a minimum specification that can be easily identified by patient/site.
d) Electronic back-up
Most of the eCOA vendors can provide an alternative electronic back-up in case the patient or caregiver is not able to enter data in the primary device because of device functioning issues (or if the device is lost). An app on BYOD may require an installation on a replacement device but it is more straightforward if web-based back-up is implemented. Please ensure all the backup options are discussed well in advance as it may impact on design, screens for submissions, training, etc.
e) Paper backup
The study team should decide if a paper backup for ePRO is justifiable for primary endpoint related data for scenarios such as a patient going on vacation with no internet access or charging facility, or a lost device.
For many reasons, the general recommendation is not to implement a paper backup. For example, a paper backup needs a separate process including separate source and justification for data change provided by the patient at a given time. In addition, Fleming (2015)14 indicated mixing paper and electronic field-based assessments is the riskiest type of mode mixing because of the significant likelihood that the two modes will not generate equivalent responses. As would be anticipated, deviations from the planned electronic collection of PRO data in a clinical trial primarily involve the study site or subject defaulting to a paper-based data collection form. This has the potential to introduce measurement errors that could diminish the ability to document true treatment benefit and should be avoided. In addition, the FDA implicitly encourages the use of ePRO when possible and discourages field-based PRO data collection using paper methods because of the inability to know when the data was entered as required; there are no electronic date and time stamps on paper-based questionnaires.
With that being said, it will be critical that the sponsor determine “whether to allow data collection on paper as a backup or to incur missing data” (Fleming, 2015).14 The justification must be documented towards any data that needs paper back up based on patient safety, study design, and endpoint need. The same level of supporting source or evidence documentation will be needed for any change request for ePRO data by the site.
It must be noted, however, that mixed methods of data collection do occur within trials, although this practice is not recommended, and so should be addressed. Strategies for transcribing PRO data from paper into the ePRO/EDC should be outlined in the DMP or equivalent document prior to implementation. This includes noting in the database that the source is paper, as the statistical analysis plan may need to allow for a sensitivity analysis to be conducted (Fleming, 2015).14
Another important point to remember, while implementing paper back-up, is that it needs to be signed by the patient or observer if the COA type is a PRO or ObsRO and the data points subsequently entered in the EDC or eCOA. Once the data is entered from paper backup in EDC, that data can be source data verified (SDV) according to the study’s clinical monitoring plan.
f) eCOA translations and linguistic validation level
Translation and linguistic validation are typically not the responsibility of a Data Management function, but it could be helpful to be aware of the timelines. The majority of the eCOA building vendors build the software with English as a primary source language and a translation vendor initiates the other language’s translation as applicable after the English Go Live. The strategy might differ depending on the first country to be enrolled.
The level of linguistic validation requirement is based on the criticality of the data and guidance around it. The study team should discuss this in detail during the planning phase so that the build timeline is not affected. Depending on the study design, requirement, population, and the eCOA, there may be a need to implement more than one linguistic validation. Ensure translations details and timelines are discussed in detail to avoid unpleasant surprises.
g) eCOA integration in the EDC
When the eCOA and EDC are the same vendor, data is stored on the same platform and integration discussion is not necessary. If the platforms are different, the study team will need to determine if there is a need to integrate eCOA data into the EDC platform and what the justification for doing so is. This process impacts the budget and timeline for Go Live. In addition, there may be a need for ongoing quality control throughout the life of the study, which may generate related costs.
Alternatively, there are multiple ways to access the eCOA data outside of the EDC system (e.g., vendor portal, data extracts, standard reports, data analytics tools). In addition, custom reports can be programmed based on functional review needs. If a data is not integrated and collected in both platforms, reconciliation is needed for data cleaning purposes.
D. eCOA in Study Conduct
a) eCOA Data Management Plan (DMP)
As with any data collection in a clinical trial, the means of collection and how it will be handled by data management should be described in the DMP or an equivalent document. Please refer to the chapter on Data Management Plan in the GCDMP for details (Lebedys, E, 2021).19 This section will focus on the eCOA-specific DMP. The eCOA DMP should describe the device and how it is expected to be used. The technical specifications, including screenshots, are usually contained in a separate document, along with the validation of the tool. This document should describe all branching attributes and any differences between the data collected at different visits. Information as to where these documents are stored and what Standard Operating Procedures (SOPs) they follow are vital for audit purposes. The DMP should include details of the following:
The eCOA device type and version used in the study (if provisioned).
A list of all validated questionnaires.
The process and frequency of how the data is transferred from the device to the eCOA database (e.g., immediately with Wi-Fi, once a day, etc.).
A process for a paper backup in case of a device failure or loss.
An acceptable period of data correction, and a data correction process.
Actions to be completed at the end of patient participation, e.g., will the website be retired, how to remove/disable the app if downloaded on BYOD, track returned devices if provided.
A defined escalation path, to be used if issues cannot be solved in a timely manner. For example, if users are to contact the Helpdesk and the Helpdesk is not able to help solve the problem then to whom and how should the issue be escalated.
A process for data change/correction must be defined in detail in the DMP for situations in which eCOA/ePRO data is incorrect and there is evidence for correct data. This process of collecting evidence may include paper, eSource, within the device, etc. The audit trail must reflect the change, the reason for the change and the originator. Guidance from the EMA (2023) states “For certain types of systems (e.g., ePRO) the data entered may not be uploaded immediately but may be temporarily stored in local memory. Such data should not be edited or changed without the knowledge of the data originator prior to saving. Any changes or edits should be acknowledged by the data originator, should be documented in an audit trail and should be part of validation procedures.” 20
The DMP should be updated throughout the study life cycle as and when there is impact due to protocol amendment, version updates, new data review, and reconciliation.
If eCOA data is to be integrated into the EDC, then UAT is required before the Go Live of both the systems and the process should be defined in the DMP, including how to handle incorrect data. If eCOA data is not integrated into EDC, then a reconciliation process should be documented.
A software modification process.
An interim lock and final eCOA database; if outside the EDC, a lock checklist and process.
Any eCOA relevant SOPs that will be implemented towards the study.
An eCOA Data Flow chart, unless part of the overall data flow chart.
Any review or reconciliation expected and associated details.
A responsibility matrix to identify accountability for reviewers for each task around eCOA, whether it is compliance of device, compliance of data, activation of visits if applicable, review, etc.
b) ePRO/COA Data Transfer Agreements (DTA)
When ePRO/COA data is not directly collected in the EDC system, this document should be finalized soon after the Go Live, including which data and the format for the data to be transferred, any mapping that is required, e.g., visit names, the frequency of the transfers, location of transfer, etc. It is critical that the test transfer occurs with UAT data immediately after the DTA finalization as a means of validating the process and ensuring that no errors, such as inversion of questions, occur. If the data is to be loaded into the EDC where various parties will be able to view the data, then this process must be tested prior to the transfer of actual data. If data is integrated into the EDC, it may be necessary to test updated data in the eCOA server that is also reflected in EDC. If any data collected in ePRO is blinded, the DTA should be prepared accordingly to ensure data is not transferred or provided to blinded team members. Ensure that the DTA is reviewed by the study programming team, eCOA vendor, and any other core team member as applicable.
c) eCOA training materials
Training material is key for eCOA data completion to ensure that all the relevant completers of eCOA understand what to expect and how to complete the assessments in different scenarios. If the device is being provisioned it is important for the site staff to be comfortable with the device, how it works, and how to get support before giving it to the patient. Time for individual training by the site staff for the patient must be factored into the relevant visit. If the device is to be used for both PRO and ObsRO, it will be necessary to train both the patient and the observer in the use of the device and on how to ensure that the correct person is entering the correct data. An information or smart card may be helpful to keep useful information, such as the web address, help desk number, and tips such as what to do when the battery fails in easy-to-follow instructions.
When the eCOA is on a patient’s own device then the site staff should check that the app has been downloaded correctly (if necessary) and that the patient has the information or link for any support they may need. The variety of devices, if BYOD used, may seem daunting to site staff, but patients are usually confident in downloading apps on their own device. The patient should be made aware that updates may get pushed to their device and to accept notifications to update when necessary. Patients should also be instructed to not change device locations, such as time zones, as this can impact programmed notifications and alerts. In many cases, patient training is available in BYOD as well when the app is downloaded. Web-based solutions usually have training online.
There is always the need to have a process for when things do not go well, such as a battery failure or loss of signal during data entry. There should be straightforward directions for the patient and site staff in such instances. Additionally, patients are likely to call sites before they call the helpdesk and sites are likely to call the sponsor before the helpdesk. Maintaining a log of frequently asked questions with answers in a shared location is recommended. Data is only as good as the entry.
d) eCOA data review & monitoring
Depending on the type of data collected in eCOA, data review and responsible core team members will need to be discussed and documented:
Principal Investigator (PI)– Review of PRO responses, e.g., mental/physical state of the patient completing the Suicide Scale that requires urgent intervention by the PI.
Site staff – follow up on patients with significantly missed entry.
CRA – review to assess compliance and notify the site to follow up.
Data Manager – review fired edit checks if integrated in EDC, review data against certain logical measures e.g., extreme changes in physical performance, repeated AEs, etc.
Clinician – possible protocol deviations, AEs of concern, incompatible data with disease e.g., consecutive days of no pain in a pain study, etc.
Since eCOA data is time-point specific, by the time the study team reviews and identifies issues, it may be too late for the change. The use of inbuilt edit checks in eCOA can help identify any illogical data right at that time point as eCOA is being completed and enhance the quality and completeness of the data. These should not be designed to shape the information received but will allow for easier data entry in the case of a lead question such as “Have you had pain today?” If “Yes” further questions appear such as intensity, if “No” the questions move to the next section such as activity. Ensuring in the design of the data collection that all questions need to be answered to move forward is an excellent way to achieve complete data.
Despite edit checks within the system, there may be data issues that can be highlighted by using risk-based monitoring software to look for outliers both for completion and quality on an individual level but also for site performance. These are useful, but similar plotting in more basic systems or programming will help with a thorough review. Aligning dates of completion with visits can also identify if missed visits occur at specific points in the visit schedule, e.g., when visits move from weekly to monthly. This might require the intervention of site staff to ensure patient motivation.
A patient profile is an additional review tool that can be scoped with the eCOA vendor to be designed in an eCOA system that allows a real-time patient profile. Alternatively, it can be scoped to be programmed separately, utilizing eCOA, EDC, and other external data via data transfers.
Regular review of the audit trail, usually by data management, must be focused and can provide not just information regarding patient/observer compliance but site performance, data changes, and exceptional activity that might require some explanation. Please refer to the SCDM White paper Audit Trail Review: A Key Tool to Ensure Data Integrity (SCDM, 2021) for further information.21
e) eCOA data correction
The process for data changes should be clear and unambiguous. Some corrections are straightforward, such as devices initiated by a site with incorrect patient ID. As far as possible the data entered by the patient should remain as is unless there is clear evidence that supports data being illogical and/or incorrect.
The process for questioning the data must be defined in the DMP based on the study team’s input, e.g., a study may use multiple QOL or validated questionnaires during the same time-point, which may share a common question, and because of licensing the sponsor is not able to delete or modify questions. In that case it will be necessary to compare the responses, especially if it is related to critical data. If it means that some data is void, then a process must be put in place to remove that data from any analysis.
“If” and “how” the data is corrected should also be considered very carefully when choosing the COA tool. As with all clinical study data, it is important for there to be a clear audit trail of data changes including by whom, when, what was changed, and why. The regulatory guidance towards changing data that is entered directly by patients is evolving and highly scrutinized. It is critical to ensure that all of the details around patient-entered data change are backed with proper evidence/source approved by the respective site principal investigator (PI) and documented in a timely manner. While the sponsor must oversee the conduct of the trial, there can be no sponsor influence in any data source or change in data source. Please refer to the JSCDM consensus paper by Delong et al. (2023).22
With respect to patient reported data changes, there may be some regulatory guidance, for example, specific to AE 24-hr recall. Reliability and validity of PRO-CTCAE® daily reporting with a 24-hour recall period (Lee, 2023),23 the decision for “patient ability to validly recall the information” should be well justified and documented in the study document. Please refer to “Recall Period” in the FDA guidance (2009),24 section 3.
f) eCOA data quality and re-training
Data quality can be directly affected by training, by motivation, and by design. If the study team encounters significant missed or illogical data entered in the eCOA there could be few solutions: A. Enhancing the training material and re-train the site staff and the users; or B. Update the eCOA software to make it user friendly; or protocol amendment (e.g. schedule of assessment, number of assessments, etc.) can play a key role in reducing user burden and increasing motivation.
Data managers should take a role in defining the sorts of training required, whether face-to-face, remote learning, interactive multimedia, or train-the-trainer. “It is important to consider that optimal training does not typically happen at a single point in time, but rather over time through repetition and the use of more than one delivery medium” (Ly et. al, 2019).25 It is too easy to dismiss training as part of the Investigator Meeting or Site Initiation Visit. Time and time again this has resulted in multiple questions, different interpretations of the questions, and inconsistent data. Change of site staff is another critical factor.
g) ePRO/COA reports and metrics
Standard and custom reports and metrics should be discussed with eCOA vendors. Depending on the standard reports available, the need for complex custom reports should be discussed early in the study and periodically to assess the study data review and compliance review need.
Compliance – these are metrics that report the completed scales/questionnaires and, more importantly, the missing scales/questionnaires. The frequency of such reviews will depend on the therapeutic area and the review cycle of the study. It is critical to detect any compliance issue as early as possible to (a) identify non-compliant patients for re-training purposes; (b) enable data correction if “recall period” is still applicable; and (c) implement a mitigation plan to prevent further data loss. The development of data visualization and data analytics tools have greatly improved the tracking of compliance. The site staff and CRA should review compliance reports and take action as necessary.
Anomalies – these are unexpected data, for example, dramatic increases or decreases of events such as narcolepsy incidents; or alternatively, an anomaly could be identified as sustained responses over a long period of time, e.g., several weeks of the same depression score. These data may be correct but are worth following up. If the device is a site device that can be accessed by several subjects, then reviewing by site may also identify potential concerns.
Transfers – if the device is not uploading immediately, it is possible to review the uploading pattern from the device. Whilst the data may be contemporaneous if the data is only loaded on the day of the visit, it can delay reviews and can hide possible issues.
h) ePRO/COA data snapshots
Whilst most snapshots will be planned in advance and at certain time-points, e.g., 50 patients reaching the 6-month visit, it is possible for data to be required at short notice. With traditional paper collection, sites would require time to transcribe the information into the EDC system. An eCOA, however, can function as a Transient Data Collector by capturing patient-reported outcomes, clinician-reported outcomes, or observer-reported outcomes in real time during clinical trials. The data is temporarily stored on electronic devices, such as smartphones, tablets, or web-based platforms, before being transferred to a central database for permanent storage and analysis. This temporary storage ensures data integrity and accuracy by reducing recall bias and by allowing for timely, remote data collection. In such a case, data may have to be downloaded or synced by the patient and then uploaded for the data to be available to the vendor/sponsor. Depending on the cleanliness required for the snapshot, the procedures for database lock will need to be followed or the data accepted as is. In both instances, planning and communication is important to enable as much data as possible to be available. The process decided for the study should also be mentioned in the DMP.
E. ePRO/COA in Study Close Out
The processes in this phase are “performed to finalize all activities across all process groups to formally close the project or phase” (PMI, 2013).26 Like with EDC, ensure that the ePRO lock checklist is met prior to ePRO database lock.
Study Close Out demands that all data collected in the eCOA is synced and uploaded to the server, that the audit trail is complete, that there are no queries outstanding for clinical data and/or non-clinical data as applicable to the protocol, that all subjects are marked completed or discontinued and unable to enter additional data. In the case of provisioned devices, Study Close Out requires that all devices are returned to the site; or in case of BYOD, that the apps are disabled or accesses revoked, etc. In addition to the system close out, ensure all documents are finalized and uploaded in the TMF and any post-lock activities are completed with final data transfer.
| End of Study Activities | eCOA vendor | CRO/Sponsor | Site | Patient |
| Inform sites and eCOA vendor of the scheduled study Database Lock date, project closeout requirements, and process. | X | |||
| Ensure all eCOA data has synced from the device and all data has been transmitted to the server. Obtain a final confirmation from eCOA vendor that syncing is complete for all the subjects, if required. | X | X | X | |
| Check if all eCOA data has been made available in the eCOA database. | X | X | X | |
| If eCOA data transfers back in EDC, any empty forms should be deactivated and any data change request by site/patient actioned, | X | |||
| Use report to find and solve data issues. | X | X | ||
| Freeze/lock eCOA data. | X | X | ||
| Limit or remove access for using the device, if applicable. | X | X | ||
| Return provisioned devices to the site, if applicable. | X | |||
| Collect devices from patients if the sponsor provided provisioned devices to use for the duration of the study. | X | X | ||
| For app-based BYOD, remove eCOA app from the patient/user’s device. | X | X | ||
| Request return of provisioned devices following eCOA vendor’s process. | X | |||
| Provide return shipping material including labels, boxes, etc. for sites if provisioned devices are used. | X | |||
| Return devices to eCOA vendor and report missing ones. | X | X | ||
| Confirm receipt of the devices. | X | |||
| Request eCOA vendor to prepare media of all sites and patients’ data and distribute to specific sites with respective data, if not collected directly into EDC. | X | |||
| Provide each site with an end of study media of their patients’ data, or instructions on how to download the data. | X | |||
| Deliver final dataset and reports. The final study data is archived to compact disc or secure FTP to sponsor. Other, more modern solutions as available from vendors. | X | X | ||
| Complete the document archiving. | X |
7) Project Management Considerations for Implementing eCOA
Building and implementing ePRO software is a critical task in clinical development to ensure data integrity, especially data collected that is related to the primary endpoint, the secondary endpoint or is efficacy/safety related. The criticality of proper ePRO implementation is massive considering that data collected in ePRO is the source data directly coming from the patients, which is time-stamp sensitive and often is collected without any site staff’s presence, unlike EDC data where the site holds source data that can be used for source data verifications. Any data not collected correctly or completely or in the correct timeframe could compromise data integrity and may not be used towards the analysis or submissions. Considering the magnitude of the impact of eCOA data, this section will provide project management considerations to clinical data managers to assist ePRO implementations.
This section defines a framework from a Guide to the Project Management Body of Knowledge fifth edition (PMBOK) (PMI, 2013),26 that divides the study phases into initiating, planning, executing, monitoring and controlling, and close out. It also discusses balancing the competing knowledge areas that apply to any project management within all the phases, which include, but are not limited to: integration management, scope management, time management, cost management, quality management, HR/resources management, communication management, risk management, procurement management, and stakeholder management. ePRO project management is an integrative undertaking that requires all the knowledge areas to be appropriately aligned and connected with many other aspects of clinical trials to facilitate coordination. The PM considerations for implementing ePRO tool, for different study phases, are discussed below. Furthermore, please refer to Appendix A.
A. Initiating
The processes in this phase are “performed to define a new study or a new phase for an existing study by obtaining authorization to start the project or phase” as defined in PMBOK 5th ed. 26 eCOA development may require a significant timeline in this phase compared with EDC. Some of the applicable eCOA development related activities are listed below.
The protocol must be as clear as possible with respect to assessment titles and assessment versions (if it is a validated tool).
The DM lead should participate in the protocol review and should highlight any possible build issues. This is a critical step for core team alignment towards data collection and cleaning throughout the study.
Obtain an official version of each assessment from the license holder (if applicable) to be used towards the initial build. If the assessment is not a validated tool, it is still critical to ensure the version used for the build is the correct one.
In global studies, it is critical to identify the countries and corresponding language(s). The translations and linguistic validation level requires thorough research as the level of linguistic validation and cognitive debrief may differ depending on criticality of data, ie, primary endpoint, secondary endpoint, exploratory endpoint, etc. per regulatory guidance or best practices. The level of validations will have a significant timeline and budget impact. It is also important to identify when certain languages will need to be implemented depending on the study site locations.
The patient population and their ease of use are important factors. The complexity of the data to be collected and the study design complexity on branching should be a primary driver in selecting the eCOA system vendor as capabilities differ widely.
B. Planning
The processes in this phase are “required to establish the scope of the project, refine the objectives, and define the course of action required to attain the objectives that the project was undertaken to achieve” (PMI, 2013).26 At this phase, the following may be applicable during the eCOA development:
Identify a list of countries participating in the study.
Identify Site Initiative Visit (SIV) and First Patient First Visit (FPFV) for each site and languages required for translations.
Define the translation validation level for each instrument as these are significant cost and timeline drivers.
Identify the translation vendor and determine who is responsible for oversight.
Decide between provisioned, BYOD, or mixed method implementation.
Draft key eCOA study plans, e.g., project management plan, data management plan, data transfer specifications, escalation plan, etc.
Understand the scope of standard training and decide if supplementary material is required, e.g., study-specific video training. This is applicable to training across various roles including patient, PI, study coordinator, CRA, etc.
Ensure the eCOA build timeline is reviewed and understood by the study team. The timeline should include all key milestones (e.g., specifications review and finalization, multiple rounds of UAT, language-specific screenshots, patient-facing document translations, site document translations, etc.).
Schedule an eCOA build kick-off meeting and regular focused meetings to monitor build activities.
Establish logistics for device shipment (initial and replacement) and equipment returns if applicable.
Decide on the implementation of interactive reports, status reports, risk/issue tracker, etc. to support the study.
Review lessons learnt from any previous eCOA implementations.
Decide on a secure location and method to transfer the files. This is critical to ensure the flow and transit of data is secured and in compliance with regulatory and country requirements, e.g., GDPR.
C. Executing
The processes in this phase are “performed to complete the work defined in the project management plan to satisfy the project specifications” (PMI, 2013).26 The eCOA development activities in this phase ensure it is implemented according to the timelines:
Draft, review, and finalize eCOA specifications. It is critical that the team performs a thorough review of the specifications, including aligning to protocol requirements, sponsor specific requirements, etc. to avoid UAT findings later in the process that may lead to significant impact on Go Live.
Once the programming is complete, a cross-functional screen review and UAT should be performed to ensure the design matches the specifications. Any UAT findings that result in a change in programming or specifications must be re-tested. This process should be repeated until there are no more findings before the software is pushed into production. Studies using validated software may only require configuration and UAT rather than programming.
The initial programming is done in the primary language. Once complete, subsequent translations may be required. The translation priority will be based on the site initiation visits.
Ship devices to the sites (if provisioned), instruct sites to download the study app or provide access to the website.
Ensure all parties are trained on the devices, apps, etc.
Train the sites on how to eCOA device activation.
Finalize all eCOA study plans before the FPFV, e.g., project management plan, data management plan, data transfer specifications, escalation plan, etc.
Ensure test transfer is complete after the DTA is finalized.
Train the study team on the implementation of interactive reports, status reports, risk/issue tracker, etc. to support the study.
D. Monitoring and Controlling
The processes in this phase are “required to track, review, and regulate the progress and performance of the project; identify any areas in which changes to the plan are required; and initiate the corresponding changes” (PMI, 2013).26 Considering that the eCOA data is directly coming from patients/caregivers/site staff, with a huge spectrum of experience, comfort, and scenarios that the team may not have considered during the build process. It is therefore very important to perform regular eCOA compliance reviews, eCOA data trend analyses, and to schedule team review meetings to determine if there is a system issue, site and/or patient/caregiver training issue, or any process gap. It is critical to understand the issue and to perform impact analysis for each finding. In case of a protocol amendment, the impact analysis must be performed in terms of design and documentation and implemented following the study process. Reports may need to be updated as necessary.
E. Close Out
The processes in this phase are “performed to finalize all activities across all process groups to formally close the project or phase” (PMI, 2013).26 Like EDC, ensure that the eCOA lock checklist, defined in eCOA DMP, is met prior to the database lock. This includes but is not limited to:
All data collected in an external eCOA database is synced and uploaded to the server.
No queries outstanding.
Paper back-up related actions fully completed if applicable.
All subjects are marked completed or discontinued.
Access is restricted to ensure additional data cannot be entered, changed or deleted.
In case of provisioned devices, all devices returned to the site or in case of app-based BYOD apps disabled.
Site and sponsor TMF completions.
Final data transfer to sites and sponsors.
Regulatory, site and sponsor file retention (archival) policies are implemented.
8) Recommended SOPs
The relevant SOP may vary from company to company. There might be an overarching SOP and then associated job aids or work instructions, or it may spread across various SOPs. However, the following areas should be covered by process document(s):
Study Startup and Clinical Database build.
Interim/Final Database lock and Study Closeout.
Data Transfer.
Reports and Metrics.
Maintenance and Data Review.
Data Management Plan.
Data Review Plan.
Data Correction/Change Plan.
User Training.
User Management and Security.
Software System Change Control.
Generation and Review of Archive Media (Other, if media is not used).
Paper backup (Paper CRF design with header information).
Device replacement process.
Helpdesk process, including escalation.
9) Literature Review
This revision is based on a systematic review of the peer-reviewed literature indexed for retrieval. The goals of literature review were to (1) identify published research results and reports of evaluation of new methods regarding eCOAs and (2) identify, evaluate, and summarize evidence capable of informing the practice of incorporating eCOAs in a clinical trial from start up, through implementation and study close out.
PRISMA* Diagram for electronic Clinical Outcome Assessment Chapter.
*PRISMA is the acronym for the Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
Seven reviewers screened all 74 abstracts for relevance to clinical trials. Disagreements were adjudicated by the writing group. Nineteen articles meeting inclusion criteria were selected for review. Seven individuals reviewed each of the 19 selected articles and three additional articles were found during the review. Each was read for mention of explicit practice recommendations or research results informing practice. Relevant findings have been included in the paper. This synthesis of the literature relevant to eCOA supports transition of this paper to an evidence-based guideline.
The following PubMed query was used:
(“electronic Clinical Outcomes Assessment” OR “eCOA” OR “electronic patient-reported outcome” OR “ePRO” AND “clinical trial” OR “clinical trials” OR “clinical study” OR “clinical studies”)
This was further narrowed by adding:
AND (“clinical trial” {Title/Abstract] OR (“clinical trials” {Title/Abstract] OR (“clinical study” {Title/Abstract] OR (“clinical studies” {Title/Abstract] OR (“registry” {Title/Abstract] OR (“registries” {Title/Abstract] OR (“observational study” {Title/Abstract] OR (“interventional study” {Title/Abstract] OR (“phase 1 study” {Title/Abstract] OR (“phase 2 study” {Title/Abstract] OR (“phase 3 study” {Title/Abstract] OR (“phase 4 study” {Title/Abstract] RO (“phase I study” {Title/Abstract] OR (“phase II study” {Title/Abstract] OR (“phase III study” {Title/Abstract] OR (“phase IV study” {Title/Abstract] OR (“first in man” {Title/Abstract] OR (“clinical research” {Title/Abstract] OR (“device study” {Title/Abstract] OR (“interventional trial” {Title/Abstract] OR (“phase 1 trial” {Title/Abstract] OR (“phase 2 trial” {Title/Abstract] OR (“phase 3 trial” {Title/Abstract] OR (“phase 4 trial” {Title/Abstract] OR (“phase I trial” {Title/Abstract] OR (“phase II trial” {Title/Abstract] OR (“phase III trial” {Title/Abstract] OR (“phase IV trial” {Title/Abstract] OR (“randomized clinical trial” {Title/Abstract] OR (“clinical research” {Title/Abstract]
The search query was customized for, and executed on, the following databases: PubMed, CINHAL), EMBASE, PsychSource, Association for Computing Machinery (ACM), and Web of Science. A total of 382 works were identified through the searches. The searches were conducted in May 2021. Search results were consolidated to obtain a list of 74 distinct articles. Because this was the first review for this paper, the searches were not restricted to any time range. Literature review and screening details are included in the PRISMA diagram for the paper, which follows the references.
Appendix A: Project Management in eCOA (PMBOK Framework)
| Knowledge Areas of Project Management | Initiating | Planning | Executing | Monitoring and Controlling | Closing |
| Project Integration management | Decision on the ePRO instrument and method of data collection in the protocol | ePRO Project Management Plan Identify Mother language for each instrument (US English vs. others) and Translation Validation plan |
Lesson learned | Lesson Learned | ePRO DBL |
| Project Scope management | Identify all the ePRO instruments, respective versions, and the official copy to be used for the ePRO build. Identify respective timepoints. Identify data collection with or without site staff. Identify languages and translation linguistic validation requirements per regulatory requirement. [high impact on cost and timeline] Identify any data collected in ePRO is blinded |
ePRO kick-off meetings (Ensure key team members attend) | Compare budgets vs. actual spending to prevent scope creep. Review protocol amendments (if any) and provide impact on the ePRO device and documents. |
Compare budgets vs. actual spending to prevent scope creep. Review protocol amendments (if any) and provide impact on the ePRO device and documents. |
|
| Project Time Management | High level timeline for ePRO database build, timeline for any major or minor modification to go live, submission ready patient facing screens, and LPLV to database lock duration. (e.g., # of weeks.) Factor in LV level for translations for local submission. |
Detailed timeline finalized and communicated to team re: ePRO specs; Finalized screens for submission; any integration in EDC; UAT; Go Live in primary language; Go Live for each language; ePRO DMP, DTA, etc. | Ensure database build activities are on time and Go Live occurs before SIV in appropriate languages. The device shipped beforehand. |
Initiate Database Lock checklist and timeline discussion | Ensure ePRO is locked, and subject data is received on time in correct format. |
| Project Cost Management | Budget review for all the components including translations vendor and translation LTV level | Compare Budget vs Actual and initiate Change Order if needed | Compare Budget vs Actual and initiate Change Order if needed | ||
| Project Quality Management | Vendor Qualification Audit/ePRO system validation audit report Decide which SOP to implement CRO or Sponsor SOPs |
Review SOPs Discuss how to manage process deviation and CAPA creation if needed |
Perform UAT Participate in review/approval of all key documents including: specs, PMP, DMP, lock process, DTA etc. Work with core team for Data cleaning expectation (if applicable) |
Perform frequent data review with core team’s input including protocol deviations. Critical to identify any device modification or site/subject training is required. Ensure agenda and minutes for meetings are filed in TMF. |
Ensure database lock checklist is applied before database lock. Ensure TMF is complete. Ensure Final data is received and filed in a data repository. Ensure devices are returned, database is decommissioned, and accesses revoked |
| Project Human Resource Management | Identify Lead ePRO PM, ePRO designer, ePRO engineer, Lead ePRO DM, and other critical team members. It is important to identify the team and sub-vendor while obtaining a license for an instrument. Review CV to ensure the assigned team is qualified for the complexity of the study. |
Identify other supporting ePRO staff and their functions. | Ensure there is a transition plan in place in case of change of key team members. Team training requirements |
Assess team’s training requirements based on issue trends. | Release team members from project after confirming no further post lock activities or scope accordingly. |
| Project Communication Management | Review Communication Plan Decide the frequency of team meetings. Finalize template for agenda and minutes. Attend/present in the Investigator Meeting(s)/format. Decide which SharePoint or similar tool will be used to transfer files and communicate instead of relying on emails |
Attend the recurring team meeting(s) for UpToDate communications. Maintain meeting agendas and minutes in TMF |
Attend the recurring team meeting(s) for UpToDate communications. Frequent data metrics and site trend communication to the team. Organize database lock kick off meeting with core team members |
Communicate database checklist items are achieved, and database locked to team members. | |
| Project Risk Management | Identify ePRO device and data related risk factors (major/minor) | Periodic risk review and mitigation | Periodic risk review and mitigation | Finalize risk register by updating & closing all risks. | |
| Project Procurement Management | Finalize 3rd party vendors including License holders and final approval to use the tools towards the study; and software vendor; translation vendor, linguistic validation vendor, etc. and SOW | Attend third party vendor kick off meetings, decide on contact information, timelines, etc. | Create and finalize change order as applicable. | Create and finalize change order as applicable | Ensure all data activities are complete and clean before the database locks. Ensure there is a confirmation of completion for all post lock activities. |
| Project stakeholder management | Identify all cross functional team members at sponsor, CRO and ePRO vendors and set up regular calls towards the build with all core team members. | Identify all the ePRO documents authored by different functions and frequency meetings. | Obtain the finalized copy of all these plans. | Gather stakeholder feedback/issues; make necessary updates. |
Abbreviation
| Abbreviation | Terminology |
| AE | Adverse Event |
| BYOD | Bring Your Own Device |
| CDISC | Clinical Data Interchange Standards Consortium |
| ClinRO | Clinical Reported Outcome |
| COA | Clinical Outcome Assessment |
| CRA | Clinical Research Associate |
| DCT | Decentralized Clinical Trial |
| DM | Data Management |
| DMP | Data Management Plan |
| DHT | Digital Health Technology |
| DTA | Data Transfer Agreement |
| eCOA | electronic Clinical Outcome Assessment |
| EDC | Electronic Data Collection |
| ePRO | electronic Patient Reported Outcome |
| FDA | Food and Drug Administration |
| FPFV | First Patient First Visit |
| GDPR | General Data Protection Regulation |
| GCDMP | Good Clinical Data Management Practice |
| HIPAA | Health Insurance Portability and Accountability Act |
| ICH | International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use |
| LPLV | Last Patient Last Visit |
| MHRA | Medicines and Healthcare Products Regulatory Agency |
| ObsRO | Observer Reported Outcome |
| PerfO | Performance Outcome |
| PI | Principal Investigator |
| PMBOK | Project Management Body of Knowledge |
| PRO | Patient Reported Outcome |
| RFI | Request for Information |
| RFP | Request for a Proposal |
| SDV | Source Data Verification |
| UAT | User Acceptance Testing |
| VAS | Visual Analog Scale |
Competing Interests
Erica Sage has 28 years of experience in data management, during which she has worked in small, medium and large CROs. Her work has included clinical trials in every phase, from small phase 1 to large scale global phase 4 studies, across all therapeutic areas, supporting companies ranging from small biotech start-ups to large pharmaceutical multinationals. Her reach extends to all aspects of data management; she has led diverse project management teams, managed global teams, and contributed to protocol development and SOP writing. She has been a contributing member of SCDM for over 10 years. Erica recently retired from her full-time employment but has continued to contribute to the field of clinical data management through publications with colleagues within the SCDM community. Dawn Edgerton has over 25 years of experience in research computing and 10 years managing clinical trial staff across all functional areas, including project management, clinical operations, data management, data standards, biostatistics, medical writing, and statistical programming. She has been an independent consultant since 2015 providing clinical data management vendor oversight to small biopharma companies. She has an MBA with concentrations in BioSciences Management and Services Management from NC State University in Raleigh, NC, and a BS in Mathematical Sciences with a Computer Science Option from the UNC, Chapel Hill, NC. Sachi Amatya is a clinical data strategy and management leader with over 20 years of experience across CROs, big pharma, and small biotech. As the founder of Amatya Data Consulting LLC, she specializes in eClinical data solutions, including eCOA, EDC, IRT, and external data integration, ensuring high-quality, regulatory-compliant data for clinical trials. With deep expertise in data governance and risk-based oversight, she has led data management functions across global clinical programs. A Certified Clinical Data Manager (CCDM), Sachi is an active contributor to industry best practices and drives innovation in clinical data strategy. She holds a Doctor of Education (Ed.D.) degree in Leadership, Policy, and Organization from Vanderbilt University.
Version History
| Version | Date | Author(s) |
| 1.0 | 20 JAN 2025 | Amatya, S., Edgerton, D., Sage, E. |
References
1. U.S. Food and Drug Administration. Clinical outcome assessments (COAs) in medical device decision making. Division of Patient-Centered Development. October 2023. Accessed January 17, 2025. https://www.fda.gov/about-fda/division-patient-centered-development/clinical-outcome-assessments-coas-medical-device-decision-making
2. Walton MK, Powers JH, Hobart J, et al. Clinical outcome assessments: Conceptual foundation—Report of the ISPOR Clinical Outcomes Assessment—Emerging Good Practices for Outcomes Research Task Force. Value Health. 2015; 18(6): 741–752. DOI: http://doi.org/10.1016/j.jval.2015.08.006
3. Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient non-compliance with paper diaries. BMJ. 2002; 324(7347): 1193–1194. DOI: http://doi.org/10.1136/bmj.324.7347.1193
4. Coons SJ, Eremenco S, Lundy JJ, et al. Capturing patient-reported outcome (PRO) data electronically: The past, present, and promise of ePRO measurement in clinical trials. Patient. 2015; 8: 301–309. DOI: http://doi.org/10.1007/s40271-014-0090-z
5. Pestronk M, Johnson D, Muthanna M, et al. Electronic data capture—Selecting an EDC system. J Soc Clin Data Manag. 2021; 1(1). DOI: http://doi.org/10.47912/jscdm.29
6. Amatya S, Edgerton D. Vendor Selection & Management. J Soc Clin Data Manag. 2021; 1(4). DOI: http://doi.org/10.47912/jscdm.118
7. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH harmonised guideline: Integrated addendum to ICH E6(R1): Guideline for good clinical practice E6(R2). Published 2016. Accessed January 17, 2025. https://database.ich.org/sites/default/files/E6_R2_Addendum.pdf
8. U.S. Food and Drug Administration. Use of electronic health record data in clinical investigations: Guidance for industry. Published 2018. Accessed January 17, 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-electronic-health-record-data-clinical-investigations-guidance-industry
9. U.S. Food and Drug Administration. Electronic source data in clinical investigations: Guidance for industry. Published 2013. Accessed January 17, 2025. https://www.fda.gov/media/83113/download
10. U.S. Food and Drug Administration. General principles of software validation; Final guidance for industry and FDA staff. U.S. Department of Health and Human Services. January 11, 2002. Accessed January 17, 2025. https://www.fda.gov/media/73141/download
11. Medicines & Healthcare products Regulatory Agency. GXP data integrity guidance and definitions. Published 2018. Accessed January 17, 2025. https://www.gov.uk/government/publications/guidance-on-gxp-data-integrity
12. U.S. Food and Drug Administration. Guidance for industry: Computerized systems used in clinical trials. U.S. Department of Health and Human Services. April 1999. Accessed January 17, 2025. https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/fda-bioresearch-monitoring-information/guidance-industry-computerized-systems-used-clinical-trials
13. U.S. Food and Drug Administration. Code of Federal Regulations, Title 21, Part 11 (21 CFR Part 11): Electronic records; electronic signatures. Published 1997. Accessed January 17, 2025. https://www.ecfr.gov/current/title-21/chapter-I/subchapter-A/part-11
14. Fleming S. Optimizing electronic capture of clinical outcome assessment data in clinical trials: The case of patient-reported endpoints. Ther Innov Regul Sci. 2015; 49(6): 797–804. DOI: http://doi.org/10.1177/2168479015609102
15. Gordon S, Crager J, Howry C, et al. Best practice recommendations: User acceptance testing for systems designed to collect clinical outcome assessment data electronically. Ther Innov Regul Sci. 2022; 56: 442–453. DOI: http://doi.org/10.1007/s43441-021-00363-z
16. Matza L, Patrick D, Riley A, et al. Pediatric patient-reported outcome instruments for research to support medical product labeling: Report of the ISPOR PRO Good Research Practices for the Assessment of Children and Adolescents Task Force. Value Health. 2013; 16(4): 461–479. DOI: http://doi.org/10.1016/j.jval.2013.04.004
17. Duracinsky M, Lalanne C, Goujard C, et al. Electronic versus paper-based assessment of health-related quality of life specific to HIV disease: Reliability study of the PROQOL-HIV questionnaire. J Med Internet Res. 2014; 16(4): e115. DOI: http://doi.org/10.2196/jmir.3330
18. Shahraz S, Pham T, Gibson M, et al. Does scrolling affect measurement equivalence of electronic patient-reported outcome measures (ePROM)? Results of a quantitative equivalence study. J Patient Rep Outcomes. 2021; 5: 23. DOI: http://doi.org/10.1186/s41687-021-00296-z
19. Lebedys E, Famatiga-Fay C, Bhatkar P, et al. Data Management Plan. J Soc Clin Data Manag. 2021; 4. DOI: http://doi.org/10.47912/jscdm.116
20. European Medicines Agency. Guideline in computerised systems and electronic data in clinical trials. Published 2023. Accessed January 17, 2025. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-computerised-systems-electronic-data-clinical-trials_en.pdf
21. Society for Clinical Data Management. Audit trail review: A key tool to ensure data integrity. Published 2021. Accessed January 17, 2025. https://scdm.org/wp-content/uploads/2024/07/2021-eCF_SCDM-ATR-Industry-Position-Paper-Version-PR1-2.pdf
22. Delong PS, Humler D, Haag T, et al. Best practice recommendations for electronic clinical outcome assessment data changes. J Soc Clin Data Manag. 2023; 2(1): 1–11. DOI: http://doi.org/10.47912/jscdm.249
23. Lee MK, Basch E, Mitchell SA, et al. Reliability and validity of PRO-CTCAE® daily reporting with a 24-hour recall period. Qual Life Res. 2023; 32(7): 2047–2058. DOI: http://doi.org/10.1007/s11136-023-03374-5
24. U.S. Food and Drug Administration. Patient-reported outcome measures: Use in medical product development to support labeling claims. December 2009. Accessed January 17, 2025. https://www.fda.gov/media/77832/download
25. Ly JJ, Crescioni M, Eremenco S, et al. Training on the use of technology to collect patient-reported outcome data electronically in clinical trials: Best practice recommendations from the ePRO Consortium. Ther Innov Regul Sci. 2019; 53: 431–440. DOI: http://doi.org/10.1177/2168479018796206
26. Project Management Institute. A guide to the project management body of knowledge (PMBOK® guide) (5th ed.). Project Management Institute; 2013. DOI: http://doi.org/10.1002/pmj.21345
Further Reading
1. Coons S. ePRO systems validation: Clearly defining the roles of clinical trial teams and ePRO system providers. Value Health. 2013; 16(4): 457–458. DOI: http://doi.org/10.1016/j.jval.2013.04.006
2. European Medicines Agency. Clinical investigation of medicinal products for the treatment of asthma – Scientific guideline; 2015. https://www.ema.europa.eu/en/clinical-investigation-medicinal-products-treatment-asthma-scientific-guideline. Accessed February 7, 2025.
3. European Medicines Agency. Appendix 2 to the guideline on the evaluation of anticancer medicinal products in man: The use of patient-reported outcome (PRO) measures in oncology studies (EMA/CHMP/292464/2014); 2016. https://www.ema.europa.eu/en/documents/other/appendix-2-guideline-evaluation-anticancer-medicinal-products-man_en.pdf. Accessed February 7, 2025.
4. Montano O, Pestronk M, Johnson D, et al. Electronic data capture—Study conduct, maintenance, and closeout. Journal of the Society for Clinical Data Management. 2021; 1(1). DOI: http://doi.org/10.47912/jscdm.31
5. Eade D, Pestronk M, Russo R, et al. Electronic data capture—Study implementation and start-up. Journal of the Society for Clinical Data Management. 2021; 1(1). DOI: http://doi.org/10.47912/jscdm.30
6. U.S. Food and Drug Administration. Clinical outcome assessment (COA) frequently asked questions: COA definition. U.S. Food and Drug Administration; December 2, 2020. https://www.fda.gov/about-fda/clinical-outcome-assessment-coa-frequently-asked-questions#COADefinition. Accessed February 7, 2025.
7. U.S. Food and Drug Administration. Value and use of patient-reported outcomes (PROs) in assessing effects of medical devices: PRO report final document. U.S. Food and Drug Administration; 2016–2017. Accessed January 17, 2025, Available at: PRO Report final document.pdf (fda.gov).
8. U.S. Food and Drug Administration. FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) resource: Glossary. 2016. Accessed January 17, 2025, Available at https://www.ncbi.nlm.nih.gov/books/NBK338448/
9. U.S. Food and Drug Administration. Submitting patient-reported outcome data in cancer clinical trials: Guidance for industry technical specifications document. U.S. Department of Health and Human Services, Center for Drug Evaluation and Research. November 2023. Accessed January 17, 2025, Available at https://www.fda.gov/media/173581/download
10. U.S. Food and Drug Administration. FDA patient-focused drug development guidance series for enhancing the incorporation of the patient’s voice in medical product development and regulatory decision making. U.S. Food and Drug Administration. Accessed January 17, 2025, Available at https://www.fda.gov/drugs/development-approval-process-drugs/fda-patient-focused-drug-development-guidance-series-enhancing-incorporation-patients-voice-medical
11. U.S. Food and Drug Administration. Ulcerative colitis: Clinical trial endpoints. Guidance for industry (Draft guidance); 2016. Accessed January 17, 2025, Available at https://www.federalregister.gov/documents/2016/08/08/2016-18716/ulcerative-colitis-clinical-trial-endpoints-draft-guidance-for-industry-availability
12. U.S. Food and Drug Administration. Digital Health Technologies for Remote Data Acquisition in Clinical Investigations. December 2023. Retrieved January 12, 2024, Available at https://www.fda.gov/regulatory-information/search-fda-guidance-documents/digital-health-technologies-remote-data-acquisition-clinical-investigations#:~:text=Digital%20Health%20Technologies%20for%20Remote%20Data%20Acquisition%20in%20Clinical%20Investigations%20December%202023&text=This%20guidance%20provides%20recommendations%20on,investigations%20that%20evaluate%20medical%20products.