Introduction
REDCapCloud.com is a unified data science platform designed for regulatory-grade usage in healthcare systems to support highly sensitive investigations, clinical trials, and translational science. The platform integrates patient engagement features such as eRecruitment, eScreening, and myREDCloud.com to facilitate remote communications. Its electronic data capture (EDC) capabilities include advanced branching logic that enables customizable study workflows controls to screen, consent, enroll, and deliver customizable surveys.
We posited that the platform’s advanced technology could make it an ideal data capture solution for an IRB-approved genomics-based study on factors affecting resilience in active-duty military service members. Notably, Department of Defense (DoD) and Defense Health Agency (DHA) policies require strict protections of service members’ data. REDCapCloud.com’s integrated secure communications, and its advanced programming – along with the fact that it was licensed for use in the DoD – made it a seemingly optimal platform for potential use in the DHA. We therefore conducted a usability study of the REDCapCloud.com platform for the collection of genetic material, research health information, and personally identifiable information, to establish its suitability for deployment in future DHA studies.
Background
Our planned genomics-based study design will require the recruitment of service members from all branches at installations throughout the continental United States and will use large-scale genomic data (LSGD) employing Mendelian randomization, a research method that uses genetic variants to study the causal relationship between modifiable risk factors and health outcomes and provide evidence equivalent to a randomized control trial.1,2 LSGD requires deoxyribonucleic acid (DNA), which can be obtained through saliva collection using specially designed tubes that stabilize DNA for extended periods of time. Saliva collection kits can be mailed or shipped to participants’ homes for convenient, non-invasive collection, and shipment requires secure collection of service members’ home addresses and telephone numbers. Our planned genomics study is designed to inform policy and practice to improve personalized medicine, which is an emerging practice of medicine that uses individuals’ genetic profiles to guide prevention, diagnosis, and treatment of disease, including mental health conditions.
Resiliency is an abstract concept that broadly refers to the ability to adapt to or recover from adverse events and is an important attribute for members of the military. Resiliency allows military members to withstand the stresses of combat and life in the military and helps military members to stay focused on their missions; it is also essential to recovering from illnesses, injuries, and emotional trauma.3,4 Generally, alcohol consumption is thought to have negative effects on resiliency and/or readiness in the military. Alcohol can impair judgment, can increase the risk of accidents, can lead to disruptive behaviors such as fighting, and can lead to health problems and addiction. Alcohol use is also associated with disorders, such as post-traumatic stress disorder and depression.5,6,7,8 However, alcohol is also commonly used in social settings. Alcohol reduces inhibition and can help people to socialize and to connect to others, which are important aspects of overall well-being and ability to cope with emotional trauma.9 Observational epidemiological studies do not produce evidence of causality and thus it remains unclear if correlations between alcohol consumption and measures of resiliency are causally related. There is a broad and general need for studies that can reach conclusions about causality and elucidate etiology, especially for situations in which conducting a randomized controlled trial is not possible for logistical or ethical reasons.
The planned use of an effective, customizable EDC system will ultimately be to investigate causal models of comorbid alcohol use and post-traumatic stress disorder using participants with specific alleles associated with alcohol flush reaction. Alcohol flush syndrome is a condition in which consuming alcohol causes symptoms such as facial redness (flush).10 Alcohol flush syndrome is caused by a deficiency of the enzyme aldehyde dehydrogenase 2 (ALDH2) and to a lesser extent the alcohol dehydrogenase (ADH) genes, which are all part of the metabolic pathway that breaks down alcohol.10,11,12 When consumed, alcohol is first metabolized into acetaldehyde by ADH1B and ADH1C, and then acetaldehyde is metabolized into acetate by ALDH2. When ALDH2 is deficient, the toxic compound acetaldehyde accumulates in the body, leading to the symptoms of alcohol flush syndrome.13 Alcohol flush is generally considered to be an unpleasant experience and individuals with alcohol flush syndrome tend to consume less alcohol on average. Alcohol flush syndrome is common in East Asian populations. The single nucleotide polymorphism rs671 in the ALDH2 gene is the common genetic cause of ALDH2 deficiency and is found in substantial frequencies in East Asian populations. For example, rs671 is present in 28% in Korean people and as high as 45% in Taiwanese people.11 Overall, approximately 8% of the world’s population carries the allele of rs671 that causes alcohol flush syndrome, many of whom will have attenuated ability to process acetaldehyde into acetate.11 Alcohol flush is also present in other populations, such as Caucasians, but has a lower prevalence.
Rather than target a specific ancestry group, the planned study will create a cohort drawn from the general service member population with the goal of recruiting service members with alcohol flush reaction (or with a first degree relative with the reaction) using a random sampling pool of up to 20,000 to be created by the Defense Manpower Data Center. Inclusion in this future study would therefore require screening for ages 21 years and older (legally permitted to drink) and self-reported confirmation of alcohol flush reaction.
To provide evidence of the platform’s suitability for use in future studies falling within the purview of the DHA, a pragmatic evaluation of the platform’s workflow processes and effectiveness of data collection using test subjects was deemed necessary.
Methods
This usability study was designed to support the data collection for the Precision Resilience Study (IRB # C.2024.105). We employed a modified hybrid moderated/unmoderated workflow usability to assess the ability of the programmed workflow. The moderated usability consisted of a facilitator guiding participants through task in real-time, whereas the unmoderated usability consisted of participants completing tasks remotely, such as surveys, without real-time supervision. After obtaining Institutional Review Board (IRB) determination as a non-research study (IRB #FWH20240061 N), EDC usability data were collected in March 2024 using a convenience sample of six 59th Medical Wing – Science and Technology, Joint Base San Antonio team members who functioned as tests subjects, or “participants”, completing a variety of actions that will typically be required of future participants enrolled in the Precision Resilience Study.
Procedures
General workflow methods were derived from usability testing frameworks.14 Five major inflection points in the study included the abilities to perform the following functions:
Conduct screening to identify study participants who meet the study’s inclusion criteria;
Allow study participants to schedule a virtual consent call and sign consent and Health Insurance Portability and Accountability Act (HIPAA) documents (necessary for genetic testing);
Email consented study participants their signed electronic consent and HIPAA forms;
Deliver 15 web-based survey instruments and collect responses, and
Request shipping address and telephone contact details if study participants consented to provide a saliva sample for DNA analysis.
A research scientist programmed the platform to perform a sequence of critical steps in workflow to recruit, consent, and collect survey data and saliva samples for genetic analyses. Branching logic and time stamps for start and end of tasks to perform these stated goals were coded. Programming to facilitate the signing of DHA Informed Consent and HIPAA forms and send signed copies to respondents via a password protected file was performed. Additional programming included communications such as appointment making for consent, reminders to complete the survey, notifications of shipping of saliva kit, notifications of compensation, and other emails.
Test subjects received a personalized email invitation with a link to the screener. Personal devices, both computers and smartphones, were used. The screener used a modified version of a validated 3-item alcohol flush reaction questionnaire and other items to determine eligibility.
Test subjects that met study eligibility were sent an email to create a myREDCapCloud.com account to continue their participation in the study. Communications to schedule a virtual consent would then be sent only to eligible participants. Consenting was conducted on REDCapCloud.com’s virtual platform by a study administrator who utilized computer/smartphone cameras and microphones for visual and audio communication. After consenting, participants were provided a document to permit engagement in a compensated research study, and a Commander Approval Memo for signature by their commander per DoD/DHA policy. In this mock workflow study, receipt of the Commander Approval Memo was confirmed, however, no Commander signature was required.
Consented test subjects then logged into a personalized REDCapCloud.com account to complete 14 validated instruments commonly used in medical research (OMB Control No. 0720-FLSH). The instruments included were 1) Facts About You—Demographics; 2) Alcohol Use Disorder (AUDIT) Screening, tobacco/caffeine use, and medications checklist; 3) Response to Stressful Events, a brief resilience scale; 4) Adverse Childhood Events (ACE); 5) Life Events Checklist (LEC); 6) Combat Exposure Scale (CES); 7) PTSD Checklist; 8) Patient Health Questionnaire (PHQ-9); 9) Generalized Anxiety Disorder (GAD-7); 10) Pittsburg Sleep Quality Index (PQSI); 11) The Patient-Reported Outcomes Measurement Information System (PROMIS)—Sleep; 12) Epworth Sleepiness Scale (ESS); 13) Insomnia Severity Index (ISI); and 14) Nightmare Disorder Index (NDI).
The system recorded responses and duration to complete for each survey. The ACE, LEC, and CES are indexes of trauma severity. Outcome measures of symptom severity of PTSD, depression, and anxiety were captured by PTSD Checklist, PHQ-9, and GAD-7. Subclinical standings on the psychological outcomes would indicate resilience, per cut-points defined by previous research. Functional, modifiable risk factors of sleep linked to mental health were measured by PSQI, PROMIS, ESS, ISS, and NDI. Confounding variables and modifiable factors were measured by tobacco and caffeine usage and medications that might interfere with the use of alcohol flush reaction as an instrumental variable. Resilience as a psychological stressor was measured by RSE, a modifiable factor.
At the completion of all surveys, the platform prompted the study participant to upload the signed copy of the Commander Approval Memo. Finally, only for participants who simulated consent to provide a saliva sample for genetic analysis, the platform requested their best address and telephone number for the shipment of the saliva test kit via Federal Express. The study concluded with an optional survey entitled “User Feedback” to help improve the survey. Because REDCapCloud.com is not an approved software for use by the DHA, this project was designed only to evaluate the workflow processes using a convenience sample consisting of researchers and team members.
Results
To simulate the email invitation that would garner interested service members to begin the screening process, we sent an invitation email containing the screening link to team members serving as simulated “participants”. The screening email contained a link to the homepage, where respondents were automatically screened to determine if they met the study criteria: 1) they are at least 21 years of age; 2) they are an active-duty service member, and 3) they have symptoms of alcohol flush or a first order family member with symptoms of alcohol flush. Branching logic was programmed to send only those respondents who met the criteria to move forward through the screening process to the virtual consent process. Test subjects who met all study criteria were prompted to create a myREDCapCloud.com account to continue through the consent and data collection phase of the study.
Screening: All study participants met the study criteria and provided an email address to continue the study and receive study materials. The mean ± Standard Error of the Mean (SEM) duration to complete the screening process was 2.00 ± 0.82 minutes. The platform was programmed to email each study participant a login request to create a myREDCapCloud.com account for the study participant to move to the next phase of the study: the Virtual Consent Scheduling and the Virtual Consent Call.
Virtual Consent Scheduling: After the platform confirmed that a study participant qualified for the study (Figure 1A), the study participants were able to schedule a date and time for a virtual consent call (Figure 1B) with the Study Administrator. At the appointed date and time, the study participants clicked on the “Join Video Call” button on the Join Virtual Video page (Figure 1C). All instruments, including the Virtual Consent call, were embedded with a REDCapCloud.com code to record the “start” and “end” time for each instrument.
REDCapCloud.com confirms if the subject meets the study criteria (Figure 1A), prompts them to schedule a date and time for their Virtual Consent Video (Figure 1B), and allows them to Join Virtual Video call with a study administrator (Figure 1C).
Virtual Consent: The Virtual Consent call contained the embedded instruments, namely, Commander Approval Memo, Informed Consent Document, and HIPAA Form. Study participants were first shown the Commander Approval Memo and were instructed that further participation in the study required a signed Commander Approval Memo to be uploaded via their myREDCapCloud.com account. Upon agreement to upload a signed Commander Approval Memo, the platform automatically sent the study participant a portable document format (PDF) of the Commander Approval Memo for signature and upload to the study participant’s myREDCapCloud.com account.
Study participants were then presented with the consent form. The study details were described to the participant by the study administrator. Questions from the study participant were addressed by the administrator. Using their myREDCapCloud.com account, the study participant selected the individual “Yes” or “No” buttons to indicate their consent to the following questions:
Do you consent to take the survey?
Do you consent to providing your saliva for genetic sequencing?
Do you consent to receiving information about other genetic findings relevant to your health?
Do you consent to be contacted about future research opportunities?
The participant then typed their name in a text box and provided an electronic signature in myREDCapCloud.com. The study administrator signed the Consent form on their REDCapCloud.com platform. This dually signed consent was time stamped. The consented participant immediately received an email stating that their consent was available for them to view and download, however, because this file was password protected, they were told that the password would arrive in a separate email. All consented participants received both the email containing the signed Consent Form and the subsequent email containing their unique password.
The HIPAA form was then reviewed by the participant and the study administrator answered any questions that the participant had concerning the HIPAA form. The participant entered their printed name and provided an electronic signature to the HIPAA form. Upon the study administrator’s acknowledgement that the form was signed, the platform automatically sent the signed HIPAA form to the participants (6/6). The mean ± SEM duration to complete the Virtual Consent Call was 9.16 ± 2.46 minutes. The mean duration to complete the review of the Commander Approval Memo, Consent, and HIPAA documentation appears in Figure 2.
The duration in minutes (mean ± SEM) of the Commander Approval Memo (1.00 ± 0.00), Consent Form (5.33 ± 0.56), and HIPAA Form (2.83 ± 0.06).
After confirmation of the email delivery of the signed Consent and HIPAA forms, the platform presented 14 sequential surveys to the participants, and they were instructed to read the items and select a response. The platform recorded responses and duration to complete for each instrument as described in the Methods section (see Figure 3). All instruments were presented in the same order with the assumption that no order effects would be evidenced. While individual surveys contained a range of questions, quantification of the participants’ individual and cumulative survey completion times provide data collection administrators the potential to identify technical difficulties or confusing instructions embedded within a survey to better optimize survey design.
The duration in minutes (mean ± SEM) of the Individual Surveys of 6 subjects. The mean duration to complete the above 14 surveys was 31.50 ± 9.96 minutes.
At the completion of all surveys, REDCapCloud.com utilized the study participants’ myREDCapCloud.com account to prompt the participants to upload the signed copy of the Commander Approval Memo. Finally, only if test subjects consented to provide a saliva sample for genetic analysis, the platform requested their mailing address for the shipment of the saliva test kit. The study concluded with an optional survey entitled “User Feedback”. The duration of each of these final tasks was recorded (see Figure 4).
The duration (mean ± SEM) of the Commander Approval Memo Upload (1.67 ± 0.42), Contact Information entry (1.17 ± 0.17), and User Feedback (2.17 ± 0.65).
Discussion
We were successful in programming REDCapCloud.com to correctly identify mock subjects who met the study criteria during the screening process. Once test subjects were identified that met the study criteria, we were able to seamlessly schedule and conduct a Virtual Consent Call to consent study participants. We observed all aspects of study workflow to proceed as programmed, with all study participants receiving pertinent study documents in a secure manner.
The EDC system performed equally as well in the one instance in which a study participant used a smartphone to perform all usability activities rather than a personal computer. After the subjects were consented, the platform allowed study participants to complete the surveys and other study tasks in their own time. Study administrators were able to observe all consented study participants’ progress throughout the study duration. Given that “start” and “end” fields were embedded in all study instruments, we were able to quantify the time it took each study participant to complete study related tasks.
Limitations of this study included a convenience sample consisting of team members who had varying familiarity of online data collection platforms, including REDCapCloud.com, SurveyMonkey, and REDCap.org. We used six testers which may be considered adequate to uncover many usability problems in qualitative studies but quantitative studies may require at least 20 users to achieve statistical (and practical) significance, according to usability experts.15 This electronic data capture system addresses common limitations associated with clinical data collection, such as manual data entry errors, inconsistent data formats, difficulties with multi-site coordination, challenges associated with informed consent documentation, data security and compliance risks, and difficulty in generating study reports. The built-in validation rules help to reduce manual data entry typographical errors and invalid data entry. Inconsistent data format errors are mitigated by utilizing standardized field types and branching logic to ensure uniform data entry across sites and users. The EDC allows multi-center access with permission controls to standardize protocols across study site locations. Informed consent documentation is achieved using an eConsent module to streamline the collection and storage of electronic informed consent documents, and the EDC incorporates HIPAA-compliant security settings, audit trails, and role-based permissions to protect sensitive data. The EDC has built-in export tools and customizable reports that can be generated at any time point in a clinical study.
Future usability testing would include conducting cognitive interviewing to ensure items were interpreted correctly, ease of use studies, and other usability aspects using participants with representative characteristics of our intended study population. Other future testing might include clinical research coordinators’ user experiences from the perspective of usability theories.
Conclusion
The REDCapCloud.com workflow study demonstrated the efficacy of a highly functional EDC system capable of expediting study recruitment, enrollment, virtual consent, and data capture, thereby broadening the impact of research within the DoD and DHA. REDCapCloud.com provides a graphical user interface for data entry, a validation component to check user data, and a deidentification component to make study participant data less identifiable. Once REDCapCloud.com becomes approved for use by the DHA, the Precision Resilience Study can be deployed and the findings used to inform policy and practice relevant to modifiable behavioral health factors within the context of personalized medicine to improve resiliency and readiness in areas that include, but are not limited to, the capacity to deploy.
DoD Disclaimer
The views expressed are those of the authors and do not reflect the official views or policy of the Department of Defense or its components. The views of REDCapCloud.com are not necessarily the official views of, or endorsed by, the U.S. Government, the Department of Defense, or the Department of the Air Force. No Federal endorsement of REDCapCloud.com is intended.
Competing Interests
The authors have no competing interests to declare.
References
1. Ference BA, Holmes MV, Smith GD. Using Mendelian Randomization to Improve the Design of Randomized Trials. Cold Spring Harbor Perspectives in Medicine. Jul 1 2021; 11(7). DOI: http://doi.org/10.1101/cshperspect.a040980
2. Treur JL, Munafò MR, Logtenberg E, Wiers RW, Verweij KJ. Using Mendelian randomization analysis to better understand the relationship between mental health and substance use: a systematic review. Psychological Medicine. 2021; 51(10):1593–1624. DOI: http://doi.org/10.1017/S003329172100180X
3. McInerney SA, Waldrep E, Benight CC. Resilience enhancing programs in the US military: An exploration of theory and applied practice. Mil Psychol. 2024; 36(3):241–252. DOI: http://doi.org/10.1080/08995605.2022.2086418
4. Tornero-Aguilera JF, Stergiou M, Rubio-Zarapuz A, Martín-Rodríguez A, Massuça LM, Clemente-Suárez VJ. Optimising Combat Readiness: Practical Strategies for Integrating Physiological and Psychological Resilience in Soldier Training. Healthcare. 2024:1160. DOI: http://doi.org/10.3390/healthcare12121160
5. Polimanti R, Peterson RE, Ong J-S, et al. Evidence of causal effect of major depression on alcohol dependence: findings from the psychiatric genomics consortium. Psychol Medicine. 2019; 49(7):1218–1226. DOI: http://doi.org/10.1017/S0033291719000667
6. Smith ND, Cottler LB. The epidemiology of post-traumatic stress disorder and alcohol use disorder. Alcohol research: current reviews. 2018; 39(2):113.
7. Wium-Andersen MK, Ørsted DD, Tolstrup JS, Nordestgaard BG. Increased alcohol consumption as a cause of alcoholism, without similar evidence for depression: a Mendelian randomization study. Int J Epidemiol. 2015; 44(2):526–539. DOI: http://doi.org/10.1093/ije/dyu220
8. Zhu C, Chen Q, Si W, Li Y, Chen G, Zhao Q. Alcohol use and depression: a Mendelian randomization study from China. Front Genet. 2020; 11:585351. DOI: http://doi.org/10.3389/fgene.2020.585351
9. Hoopsick RA, Homish DL, Vest BM, Bartone PT, Homish GG. Resilience to hazardous drinking among never-deployed male United States Army Reserve and National Guard soldiers. Alcohol Clin Exp Res. 2021; 45(3):566–576. DOI: http://doi.org/10.1111/acer.14561
10. Zeng HY, McAloon M. The “Asian flush syndrome”: ALDH2 Deficiency and Long-term Health Consequences. Journal of Student Research. 2023; 12(1). DOI: http://doi.org/10.47611/jsrhs.v12i1.4048
11. Chang JS, Hsiao J-R, Chen C-H. ALDH2 polymorphism and alcohol-related cancers in Asians: a public health perspective. J Biomed Sci. 2017; 24:1–10. DOI: http://doi.org/10.1186/s12929-017-0327-y
12. Chen C-H, Kraemer BR, Mochly-Rosen D. ALDH2 variance in disease and populations. Dis Model Mech. 2022; 15(6):dmm049601. DOI: http://doi.org/10.1242/dmm.049601
13. Eng MY, Luczak SE, Wall TL. ALDH2, ADH1B, and ADH1C genotypes in Asians: a literature review. Alcohol Res Health. 2007; 30(1):22.
14. Lewis JR. Usability: lessons learned… and yet to be learned. International Journal of Human-Computer Interaction. 2014; 30(9):663–684. DOI: http://doi.org/10.1080/10447318.2014.930311
15. Nielsen J. How Many Test Users in a Usability Study? Accessed October 18, 2024, https://www.nngroup.com/articles/how-many-test-users/.