Members of clinical trial teams are expected to possess or acquire data literacy skills necessary to leverage available data in collaborative trial management. Data leads, such as data managers, data scientists, and statisticians, are expected to generate visualizations to support collaborative trial management decision-making. We consider these skills necessary for organizations to benefit from the digital evolution of the drug development industry. As such, we describe the design and implementation of a data learning series to increase data literacy skills within a contract research organization. Results of the data learning series demonstrate participant engagement and satisfaction, as well as modest improvements in data literacy, as evidenced by pre- and post-testing of the participants. Potential changes in behavior and culture are typically measured over a period longer than this initial evaluation. Measured by the project’s early outcomes, we conclude that an opportunity to increase skills across the spectrum of data literacy has the potential to increase user proficiency and job satisfaction. The successful planning and development of such a program is highly dependent on leadership support through a direct link to organizational business objectives.
Clinical trials have grown in operational complexity, or “the aspects of a clinical trial that may be difficult to implement according to the timeline or procedures outlined.”
Enhanced data capabilities, such as real time access to information that provides useful insights into trial progress, may increase clinical trial success as measured by time, cost, and quality. Dashboards have been developed and implemented for decision making with varying results in both effectiveness and adoption.
In clinical trials, decision making has been studied from the trial team perspective.
However, a user’s level of data and graphical literacy challenges the effectiveness of providing information through visualization techniques. While the clinical research data management profession has continued to examine and refine their competencies and professional certification,
Evidence exists demonstrating the benefit of data training, whether it is specific to data analytics or data literacy in general.
In the healthcare sector in general, it has been recognized that data literacy and analytics enables the use of multiple forms of organizational data and enhances quality and risk management scoring within those organizations.
The need for a data learning program intervention stemmed from our organization’s Data Governance Council (the Council) (Figure
Data Governance Council Structure.
Note: Gray boxed denote DLS team members.
A companion instruction strategy incorporated standard data related materials from a concept and theory perspective and then organized and arranged the materials to best relate them to the work being done within a CRO. Current circumstances, mainly related to the global pandemic, dictated that the program be built as a virtual offering.
To establish a baseline for data literacy within the organization, the first class of participants was randomly chosen by function in proportion to the size of each department. The participant list was prepared by the organization’s human resources department using a funnel technique that initially targeted all global employees that were members of a function supporting clinical development services. The list was narrowed down to a proposed class size of 100 participants using the parameters of equal distribution between regions while keeping the number of participants proportional to function size. No other criteria, such as time in organization, gender, level of education, or hierarchy, were considered. The selection methodology allowed for a sample of participants that mirrored the organization’s structure and composition. Classes after the pilot session were designed to be filled by a nomination process.
The pilot program emerged as a six-week, six-module virtual offering (Table
Data Learning Series Module Summary.
Module | Objectives |
---|---|
Module 1: Data and Organizational Value | Understanding the drivers of clinical transformation Measuring value from the customer and patient perspective How we measure quality How analytics can transform the industry |
Module 2: The Data Driven Organization | Data and knowledge Information as an asset Effective analysis of performance Attributes of quality |
Module 3: Understanding the Mechanics of Data | Contextual properties of data Data types Introductory statistical measures Numerical measures Graphical representations of data important to determining our success |
Module 4: Systems and Technology Supporting a Data Infrastructure | Data and systems terminology Data attributes The process of data analysis Data bases and data types common in the clinical research environment The data lifecycle in a Contract Research Organization Data visualizations to evaluate study progress |
Module 5: Using Data to Measure Effectiveness | Key Performance Indicators Benchmarking Measuring process cycle times in clinical research Process methodologies common in the industry |
Module 6: Communicating with Data to Achieve Intended Outcomes | Effective data displays Achieving outcomes using data Communication for different audiences using data |
The team used the analyze, design, develop, implement, and evaluate (ADDIE) model, traditionally used by instructional designers and training developers, to build the program.
describe the value of the organization’s work using performance data,
explain the hierarchy of data, data quality attributes, data governance, data context, and relevance,
discuss the role of a data analyst, database structures, warehouses, and data architecture,
practice using data types and statistical terms and recognize common patterns in data visualizations,
apply skills in interpreting and communicating with data.
Evaluation and assessment of our program effectiveness was performed using the Kirkpatrick model (Figure
Measurement of Success. Figure adapted from the Kirkpatrick Model of Evaluation.
Level three involves behaviour change, which is largely dependent on targeted manager feedback or observations, such as team members using data to support recommendations. Level four involves organizational measures that are indicative of a broader change and are associated with a positive impact on culture and specific business goals. An example of level four results is Brynjolfsson and colleagues’ paper indicating that organizations that employ data driven decision making demonstrate a productivity rate 5–6% higher than other firms.
Careful consideration must go into linking program goals to organizational outcomes, along with how those will be measured and compared to the organization’s business goals. A formal return on investment (ROI) analysis was completed, which enabled the development team to focus on which measurements would be monitored over a specific period to assess the impact of the program. The evaluation of the program is continuous and in sync with the completion of each class. Classes are expected to be delivered once per quarter, leaving time for evaluation and adjustment if necessary
During the pilot period, 276 participants across 3 classes were enrolled in the 6-week course. Of those, 172 (62%) completed all 6 weeks, with 143 (52%) completing their final activity. Each module of the course built on the previous one and included a synchronous application session. If participants missed sessions due to vacation or work requirements it was difficult to make that portion up and to continue in the course. The original class (randomly chosen) was representative of the departmental proportion in the company, but classes two and three (nominated) had strong representation from the clinical and biometrics departments based on the requirements of those positions. Class representation was balanced across age groups and years of experience and proportionally by size of office in each region (North America and Europe) (Figure
Number of Program Participants by Age Category, Region, and Years of Experience.
To capture level one (reaction) outcomes, a survey (Figure
Participant Survey-Data Learning Program.
Word Cloud Data Analysis Illustrating Participant Opinion of Most Liked Course Elements. (Larger type represents most mentioned topics. Color is added for visual interest).
Word Cloud Data Analysis Illustrating Participant Opinion on Course Improvement. (Larger type represents most mentioned topics. Color is added for visual interest).
Average Scoring by Category by Module.
Level two (learning) evaluation of the program consisted of the six end-of-module knowledge checks, the pre- and post-course assessment, and the final case study written activity. Details for the pre- and post-course assessment included a validated numeracy skill level measurement from the health literature from both objective and subjective perspectives.
Relationship Between Subjective and Objective Scores.
Level three (behaviour) evaluation relied on line manager participation. Engagement was accomplished through an initial meeting between the program owners and line managers to review course details. Weekly guided learning agendas were provided detailing the objectives for each module along with examples and suggested activities to reinforce learning. Example questions for managers to use included the following: “What was your biggest takeaway from this week’s module?” “What is new learning for you from this module?” “How are you applying your learning?” Included in the agendas were knowledge check questions for line managers to check their own knowledge. The managers were also provided access to the data learning series SharePoint site where they could review materials, engage in discussion groups with participants, or take advantage of the private manager area on the site that contained tools and materials relevant to supporting the participants in their learning journey. Line manager feedback was captured through both a periodic survey and a post-course focus group interview. The survey asked about behaviour change, such as using data to make recommendations or to solve problems. The focus group feedback centred on the program participants’ desire to have content learning relate directly to on-the-job application. Additional feedback included praise for the program and its value to individuals and the organization. Variation in manager engagement was not examined but would be an interesting outcome to measure against improvement in course participant scores or career progress within the organization.
Finally, level four (results) evaluation is based on measures included in the formal ROI. In general, level four is captured over a period that exceeds the time frame during which the written description of the project was completed. Level four measures are also difficult to isolate as responses to one single intervention. The measures we will periodically examine include a reduction in employee turnover, improvement in employee data presentation skills at key meetings related to business outcomes, dashboard user metrics, and quality deliverable measurements, such as project key performance indicators (KPIs).
The pilot program has prompted some revisions based on timing, participant feedback, workload, and effectiveness of the materials as determined by the module knowledge checks and assessments. For example, due to time off conflicts, summer is typically not the best time for employees to enrol in a six-week program. Additionally, module content has been re-evaluated and either reduced or expanded depending on course outcomes and to better accommodate participants’ work schedules. It is expected that after several iterations the course will still be evaluated but less frequently and with fewer adjustments, depending on the direction of the industry in general. Regular course adjustments also mean each class can only be evaluated in isolation. Once the course is standardized and not experiencing measurable change, we anticipate being able to combine class evaluations into a larger data set. We also anticipate future development of advanced coursework based on employee requests and job performance.
Due to the nature and necessity of delivering a self-directed and online program, completion rates with this learning program matched with our team’s previous combined years of industry learning and development experience and can be expected to hover somewhere around 60%. Based on their workload or other competing priorities, both the randomly selected pilot class and the subsequent self-selected or nominated participants struggled in some cases to keep up with the program or to complete the program as designed. While a self-paced on-demand program may prove to be the most reasonable future direction, the sacrifice comes with the rapidly evolving digitization of the industry and employees’ abilities to keep up with that change to remain relevant in their roles. We found that participants varied in their valuation of the program, with some declining to participate or dropping out early, while others were willing to keep up by completing modules outside of work hours, including during time off. Value perception did not necessarily align with any demographic or role across participants, with most functions represented in all non-completion reason categories (Figure
Data Learning Program Functional Group Non-completers by Reason as Percentage of Whole.
Using preliminary data captured based on the Kirkpatrick model, we found good acceptance based on level one (reaction) as shown in Figure
Level two (learning) data clarified that the pre- and post-course assessment measured data literacy incompletely because participants scored high from the outset. The final activity, which required both synthesis of information and communication skills, indicated an opportunity for future course work and suggested a more relevant pre- and post-course assessment was required.
Level three (behaviour) data has been somewhat sparse and difficult to obtain at this stage in the program. Less than one third of the managers surveyed or invited to the focus groups participated. Although we did establish a process for measuring behaviour change (explained in the Methods section), we do not yet have enough data to come to a reliable conclusion other than a weak positive trend based on manager comments. It is unknown whether the lack of robust feedback was due to increasing demands for time in their roles or to managers who were not invested in the coursework. We have had a substantial increase in nominations and requests for additional classes. As part of our future direction, we will conduct an evaluation of our manager engagement process to improve outcomes data collection. Another important consideration for improving feedback related to participant behaviours would be prioritization of the program administration to line managers. Tying the program to increases in data maturity index levels would demonstrate a positive effect. For example, capturing awareness of the program and increasing numbers of managers and leaders enrolling and completing the program is a measurement that would contribute to specific areas of the maturity index. Our intention was to balance time required to attend the training with a focus on prioritization of increasing data skills across the organization, but we may have overestimated either the level of data expertise or the ability to support an evolution of skill sets within the industry with those we depended on for program support.
Level four (results) will require significantly more time as we monitor the business goals and organizational outcomes tied to the program’s ROI. It is expected that career growth and development opportunities could increase employee satisfaction and engagement and thus reduce turnover. The cost of the training program will be compared to an expected reduction in turnover, offset against the costs of onboarding and training a new employee. Participants in the program will be tracked over time for both tenure and promotion opportunities. Other level four measurements are related to (1) the ability of project teams to communicate key study progress using analytics and (2) the cost of hiring an experienced data analyst versus training existing personnel, including the cost of recruitment, disruption in services, and continuity of the team’s mission. More intangible effects include culture change and the increase in client opportunities related to data experience and expertise.
The program must support all employees globally, either through adjustments, compromise in working hours, or through alternative means, such as recorded sessions with support available during working hours (Figure
Participant Location by Enrollment.
In addition to different geographies, program participants were comprised of employees from different functions, each bringing different perspectives to the series. The interactive application sessions enabled cross-functional discussion and debate that enriched project team interactions. Special attention and planning were given to the session activities to promote making the connection with everyday tasks. For example, one of the first activities involved understanding how value is generated through the work being done within a CRO, along with how that value is measured. The participants were provided with maps of the drug development process and an oncology patient journey. Participants were then asked to consider their contribution to each of these processes from the perspective of the customer and the patient. The high value points in each process were considered from a data generation and measurement standpoint, with subsequent discussion on how teams could improve that value for the customer and the patient. As the modules progressed, activities became more granular, such as determining what type of data visualization is most useful for which type of data, before circling back to looking at data sets in the aggregate and making a decision or recommendation, such as whether to incorporate telemedicine into the clinical trial process.
Achieving the desired outcomes from a data learning initiative can be challenging no matter how well thought out the process is. A critical success factor is executive support that serves as a champion for the project. The champion must have a clear understanding of the investment and return expected and be willing to bear both the costs of the program and the time participants spend on the learning activities in exchange for establishing a data driven culture. When soliciting the support of a senior executive, communication and timing are key. Having a clear understanding of the organization’s goals, strategy, and other demands on the executive’s time are important in tailoring a message.
Beyond the program champion, internal expertise is necessary from a data, information, and technology perspective. There are multiple touchpoints within a CRO that involve the data lifecycle, and it is to the advantage of the program to both incorporate those concepts and be supported by those involved to promote a data driven culture. Perhaps most important, expertise from a learning and development, leadership, and communication perspective is necessary to ensure the program is both promoted appropriately and measured and monitored for continuous improvement.
In summary, while the industry continues to evolve rapidly with new technology and data offerings coming from non-traditional sectors, we have seen early success with our data learning series program that could help fill a gap in data knowledge and support the changing work structure that is emerging across our industry. The program is meeting the goals and objectives outlined in the business proposal for the program. We anticipate seeing additional value in the upcoming results of a recent formal organizational engagement survey. While we have made every attempt to develop this program using established principles for instructional design, including outcomes measures, it is important to obtain agreement and funding for this approach prior to embarking on the task. Time commitment is a critical priority. Future considerations include offering the program externally or offering collaborative training across organizations or through a consortium to support standardization. A limitation to this approach would be consensus on training content and support in the form of resources for program development. Scalability related to need is an additional point to consider. While the program assessed and served participants from only one organization, it is not uncommon for resources to transition across organizations, indicating that the need is likely more widespread.
The authors express deepest gratitude to the following people for their work in development and implementation of the data learning series program: LaRae Bennet, Debra Jendrasek, Priscilla Pierre, Brianna Brewer, William Griffin, Megan Peters, and Diana Ritchie. The authors also express appreciation for the organization’s leadership, particularly Michael O. Wilkinson, for his advice and support of the program.
The authors are currently employed at the organization of interest in senior leadership roles within Clinical Development.