Blog

ePRO Clinical Trials: Improving Outcomes Capture

Introduction

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences. In recognition of the importance of the patient perspective, the FDA’s industry guidance, entitled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (a.k.a. “PRO Guidance”), emphasizes the use of PRO instruments when measuring concepts best known by the patient or best measured from the patient’s perspective (Patient-Reported Outcome Measures). The “e” in ePRO then facilitates this process by accepting digital, rather than paper-based, reporting.  

The outcomes selected for evaluation in a clinical trial must address the trial objectives; but, they must also be recognized as meaningful to end-users. As per the U.S. Department of Health and Human Services, for an outcome to be meaningful, it should reflect or describe how a person feels, functions, and survives (Multiple Endpoints).

The effort to understand a patient’s experience isn't merely a matter of empathy; it contributes significantly to the holistic evaluation of a treatment's effectiveness. ePRO gives weight to both subjective and objective measures of health, providing a more comprehensive view of how a treatment impacts patients' lives. An important aspect of this holistic evaluation is the consideration of the patients’ quality of life, an assessment that ePRO is instrumental in measuring.

The Many Benefits of ePRO in Clinical Trials

The digital nature of ePRO brings significant advantages, enabling patients to report on events as they occur in their day-to-day lives—from wherever they may be. This means patients can capture both planned and episodic events in real time. Perhaps they are being asked to journal each morning when they take their medication or during other daily events. Or, in other cases, perhaps they need to log more punctual or episodic symptoms like headaches or nausea. These electronic diaries, conveniently logged by patients from their homes or workplaces, provide study coordinators and sponsors with a clearer line of sight into the patient experience than they would otherwise have had access to.  

Therapeutic development is no longer about just saving lives; it's about improving the quality of those saved lives. And, ePRO is the best way to measure improvement or deterioration in quality of life. For example, patients are likely to report symptoms—such as vomiting, diarrhea, or fatigue—earlier and more often than clinicians. Importantly, there is also growing correlation between quality of life and lives saved. In an oncology study, early symptom detection through ePRO showed increased survival rates (Basch et al.).

ePRO also aids in improving patient compliance. The digital platform that deploys the ePRO can also facilitate patient engagement, enhancing protocol compliance and data quality (Gwaltney et al.). This timely and complete data collection translates to more reliable outcomes.

The Challenges of ePRO Implementation

Despite the many advantages associated with the incorporation of patient diaries into clinical trials, the implementation of ePRO comes with challenges. Patients are not always diligent in completing their paper diaries, and this can be true for ePRO as well.

Patients can become demotivated when ePRO clinical trial tasks are too long or complicated, and this confusion can lead to both decreased compliance and reduced reporting accuracy (Coons et al.). Anything that creates friction in the reporting journey, whether it’s more time required or less clarity, will have a negative impact on engagement.  

Patients can often feel frustrated by long and redundant questions in ePRO surveys (Stone et al.). It is natural to feel annoyed when you are asked to answer what appears to be the same or similar question multiple times, even if those questions are embedded in different tasks. Task definitions can also sometimes be perceived as confusing. For example, if a patient is asked to report on his or her symptom of dizziness, nausea, or vertigo without clear definitions of each, he or she may misinterpret the questions and will likely provide contradictory responses.  

Another hurdle to engagement can arise when questions are perceived to be irrelevant to the patient's condition. It is important for patients to understand why certain questions are being asked so that they can feel more invested in answering.  

Finally, and importantly, patients often neglect to report on time, resulting in recall bias. It can be difficult to remember, hours later, how many times one coughed or sneezed. Similarly with pain: The ability to remember how severe pain may have felt is inversely proportional to the time that goes by between the event and its reporting.  

Optimizing ePRO for Better Outcomes in Clinical Trials

To improve ePRO implementation, a more patient-centric approach is required. A successful ePRO strategy should increase patient understanding and motivation while correspondingly reducing burden. Efforts should also be made to encourage and assist patients in real-time reporting whenever possible.  

More Convenient ePRO

There are multiple ways to reduce patient burden, starting with enabling patients to use their own mobile devices for ePRO. BYOD (“bring your own device”) ePRO eliminates the need to carry multiple devices. Instead, patients use an app downloaded onto their smartphones, which naturally integrates more seamlessly into their everyday lives because they are more comfortable on their own devices.

Simplification should be built into the study design from the outset. Clinical research teams should be on the lookout for complex, long, and redundant questions, particularly those that may not be necessary to achieve primary endpoints. The National Institutes of Health's Patient-Reported Outcomes Measurement Information System (PROMIS) measures are a good example of an initiative that has helped to align the industry around simplified scales (National Institutes of Health).  

Another important contributor to patient convenience is through UX design. Good ePRO design should incorporate user-friendly scales and interactive functions to make data capture quick and intuitive. One-tap answers and easy scrolling enable participant responses on the go. For instance, hot flash or pain buttons can be tapped to quickly log symptoms for menopause or migraine trials. When questions are long, they can be broken down into multiple screens to make them more digestible. The easier it is for patients to access and complete tasks, the better their compliance will be.

In some cases, there may also be opportunities to enrich or replace labor-intensive ePRO with devices to capture continuous, more objective, or more detailed information about a symptom or patient activity. For example, rather than asking patients to take their temperature, they can use a device or even a patch that can relay temperature information directly to a data repository.  

More Real-Time ePRO

The closer to real time patients can report an event, the better. BYOD and intuitive UX can get patients part of the way there. Automated notifications and reminders are an additional way to aid patients, providing helpful alerts that nudge them towards on-time compliance. Gamification can also boost compliance for certain types of patients who are motivated by a competitive spirit. Certain trials also benefit from virtual study teams who monitor both compliance and safety, ensuring that any issues are flagged and addressed promptly.

Patients are not always in a place where they have access to the internet or to phone service; so, it is also important that technology enable them to log time-stamped events even when they are not connected.  

ePRO Relevance and Comprehension

Given the importance of patient comprehension for the successful completion of trials, training is a critical component to ensuring clear understanding of study objectives, the protocol, study design, and patient responsibilities. Multi-media, on-demand training modules and virtual site team support can be instrumental in enhancing patient comprehension and engagement. Training materials are also an opportunity to highlight the aspects of the trial that they might find the most personally compelling, as well as the ways in which the trial can contribute positively to their health and quality of life.  

Whenever possible, communication channels and content should be personalized to the specific patient population in the trial. Personalization needs can take many forms and should consider demographics and clinical constraints as well as comfort with technology. Adjustments might include anything from increased font size for older participants to culturally sensitive translations to multiple options for reaching out to site staff, in recognition of the fact that some patients may prefer phone or email to chat or video calls.  

Novel ePRO Approaches

While validated ePRO instruments are beneficial, they may not always suffice. There are times when novel ePRO approaches are necessary. For example, a study was run in 2022 to explore the new ePRO data capture system, PROMIS, and its ability to provide deeper understanding of treatment tolerability for patients with immune-mediated inflammatory disease (Iaquinta et al.).  

There may be opportunities to augment ePRO through image, audio, and video capture technologies to enhance data collection. Using expert raters to score this media can provide more objective outcomes to complement subjective ePRO data. Of course, creating and validating new instruments requires careful consideration, thorough testing, and validation to ensure they accurately and reliably capture the data intended.

Conclusion

ePRO plays a crucial role in clinical trials, offering a comprehensive and meaningful way to capture patient experiences. It contributes significantly to the development of treatments that are not only effective but also enhance patients' quality of life because therapeutics have considered the patient’s experience.  

However, it's important to remember that because ePRO is subjective, it relies on human behaviors. In order to optimize its collection, researchers must minimize the friction and constraints that impede reporting. More patient-centric ePRO solutions are thus better ePRO, and designing this patient centricity into our instruments requires the optimization of current practices.  

As we move forward into a digital future, it's clear that ePRO solutions will continue to play an integral part in clinical trials. As such, it's crucial to find the ePRO solution that best fits the unique needs of each study. This will require continued exploration of innovative approaches to improve patient engagement and data capture. With the right approach, and technological support, ePRO can serve as a powerful tool, driving better outcomes in clinical trials and ultimately leading to treatments that better serve patients' needs.

References

Basch, Ethan, et al. “Symptom Monitoring with Patient-Reported Outcomes during Routine Cancer Treatment: A Randomized Controlled Trial.” Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, vol. 34, no. 6, 2016, pp. 557–65, www.ncbi.nlm.nih.gov/pubmed/26644527, https://doi.org/10.1200/JCO.2015.63.0830. Accessed 22 May 2023.

Coons, Stephen Joel, et al. “Recommendations on Evidence Needed to Support Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR EPRO Good Research Practices Task Force Report.” Value in Health, vol. 12, no. 4, June 2009, pp. 419–429, https://doi.org/10.1111/j.1524-4733.2008.00470.x. Accessed 21 May 2023.

Gwaltney, Chad J., et al. “Equivalence of Electronic and Paper-And-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review.” Value in Health, vol. 11, no. 2, Mar. 2008, pp. 322–333, https://doi.org/10.1111/j.1524-4733.2007.00231.x. Accessed 22 May 2023.

Iaquinta, Francesco Salvatore, et al. “Fatigue and Associated Factors in an Immune-Mediated Inflammatory Disease Population: A Cross-Sectional Study.” Journal of Clinical Medicine, vol. 11, no. 9, 1 Jan. 2022, p. 2455, www.mdpi.com/2077-0383/11/9/2455, https://doi.org/10.3390/jcm11092455.Accessed 21 May 2023.

National Institutes of Health. “PROMIS: Clinical Outcomes Assessment.” Nih.gov, National Institutes of Health, 29 Jan. 2019, commonfund.nih.gov/promis/index. Accessed 20 May 2023.

Stone, Arthur A., et al. “Patient Compliance with Paper and Electronic Diaries.” Controlled Clinical Trials, vol. 24, no. 2, Apr. 2003, pp. 182–199, https://doi.org/10.1016/s0197-2456(02)00320-3. Accessed 21 May 2023.

United States, U.S. Department of Health and Human Services, Food and Drug Administration. Multiple Endpoints in Clinical Trials: Guidance for Industry. January 2017, https://www.fda.gov/files/drugs/published/Multiple-Endpoints-in-Clinical-Trials-Guidance-for-Industry.pdf. 

---. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. December 2009, www.fda.gov/media/77832/download. 

Blog

ePRO Clinical Trials: Improving Outcomes Capture

No items found.

Introduction

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences. In recognition of the importance of the patient perspective, the FDA’s industry guidance, entitled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (a.k.a. “PRO Guidance”), emphasizes the use of PRO instruments when measuring concepts best known by the patient or best measured from the patient’s perspective (Patient-Reported Outcome Measures). The “e” in ePRO then facilitates this process by accepting digital, rather than paper-based, reporting.  

The outcomes selected for evaluation in a clinical trial must address the trial objectives; but, they must also be recognized as meaningful to end-users. As per the U.S. Department of Health and Human Services, for an outcome to be meaningful, it should reflect or describe how a person feels, functions, and survives (Multiple Endpoints).

The effort to understand a patient’s experience isn't merely a matter of empathy; it contributes significantly to the holistic evaluation of a treatment's effectiveness. ePRO gives weight to both subjective and objective measures of health, providing a more comprehensive view of how a treatment impacts patients' lives. An important aspect of this holistic evaluation is the consideration of the patients’ quality of life, an assessment that ePRO is instrumental in measuring.

The Many Benefits of ePRO in Clinical Trials

The digital nature of ePRO brings significant advantages, enabling patients to report on events as they occur in their day-to-day lives—from wherever they may be. This means patients can capture both planned and episodic events in real time. Perhaps they are being asked to journal each morning when they take their medication or during other daily events. Or, in other cases, perhaps they need to log more punctual or episodic symptoms like headaches or nausea. These electronic diaries, conveniently logged by patients from their homes or workplaces, provide study coordinators and sponsors with a clearer line of sight into the patient experience than they would otherwise have had access to.  

Therapeutic development is no longer about just saving lives; it's about improving the quality of those saved lives. And, ePRO is the best way to measure improvement or deterioration in quality of life. For example, patients are likely to report symptoms—such as vomiting, diarrhea, or fatigue—earlier and more often than clinicians. Importantly, there is also growing correlation between quality of life and lives saved. In an oncology study, early symptom detection through ePRO showed increased survival rates (Basch et al.).

ePRO also aids in improving patient compliance. The digital platform that deploys the ePRO can also facilitate patient engagement, enhancing protocol compliance and data quality (Gwaltney et al.). This timely and complete data collection translates to more reliable outcomes.

The Challenges of ePRO Implementation

Despite the many advantages associated with the incorporation of patient diaries into clinical trials, the implementation of ePRO comes with challenges. Patients are not always diligent in completing their paper diaries, and this can be true for ePRO as well.

Patients can become demotivated when ePRO clinical trial tasks are too long or complicated, and this confusion can lead to both decreased compliance and reduced reporting accuracy (Coons et al.). Anything that creates friction in the reporting journey, whether it’s more time required or less clarity, will have a negative impact on engagement.  

Patients can often feel frustrated by long and redundant questions in ePRO surveys (Stone et al.). It is natural to feel annoyed when you are asked to answer what appears to be the same or similar question multiple times, even if those questions are embedded in different tasks. Task definitions can also sometimes be perceived as confusing. For example, if a patient is asked to report on his or her symptom of dizziness, nausea, or vertigo without clear definitions of each, he or she may misinterpret the questions and will likely provide contradictory responses.  

Another hurdle to engagement can arise when questions are perceived to be irrelevant to the patient's condition. It is important for patients to understand why certain questions are being asked so that they can feel more invested in answering.  

Finally, and importantly, patients often neglect to report on time, resulting in recall bias. It can be difficult to remember, hours later, how many times one coughed or sneezed. Similarly with pain: The ability to remember how severe pain may have felt is inversely proportional to the time that goes by between the event and its reporting.  

Optimizing ePRO for Better Outcomes in Clinical Trials

To improve ePRO implementation, a more patient-centric approach is required. A successful ePRO strategy should increase patient understanding and motivation while correspondingly reducing burden. Efforts should also be made to encourage and assist patients in real-time reporting whenever possible.  

More Convenient ePRO

There are multiple ways to reduce patient burden, starting with enabling patients to use their own mobile devices for ePRO. BYOD (“bring your own device”) ePRO eliminates the need to carry multiple devices. Instead, patients use an app downloaded onto their smartphones, which naturally integrates more seamlessly into their everyday lives because they are more comfortable on their own devices.

Simplification should be built into the study design from the outset. Clinical research teams should be on the lookout for complex, long, and redundant questions, particularly those that may not be necessary to achieve primary endpoints. The National Institutes of Health's Patient-Reported Outcomes Measurement Information System (PROMIS) measures are a good example of an initiative that has helped to align the industry around simplified scales (National Institutes of Health).  

Another important contributor to patient convenience is through UX design. Good ePRO design should incorporate user-friendly scales and interactive functions to make data capture quick and intuitive. One-tap answers and easy scrolling enable participant responses on the go. For instance, hot flash or pain buttons can be tapped to quickly log symptoms for menopause or migraine trials. When questions are long, they can be broken down into multiple screens to make them more digestible. The easier it is for patients to access and complete tasks, the better their compliance will be.

In some cases, there may also be opportunities to enrich or replace labor-intensive ePRO with devices to capture continuous, more objective, or more detailed information about a symptom or patient activity. For example, rather than asking patients to take their temperature, they can use a device or even a patch that can relay temperature information directly to a data repository.  

More Real-Time ePRO

The closer to real time patients can report an event, the better. BYOD and intuitive UX can get patients part of the way there. Automated notifications and reminders are an additional way to aid patients, providing helpful alerts that nudge them towards on-time compliance. Gamification can also boost compliance for certain types of patients who are motivated by a competitive spirit. Certain trials also benefit from virtual study teams who monitor both compliance and safety, ensuring that any issues are flagged and addressed promptly.

Patients are not always in a place where they have access to the internet or to phone service; so, it is also important that technology enable them to log time-stamped events even when they are not connected.  

ePRO Relevance and Comprehension

Given the importance of patient comprehension for the successful completion of trials, training is a critical component to ensuring clear understanding of study objectives, the protocol, study design, and patient responsibilities. Multi-media, on-demand training modules and virtual site team support can be instrumental in enhancing patient comprehension and engagement. Training materials are also an opportunity to highlight the aspects of the trial that they might find the most personally compelling, as well as the ways in which the trial can contribute positively to their health and quality of life.  

Whenever possible, communication channels and content should be personalized to the specific patient population in the trial. Personalization needs can take many forms and should consider demographics and clinical constraints as well as comfort with technology. Adjustments might include anything from increased font size for older participants to culturally sensitive translations to multiple options for reaching out to site staff, in recognition of the fact that some patients may prefer phone or email to chat or video calls.  

Novel ePRO Approaches

While validated ePRO instruments are beneficial, they may not always suffice. There are times when novel ePRO approaches are necessary. For example, a study was run in 2022 to explore the new ePRO data capture system, PROMIS, and its ability to provide deeper understanding of treatment tolerability for patients with immune-mediated inflammatory disease (Iaquinta et al.).  

There may be opportunities to augment ePRO through image, audio, and video capture technologies to enhance data collection. Using expert raters to score this media can provide more objective outcomes to complement subjective ePRO data. Of course, creating and validating new instruments requires careful consideration, thorough testing, and validation to ensure they accurately and reliably capture the data intended.

Conclusion

ePRO plays a crucial role in clinical trials, offering a comprehensive and meaningful way to capture patient experiences. It contributes significantly to the development of treatments that are not only effective but also enhance patients' quality of life because therapeutics have considered the patient’s experience.  

However, it's important to remember that because ePRO is subjective, it relies on human behaviors. In order to optimize its collection, researchers must minimize the friction and constraints that impede reporting. More patient-centric ePRO solutions are thus better ePRO, and designing this patient centricity into our instruments requires the optimization of current practices.  

As we move forward into a digital future, it's clear that ePRO solutions will continue to play an integral part in clinical trials. As such, it's crucial to find the ePRO solution that best fits the unique needs of each study. This will require continued exploration of innovative approaches to improve patient engagement and data capture. With the right approach, and technological support, ePRO can serve as a powerful tool, driving better outcomes in clinical trials and ultimately leading to treatments that better serve patients' needs.

References

Basch, Ethan, et al. “Symptom Monitoring with Patient-Reported Outcomes during Routine Cancer Treatment: A Randomized Controlled Trial.” Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, vol. 34, no. 6, 2016, pp. 557–65, www.ncbi.nlm.nih.gov/pubmed/26644527, https://doi.org/10.1200/JCO.2015.63.0830. Accessed 22 May 2023.

Coons, Stephen Joel, et al. “Recommendations on Evidence Needed to Support Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR EPRO Good Research Practices Task Force Report.” Value in Health, vol. 12, no. 4, June 2009, pp. 419–429, https://doi.org/10.1111/j.1524-4733.2008.00470.x. Accessed 21 May 2023.

Gwaltney, Chad J., et al. “Equivalence of Electronic and Paper-And-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review.” Value in Health, vol. 11, no. 2, Mar. 2008, pp. 322–333, https://doi.org/10.1111/j.1524-4733.2007.00231.x. Accessed 22 May 2023.

Iaquinta, Francesco Salvatore, et al. “Fatigue and Associated Factors in an Immune-Mediated Inflammatory Disease Population: A Cross-Sectional Study.” Journal of Clinical Medicine, vol. 11, no. 9, 1 Jan. 2022, p. 2455, www.mdpi.com/2077-0383/11/9/2455, https://doi.org/10.3390/jcm11092455.Accessed 21 May 2023.

National Institutes of Health. “PROMIS: Clinical Outcomes Assessment.” Nih.gov, National Institutes of Health, 29 Jan. 2019, commonfund.nih.gov/promis/index. Accessed 20 May 2023.

Stone, Arthur A., et al. “Patient Compliance with Paper and Electronic Diaries.” Controlled Clinical Trials, vol. 24, no. 2, Apr. 2003, pp. 182–199, https://doi.org/10.1016/s0197-2456(02)00320-3. Accessed 21 May 2023.

United States, U.S. Department of Health and Human Services, Food and Drug Administration. Multiple Endpoints in Clinical Trials: Guidance for Industry. January 2017, https://www.fda.gov/files/drugs/published/Multiple-Endpoints-in-Clinical-Trials-Guidance-for-Industry.pdf. 

---. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. December 2009, www.fda.gov/media/77832/download. 

Blog

ePRO Clinical Trials: Improving Outcomes Capture

No items found.

Introduction

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences. In recognition of the importance of the patient perspective, the FDA’s industry guidance, entitled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (a.k.a. “PRO Guidance”), emphasizes the use of PRO instruments when measuring concepts best known by the patient or best measured from the patient’s perspective (Patient-Reported Outcome Measures). The “e” in ePRO then facilitates this process by accepting digital, rather than paper-based, reporting.  

The outcomes selected for evaluation in a clinical trial must address the trial objectives; but, they must also be recognized as meaningful to end-users. As per the U.S. Department of Health and Human Services, for an outcome to be meaningful, it should reflect or describe how a person feels, functions, and survives (Multiple Endpoints).

The effort to understand a patient’s experience isn't merely a matter of empathy; it contributes significantly to the holistic evaluation of a treatment's effectiveness. ePRO gives weight to both subjective and objective measures of health, providing a more comprehensive view of how a treatment impacts patients' lives. An important aspect of this holistic evaluation is the consideration of the patients’ quality of life, an assessment that ePRO is instrumental in measuring.

The Many Benefits of ePRO in Clinical Trials

The digital nature of ePRO brings significant advantages, enabling patients to report on events as they occur in their day-to-day lives—from wherever they may be. This means patients can capture both planned and episodic events in real time. Perhaps they are being asked to journal each morning when they take their medication or during other daily events. Or, in other cases, perhaps they need to log more punctual or episodic symptoms like headaches or nausea. These electronic diaries, conveniently logged by patients from their homes or workplaces, provide study coordinators and sponsors with a clearer line of sight into the patient experience than they would otherwise have had access to.  

Therapeutic development is no longer about just saving lives; it's about improving the quality of those saved lives. And, ePRO is the best way to measure improvement or deterioration in quality of life. For example, patients are likely to report symptoms—such as vomiting, diarrhea, or fatigue—earlier and more often than clinicians. Importantly, there is also growing correlation between quality of life and lives saved. In an oncology study, early symptom detection through ePRO showed increased survival rates (Basch et al.).

ePRO also aids in improving patient compliance. The digital platform that deploys the ePRO can also facilitate patient engagement, enhancing protocol compliance and data quality (Gwaltney et al.). This timely and complete data collection translates to more reliable outcomes.

The Challenges of ePRO Implementation

Despite the many advantages associated with the incorporation of patient diaries into clinical trials, the implementation of ePRO comes with challenges. Patients are not always diligent in completing their paper diaries, and this can be true for ePRO as well.

Patients can become demotivated when ePRO clinical trial tasks are too long or complicated, and this confusion can lead to both decreased compliance and reduced reporting accuracy (Coons et al.). Anything that creates friction in the reporting journey, whether it’s more time required or less clarity, will have a negative impact on engagement.  

Patients can often feel frustrated by long and redundant questions in ePRO surveys (Stone et al.). It is natural to feel annoyed when you are asked to answer what appears to be the same or similar question multiple times, even if those questions are embedded in different tasks. Task definitions can also sometimes be perceived as confusing. For example, if a patient is asked to report on his or her symptom of dizziness, nausea, or vertigo without clear definitions of each, he or she may misinterpret the questions and will likely provide contradictory responses.  

Another hurdle to engagement can arise when questions are perceived to be irrelevant to the patient's condition. It is important for patients to understand why certain questions are being asked so that they can feel more invested in answering.  

Finally, and importantly, patients often neglect to report on time, resulting in recall bias. It can be difficult to remember, hours later, how many times one coughed or sneezed. Similarly with pain: The ability to remember how severe pain may have felt is inversely proportional to the time that goes by between the event and its reporting.  

Optimizing ePRO for Better Outcomes in Clinical Trials

To improve ePRO implementation, a more patient-centric approach is required. A successful ePRO strategy should increase patient understanding and motivation while correspondingly reducing burden. Efforts should also be made to encourage and assist patients in real-time reporting whenever possible.  

More Convenient ePRO

There are multiple ways to reduce patient burden, starting with enabling patients to use their own mobile devices for ePRO. BYOD (“bring your own device”) ePRO eliminates the need to carry multiple devices. Instead, patients use an app downloaded onto their smartphones, which naturally integrates more seamlessly into their everyday lives because they are more comfortable on their own devices.

Simplification should be built into the study design from the outset. Clinical research teams should be on the lookout for complex, long, and redundant questions, particularly those that may not be necessary to achieve primary endpoints. The National Institutes of Health's Patient-Reported Outcomes Measurement Information System (PROMIS) measures are a good example of an initiative that has helped to align the industry around simplified scales (National Institutes of Health).  

Another important contributor to patient convenience is through UX design. Good ePRO design should incorporate user-friendly scales and interactive functions to make data capture quick and intuitive. One-tap answers and easy scrolling enable participant responses on the go. For instance, hot flash or pain buttons can be tapped to quickly log symptoms for menopause or migraine trials. When questions are long, they can be broken down into multiple screens to make them more digestible. The easier it is for patients to access and complete tasks, the better their compliance will be.

In some cases, there may also be opportunities to enrich or replace labor-intensive ePRO with devices to capture continuous, more objective, or more detailed information about a symptom or patient activity. For example, rather than asking patients to take their temperature, they can use a device or even a patch that can relay temperature information directly to a data repository.  

More Real-Time ePRO

The closer to real time patients can report an event, the better. BYOD and intuitive UX can get patients part of the way there. Automated notifications and reminders are an additional way to aid patients, providing helpful alerts that nudge them towards on-time compliance. Gamification can also boost compliance for certain types of patients who are motivated by a competitive spirit. Certain trials also benefit from virtual study teams who monitor both compliance and safety, ensuring that any issues are flagged and addressed promptly.

Patients are not always in a place where they have access to the internet or to phone service; so, it is also important that technology enable them to log time-stamped events even when they are not connected.  

ePRO Relevance and Comprehension

Given the importance of patient comprehension for the successful completion of trials, training is a critical component to ensuring clear understanding of study objectives, the protocol, study design, and patient responsibilities. Multi-media, on-demand training modules and virtual site team support can be instrumental in enhancing patient comprehension and engagement. Training materials are also an opportunity to highlight the aspects of the trial that they might find the most personally compelling, as well as the ways in which the trial can contribute positively to their health and quality of life.  

Whenever possible, communication channels and content should be personalized to the specific patient population in the trial. Personalization needs can take many forms and should consider demographics and clinical constraints as well as comfort with technology. Adjustments might include anything from increased font size for older participants to culturally sensitive translations to multiple options for reaching out to site staff, in recognition of the fact that some patients may prefer phone or email to chat or video calls.  

Novel ePRO Approaches

While validated ePRO instruments are beneficial, they may not always suffice. There are times when novel ePRO approaches are necessary. For example, a study was run in 2022 to explore the new ePRO data capture system, PROMIS, and its ability to provide deeper understanding of treatment tolerability for patients with immune-mediated inflammatory disease (Iaquinta et al.).  

There may be opportunities to augment ePRO through image, audio, and video capture technologies to enhance data collection. Using expert raters to score this media can provide more objective outcomes to complement subjective ePRO data. Of course, creating and validating new instruments requires careful consideration, thorough testing, and validation to ensure they accurately and reliably capture the data intended.

Conclusion

ePRO plays a crucial role in clinical trials, offering a comprehensive and meaningful way to capture patient experiences. It contributes significantly to the development of treatments that are not only effective but also enhance patients' quality of life because therapeutics have considered the patient’s experience.  

However, it's important to remember that because ePRO is subjective, it relies on human behaviors. In order to optimize its collection, researchers must minimize the friction and constraints that impede reporting. More patient-centric ePRO solutions are thus better ePRO, and designing this patient centricity into our instruments requires the optimization of current practices.  

As we move forward into a digital future, it's clear that ePRO solutions will continue to play an integral part in clinical trials. As such, it's crucial to find the ePRO solution that best fits the unique needs of each study. This will require continued exploration of innovative approaches to improve patient engagement and data capture. With the right approach, and technological support, ePRO can serve as a powerful tool, driving better outcomes in clinical trials and ultimately leading to treatments that better serve patients' needs.

References

Basch, Ethan, et al. “Symptom Monitoring with Patient-Reported Outcomes during Routine Cancer Treatment: A Randomized Controlled Trial.” Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, vol. 34, no. 6, 2016, pp. 557–65, www.ncbi.nlm.nih.gov/pubmed/26644527, https://doi.org/10.1200/JCO.2015.63.0830. Accessed 22 May 2023.

Coons, Stephen Joel, et al. “Recommendations on Evidence Needed to Support Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR EPRO Good Research Practices Task Force Report.” Value in Health, vol. 12, no. 4, June 2009, pp. 419–429, https://doi.org/10.1111/j.1524-4733.2008.00470.x. Accessed 21 May 2023.

Gwaltney, Chad J., et al. “Equivalence of Electronic and Paper-And-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review.” Value in Health, vol. 11, no. 2, Mar. 2008, pp. 322–333, https://doi.org/10.1111/j.1524-4733.2007.00231.x. Accessed 22 May 2023.

Iaquinta, Francesco Salvatore, et al. “Fatigue and Associated Factors in an Immune-Mediated Inflammatory Disease Population: A Cross-Sectional Study.” Journal of Clinical Medicine, vol. 11, no. 9, 1 Jan. 2022, p. 2455, www.mdpi.com/2077-0383/11/9/2455, https://doi.org/10.3390/jcm11092455.Accessed 21 May 2023.

National Institutes of Health. “PROMIS: Clinical Outcomes Assessment.” Nih.gov, National Institutes of Health, 29 Jan. 2019, commonfund.nih.gov/promis/index. Accessed 20 May 2023.

Stone, Arthur A., et al. “Patient Compliance with Paper and Electronic Diaries.” Controlled Clinical Trials, vol. 24, no. 2, Apr. 2003, pp. 182–199, https://doi.org/10.1016/s0197-2456(02)00320-3. Accessed 21 May 2023.

United States, U.S. Department of Health and Human Services, Food and Drug Administration. Multiple Endpoints in Clinical Trials: Guidance for Industry. January 2017, https://www.fda.gov/files/drugs/published/Multiple-Endpoints-in-Clinical-Trials-Guidance-for-Industry.pdf. 

---. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. December 2009, www.fda.gov/media/77832/download. 

Blog

ePRO Clinical Trials: Improving Outcomes Capture

No items found.

Introduction

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences. In recognition of the importance of the patient perspective, the FDA’s industry guidance, entitled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (a.k.a. “PRO Guidance”), emphasizes the use of PRO instruments when measuring concepts best known by the patient or best measured from the patient’s perspective (Patient-Reported Outcome Measures). The “e” in ePRO then facilitates this process by accepting digital, rather than paper-based, reporting.  

The outcomes selected for evaluation in a clinical trial must address the trial objectives; but, they must also be recognized as meaningful to end-users. As per the U.S. Department of Health and Human Services, for an outcome to be meaningful, it should reflect or describe how a person feels, functions, and survives (Multiple Endpoints).

The effort to understand a patient’s experience isn't merely a matter of empathy; it contributes significantly to the holistic evaluation of a treatment's effectiveness. ePRO gives weight to both subjective and objective measures of health, providing a more comprehensive view of how a treatment impacts patients' lives. An important aspect of this holistic evaluation is the consideration of the patients’ quality of life, an assessment that ePRO is instrumental in measuring.

The Many Benefits of ePRO in Clinical Trials

The digital nature of ePRO brings significant advantages, enabling patients to report on events as they occur in their day-to-day lives—from wherever they may be. This means patients can capture both planned and episodic events in real time. Perhaps they are being asked to journal each morning when they take their medication or during other daily events. Or, in other cases, perhaps they need to log more punctual or episodic symptoms like headaches or nausea. These electronic diaries, conveniently logged by patients from their homes or workplaces, provide study coordinators and sponsors with a clearer line of sight into the patient experience than they would otherwise have had access to.  

Therapeutic development is no longer about just saving lives; it's about improving the quality of those saved lives. And, ePRO is the best way to measure improvement or deterioration in quality of life. For example, patients are likely to report symptoms—such as vomiting, diarrhea, or fatigue—earlier and more often than clinicians. Importantly, there is also growing correlation between quality of life and lives saved. In an oncology study, early symptom detection through ePRO showed increased survival rates (Basch et al.).

ePRO also aids in improving patient compliance. The digital platform that deploys the ePRO can also facilitate patient engagement, enhancing protocol compliance and data quality (Gwaltney et al.). This timely and complete data collection translates to more reliable outcomes.

The Challenges of ePRO Implementation

Despite the many advantages associated with the incorporation of patient diaries into clinical trials, the implementation of ePRO comes with challenges. Patients are not always diligent in completing their paper diaries, and this can be true for ePRO as well.

Patients can become demotivated when ePRO clinical trial tasks are too long or complicated, and this confusion can lead to both decreased compliance and reduced reporting accuracy (Coons et al.). Anything that creates friction in the reporting journey, whether it’s more time required or less clarity, will have a negative impact on engagement.  

Patients can often feel frustrated by long and redundant questions in ePRO surveys (Stone et al.). It is natural to feel annoyed when you are asked to answer what appears to be the same or similar question multiple times, even if those questions are embedded in different tasks. Task definitions can also sometimes be perceived as confusing. For example, if a patient is asked to report on his or her symptom of dizziness, nausea, or vertigo without clear definitions of each, he or she may misinterpret the questions and will likely provide contradictory responses.  

Another hurdle to engagement can arise when questions are perceived to be irrelevant to the patient's condition. It is important for patients to understand why certain questions are being asked so that they can feel more invested in answering.  

Finally, and importantly, patients often neglect to report on time, resulting in recall bias. It can be difficult to remember, hours later, how many times one coughed or sneezed. Similarly with pain: The ability to remember how severe pain may have felt is inversely proportional to the time that goes by between the event and its reporting.  

Optimizing ePRO for Better Outcomes in Clinical Trials

To improve ePRO implementation, a more patient-centric approach is required. A successful ePRO strategy should increase patient understanding and motivation while correspondingly reducing burden. Efforts should also be made to encourage and assist patients in real-time reporting whenever possible.  

More Convenient ePRO

There are multiple ways to reduce patient burden, starting with enabling patients to use their own mobile devices for ePRO. BYOD (“bring your own device”) ePRO eliminates the need to carry multiple devices. Instead, patients use an app downloaded onto their smartphones, which naturally integrates more seamlessly into their everyday lives because they are more comfortable on their own devices.

Simplification should be built into the study design from the outset. Clinical research teams should be on the lookout for complex, long, and redundant questions, particularly those that may not be necessary to achieve primary endpoints. The National Institutes of Health's Patient-Reported Outcomes Measurement Information System (PROMIS) measures are a good example of an initiative that has helped to align the industry around simplified scales (National Institutes of Health).  

Another important contributor to patient convenience is through UX design. Good ePRO design should incorporate user-friendly scales and interactive functions to make data capture quick and intuitive. One-tap answers and easy scrolling enable participant responses on the go. For instance, hot flash or pain buttons can be tapped to quickly log symptoms for menopause or migraine trials. When questions are long, they can be broken down into multiple screens to make them more digestible. The easier it is for patients to access and complete tasks, the better their compliance will be.

In some cases, there may also be opportunities to enrich or replace labor-intensive ePRO with devices to capture continuous, more objective, or more detailed information about a symptom or patient activity. For example, rather than asking patients to take their temperature, they can use a device or even a patch that can relay temperature information directly to a data repository.  

More Real-Time ePRO

The closer to real time patients can report an event, the better. BYOD and intuitive UX can get patients part of the way there. Automated notifications and reminders are an additional way to aid patients, providing helpful alerts that nudge them towards on-time compliance. Gamification can also boost compliance for certain types of patients who are motivated by a competitive spirit. Certain trials also benefit from virtual study teams who monitor both compliance and safety, ensuring that any issues are flagged and addressed promptly.

Patients are not always in a place where they have access to the internet or to phone service; so, it is also important that technology enable them to log time-stamped events even when they are not connected.  

ePRO Relevance and Comprehension

Given the importance of patient comprehension for the successful completion of trials, training is a critical component to ensuring clear understanding of study objectives, the protocol, study design, and patient responsibilities. Multi-media, on-demand training modules and virtual site team support can be instrumental in enhancing patient comprehension and engagement. Training materials are also an opportunity to highlight the aspects of the trial that they might find the most personally compelling, as well as the ways in which the trial can contribute positively to their health and quality of life.  

Whenever possible, communication channels and content should be personalized to the specific patient population in the trial. Personalization needs can take many forms and should consider demographics and clinical constraints as well as comfort with technology. Adjustments might include anything from increased font size for older participants to culturally sensitive translations to multiple options for reaching out to site staff, in recognition of the fact that some patients may prefer phone or email to chat or video calls.  

Novel ePRO Approaches

While validated ePRO instruments are beneficial, they may not always suffice. There are times when novel ePRO approaches are necessary. For example, a study was run in 2022 to explore the new ePRO data capture system, PROMIS, and its ability to provide deeper understanding of treatment tolerability for patients with immune-mediated inflammatory disease (Iaquinta et al.).  

There may be opportunities to augment ePRO through image, audio, and video capture technologies to enhance data collection. Using expert raters to score this media can provide more objective outcomes to complement subjective ePRO data. Of course, creating and validating new instruments requires careful consideration, thorough testing, and validation to ensure they accurately and reliably capture the data intended.

Conclusion

ePRO plays a crucial role in clinical trials, offering a comprehensive and meaningful way to capture patient experiences. It contributes significantly to the development of treatments that are not only effective but also enhance patients' quality of life because therapeutics have considered the patient’s experience.  

However, it's important to remember that because ePRO is subjective, it relies on human behaviors. In order to optimize its collection, researchers must minimize the friction and constraints that impede reporting. More patient-centric ePRO solutions are thus better ePRO, and designing this patient centricity into our instruments requires the optimization of current practices.  

As we move forward into a digital future, it's clear that ePRO solutions will continue to play an integral part in clinical trials. As such, it's crucial to find the ePRO solution that best fits the unique needs of each study. This will require continued exploration of innovative approaches to improve patient engagement and data capture. With the right approach, and technological support, ePRO can serve as a powerful tool, driving better outcomes in clinical trials and ultimately leading to treatments that better serve patients' needs.

References

Basch, Ethan, et al. “Symptom Monitoring with Patient-Reported Outcomes during Routine Cancer Treatment: A Randomized Controlled Trial.” Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, vol. 34, no. 6, 2016, pp. 557–65, www.ncbi.nlm.nih.gov/pubmed/26644527, https://doi.org/10.1200/JCO.2015.63.0830. Accessed 22 May 2023.

Coons, Stephen Joel, et al. “Recommendations on Evidence Needed to Support Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR EPRO Good Research Practices Task Force Report.” Value in Health, vol. 12, no. 4, June 2009, pp. 419–429, https://doi.org/10.1111/j.1524-4733.2008.00470.x. Accessed 21 May 2023.

Gwaltney, Chad J., et al. “Equivalence of Electronic and Paper-And-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review.” Value in Health, vol. 11, no. 2, Mar. 2008, pp. 322–333, https://doi.org/10.1111/j.1524-4733.2007.00231.x. Accessed 22 May 2023.

Iaquinta, Francesco Salvatore, et al. “Fatigue and Associated Factors in an Immune-Mediated Inflammatory Disease Population: A Cross-Sectional Study.” Journal of Clinical Medicine, vol. 11, no. 9, 1 Jan. 2022, p. 2455, www.mdpi.com/2077-0383/11/9/2455, https://doi.org/10.3390/jcm11092455.Accessed 21 May 2023.

National Institutes of Health. “PROMIS: Clinical Outcomes Assessment.” Nih.gov, National Institutes of Health, 29 Jan. 2019, commonfund.nih.gov/promis/index. Accessed 20 May 2023.

Stone, Arthur A., et al. “Patient Compliance with Paper and Electronic Diaries.” Controlled Clinical Trials, vol. 24, no. 2, Apr. 2003, pp. 182–199, https://doi.org/10.1016/s0197-2456(02)00320-3. Accessed 21 May 2023.

United States, U.S. Department of Health and Human Services, Food and Drug Administration. Multiple Endpoints in Clinical Trials: Guidance for Industry. January 2017, https://www.fda.gov/files/drugs/published/Multiple-Endpoints-in-Clinical-Trials-Guidance-for-Industry.pdf. 

---. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. December 2009, www.fda.gov/media/77832/download. 

Blog

ePRO Clinical Trials: Improving Outcomes Capture

No items found.

Introduction

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences. In recognition of the importance of the patient perspective, the FDA’s industry guidance, entitled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (a.k.a. “PRO Guidance”), emphasizes the use of PRO instruments when measuring concepts best known by the patient or best measured from the patient’s perspective (Patient-Reported Outcome Measures). The “e” in ePRO then facilitates this process by accepting digital, rather than paper-based, reporting.  

The outcomes selected for evaluation in a clinical trial must address the trial objectives; but, they must also be recognized as meaningful to end-users. As per the U.S. Department of Health and Human Services, for an outcome to be meaningful, it should reflect or describe how a person feels, functions, and survives (Multiple Endpoints).

The effort to understand a patient’s experience isn't merely a matter of empathy; it contributes significantly to the holistic evaluation of a treatment's effectiveness. ePRO gives weight to both subjective and objective measures of health, providing a more comprehensive view of how a treatment impacts patients' lives. An important aspect of this holistic evaluation is the consideration of the patients’ quality of life, an assessment that ePRO is instrumental in measuring.

The Many Benefits of ePRO in Clinical Trials

The digital nature of ePRO brings significant advantages, enabling patients to report on events as they occur in their day-to-day lives—from wherever they may be. This means patients can capture both planned and episodic events in real time. Perhaps they are being asked to journal each morning when they take their medication or during other daily events. Or, in other cases, perhaps they need to log more punctual or episodic symptoms like headaches or nausea. These electronic diaries, conveniently logged by patients from their homes or workplaces, provide study coordinators and sponsors with a clearer line of sight into the patient experience than they would otherwise have had access to.  

Therapeutic development is no longer about just saving lives; it's about improving the quality of those saved lives. And, ePRO is the best way to measure improvement or deterioration in quality of life. For example, patients are likely to report symptoms—such as vomiting, diarrhea, or fatigue—earlier and more often than clinicians. Importantly, there is also growing correlation between quality of life and lives saved. In an oncology study, early symptom detection through ePRO showed increased survival rates (Basch et al.).

ePRO also aids in improving patient compliance. The digital platform that deploys the ePRO can also facilitate patient engagement, enhancing protocol compliance and data quality (Gwaltney et al.). This timely and complete data collection translates to more reliable outcomes.

The Challenges of ePRO Implementation

Despite the many advantages associated with the incorporation of patient diaries into clinical trials, the implementation of ePRO comes with challenges. Patients are not always diligent in completing their paper diaries, and this can be true for ePRO as well.

Patients can become demotivated when ePRO clinical trial tasks are too long or complicated, and this confusion can lead to both decreased compliance and reduced reporting accuracy (Coons et al.). Anything that creates friction in the reporting journey, whether it’s more time required or less clarity, will have a negative impact on engagement.  

Patients can often feel frustrated by long and redundant questions in ePRO surveys (Stone et al.). It is natural to feel annoyed when you are asked to answer what appears to be the same or similar question multiple times, even if those questions are embedded in different tasks. Task definitions can also sometimes be perceived as confusing. For example, if a patient is asked to report on his or her symptom of dizziness, nausea, or vertigo without clear definitions of each, he or she may misinterpret the questions and will likely provide contradictory responses.  

Another hurdle to engagement can arise when questions are perceived to be irrelevant to the patient's condition. It is important for patients to understand why certain questions are being asked so that they can feel more invested in answering.  

Finally, and importantly, patients often neglect to report on time, resulting in recall bias. It can be difficult to remember, hours later, how many times one coughed or sneezed. Similarly with pain: The ability to remember how severe pain may have felt is inversely proportional to the time that goes by between the event and its reporting.  

Optimizing ePRO for Better Outcomes in Clinical Trials

To improve ePRO implementation, a more patient-centric approach is required. A successful ePRO strategy should increase patient understanding and motivation while correspondingly reducing burden. Efforts should also be made to encourage and assist patients in real-time reporting whenever possible.  

More Convenient ePRO

There are multiple ways to reduce patient burden, starting with enabling patients to use their own mobile devices for ePRO. BYOD (“bring your own device”) ePRO eliminates the need to carry multiple devices. Instead, patients use an app downloaded onto their smartphones, which naturally integrates more seamlessly into their everyday lives because they are more comfortable on their own devices.

Simplification should be built into the study design from the outset. Clinical research teams should be on the lookout for complex, long, and redundant questions, particularly those that may not be necessary to achieve primary endpoints. The National Institutes of Health's Patient-Reported Outcomes Measurement Information System (PROMIS) measures are a good example of an initiative that has helped to align the industry around simplified scales (National Institutes of Health).  

Another important contributor to patient convenience is through UX design. Good ePRO design should incorporate user-friendly scales and interactive functions to make data capture quick and intuitive. One-tap answers and easy scrolling enable participant responses on the go. For instance, hot flash or pain buttons can be tapped to quickly log symptoms for menopause or migraine trials. When questions are long, they can be broken down into multiple screens to make them more digestible. The easier it is for patients to access and complete tasks, the better their compliance will be.

In some cases, there may also be opportunities to enrich or replace labor-intensive ePRO with devices to capture continuous, more objective, or more detailed information about a symptom or patient activity. For example, rather than asking patients to take their temperature, they can use a device or even a patch that can relay temperature information directly to a data repository.  

More Real-Time ePRO

The closer to real time patients can report an event, the better. BYOD and intuitive UX can get patients part of the way there. Automated notifications and reminders are an additional way to aid patients, providing helpful alerts that nudge them towards on-time compliance. Gamification can also boost compliance for certain types of patients who are motivated by a competitive spirit. Certain trials also benefit from virtual study teams who monitor both compliance and safety, ensuring that any issues are flagged and addressed promptly.

Patients are not always in a place where they have access to the internet or to phone service; so, it is also important that technology enable them to log time-stamped events even when they are not connected.  

ePRO Relevance and Comprehension

Given the importance of patient comprehension for the successful completion of trials, training is a critical component to ensuring clear understanding of study objectives, the protocol, study design, and patient responsibilities. Multi-media, on-demand training modules and virtual site team support can be instrumental in enhancing patient comprehension and engagement. Training materials are also an opportunity to highlight the aspects of the trial that they might find the most personally compelling, as well as the ways in which the trial can contribute positively to their health and quality of life.  

Whenever possible, communication channels and content should be personalized to the specific patient population in the trial. Personalization needs can take many forms and should consider demographics and clinical constraints as well as comfort with technology. Adjustments might include anything from increased font size for older participants to culturally sensitive translations to multiple options for reaching out to site staff, in recognition of the fact that some patients may prefer phone or email to chat or video calls.  

Novel ePRO Approaches

While validated ePRO instruments are beneficial, they may not always suffice. There are times when novel ePRO approaches are necessary. For example, a study was run in 2022 to explore the new ePRO data capture system, PROMIS, and its ability to provide deeper understanding of treatment tolerability for patients with immune-mediated inflammatory disease (Iaquinta et al.).  

There may be opportunities to augment ePRO through image, audio, and video capture technologies to enhance data collection. Using expert raters to score this media can provide more objective outcomes to complement subjective ePRO data. Of course, creating and validating new instruments requires careful consideration, thorough testing, and validation to ensure they accurately and reliably capture the data intended.

Conclusion

ePRO plays a crucial role in clinical trials, offering a comprehensive and meaningful way to capture patient experiences. It contributes significantly to the development of treatments that are not only effective but also enhance patients' quality of life because therapeutics have considered the patient’s experience.  

However, it's important to remember that because ePRO is subjective, it relies on human behaviors. In order to optimize its collection, researchers must minimize the friction and constraints that impede reporting. More patient-centric ePRO solutions are thus better ePRO, and designing this patient centricity into our instruments requires the optimization of current practices.  

As we move forward into a digital future, it's clear that ePRO solutions will continue to play an integral part in clinical trials. As such, it's crucial to find the ePRO solution that best fits the unique needs of each study. This will require continued exploration of innovative approaches to improve patient engagement and data capture. With the right approach, and technological support, ePRO can serve as a powerful tool, driving better outcomes in clinical trials and ultimately leading to treatments that better serve patients' needs.

References

Basch, Ethan, et al. “Symptom Monitoring with Patient-Reported Outcomes during Routine Cancer Treatment: A Randomized Controlled Trial.” Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, vol. 34, no. 6, 2016, pp. 557–65, www.ncbi.nlm.nih.gov/pubmed/26644527, https://doi.org/10.1200/JCO.2015.63.0830. Accessed 22 May 2023.

Coons, Stephen Joel, et al. “Recommendations on Evidence Needed to Support Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR EPRO Good Research Practices Task Force Report.” Value in Health, vol. 12, no. 4, June 2009, pp. 419–429, https://doi.org/10.1111/j.1524-4733.2008.00470.x. Accessed 21 May 2023.

Gwaltney, Chad J., et al. “Equivalence of Electronic and Paper-And-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review.” Value in Health, vol. 11, no. 2, Mar. 2008, pp. 322–333, https://doi.org/10.1111/j.1524-4733.2007.00231.x. Accessed 22 May 2023.

Iaquinta, Francesco Salvatore, et al. “Fatigue and Associated Factors in an Immune-Mediated Inflammatory Disease Population: A Cross-Sectional Study.” Journal of Clinical Medicine, vol. 11, no. 9, 1 Jan. 2022, p. 2455, www.mdpi.com/2077-0383/11/9/2455, https://doi.org/10.3390/jcm11092455.Accessed 21 May 2023.

National Institutes of Health. “PROMIS: Clinical Outcomes Assessment.” Nih.gov, National Institutes of Health, 29 Jan. 2019, commonfund.nih.gov/promis/index. Accessed 20 May 2023.

Stone, Arthur A., et al. “Patient Compliance with Paper and Electronic Diaries.” Controlled Clinical Trials, vol. 24, no. 2, Apr. 2003, pp. 182–199, https://doi.org/10.1016/s0197-2456(02)00320-3. Accessed 21 May 2023.

United States, U.S. Department of Health and Human Services, Food and Drug Administration. Multiple Endpoints in Clinical Trials: Guidance for Industry. January 2017, https://www.fda.gov/files/drugs/published/Multiple-Endpoints-in-Clinical-Trials-Guidance-for-Industry.pdf. 

---. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. December 2009, www.fda.gov/media/77832/download. 

Blog

ePRO Clinical Trials: Improving Outcomes Capture

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences.

No items found.

Introduction

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences. In recognition of the importance of the patient perspective, the FDA’s industry guidance, entitled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (a.k.a. “PRO Guidance”), emphasizes the use of PRO instruments when measuring concepts best known by the patient or best measured from the patient’s perspective (Patient-Reported Outcome Measures). The “e” in ePRO then facilitates this process by accepting digital, rather than paper-based, reporting.  

The outcomes selected for evaluation in a clinical trial must address the trial objectives; but, they must also be recognized as meaningful to end-users. As per the U.S. Department of Health and Human Services, for an outcome to be meaningful, it should reflect or describe how a person feels, functions, and survives (Multiple Endpoints).

The effort to understand a patient’s experience isn't merely a matter of empathy; it contributes significantly to the holistic evaluation of a treatment's effectiveness. ePRO gives weight to both subjective and objective measures of health, providing a more comprehensive view of how a treatment impacts patients' lives. An important aspect of this holistic evaluation is the consideration of the patients’ quality of life, an assessment that ePRO is instrumental in measuring.

The Many Benefits of ePRO in Clinical Trials

The digital nature of ePRO brings significant advantages, enabling patients to report on events as they occur in their day-to-day lives—from wherever they may be. This means patients can capture both planned and episodic events in real time. Perhaps they are being asked to journal each morning when they take their medication or during other daily events. Or, in other cases, perhaps they need to log more punctual or episodic symptoms like headaches or nausea. These electronic diaries, conveniently logged by patients from their homes or workplaces, provide study coordinators and sponsors with a clearer line of sight into the patient experience than they would otherwise have had access to.  

Therapeutic development is no longer about just saving lives; it's about improving the quality of those saved lives. And, ePRO is the best way to measure improvement or deterioration in quality of life. For example, patients are likely to report symptoms—such as vomiting, diarrhea, or fatigue—earlier and more often than clinicians. Importantly, there is also growing correlation between quality of life and lives saved. In an oncology study, early symptom detection through ePRO showed increased survival rates (Basch et al.).

ePRO also aids in improving patient compliance. The digital platform that deploys the ePRO can also facilitate patient engagement, enhancing protocol compliance and data quality (Gwaltney et al.). This timely and complete data collection translates to more reliable outcomes.

The Challenges of ePRO Implementation

Despite the many advantages associated with the incorporation of patient diaries into clinical trials, the implementation of ePRO comes with challenges. Patients are not always diligent in completing their paper diaries, and this can be true for ePRO as well.

Patients can become demotivated when ePRO clinical trial tasks are too long or complicated, and this confusion can lead to both decreased compliance and reduced reporting accuracy (Coons et al.). Anything that creates friction in the reporting journey, whether it’s more time required or less clarity, will have a negative impact on engagement.  

Patients can often feel frustrated by long and redundant questions in ePRO surveys (Stone et al.). It is natural to feel annoyed when you are asked to answer what appears to be the same or similar question multiple times, even if those questions are embedded in different tasks. Task definitions can also sometimes be perceived as confusing. For example, if a patient is asked to report on his or her symptom of dizziness, nausea, or vertigo without clear definitions of each, he or she may misinterpret the questions and will likely provide contradictory responses.  

Another hurdle to engagement can arise when questions are perceived to be irrelevant to the patient's condition. It is important for patients to understand why certain questions are being asked so that they can feel more invested in answering.  

Finally, and importantly, patients often neglect to report on time, resulting in recall bias. It can be difficult to remember, hours later, how many times one coughed or sneezed. Similarly with pain: The ability to remember how severe pain may have felt is inversely proportional to the time that goes by between the event and its reporting.  

Optimizing ePRO for Better Outcomes in Clinical Trials

To improve ePRO implementation, a more patient-centric approach is required. A successful ePRO strategy should increase patient understanding and motivation while correspondingly reducing burden. Efforts should also be made to encourage and assist patients in real-time reporting whenever possible.  

More Convenient ePRO

There are multiple ways to reduce patient burden, starting with enabling patients to use their own mobile devices for ePRO. BYOD (“bring your own device”) ePRO eliminates the need to carry multiple devices. Instead, patients use an app downloaded onto their smartphones, which naturally integrates more seamlessly into their everyday lives because they are more comfortable on their own devices.

Simplification should be built into the study design from the outset. Clinical research teams should be on the lookout for complex, long, and redundant questions, particularly those that may not be necessary to achieve primary endpoints. The National Institutes of Health's Patient-Reported Outcomes Measurement Information System (PROMIS) measures are a good example of an initiative that has helped to align the industry around simplified scales (National Institutes of Health).  

Another important contributor to patient convenience is through UX design. Good ePRO design should incorporate user-friendly scales and interactive functions to make data capture quick and intuitive. One-tap answers and easy scrolling enable participant responses on the go. For instance, hot flash or pain buttons can be tapped to quickly log symptoms for menopause or migraine trials. When questions are long, they can be broken down into multiple screens to make them more digestible. The easier it is for patients to access and complete tasks, the better their compliance will be.

In some cases, there may also be opportunities to enrich or replace labor-intensive ePRO with devices to capture continuous, more objective, or more detailed information about a symptom or patient activity. For example, rather than asking patients to take their temperature, they can use a device or even a patch that can relay temperature information directly to a data repository.  

More Real-Time ePRO

The closer to real time patients can report an event, the better. BYOD and intuitive UX can get patients part of the way there. Automated notifications and reminders are an additional way to aid patients, providing helpful alerts that nudge them towards on-time compliance. Gamification can also boost compliance for certain types of patients who are motivated by a competitive spirit. Certain trials also benefit from virtual study teams who monitor both compliance and safety, ensuring that any issues are flagged and addressed promptly.

Patients are not always in a place where they have access to the internet or to phone service; so, it is also important that technology enable them to log time-stamped events even when they are not connected.  

ePRO Relevance and Comprehension

Given the importance of patient comprehension for the successful completion of trials, training is a critical component to ensuring clear understanding of study objectives, the protocol, study design, and patient responsibilities. Multi-media, on-demand training modules and virtual site team support can be instrumental in enhancing patient comprehension and engagement. Training materials are also an opportunity to highlight the aspects of the trial that they might find the most personally compelling, as well as the ways in which the trial can contribute positively to their health and quality of life.  

Whenever possible, communication channels and content should be personalized to the specific patient population in the trial. Personalization needs can take many forms and should consider demographics and clinical constraints as well as comfort with technology. Adjustments might include anything from increased font size for older participants to culturally sensitive translations to multiple options for reaching out to site staff, in recognition of the fact that some patients may prefer phone or email to chat or video calls.  

Novel ePRO Approaches

While validated ePRO instruments are beneficial, they may not always suffice. There are times when novel ePRO approaches are necessary. For example, a study was run in 2022 to explore the new ePRO data capture system, PROMIS, and its ability to provide deeper understanding of treatment tolerability for patients with immune-mediated inflammatory disease (Iaquinta et al.).  

There may be opportunities to augment ePRO through image, audio, and video capture technologies to enhance data collection. Using expert raters to score this media can provide more objective outcomes to complement subjective ePRO data. Of course, creating and validating new instruments requires careful consideration, thorough testing, and validation to ensure they accurately and reliably capture the data intended.

Conclusion

ePRO plays a crucial role in clinical trials, offering a comprehensive and meaningful way to capture patient experiences. It contributes significantly to the development of treatments that are not only effective but also enhance patients' quality of life because therapeutics have considered the patient’s experience.  

However, it's important to remember that because ePRO is subjective, it relies on human behaviors. In order to optimize its collection, researchers must minimize the friction and constraints that impede reporting. More patient-centric ePRO solutions are thus better ePRO, and designing this patient centricity into our instruments requires the optimization of current practices.  

As we move forward into a digital future, it's clear that ePRO solutions will continue to play an integral part in clinical trials. As such, it's crucial to find the ePRO solution that best fits the unique needs of each study. This will require continued exploration of innovative approaches to improve patient engagement and data capture. With the right approach, and technological support, ePRO can serve as a powerful tool, driving better outcomes in clinical trials and ultimately leading to treatments that better serve patients' needs.

References

Basch, Ethan, et al. “Symptom Monitoring with Patient-Reported Outcomes during Routine Cancer Treatment: A Randomized Controlled Trial.” Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, vol. 34, no. 6, 2016, pp. 557–65, www.ncbi.nlm.nih.gov/pubmed/26644527, https://doi.org/10.1200/JCO.2015.63.0830. Accessed 22 May 2023.

Coons, Stephen Joel, et al. “Recommendations on Evidence Needed to Support Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR EPRO Good Research Practices Task Force Report.” Value in Health, vol. 12, no. 4, June 2009, pp. 419–429, https://doi.org/10.1111/j.1524-4733.2008.00470.x. Accessed 21 May 2023.

Gwaltney, Chad J., et al. “Equivalence of Electronic and Paper-And-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review.” Value in Health, vol. 11, no. 2, Mar. 2008, pp. 322–333, https://doi.org/10.1111/j.1524-4733.2007.00231.x. Accessed 22 May 2023.

Iaquinta, Francesco Salvatore, et al. “Fatigue and Associated Factors in an Immune-Mediated Inflammatory Disease Population: A Cross-Sectional Study.” Journal of Clinical Medicine, vol. 11, no. 9, 1 Jan. 2022, p. 2455, www.mdpi.com/2077-0383/11/9/2455, https://doi.org/10.3390/jcm11092455.Accessed 21 May 2023.

National Institutes of Health. “PROMIS: Clinical Outcomes Assessment.” Nih.gov, National Institutes of Health, 29 Jan. 2019, commonfund.nih.gov/promis/index. Accessed 20 May 2023.

Stone, Arthur A., et al. “Patient Compliance with Paper and Electronic Diaries.” Controlled Clinical Trials, vol. 24, no. 2, Apr. 2003, pp. 182–199, https://doi.org/10.1016/s0197-2456(02)00320-3. Accessed 21 May 2023.

United States, U.S. Department of Health and Human Services, Food and Drug Administration. Multiple Endpoints in Clinical Trials: Guidance for Industry. January 2017, https://www.fda.gov/files/drugs/published/Multiple-Endpoints-in-Clinical-Trials-Guidance-for-Industry.pdf. 

---. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. December 2009, www.fda.gov/media/77832/download. 

Blog

ePRO Clinical Trials: Improving Outcomes Capture

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences.

Introduction

ePROs, or electronic patient-reported outcomes, play an essential role in clinical trials, illuminating patients' lived experiences. In recognition of the importance of the patient perspective, the FDA’s industry guidance, entitled Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (a.k.a. “PRO Guidance”), emphasizes the use of PRO instruments when measuring concepts best known by the patient or best measured from the patient’s perspective (Patient-Reported Outcome Measures). The “e” in ePRO then facilitates this process by accepting digital, rather than paper-based, reporting.  

The outcomes selected for evaluation in a clinical trial must address the trial objectives; but, they must also be recognized as meaningful to end-users. As per the U.S. Department of Health and Human Services, for an outcome to be meaningful, it should reflect or describe how a person feels, functions, and survives (Multiple Endpoints).

The effort to understand a patient’s experience isn't merely a matter of empathy; it contributes significantly to the holistic evaluation of a treatment's effectiveness. ePRO gives weight to both subjective and objective measures of health, providing a more comprehensive view of how a treatment impacts patients' lives. An important aspect of this holistic evaluation is the consideration of the patients’ quality of life, an assessment that ePRO is instrumental in measuring.

The Many Benefits of ePRO in Clinical Trials

The digital nature of ePRO brings significant advantages, enabling patients to report on events as they occur in their day-to-day lives—from wherever they may be. This means patients can capture both planned and episodic events in real time. Perhaps they are being asked to journal each morning when they take their medication or during other daily events. Or, in other cases, perhaps they need to log more punctual or episodic symptoms like headaches or nausea. These electronic diaries, conveniently logged by patients from their homes or workplaces, provide study coordinators and sponsors with a clearer line of sight into the patient experience than they would otherwise have had access to.  

Therapeutic development is no longer about just saving lives; it's about improving the quality of those saved lives. And, ePRO is the best way to measure improvement or deterioration in quality of life. For example, patients are likely to report symptoms—such as vomiting, diarrhea, or fatigue—earlier and more often than clinicians. Importantly, there is also growing correlation between quality of life and lives saved. In an oncology study, early symptom detection through ePRO showed increased survival rates (Basch et al.).

ePRO also aids in improving patient compliance. The digital platform that deploys the ePRO can also facilitate patient engagement, enhancing protocol compliance and data quality (Gwaltney et al.). This timely and complete data collection translates to more reliable outcomes.

The Challenges of ePRO Implementation

Despite the many advantages associated with the incorporation of patient diaries into clinical trials, the implementation of ePRO comes with challenges. Patients are not always diligent in completing their paper diaries, and this can be true for ePRO as well.

Patients can become demotivated when ePRO clinical trial tasks are too long or complicated, and this confusion can lead to both decreased compliance and reduced reporting accuracy (Coons et al.). Anything that creates friction in the reporting journey, whether it’s more time required or less clarity, will have a negative impact on engagement.  

Patients can often feel frustrated by long and redundant questions in ePRO surveys (Stone et al.). It is natural to feel annoyed when you are asked to answer what appears to be the same or similar question multiple times, even if those questions are embedded in different tasks. Task definitions can also sometimes be perceived as confusing. For example, if a patient is asked to report on his or her symptom of dizziness, nausea, or vertigo without clear definitions of each, he or she may misinterpret the questions and will likely provide contradictory responses.  

Another hurdle to engagement can arise when questions are perceived to be irrelevant to the patient's condition. It is important for patients to understand why certain questions are being asked so that they can feel more invested in answering.  

Finally, and importantly, patients often neglect to report on time, resulting in recall bias. It can be difficult to remember, hours later, how many times one coughed or sneezed. Similarly with pain: The ability to remember how severe pain may have felt is inversely proportional to the time that goes by between the event and its reporting.  

Optimizing ePRO for Better Outcomes in Clinical Trials

To improve ePRO implementation, a more patient-centric approach is required. A successful ePRO strategy should increase patient understanding and motivation while correspondingly reducing burden. Efforts should also be made to encourage and assist patients in real-time reporting whenever possible.  

More Convenient ePRO

There are multiple ways to reduce patient burden, starting with enabling patients to use their own mobile devices for ePRO. BYOD (“bring your own device”) ePRO eliminates the need to carry multiple devices. Instead, patients use an app downloaded onto their smartphones, which naturally integrates more seamlessly into their everyday lives because they are more comfortable on their own devices.

Simplification should be built into the study design from the outset. Clinical research teams should be on the lookout for complex, long, and redundant questions, particularly those that may not be necessary to achieve primary endpoints. The National Institutes of Health's Patient-Reported Outcomes Measurement Information System (PROMIS) measures are a good example of an initiative that has helped to align the industry around simplified scales (National Institutes of Health).  

Another important contributor to patient convenience is through UX design. Good ePRO design should incorporate user-friendly scales and interactive functions to make data capture quick and intuitive. One-tap answers and easy scrolling enable participant responses on the go. For instance, hot flash or pain buttons can be tapped to quickly log symptoms for menopause or migraine trials. When questions are long, they can be broken down into multiple screens to make them more digestible. The easier it is for patients to access and complete tasks, the better their compliance will be.

In some cases, there may also be opportunities to enrich or replace labor-intensive ePRO with devices to capture continuous, more objective, or more detailed information about a symptom or patient activity. For example, rather than asking patients to take their temperature, they can use a device or even a patch that can relay temperature information directly to a data repository.  

More Real-Time ePRO

The closer to real time patients can report an event, the better. BYOD and intuitive UX can get patients part of the way there. Automated notifications and reminders are an additional way to aid patients, providing helpful alerts that nudge them towards on-time compliance. Gamification can also boost compliance for certain types of patients who are motivated by a competitive spirit. Certain trials also benefit from virtual study teams who monitor both compliance and safety, ensuring that any issues are flagged and addressed promptly.

Patients are not always in a place where they have access to the internet or to phone service; so, it is also important that technology enable them to log time-stamped events even when they are not connected.  

ePRO Relevance and Comprehension

Given the importance of patient comprehension for the successful completion of trials, training is a critical component to ensuring clear understanding of study objectives, the protocol, study design, and patient responsibilities. Multi-media, on-demand training modules and virtual site team support can be instrumental in enhancing patient comprehension and engagement. Training materials are also an opportunity to highlight the aspects of the trial that they might find the most personally compelling, as well as the ways in which the trial can contribute positively to their health and quality of life.  

Whenever possible, communication channels and content should be personalized to the specific patient population in the trial. Personalization needs can take many forms and should consider demographics and clinical constraints as well as comfort with technology. Adjustments might include anything from increased font size for older participants to culturally sensitive translations to multiple options for reaching out to site staff, in recognition of the fact that some patients may prefer phone or email to chat or video calls.  

Novel ePRO Approaches

While validated ePRO instruments are beneficial, they may not always suffice. There are times when novel ePRO approaches are necessary. For example, a study was run in 2022 to explore the new ePRO data capture system, PROMIS, and its ability to provide deeper understanding of treatment tolerability for patients with immune-mediated inflammatory disease (Iaquinta et al.).  

There may be opportunities to augment ePRO through image, audio, and video capture technologies to enhance data collection. Using expert raters to score this media can provide more objective outcomes to complement subjective ePRO data. Of course, creating and validating new instruments requires careful consideration, thorough testing, and validation to ensure they accurately and reliably capture the data intended.

Conclusion

ePRO plays a crucial role in clinical trials, offering a comprehensive and meaningful way to capture patient experiences. It contributes significantly to the development of treatments that are not only effective but also enhance patients' quality of life because therapeutics have considered the patient’s experience.  

However, it's important to remember that because ePRO is subjective, it relies on human behaviors. In order to optimize its collection, researchers must minimize the friction and constraints that impede reporting. More patient-centric ePRO solutions are thus better ePRO, and designing this patient centricity into our instruments requires the optimization of current practices.  

As we move forward into a digital future, it's clear that ePRO solutions will continue to play an integral part in clinical trials. As such, it's crucial to find the ePRO solution that best fits the unique needs of each study. This will require continued exploration of innovative approaches to improve patient engagement and data capture. With the right approach, and technological support, ePRO can serve as a powerful tool, driving better outcomes in clinical trials and ultimately leading to treatments that better serve patients' needs.

References

Basch, Ethan, et al. “Symptom Monitoring with Patient-Reported Outcomes during Routine Cancer Treatment: A Randomized Controlled Trial.” Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, vol. 34, no. 6, 2016, pp. 557–65, www.ncbi.nlm.nih.gov/pubmed/26644527, https://doi.org/10.1200/JCO.2015.63.0830. Accessed 22 May 2023.

Coons, Stephen Joel, et al. “Recommendations on Evidence Needed to Support Measurement Equivalence between Electronic and Paper-Based Patient-Reported Outcome (PRO) Measures: ISPOR EPRO Good Research Practices Task Force Report.” Value in Health, vol. 12, no. 4, June 2009, pp. 419–429, https://doi.org/10.1111/j.1524-4733.2008.00470.x. Accessed 21 May 2023.

Gwaltney, Chad J., et al. “Equivalence of Electronic and Paper-And-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review.” Value in Health, vol. 11, no. 2, Mar. 2008, pp. 322–333, https://doi.org/10.1111/j.1524-4733.2007.00231.x. Accessed 22 May 2023.

Iaquinta, Francesco Salvatore, et al. “Fatigue and Associated Factors in an Immune-Mediated Inflammatory Disease Population: A Cross-Sectional Study.” Journal of Clinical Medicine, vol. 11, no. 9, 1 Jan. 2022, p. 2455, www.mdpi.com/2077-0383/11/9/2455, https://doi.org/10.3390/jcm11092455.Accessed 21 May 2023.

National Institutes of Health. “PROMIS: Clinical Outcomes Assessment.” Nih.gov, National Institutes of Health, 29 Jan. 2019, commonfund.nih.gov/promis/index. Accessed 20 May 2023.

Stone, Arthur A., et al. “Patient Compliance with Paper and Electronic Diaries.” Controlled Clinical Trials, vol. 24, no. 2, Apr. 2003, pp. 182–199, https://doi.org/10.1016/s0197-2456(02)00320-3. Accessed 21 May 2023.

United States, U.S. Department of Health and Human Services, Food and Drug Administration. Multiple Endpoints in Clinical Trials: Guidance for Industry. January 2017, https://www.fda.gov/files/drugs/published/Multiple-Endpoints-in-Clinical-Trials-Guidance-for-Industry.pdf. 

---. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. December 2009, www.fda.gov/media/77832/download.