Clinical Vice President, Internal Medicine, Signant Health

Anthony Everhart, MD

Recent FDA draft guidance on core patient-reported outcomes (PROs) in oncology clinical trials

 lays out a framework for effective patient-reported outcome measure (PROM) selection and implementation. This may be important in increasing the use of PRO data in oncology medication labeling in the US, where, historically, labeling claims based on PRO data have rarely been included. The reporting of PROM endpoints on drug labeling is valuable to both the patient and the prescribing physician in informing risk/benefit discussions beyond tumor response and survival endpoints. The lack of PRO-related labeling is clearly illustrated by the fact that only three out of 85 drug approvals in oncology submitted to FDA between 2010 and 2016 included PROM-related labeling claims.

While this may be related to trial design limitations (e.g., single-arm and open-label studies), it may also reflect weaknesses in clinical outcome assessment (COA) measurement strategies typically used in oncology research.

FDA’s draft guidance addresses two areas of potential weakness in current COA measurement strategies for oncology trials.

In oncology trials, PROMs are often implemented at clinic visits just ahead of starting each cycle of treatment. While administratively convenient, this is the time at which patients are feeling well enough to receive the next cycle of treatment, so this strategy may fail to measure the impact of treatment in the earlier stages of each cycle when treatment-related side effects are likely to be most pronounced. Just measuring at cycle start may provide a biased view of treatment toxicity.

The draft guidance stresses the importance of selecting measures that are highly specific so that treatment effects can be understood more granularly. FDA identifies five core measurement domains of interest: disease-related symptoms, symptomatic adverse events, an overall side-effect impact measure (single item), physical function (aspects such as walking, lifting, and reaching that are considered important for independent functioning), and role function (impact of a treatment on the ability to work and carry out daily activities). Not all commonly used instruments provide scores or subscores that provide the level of specificity required to separately assess these core areas.

In this article, we explore these two areas in more detail.

The draft guidance recommends more frequent measurement during initial cycles, with fewer later on (see Figure 1 below). In particular, FDA indicates that symptomatic adverse events, the overall impact of side effects, and physical function should be measured frequently at the start of treatment, before moving to less frequent cadences later in the treatment and subsequent follow-up periods. This schedule helps to mitigate chances of missing the incidence and severity of items associated with these domains, as may be the case if measuring only at the start of new treatment cycles.

This increased frequency of assessments raises several practical questions:

If these domains are measured by subscales of existing instruments, is it valid to use only the subscale items at these more frequent timepoints?

To enable this more frequent measurement schedule, patients would need to complete PROMs at home. Is this practical when patients are experiencing challenging treatment-related symptoms and side effects?

Should measurement timings be adjusted if a treatment cycle is delayed due to side effects?

It would be burdensome for the patient if we implemented the full 30-item EORTC QLQ-C30 questionnaire (a frequently used general oncology PROM) on a weekly basis when we only need to measure the physical function subscale (five items). However, discarding the other items within an instrument might impact the psychometric properties of the subscale, so it will always be important to verify that this approach is acceptable and valid with the instrument owner.

In the case of the QLQ-C30, the subscales are valid for independent use as discrete “item lists.” Therefore, referring to the schedule of assessments in Figure 1, the five physical function items from the QLQ-C30 could be applied independently on a weekly basis to enable more frequent assessment of this core domain. When doing so, it will be important to include the complete set of physical function subscale items, represent them in the same order as within the full instrument, and apply the same scoring and missing data rules as defined for the subscale within the full instrument.

methods to lessen patient burden should be explored, including use of electronic PRO capture that may allow for assessments outside of the clinic,”the more frequent assessment schedule may be perceived as burdensome when the effects of treatment are particularly compromising. Could this lead to increased missing assessments? In a qualitative interview study of oncology patients, it was identified that certain days within typical treatment cycles are the most difficult, with one patient saying: “Days three to six were my ‘dark’ days, and I did not leave my bed, speak to anyone, or even eat any food.”

Despite this, participants in this qualitative study did not feel that this would deter or inhibit them from completing PROM instruments on electronic devices. For example, one patient stated, “You can be bad, but not so bad that you can’t use nothing at all...I would make the effort.”

Being sensitive to this is important, and we should consider guiding principles for ePRO implementation to limit burden and aid completion rates, including:

Ensure instruments are as short as possible to measure the domains of interest. Computerized adaptive tests may provide an approach to minimize the number of questions.

Implement multi-day completion windows, so that patients can elect to complete assessments on a subsequent day if they do not feel able to complete their PROM immediately.

If implementing more than one PROM, enable patients to come back later to complete the second instrument.

Consider a bring-your-own-device (BYOD) option. On days when the patient is feeling unwell, they may be interacting with their own mobile device but are less likely to have the trial device on hand.

Collect reasons for missing assessments so that intercurrent events, such as feeling too unwell to complete the PROMs at any given time, can be taken into account in the analysis plan, the interpretation of the results, and the optimization of future trials.

Implement PROM completion compliance strategies, such as patient alarms/reminders, site notifications, and reports, to monitor and proactively encourage completion behavior.

In oncology trials, it is common for physicians to implement delays in treatment when a patient has not recovered sufficiently from the previous cycle of treatment. When collecting PROMs at the site, this typically means postponing PROM completion to coincide with the new cycle start date. However, with the change in focus to more frequent measurement, we recommend that the trial objectives will most likely be met when the measurement schedule is not adjusted to allow for changes in the start times of subsequent treatment cycles, especially when the period of frequent measurement is long enough to comfortably cover several cycles. 

Separate and independent assessments of each of the five core domains requires careful appraisal and selection of the measures to be used. To measure each domain using a single scale, or a subscale of a broader instrument, questions should all be related to the domain and comprehensive enough to encompass all the meaningful elements of that domain.

While this sounds straightforward, nuances in the definitions of domains measured by different PROMs mean that researchers should carefully inspect instruments to ensure the FDA-defined domains are accounted for. Let’s consider two examples.

Example 1. NSCLC-SAQ (Critical Path Institute, Tucson, AZ)

The NSCLC-SAQ is a seven-item symptom assessment questionnaire for non-small cell lung cancer (NSCLC)

, developed specifically to assess disease-related symptoms across five concept areas (see Figure 2 below).

The qualitative work used to identify the set of disease-related symptoms that are meaningful to NSCLC patients

 provides strong evidence of the measure’s relevance and content validity. The measure only focuses on disease-related symptoms, so in addition to content validity, it also satisfies the requirement for specificity. For this reason, it is not surprising that this instrument was cited as an example of a suitable instrument to measure disease-related symptoms in the FDA draft guidance.

Example 2. FACT-B, Breast Cancer (FACIT Group, Ponte Vedra Beach, FL)

The Functional Assessment of Cancer Therapy - Breast (FACT-B) is a 37-item questionnaire that is used to measure five domains of health-related quality of life in breast cancer patients: physical well-being, social/family well-being, emotional well-being, functional well-being, and a breast cancer subscale.

 The naming of these domains differs from FDA core domains, and they represent slightly different concepts. For example, physical well-being is not the same as physical function.

If we examine the subdomain of functional well-being, we find that the first three items of the subscale measure one’s ability to work and whether that work is fulfilling, as well as one’s ability to enjoy life. Some of these map to FDA-defined domains—for example, ability to work may be a component of a measure of role function. However, the ways that the current subscales and their scoring are defined do not facilitate specific measures of one or more core domains as identified by FDA.

However, the FACT instruments are well validated and widely used. With that in mind, it may be possible and desirable to reorganize their subscale structure to measure FDA-defined core domains. This takes us to considerations around instrument adaptation.

As described, to meet FDA draft guidance around specificity to measure the five core domain areas, it is important to select PROMs carefully. In some cases, it may not be as simple as implementing an instrument in its off-the-shelf format. Instead, it may require some degree of adaptation. We discuss two examples ahead.

As described, while the subscale construction of the FACT-B does not appear to map directly to the FDA’s core domains, the instrument and its items are well-validated. To satisfy the draft FDA guidance, researchers may consider working with the scale author to validate an alternative subscale based on existing items within the instrument. As a hypothetical example, we might consider developing a specific measure of disease-related symptoms by grouping items from the physical function subdomain (e.g., lack of energy, pain, and nausea) along with items from other subdomains (e.g., shortness of breath, swelling/tenderness of the arms). This would require working with the scale author and evaluating the psychometrics of the new subscale.

Adding to existing instruments using item banks

Item banks are databases of individual validated PROM items. It is possible to use items banks to assemble tailored PROMs more rapidly for specific populations and measurement domains. Most applicable to oncology research are the PROMIS (Northwestern University, IL) and EORTC item banks. The latter, for example, contains around 1,000 items, with many translated in up to 100 languages.

A good illustration of this approach is the use of the EORTC item bank to supplement items in the EORTC’s core instrument (QLQ-C30), to ensure full coverage of all important items in order to measure drug benefit in a set of rare hematological stem cell disorders.

 Instrument developers identified the symptoms and impacts important to patients with these diseases, and used the EORTC item bank to map these to concepts that were not contained within the QLQ-C30. They conducted further testing in patients to confirm the suitability and content validity of these additional selected items. We may see this type of approach more often as we carefully and strategically consider PROM selection for future oncology trials in light of the FDA draft guidance.

Core Patient-Reported Outcomes in Cancer Clinical Trials 

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/core-patient-reported-outcomes-cancer-clinical-trials

Gnanasakthy, A.; DeMuro, C.; Clark, M.; et al. Patient-reported Outcomes Labeling for Products Approved by the Office of Hematology and Oncology Products of the US Food and Drug Administration (2010-2014). 

https://pubmed.ncbi.nlm.nih.gov/27069082/

Gnanasakthy, A.; Barrett, A.; Evans, E; et al. A Review of Patient-reported Outcomes Labeling for Oncology Drugs Approved by the FDA and the EMA (2012-2016). 

https://pubmed.ncbi.nlm.nih.gov/30711065/

Mowlem, F.D.; Sanderson, B.; Platko J.V.; et al. Optimizing Electronic Capture of Patient-reported Outcome Measures in Oncology Clinical Trials: Lessons Learned from a Qualitative Study. 

https://pubmed.ncbi.nlm.nih.gov/33274651/

McCarrier, K.P.; Atkinson, T.M.; DeBusk, K.P.; et al. Qualitative Development and Content Validity of the Non-small Cell Lung Cancer Symptom Assessment Questionnaire (NSCLC-SAQ), a Patient-reported Outcome Instrument.

https://pubmed.ncbi.nlm.nih.gov/27041408/

Bushnell, D.M.; Atkinson, T.M.; McCarrier, K.P.; et al. Non-Small Cell Lung Cancer Symptom Assessment Questionnaire: Psychometric Performance and Regulatory Qualification of a Novel Patient-Reported Symptom Measure. 

https://pubmed.ncbi.nlm.nih.gov/34567289/

Functional Assessment of Cancer Therapy - Breast. FACIT. 

EORTC Quality of Life Group Item Library User Guidelines, First Edition. 2022, 

Bell, J.A.; Galaznik, A.; Pompilus, F.; et al. A Pragmatic Patient-reported Outcome Strategy for Rare Disease Clinical Trials: Application of the EORTC Item Library to Myelodysplastic Syndromes, Chronic Myelomonocytic Leukemia, and Acute Myeloid Leukemia. 

https://pubmed.ncbi.nlm.nih.gov/31218454/

Articles

Examining the two areas of weakness cited in FDA draft guidance.

Optimal COA Measurement Strategy in Modern Oncology Trials

For years, physicians and medical students, most of them people of color, have tried to highlight the scarcity of images of black and brown people in medical curricula. A study published in 2018 that reviewed over 4,000 images in textbooks that are commonly used in top medical schools, such as 

, found that only 4.5% of images represented dark skin tones.

 As such, while the formal medical curriculum clearly emphasizes equality of care, this message is undermined by the poor representation of people of color in textbooks, case studies, and training materials. Critically, this uneven representation that places a white person as the normative patient has been identified as a significant contributor to racial inequality in health care experience, treatment, and patient outcomes.

Although the underrepresentation of images of people of color in the medical curriculum is an issue that affects training in every branch of medicine, it is especially problematic for dermatology where diagnosis and assessment of the severity of a condition depends almost entirely on the appearance of the skin. In the US, coverage of dark skin at national meetings and in images included in core dermatology educational resources is limited and variable.

 A study published this year focusing on dermatology textbooks found that only 11.5% of images showed dark skin.

 This is echoed by dermatologists and dermatology residents in the US, who report that they have been inadequately trained in identifying conditions on dark skin, and see the need for greater exposure to images of people of color in training materials.

Recently, there has been a growing movement online aimed at tackling this issue. Brown Skin Matters (

) is an Instagram account with reference images of dermatological conditions on non-white skin, aiming to help people of color identify skin conditions. It was launched in 2019 by Ellen Buchanan Weiss after her son, who is mixed race, broke out in a rash and she could not find any images online that matched what she was seeing on his skin. Last summer, Malone Mukwende, a second-year medical student from St. George’s Medical School in London, cowrote a handbook named 

, which aims to raise awareness on how symptoms and signs of conditions can manifest differently on dark skin. Multiple articles featuring these initiatives as well as highlighting the need for better representation of dark skin in dermatology training have followed, including pieces in 

This online movement demonstrates that, unlike other pervasive economic,

 to health for people of color, the underrepresentation of dark skin in training materials can be remedied. By improving the tools and education provided to clinicians, they can be better positioned in diagnosing and treating skin conditions in people of color. In addition to improving standard of care, diversifying training can provide significant benefits to clinical research in dermatology. For a trial to yield the most meaningful results that are applicable across the entire patient population, assessments supporting endpoints need to be measured in the most accurate and consistent manner.

Atopic dermatitis is a common chronic skin condition that typically presents during early childhood but can persist into adulthood. Due to a complex interplay of genetic and environmental factors, the immune system becomes overactive, triggering inflammation that damages the skin barrier. This leaves the skin dry, itchy, and scaly. Atopic dermatitis can often create a vicious cycle where children itch and scratch and itch and scratch, causing further inflammation and damaging the skin. Black skin is significantly associated with the presence of this condition,

 and Black children are more often seen by clinicians for a diagnosis of atopic dermatitis compared to white children,

 suggesting either an increased prevalence or severity.

In clinical trials the effectiveness of a medication in treating atopic dermatitis is determined using clinician-reported outcome measures, such as the Investigator Global Assessment (IGA) and the Eczema Area and Severity Index (EASI). For the IGA, the clinician scores the overall appearance of the skin affected by selecting one of five severity grades. Achievement of IGA 0 (clear) or 1 (almost clear) has been the primary definition of treatment success. The scoring process for the EASI is more extensive. For four different body areas (head, trunk, arms, and legs) the clinician evaluates how much of the skin is affected and how intense the characteristic signs of the condition are. These signs are erythema (redness), induration (thickening), excoriation (most likely due to scratching) and lichenification (i.e. how leathery, akin to a lichen, the skin appears). A total score is then calculated by taking into account the area affected and the severity of the signs. What is common in these and other clinician-reported outcome measures in dermatology is that they very much depend on the clinician’s judgement.

The inherent subjectivity of these measures often results in disagreement between clinicians’ ratings on the severity of a patient’s condition. Hanifin and colleagues (2001)

 tested the reliability of the EASI by exploring score agreement between clinicians (inter-rater variability) and within the same clinician across time (intra-rater variability). To achieve this, 15 dermatologists assessed 20 children and adult patients with atopic dermatitis twice over two days. Overall, inter- and intra-rater variability for the EASI total score was found to be fair-to-good. However, when exploring the scores for each individual sign of atopic dermatitis independently, agreement between the raters was evidently poorer: thickness, scratching, and lichenification showed poor or fair inter-rater variability, with only erythema crossing the threshold to good for both assessment timings. Other studies exploring the reliability of clinician reported measures for atopic dermatitis have found similar results, with scores on the most commonly used assessments in clinical trials showing either poor or moderate agreement between clinicians (e.g. Silverberg et al. (2021)).

Studies have specifically focused on how agreement in clinicians’ scores on dermatology assessments differ across skin types. When it comes specifically to patients of color, rater variability has been found to be disproportionately high for dark skin. For example, Zhao and colleagues (2015)

 compared the reliability of clinician scores on the most common assessments used in clinical trials for atopic dermatitis, including the EASI. They found that across all assessments, inter-rater variability was higher the darker the skin of the patient being assessed, and that reliability was poorest for patients with highly pigmented skin. A significant part of this disagreement was due to a high degree of variability in clinicians’ scores of erythema on highly pigmented skin. When erythema was excluded from the assessment of disease severity and the score was adjusted, disease was found to be significantly more severe. These results echoed the well-reported challenge that dermatologists have detecting erythema in heavily pigmented skin,

 resulting in patients with dark skin that have cutaneous diseases such atopic dermatitis being underdiagnosed and undertreated.

For clinical research in dermatology to yield meaningful results applicable across the disease population, the most relevant outcomes–such as the EASI in atopic dermatitis–need to be measured in the most accurate and consistent manner. Critical to this is ensuring that every clinician rating disease severity understands the scores in the same way and administers the assessments in the same way every single time. Training raters who will administer the assessments prior to the initiation of a clinical trial can ensure their scores of patients’ disease severity during the trial are calibrated, reducing variability. Rater training has been used extensively for clinicians assessing patients in clinical trials on central nervous system disorders, where outcomes are often semi-structured interviews that rely heavily on clinical judgement,

 but only recently has been identified as important for endpoint quality in dermatology clinical trials.

In a recent study, Olt and colleagues (2019)

 investigated the impact of standardizing training on the EASI for clinicians participating in a US nationwide trial. Clinicians were trained on a detailed explanation of the assessment, along with consensus building exercises with expert trainers that involved example images with varying severity of atopic dermatitis signs. The participating clinicians then took part in a test-retest certification test aimed to establish competence and determine inter-rater variability. This certification exercise consisted of two sessions, 5-10 days apart, where a set of case studies with different disease severity were presented, and the clinicians were asked to rate them. The findings demonstrated that variability in clinician’s EASI scores were significantly reduced after they received training, clearly showing that standardized rater training can align clinical judgement thus improving data reliability and quality.

Given the underrepresentation of dark skin in the medical curriculum,

 and the increased rater variability observed when assessing dark skin,

 it is important to ensure that rater training materials for dermatology clinical trials are diverse and sufficiently represent all skin types. This is vital for making sure disease severity and treatment effects are accurately captured for patients of color participating in clinical research. However, although there are textbooks such as the 

 that contain high-quality images of dermatological conditions on darker skin, there is no standardized image library for rater training that illustrates how signs manifest and progress with disease severity across different skin types. Issues with images currently used in rater training range from poor image resolution and artefacts obstructing the affected area, to limited depiction of key anatomical areas affected by disease.

To address this issue, Signant Health gathered a multidisciplinary team of graphic artists, scientists, and expert clinicians to develop a set of training images for clinical research. The team developed high resolution, photo-realistic images of the different signs of atopic dermatitis across severity levels and skin types which can be seen below. These images capture the progression of the signs of atopic dermatitis in common anatomical areas affected by the condition (e.g. inside of the elbow and back of the knee), with the affected area being represented without obstruction. By using this set of images, clinicians can be trained to identify and rate otherwise difficult signs such as erythema on darker skin. Training on such a diverse set of images has the potential to increase the reliability of clinician reported measures in dermatology clinical trials.

Clinical trials are essential in contemporary evidence-based medicine, as they generate evidence on the safety and efficacy of new therapies and treatments. Diversity of participants in clinical trials is critical to ensure that the trial population is representative of the patients who will be using the treatment being tested, and that the findings are generalizable. Although historically racial and ethnic minorities have been underrepresented in clinical research across therapeutic indications,

 major efforts have been made in the US by the FDA and the Revitalization Act of 1993. This has led to a steady increase of dermatological trials involving ethnic minorities.

 As such, diversity in training materials for clinicians taking part in a clinical trial, ensuring they are well positioned to judge treatment effectiveness across skin types, is critical.

Louie, P. & Wilkes, R. (2018). Representations of race and skin tone in medical textbook imagery. 

Martin, G. C., Kirgis, J., Sid, E., & Sabin, J. A. (2016). Equitable imagery in the preclinical medical school curriculum: Findings from one medical school.

Ebede, T., & Papier, A. (2006). Disparities in dermatology educational resources. 

Journal of the American Academy of Dermatology

Adelekum, A., Onyekaba, G., & Lipoff, J. B. (2020). Skin colour in dermatology textbooks: An updated evaluation and analysis. 

Buster, K. J., Yang, L., & Elmets, C. A. (2011). Are dermatologists confident in treating skin disease in African Americans? 

Journal of Investigative Dermatology: Meeting Abstracts

Williams, D. R., Yu, Y., Jackson, J. S., & Anderson, N. B. (1997). Racial differences in physical and mental health: Socio-economic status, stress and discrimination. 

Dressler, W. W., Oths, K. S., Gravlee, C. C. (2005). RACE AND ETHNICITY IN PUBLIC HEALTH RESEARCH: Models to Explain Health Disparities. 

Brondolo, E., & Gallo, L. C. (2009). Race, racism, and health: Disparities, mechanisms, and interventions. 

Shaw T. E., Currie G. P., Koudelka C. W., & Simpson E. L. (2011). Eczema prevalence in the United States: data from the 2003 National Survey of Children’s Health. 

Horii K. A., Simon S. D., Liu D. Y., & Sharma V. (2007). Atopic dermatitis in children in the United States, 1997-2004: visit trends, patient and provider characteristics, and prescribing patterns. 

Janumpally S. R., Feldman S. R., Gupta A. K., & Fleischer A. B., Jr. (2002). In the United States, blacks and Asian/Pacific Islanders are more likely than whites to seek medical care for atopic dermatitis. 

Hanifin, J. M., Thurston, M., Omoto, M., Cherill, R., Tofte, S. J., & the EASI Evaluator Group. (2001). The eczema area and severity index (EASI): Assessment of reliability in atopic dermatitis. 

Silverberg, J.I., Lei, D., Yousaf, M., Janmohamed, S. R., Vakharia, P. P., Chopra, R., Chavda, R., Gabriel, S., Patel, K. R., Singam, V., Kantor, R., & Hsu, D. Y. (2021). Measurement properties of the Rajka-Langeland severity score in children and adults with atopic dermatitis. 

Zhao, C. Y., Wijayanti, A., Doria, M. C., Harris, A. G., Jain, S. V., Legaspi, K. N., Dlova, N. C., Law, M. G., & Murrell, D. F. (2015). The reliability and validity of outcome measures for atopic dermatitis in patients with pigmented skin: A grey area. 

International Journal of Women’s Dermatology

Chansky, P. B., Mittal, L., & Werth, V. P. (2018). Dermatological evaluation in patients with skin of colour: The effect of erythema on outcome measures in atopic dermatitis. 

Ben-Gashir, M. A., & Hay, R. J. (2002). Reliance on erythema scores may mask severe atopic dermatitis in black children compared with their white counterparts. 

Becker, R. E., Greig, N. H., & Giacobini, E. (2008). Why do so many drugs for Alzheimer’s disease fail in development? Time for new methods and new practices? 

Olt, J., Khan, R., Sanderson, B., Feaster, T., Solomon, T. S., Barbone, J. M., Platko, J. V., Bodart, S., Garner, K., & Byrom, B. (2019). Clinician training reduces variability in clinician-reported outcomes of skin surface symptoms in a dermatology study. 

Advani, A. S., Atkeson, B., Brown, C. L., Peterson, B. L., Fish, L., Johnson, J. L., Gockerman, J. P., & Gautier, M. (2003). Barrier to the participation of African-American patients with cancer in clinical trials. 

Spears, C. R., Nolan, B. V., O’Neill, J. L., Arcury, T. A., Grzywacz, J. G., & Feldman, S. R. (2011). Recruiting underserved populations to dermatologic research: A systematic review. 

Clinical Vice President, Internal Medicine, Signant Health, 

Poor representation of dark skin tones in textbooks emphasizes need for change.

Anthony Everhart, MD

Articles

Optimal COA Measurement Strategy in Modern Oncology Trials

Properly Training Physicians in Dermatology Trials