Discordance Between BICR Readers - Applied Clinical Trials

home: : subscribe: :

Facebook

Twitter

Quizzes: : about us: :

rss feeds/xml

See our 2009 Buyers Guide Digital Edition.

Discordance Between BICR ReadersUnderstanding the causes and implementing processes to mitigate preventable sources of discordance.

Publish date: Nov 1, 2010
By: Kristin Borradaile, Robert Ford, MD, Michael O'Neal, MD, Kevin Byrne, MD
Source: Applied Clinical Trials

Email Print

Save License

12 3 4

Blinded independent central review (BICR) is the process by which radiographic exams and selected clinical data, performed as part of a clinical trial protocol, are submitted to a central location for independent review. The Food & Drug Administration (FDA) advocates BICR of radiographic exams for registrational oncology studies when the primary study endpoint is based on tumor measurements, such as progression-free survival (PFS), time to progression (TTP), or objective response rate (ORR).¹ Current FDA guidance recommends multiple independent reviewers evaluating each subject.² One consequence of a multiple-reviewer paradigm is the potential for discordance between readers on the outcome of the subjects. Discordance between readers is adjudicated by a third reader to determine the final outcome. Adjudication rates concern sponsors and regulators because the reasons are poorly understood and there are few published metrics regarding the cause of discordance between BICR radiologists. This article discusses the causes of discordance between BICR readers, outlines the frequency of contributing factors, and suggests steps to mitigate preventable sources of discordance. This data was previously presented at the 2010 American Society of Clinical Oncology (ASCO) Annual Meeting.³

Understanding the causes of discordance

Table 1. Differences between readers in lesion selection was the leading cause of discordance.

Based on BICR data from 40 oncology clinical trials in 12 indications including 12,299 subjects, we determined that two readers did not agree 23 percent of the time when determining the best overall response and 31 percent of the time when determining the date of progression. In a subset analysis, data from only breast cancer clinical trials involving 876 subjects was blinded and pooled to identify cases wherein two primary readers were discordant in outcome. Discordance was defined as a difference in the overall best response, response date, or date of progression assessed between the two reviewers. There were 459 incidences of discordance identified. The radiographs were reviewed to determine the cause of discordance between readers and to establish if the discordance resulted from a justifiable interpretation difference where neither reader was incorrect in their assessment, or if it resulted from an assessment error by one reader. We acknowledge there may have been bias introduced into this process, as the "correctness" of the interpretation was judged by different radiologists from the same facility as the original reviewers.

Figure 1. The causes of discordance and classification of justifiable versus interpretive error.

We identified the reasons for discordance including differences in lesion selection (37%), the perception of new lesions (30%), the perception of non-target disease progression (13%), image quality or missing image data (11%), differences in lesion measurements (9%), and missing clinical information (<1%). In total, 77 percent of discordant cases were deemed to be the result of justifiable perception differences between the two readers, while 23 percent of discordant cases were thought to be due to an interpretive error made by one of the readers. The distribution of reasons for discordance and the classification of justifiable perception differences versus interpretive errors is shown in Table 1 and Figure 1.

Table 2. Discordance resulted from a justifiable difference between readers in lesion selection.

Lesion selection. When radiologists function as independent reviewers, they can potentially identify different target and non-target lesions that each considers representative of the subject's overall extent of disease. There is also the potential for the reviewers to identify the same lesions but classify them differently between target and non-target lesions. In a study by Hopper et al., two experienced radiologists identified the same lesions only 28 percent of the time.⁴ Since lesions do not respond or progress at the same rate, differences between reviewers with regard to response or progression of lesions can be observed. For example, in a RECIST (Response Evaluation Criteria in Solid Tumors) study, a percent change in the sum of the lesion diameters of target lesions may indicate an 18 percent increase (stable disease) by one reader and a 20 percent increase (progressive disease) by the second reader. In this example, it is likely both readers are correct and the results differ because the lesions chosen by Reader 1 changed at a different rate than the lesions chosen by Reader 2. Nonetheless, the outcomes are discordant, resulting in adjudication. There are multiple other examples where adjudication is forced by attempting to classify radiologist performance into categorical variables of complete response, partial response, stable disease, and progressive disease. One such example is illustrated in Table 2.

In Table 2, there was discordance in the date of response between reviewers (R1 and R2) due to a justifiable difference in lesion selection. R2 chose more lung disease and R1 identified hilar adenopathy. Because the thoracic lesions, classified as target disease by R2, responded at a more rapid rate than the liver lesions, R2 confirmed a partial response (PR) one time point (TP) before R1. The best response and date of progression between reviewers were concordant, but a justifiable difference in number and classification of lesions resulted in a response date discordance by one time point.

12 3 4