Mimmo Garibbo

	Endpoint adjudication is gaining importance in the industry, however, no guidance on endpoint adjudication exists. In order to shed light on endpoint adjudication, the 

 generated an Events Charter Template for endpoint adjudication to assist those in need of guidance.

	The Adjudication Committee (AC) Chair and Members submit their dated signature to ensure that they have read and approved of the following Charter and its contents.

2. Study Abstract/Proposal/Introduction Page

	Description of the Clinical Trial related to Adjudication, Study objectives (primary, secondary, exploratory) and the rationale behind the Endpoints Assessments adoption.

	The objective of an adjudication is that a well-recognized group of experts in the area of the <study indication> confirms that an event took place and that for each event it fulfills the per protocol definitions. The members of the committee are not participating in the trial as principal or coinvestigators and their review ensure well reported, unbiased, blinded to treatment, count of events. When an adjudicator also participates - directly or indirectly - in the study, due diligence needs to be in place by the sponsor to avoid either adjudication of self-reports and unintentional unblinding of that adjudicator. The process should take place on a continuous timely fashion. Source documents need to be submitted on time together with the report shortly thereafter.

	The Adjudication Committee is responsible for reviewing each event as defined in the protocol. Their work is independent and impartial. Such type of review is to ensure accurate counting, assessing, rating, etc. of protocol defined "events".

	Unless defined by the Charter and supported by the composition of the Committee review of safety signals or of benefit - risk assessments are out of the scope of the Adjudication Committee.

	Identification of the Committee Chair along with his/her required expertise and related experiences should be described in detail. Roles and responsibilities of the Committee Chair should be described.

	The chair should as a minimum write/update or, respectively, oversee writing/updating the Charter and the decision rules. S/he chairs the Adjudication Committee meetings, arbitrates, documents, reviews and approves meeting minutes, decisions and results, and communicates with the sponsor/Steering Committee and clinical sites, as appropriate.

	The Charter will include a list of the Adjudication Committee Members including their contact information (work address, telephone number, fax number or email address), credentials and experience qualifying them for their assigned role(s). Adjudication Committee members will sign a potential conflict of interests declaration.

	The roles of the Adjudication Members are to be summarized, including their responsibilities throughout the course of the clinical trial, timelines, and participation in the Adjudication Committee meetings.
	
	Procedures are to be in place in the event that it is necessary to replace a Committee Member. The Charter keeps on file the reasons for replacing an Adjudication Committee member, such as inadequate performance and lack of communication with other members, lack of participation in consensus meetings, or inability to meet the responsibilities detailed in the Charter. Decision and communication are to be kept internal but are made available to auditors and regulatory inspectors.

	An adequate description of the training processes for new members of the Committee to adjudicate clinical endpoints will include how system access controls and blinding of reviewers will be addressed. Training will include processes, provisions and measures to prevent intentional or unintentional unblinding of trial participants/data.

	A description of the Endpoint Adjudication software to be used throughout the life-cycle of the clinical trial including how systems' security (user ID, electronic signature, passwords to access), customization, and validation controls have been addressed.

	This section is to include the following elements:

		Endpoint definition(s); what constitutes an endpoint, methods of endpoint identification.

Note: an event can be an endpoint (e.g., classification of the severity of a cardiac event) or a measurement of a tracing (e.g., ECG) or an image (e.g., tumor size). In the following the term event will be used to describe both types of adjudication processes/tasks.

		Is the event a primary or secondary endpoint.

		What are the event sources, on what source documents will the Adjudication Committee deliberate.

		Event identification codes for making the events anonymous.

		Blinding provisions to clinical trial center team members and sponsors.

		Management of participants' identifiers and relation to allocation of treatment.

		A statement that adjudication of an event where the rater had any involvement is nullified.

		If used, the process for post-hoc analyses or data re-analysis using an adjudication process and how this will be managed within the study planning.

		The use of an adjudicated result will be clarified as to whether this is planned as the primary data analysis, is concurrent and in addition to or separate from analysis and publication of primary results or original results (such as intent-to-treat).

		Adjudication committee members' roles in authorship or acknowledgment in a publication is to be pre-specified and defined in the Adjudication Committee Charter.

	The Assessment process consists of the following steps:

		Description of the event submission procedure and rules applied for assignment to a specific reviewer.

		Provisions made to avoid adjudication of self-events.

		The coordinating center will ensure that each event is complete as a package including the reporting CRF and all relevant supporting source documents. This can be a paper or an electronic process. Packages are to reach the designated adjudicators independently via courier/or/electronic password protected workflows. Adjudicators might send back their decisions also by courier or register themselves in the electronic "adjudication space" especially created for that.

		The workflow shall include a process by which the rater can ask for additional information/data, e.g., a request form for the adjudicator to request. The complete package would then consist of all original information/data plus the supplemental information/documents/data).

		The workflow may support the raters with automated edit checks, fill-in instructions.

		Description of the procedures to handle Disagreements among Reviewers.

		Classification of disagreement (example: major or minor, event or not event) and respective resolution procedures. As mentioned in paragraph 8.1, if in doubt the rater needs to request more information to make an unequivocal decision.

		When a site re-submits an event and this has not been adjudicated yet, only the resubmission goes to adjudication.

		When the re-submitted event has been already adjudicated, the last information should be reanalyzed anew and the re-adjudication result superseeds any prior adjudication.

		The Chair can review the entire reporting, if deemed necessary.

	The following are examples of deliverables of the Adjudication Process:

		Format and content of the Adjudication Assessment Report by rater and consolidated finalized Adjudication Assessment Report as well as any periodic/final summary reports.

		Format of data and meta-data transfers/exports, e.g., as SAS, XML or other formats.

		Statement of preservation of blindness of sponsor and clinical trial center staff members, if applicable.

	A list of people involved in the adjudication process with their respective role (e.g., Adjudication Coordinator, Chair of the Adjudication Committee, Adjudication Committee Member), their responsibility and contact information will be maintained.

 is maintained to define upfront the following scenarios/requirements:

		Description of the situations that trigger an exchange of communication between sponsor staff and Committee Members, between Committee members, etc.

		It is recommended to develop and use communication templates such as standard meeting teleconference agendas and minutes that is as standard agenda items topics that are of particular importance to the adjudication process.

		The Adjudication status will be tracked by means of periodical reports, and according to the needs of the study, indicating, for example, how many of the total reported events have been adjudicated, how many were reported but are not complete yet, and number of refuted  from the completed ones.

	A description and flowchart of the Endpoint Assessment Standard Procedures and Workflows should be prepared upfront and these should take into consideration key study milestones such as submission and assessment timelines as well as any other milestones defined in the trial protocol.

	The sponsor's coordinator should actively manage these timelines and ensure that the adjudication process runs according to the agreed plan and, where required, issue reminders to the adjudication committee members.

13. Quality Control: Data Sources, Procedures, Analysis

	A complete list of data sources and their specified format is to be defined, for example the following aspects should be addressed:

		Data points where the CRF is the source of the entry, i.e., CRF is the source document.

		Option of .pdf upload of scanned data forms or .pdf upload of medical record elements, upload of CT or MRI imaging

		Roles and responsibilities in transferring adjudication information/assessments into the CRF or clinical database, i.e., is information related to adjudication entered into the CRF, a separate document, a separate data collection platform. Who performs these data entry tasks: sponsor representatives, adjudicators or who else.

		It is recommended to define media involved in the adjudication process (either as data capture or storage media), roles and responsibilities in entering data onto these media in a matrix table identifying explicit roles and accountabilities.

Articles

Endpoint adjudication is gaining importance in pharma today, however, no guidance on endpoint adjudication currently exists. The LinkedIn Endpoint Adjudication Community has generated an Events Charter Template to shed light on the subject for those in need of guidance.

Endpoint Adjudication of Events Charter Template: What Should Be Included?

	Sentiment analysis can give health care organizations a competitive edge in understanding what customers think about their healthcare experience, to help reduce costs and improve care service and to lead to new clinical research and treatments. In addition, it taps into a new channel of pharmacovigilance input information that can enable Marketing Authorization Holders to keep abreast of opinions on the safety of their products in real time.

	In the context of medicinal products and devices, sentiment can be referred to an adverse event experience but also a positive treatment outcome.

	The sentiment can be deducted as final output of a technique that includes the massive collection of some unstructured information from any source selected as relevant and their processing aimed to identify and extract the implicit subjective judgment or evaluation.   

	Proof of concept: Roche-GSK competition on melanoma therapy

	The main goal of the proof of concept (PoC) is to show the applicability of a sentiment analysis approach to clinical data, in the context of social media monitoring, data analysis and reporting. In order to obtain a sensible quantitative analysis we collected data from different sources. It should be noted that because this is a proof of concept we have planned for a limited data investigation. Therefore, by increasing numbers in a production environment we will gain remarkable quantitative improvements and analysis refinements.

	This proof of concept focuses on sentiment analysis of opinions shared on the Web about two products for the treatment of melanoma marketed by GSK and Roche. Our research involved Roche’s single-agent Zelboraf (vemurafenib) and GSK's combination of Mekinist (Trametinib) and Tafinlar (dabrafenib).

	In this study we gathered users comments from three different types of data sources, i.e., online patient fora, Facebook public fan pages

 and the micro-blogging Twitter from which we have been collecting tweets daily for a one-month period (to overcome Twitter’s API Search limitations on the number and date of tweets that can be collected).

	After a preliminary standard analysis, we noticed that in some cases the standard emotional valences associated with particular words were not correct in this specific domain. As an example, in colloquial English the words “positive” and “negative” almost always carry respectively a positive and negative valence, while if they refer to a treatment or a disease in a clinical context, the two words might have a completely different meaning. For this purpose, we conducted a customization of the sentiment analysis algorithm using rules specialized to the healthcare domain.

	For future analyses, we plan for a further customization refinement in order to better deal with the social media based unstructured terminology (abbreviations, slang, spelling mistakes, etc.) both in the general sentiment analysis of texts and in adverse event descriptions.

	We have been collecting tweets about Roche and GSK’s therapies for a one-month period and then performed sentiment analysis using the following procedure:

		A data collection task enabled us to extract tweets searching them by keywords (in particular, we used INNs for all drugs involved as well as the trade names). Search for abbreviations, vernacular designations, acronyms, etc. of the products was out of scope in the PoC step.
		 

		We used a general sentiment analysis algorithm of tweet texts to find out the general tone expressed by users
		 

		Entity extraction enabled us to point out sentiments associated to each of the two companies and relative treatments

	Finally, we visualized results using dashboards and we created reports to compare the results in tabular or other descriptive form.

	Once we have created the workflows we can re-run them multiple times to repeat the analysis of the tweets in order to progressively monitor opinions expressed on Twitter about the two companies.

	We collected tweets about GSK and Roche’s treatments using as search terms both the trade names (Zelboraf, Tafinlar) and the INNs (vemurafenib, dabrafenib). In order to improve the analysis we added the Bristol-Myers Squibb’s treatment for metastatic melanoma Yervoy (Ipilimumab) to the search terms.

	We then created a bar chart showing general sentiment polarity percentages of the tweets (Figure 1). The numbers on the horizontal axis represent the sentiment polarities assigned to the tweets by the sentiment classification algorithm; in particular, a positive number indicates a positive sentiment, a negative number indicates a negative sentiment while 0.0 indicates a neutral sentiment. The higher a positive number is the better the opinion, while the higher a negative number is the worse the opinion. Percentages on the bars indicate respectively the number of tweets classified with the corresponding polarity compared to the total of the tweets. We summed the percentages corresponding to positive numbers to get the total of positive results and then we did the same for negative cases.

	It turned out that the overall tone of texts was neutral (as shown by the 62.04% peak at 0.0 in Figure 1) followed by positive comments (33.54%), while only negative ones were a very small number (4.44%). 

	The algorithm of Entity Extraction returned a list of entities, i.e. nouns of products, brands, companies, etc. that are visualized using tag clouds (Figure 2). These interactive tools are used to show the most relevant entities and the importance of each tag is shown with font size or color. As an example, in Figure 3 and in Figure 4 respectively, we showed sentiment polarities associated to the entities “Vemurafenib” and “Dabrafenib” using the same type of chart as in Figure 1.             

 from three online patient fora, respectively the Macmillan's Online Community's forum, the Cancer Survivors Network discussion board and the DailyStrength health-related social network, in order to recollect patients’ experiences with the two treatments. Specifically, we carried out the following tasks:

		Data collection using a Web extractor to gather comments containing the arguments of interest;
		 

		Sentiment analysis to classify texts into three groups in terms of positive, negative or neutral polarity;
		 

		Entity extraction enabled us to retrieve treatments mentions and classify the sentiment associated to them and, more important, to monitor side effect mentioned.

	We then visualized results using dashboards and created reports.

	We can compare the results to understand which of the two companies (treatments) are the most appreciated and why (extracting sentiment words).

	We used sentiment analysis on the collected texts and then we showed the results using bar charts (Figure 5 and Figure 6).  Same as above, the numbers on the horizontal axis represent the sentiment polarities assigned to the texts by the sentiment classification algorithm; in particular, a positive number indicates a positive sentiment, a negative number indicates a negative sentiment while 0.0 indicates a neutral sentiment. Percentages on the bars indicate respectively the number of texts classified with the corresponding polarity compared to the total of the texts. We totaled the percentages corresponding to positive numbers to get the total of positive results and then we did the same for negative cases.

	Using sentiment analysis on comment texts about Zelboraf treatment it resulted that the overall tone was positive (40%) even if there was also a small number of high negative cases (-10, -8.5), while the neutral cases corresponded to the 23.85% (Figure 5)

	Doing the same analysis on the comments about the GSK therapy we retrieved similar results since positive comments were still the majority (49%) followed by negatives (32%) and neutral comments (18.89%) (Figure 6).

	This positive attitude is confirmed by predominance of positive sentiment words these are words expressing the sentiments retrieved in the texts, which could be visualized using a tag cloud (Figure 7).

	Same as in the Twitter example, the algorithm of Entity Extraction returned a list of entities that included the different therapies (“Zelboraf”, “Dabrafenib”, “Pazopanib”, “Interferon”) and we retrieved several side effects such as “fevers”, “back pain”, “skin rash”, associated with the entity “Dabrafenib” and “nausea”, “fatigue” “rash”, “cough”, “headache” associated with the entity “Zelboraf”.

	We extracted data from three no-profit melanoma organization Facebook fan pages that are Facebook public pages where each organization shares news and information about Melanoma and other users are able to leave comments to these posts.

	More specifically, we extracted a total of 836 user comments associated to the messages posted by the organization, in order to collect melanoma experiences shared in the comments.

	We used the following procedure of analysis:

		Collected data using a specific task to extract posts and comment messages searching them from the specific Facebook fan pages

		Used a general sentiment analysis algorithm on the comments’ text to find out the general tone expressed by users

		Retrieved references to the therapies of interest using entity extraction

	We used sentiment analysis on the comment texts and then we showed the results using a bar chart (Figure 7). The numbers on the vertical axis represent the sentiment polarities assigned to the texts by the sentiment classification algorithm; in particular, a positive number indicates a positive sentiment, a negative number indicates a negative sentiment while 0.0 indicates a neutral sentiment. Percentages on the bars indicate respectively the number of texts classified with the corresponding polarity compared to the total of the texts. We summed the percentages corresponding to positive numbers to get the total of positive results and then we did the same for negative cases. As we can see from the bar chart in Figure 7, the overall tone associated to comments is neutral, as shown by the 47.74% peak at 0.0. Differently to the previous examples, in this case the range of the sentiment polarities is more scattered with more fluctuation of high positive and high negative sentiment scores (9.5, 10, -12, -15, -14).

 Sentiment analysis can give health care organizations a competitive edge in understanding what customers think about their healthcare experience, to help reduce costs and improve care service and to lead to new clinical research and treatments. In addition, it taps into a new channel of pharmacovigilance input information that can enable Marketing Authorization Holders to keep abreast of opinions on the safety of their products in real time.

Mimmo Garibbo

Articles

Endpoint Adjudication of Events Charter Template: What Should Be Included?

Sentiment Analysis: Understand Your Healthcare Customers