Michael E. Marcarelli

The goal of the Probability Sampling Study (P.S.S.) is to evaluate risk factors that contribute to the state of regulatory compliance for medical device clinical investigations. The results of the P.S.S. can be used to direct limited FDA resources into areas where there may be enhanced concern for human subject protection and/or clinical data quality systems. The data from the probability sample can be extrapolated to the entire universe of medical device clinical investigators.

FDA inspected 200 randomly selected clinical investigators as part of the P.S.S. between August 4, 2003, and March 20, 2007. Ultimately, FDA issued eight Warning Letters (WLs)

  to clinical investigators. At the eight investigator sites receiving WLs, a statistical analysis of variables demonstrates a strong correlation between: the site being inspected for the first time by FDA, investigators that did not receive sufficient training on the conduct of the study, and the site being monitored by the sponsor of the investigation.

This correlation and relationship to WLs supports the need to increase education and training for both clinical investigators and sponsors. 

In January 2001 the FDA's Center for Devices and Radiological Health (CDRH) Bioresearch Monitoring Division (BIMO) and the Office of Surveillance and Biometrics (OSB) applied for and received additional needs funding to develop the Probability Sampling Study (P.S.S.) for the Center's clinical investigator inspection program.

BIMO implements its device investigator inspection program in several ways. One of the approaches involves the inspection of clinical investigators once the study is completed and the Premarket Approval Application (PMA)

  is submitted to the FDA. This type of inspection generally focuses on data verification.

A second approach, the early intervention program, is designed to inspect clinical investigators of on-going Investigational Device Exemptions (IDEs). Early intervention inspections focus on human subject protection, protocol adherence, and data quality. CDRH BIMO believes that early intervention inspections will result in higher quality submissions through the identification, correction, and prevention of regulatory deviations.

The P.S.S. focused exclusively on inspecting clinical trial investigators that were conducting on-going IDEs through the early intervention program.

Probability sampling is defined as any valid method of sampling that uses some form of random selection. A sample analysis is done after random selection to draw well-founded conclusions, which are representative of the whole population. The P.S.S. data can be used to assess the device research community's compliance strengths and weaknesses.

Using the identified trends, BIMO believes that the device research community will be better equipped to identify and implement improvements to their Good Clinical Practice (GCP) systems.

  BIMO has also used the information gathered from the P.S.S. to develop educational outreach for clinical investigators to focus on the areas of greatest need.

Further, FDA can potentially assess the effectiveness of "early inspections" by FDA field investigators on the quality of device marketing applications to determine if the inspections reduced the need for additional BIMO inspections at the marketing phase (i.e., PMA or 510(k) Submission).

CDRH estimated that a total of approximately 20,000 investigators are associated with the 1100 active IDE applications, based upon the assumption that there are approximately 20 sites for every IDE. Westat, a statistical services firm that CDRH contracted to draw the random samples, selected 158 out of the 1100 IDE files, and extracted the names and locator information of 974 physicians conducting the clinical investigations. The information gathered was entered into a database, which generated random numbers for each investigator.

A two-phase stratified random sampling, sometimes called proportional or quota random sampling, was used.

This sampling method divided the IDE submissions into homogeneous subgroups and then took a simple random selection from each subgroup. The first phase involved stratifying the IDEs by two subgroups: device area and participating clinical investigators. A sampling of IDEs within each device area was selected to ensure that the probability sample was allocated across all types of medical devices in a proportionally appropriate manner. A sampling of investigators was selected to ensure variability control in the estimates if investigators are homogeneous within an IDE.

The second phase sample was conducted after all investigators within the sampled IDEs were selected. The number of investigators was stratified by region and different sampling rates were applied in each region to equalize the workloads for the FDA inspectors. The sample was stratified geographically between 19 district office locations (see Figure 1) and 11 device types. Ultimately, the sample encompassed 74 IDE applications and 200 investigators.

The inspectional assignments issued as part of the random sample were identical to all other normal BIMO inspectional procedures in order to appropriately compare the results from the random sample to the results of the nonrandomly sampled inspections conducted during the same time period.

The results of the P.S.S. demonstrate that there is a strong correlation between initial FDA inspection, lack of sufficient training (e.g., GCP), and in-house sponsor monitoring (for more detailed information, see Figure 2). All eight WLs issued to sites under the P.S.S. had these characteristics.

Notably, seven of the eight WLs were issued to investigators that had no reported GCP training. Of the 200 random sampled clinical investigators, 52 (26%) did not report any GCP training and 180 (90%) were monitored by a third party. Moreover, the eight WLs issued to investigators that were monitored in-house by the sponsor, represented eight out of only 20 in-house sponsor monitored IDEs in the entire P.S.S.

That translates into a 40% noncompliance rate. This suggests the need for sponsor training/guidance on proper monitoring, or the need for better sponsor assessment of in-house core competencies.

None of the 48 out of 200 clinical investigators inspected in the P.S.S., who were previously inspected by FDA, received a WL, which seems to indicate that early FDA intervention helps assure that investigators are in compliance with FDA's regulatory requirements.

Fourteen percent of the nonrandom sampled BIMO inspections conducted during the same time period as the P.S.S. resulted in a WL, whereas only 4% of the P.S.S. inspections resulted in the issuance of a WL (see Figure 3).

Nonrandom sampled BIMO inspections are typically triggered by an in-house marketing application or a complaint-based inspection (aka, For Cause). This comparison reveals more than a three-fold increase in the number of regulatory actions taken for the nonrandom sampled BIMO investigator inspections versus sites inspected as part of the P.S.S.

Conversely, outcomes of inspections that resulted in either a Firm in Compliance, Information Letter or Untitled Letter

  were comparable between the P.S.S. and nonrandom sampled BIMO inspections conducted during the same time interval.

The 200 clinical investigators that BIMO inspected under the P.S.S. represents a total of 74 IDEs that Westat selected as part of the random sample of 158 IDEs. At the time of this analysis, approximately seven (11%) of the 74 IDEs were subsequently used to support a PMA approval, and one was used to support a 510(k). One IDE in the P.S.S., for which an investigator had received a WL, was used to support a PMA approval after the investigator and sponsor were able to make early corrections to the noted GCP violations.

As described in the previous section, the results of the P.S.S. demonstrate that among the eight WLs issued, there is a strong correlation with initial FDA inspections, in-house sponsor monitoring, and lack of investigator training. Accordingly, these results support the need for increased attention to effective sponsor monitoring and investigator training.

Training is available from FDA workshops, industry consultants, academic institutions, and other professional organizations. Some training can also be augmented by increased use of voluntary industry certifications or accreditations. Further, there is a CDRH BIMO Learn Module available online at: 

, which is designed to teach sponsors, clinical investigators, IRBs, and others involved in medical device research some of the basic FDA regulatory requirements for conducting medical device clinical trials involving human subjects.

FDA/CDRH contribute in other ways to educating and training sponsors and investigators. Currently, CDRH conducts early intervention inspections such as the ones conducted under the P.S.S. Early intervention inspections allow CDRH to go out during the active device research phase to sensitize investigators and sponsors to the importance of data quality and human subject protection while allowing them time to make necessary corrections before it can jeopardize subject safety or data integrity. CDRH expects that such early intervention and corrections will enhance the quality of data that supports safety and effectiveness.

No investigator in the P.S.S. monitored by a third-party was issued a WL. Based on our P.S.S. results, outsourcing study functions to a contractor beyond the sponsor's in-house capabilities (e.g., monitoring) may lessen the probability of receiving a WL. Sponsors may consider assessing their own in-house core competencies to determine any need for outside assistance; then, utilize a selection process to ensure that those needs can be met by the contractor and a risk-based oversight process to manage that contractor.

FDA further expects to apply the results of the P.S.S. to increase its GCP education and outreach efforts, with focus on sponsor monitoring and investigator training. These FDA outreach efforts are anticipated to focus on collaboration with other professional organizations, institutions, and private–public partnerships to promote understanding and compliance with GCP requirements.

A comparison of the WLs issued under the existing BIMO program and the P.S.S. demonstrates that medical device clinical investigations are generally in a state of substantial compliance. Only 4% of the investigators in the P.S.S. received a WL versus a WL rate of 14% for all other BIMO inspections conducted over the same time-period.

The higher percentage of WLs issued under the current CDRH BIMO program can be attributed to its risk-based model to identify clinical sites for inspection. This model focuses on high-risk indicators, which target inspectional areas that can yield a higher probability of identifying areas of regulatory noncompliance. FDA anticipates that use of the risk-based factors identified in the P.S.S., such as initial inspections and sponsor-monitored studies, may be added to BIMO's risk-based inspection model.

Furthermore, some sponsors develop risk-based measures for overseeing studies that include mock audits of investigators to prepare them for FDA inspections. Our findings suggest that this approach may have some merit since all WLs issued under the P.S.S. were issued to investigators undergoing their initial FDA inspection.

FDA publishes compliance program guidance manuals online, which outline areas of interest to FDA during a BIMO inspection, and are a good tool to use for this type of continuous process improvement activity. Lastly, P.S.S. results underscore the importance of hiring researchers and other third-parties with adequate training and experience in conducting or overseeing clinical studies.

Special thanks go to Gloria Irons and Dolores Bernato for their assistance in developing this article.

Articles

A probability sampling assessment by FDA takes a look at compliance in the medical device world.

Michael E. Marcarelli

Articles

Medical Device Compliance