Skip to main content
Research and Evaluation
Back to Case Studies

National Institutes of Health, Office of Extramural Research (NIH OER) Peer Review of Human Subjects Research Protection (HSRP) Grants

Client

Human Research Protections Program, Office of Extramural Research, Office of Extramural Programs/National Institutes of Health (HRPP OER OEP/NIH)

Challenge

Among research applications submitted to NIH annually, about 40 percent involve human subjects research. NIH relies on peer reviewers to fulfill its regulatory obligation to evaluate applications proposing human subjects research. Identifying and resolving human subjects concerns prior to funding reduces risk of harm to subjects participating in NIH-supported research and helps engender trust in the research enterprise.

NOVA Approach

NOVA assisted in development of analysis methods, conducted the analyses, and made substantial contributions to interpretation of evaluation results.

Task 1: Descriptive Analysis of Outcomes of Peer Review of Human Subjects—NOVA staff conducted a descriptive analysis of NIH extramural grant applications with respect to human subjects, including outcomes of peer review. Most evaluation variables were extracted directly from IMPAC II. Statistical analyses (most frequently, chi-square tests) were conducted to assess differences in peer review related to variables of interest. For analysis of peer review outcomes by study section workload, an analysis of variance (ANOVA) was conducted with human subjects research category as the independent variable and study section workload as the dependent variable. Post-hoc analysis using Bonferroni procedure was conducted to identify significant differences.

Task 2: Analysis of Types of Human Subjects Concerns According to Key Variables—NOVA assessed types of human subjects concerns identified by peer reviewers. Concerns were coded using an HRPP coding scheme. Both awarded and unawarded applications were included. NOVA identified a sample of 400 unawarded applications rated as unacceptable by Study Review Groups (SRGs) using Stata randomization functions. A team of NOVA evaluators assigned human subjects concerns codes to unawarded, unacceptable applications based on review of summary statements.

Task 3: Evaluation of Accuracy of Peer Review Assessment and Coding of Human Subjects Concerns—NOVA evaluators reviewed grant applications to assess accuracy of peer review of human subjects protections. The goal of the evaluation of acceptable nonexempt applications was to determine types of human subjects concerns missed by peer reviewers. NOVA evaluators reviewed 495 applications rated by SRGs as acceptable nonexempt human subjects research and determined which human subject code they thought should have been applied to each application. NOVA evaluators agreed with the SRG codes for nearly three-quarters of applications.

The goal of the analysis of applications rated as acceptable exempt by peer reviewers was to determine whether peer reviewers and applicants understood and applied exemptions in accordance with human subjects protection regulations. The analysis focused on the four exemptions most commonly assigned to NIH research grant applications. NOVA evaluators reviewed 95 applications rated by SRGs as acceptable exempt human subjects research and determined which human subjects protection code they thought should have been applied to each application. NOVA staff selected a code different from what was originally assigned by the SRG for 67 applications (70.5%).

Task 4: Assess Adequacy of Documentation of Human Subjects Concerns in Summary Statements of Applications Coded as Unacceptable for Human Subjects—NOVA evaluators reviewed summary statements of 400 unawarded, unacceptable applications to determine compliance with NIH standards. Discussion status of each application was recorded (discussed/not discussed by SRG). For discussed applications, the SRG resume of discussion was reviewed for inclusion of an appropriate human subjects label (acceptable/unacceptable) and presence of a summary of human subjects concerns discussed. For applications not discussed by the SRG, individual reviewer critiques were reviewed to determine whether at least one reviewer identified a human subjects protection concern.

Results

  • Evaluation results allowed OER/OEP to gauge how well peer review was functioning in evaluating NIH applications proposing human subjects research.
  • More than 90 percent of summary statements were found to be compliant with NIH standards.
  • Among applications for which NOVA evaluators identified concerns, the most common concern related to general deficiencies in the Protection of Human Subjects section.