Abstract
Objectives. To evaluate whether the Operation Peacemaker Fellowship, an innovative firearm violence-prevention program implemented in Richmond, California, was associated with reductions in firearm and nonfirearm violence.
Methods. We compiled city- and jurisdiction-level quarterly counts of violent firearm and nonfirearm incidents from statewide records of deaths from and hospital visits for homicide and assault (2005–2016) and from nationwide crime records of homicides and aggravated assaults (1996–2015). We applied a generalization of the synthetic control method to compare observed patterns in firearm and nonfirearm violence after implementation of the program (June 2010) to those predicted in the absence of the program, using a weighted combination of comparison cities or jurisdictions.
Results. The program was associated with reductions in firearm violence (annually, 55% fewer deaths and hospital visits, 43% fewer crimes) but also unexpected increases in nonfirearm violence (annually, 16% more deaths and hospital visits, 3% more crimes). These associations were unlikely to be attributable to chance for all outcomes except nonfirearm homicides and assaults in crime data.
Conclusions. The Operation Peacemaker Fellowship may have been effective in reducing firearm violence in Richmond but may have increased nonfirearm violence.
Interpersonal violence is a major driver of population health and health disparities in the United States. In 2016, homicides and assaults caused approximately 20 000 deaths and 1.7 million injuries.1 Firearm-related violence is particularly concerning because it is highly fatal and disproportionately affects young, Black men.1 However, there are few community-based violence-prevention programs, firearms centered or otherwise, that are scientifically supported. Strategies such as Ceasefire2,3 and Cure Violence,4–6 which typically involve community mobilization, street outreach, and partnerships among frontline staff in police, probation, corrections, and social services sectors, have been tested in cities nationwide. Although they are promising,7,8 additional tools to combat community violence would be valuable. Identifying other effective interventions to address firearm violence is a top priority for public health and public policy researchers and practitioners.9,10
One novel program merits evaluation: the Operation Peacemaker Fellowship implemented in Richmond, California. Richmond is a racially/ethnically diverse city of approximately 100 000 residents in the San Francisco Bay Area. In the mid-2000s, it was one of the most violent cities in the nation, with a homicide rate of 46 per 100 000, versus 5 per 100 000 in similarly sized California cities.11,12 Facing mounting pressure from residents and community leaders, in 2007 the city council moved to create the Office of Neighborhood Safety (ONS) to reduce firearm violence.13 Initial activities included street-level conflict mediation and intensive mentoring for at-risk youths in “hotspot” neighborhoods, but after an uptick in homicides in 2009, ONS shifted focus to the 30 community-dwelling individuals that the police department believed were responsible for most of Richmond’s firearm crimes.
Versus other programs,2,4 ONS uniquely invited participation in an intensive 18-month fellowship (hereafter, “Operation Peacemaker”). The core components of Operation Peacemaker are individually tailored mentorship, 24-hour case management, cognitive behavioral therapy, internship opportunities, social service navigation, substance abuse treatment, excursions, and stipends up to $1000 per month for successful completion of specific goals set by the fellowship and ONS staff, including nonparticipation in firearm violence (a conditional cash transfer).14,15 Although no aspect of the program specifically targeted firearm availability, acquisition, or use, the program delivered a set of socioeconomic and behavioral interventions to prevent involvement in firearm-related criminal activity. Additional information on the program was published previously.15
Operation Peacemaker has received nationwide attention for its unique approach and apparent success; firearm homicides and assaults declined 18% from program implementation to 2015.11,12 A detailed process evaluation documented high program uptake and positive outcomes for participants, including improved access to services, higher quality of life, and low rates of deaths, injuries, and crime perpetration.14,15 However, to our knowledge, there are no quantitative studies of the population-level outcomes of the program that control for other factors that may have contributed to declines (e.g., improvements in economic climate and broader crime reductions). We addressed this gap using a quasiexperimental design and comprehensive health and crime data on Richmond and comparison cities to quantify the association between Operation Peacemaker and population-level reductions in violence. Although firearm violence was the focus of Operation Peacemaker, the program addressed fundamental determinants of all forms of violence (e.g., substance use); therefore, we assessed both firearm and nonfirearm violence.
We used a generalization of the synthetic control method16,17 (SCM) to compare observed postintervention patterns in firearm and nonfirearm violence in Richmond to those predicted in the absence of the program. We measured firearm and nonfirearm violence using city- or town-level statewide health records of deaths from and hospital visits for homicide and assault and jurisdiction-level nationwide crime records of homicides and aggravated assaults. For each outcome and source, we identified the weighted combination of comparison units, selected from all those available, whose outcomes optimally predicted outcomes in Richmond in the preintervention period. We then used this weighted combination to predict “counterfactual” postintervention outcomes in Richmond. This method is well-suited to situations involving 1 intervention unit, and many controls and may better approximate counterfactual postintervention outcomes than using any single control or an evenly weighted combination of controls. The approach controls unobserved confounders (e.g., poverty) by assuming that the weighted combination of controls that can best predict the preintervention trends will continue to predict those trends in the postintervention period. It also controls for secular trends that are common across places.
First, we used death records from the California Department of Public Health Vital Records and emergency department and inpatient hospitalization discharge records from California’s Office of Statewide Health Planning and Development. Records included all deaths and hospital visits statewide, by location of residence of the patient or decedent. We identified deaths from homicide and injuries from assault using International Classification of Diseases Versions, Ninth Revision and International Classification of Diseases Versions, Tenth Revision external cause of death or injury codes (see the Appendix, available as a supplement to the online version of this article at http://www.ajph.org). External cause of injury coding in California’s hospital discharge records is compulsory, has ongoing quality assurance, and is regarded as 100% complete.18 Previous research also indicates completeness and validity of homicide e-codes in death data.19
Second, we used nationwide crime records from the Return A Record Card Master Files and Supplemental Homicide Reports. Data are voluntarily reported by law enforcement agencies and compiled by the Federal Bureau of Investigation Uniform Crime Reports system. Records included aggravated assaults and homicides by jurisdiction or agency logging the crime and incident month. Although subject to variable reporting over the study period,20,21 these data capture incidents that do not appear in health data, for example, assaults or shootings that do not involve hospital visits or deaths. The health data include homicide and assault injuries not reported as crimes; thus, the 2 sources are complementary.
Because of limited data availability, we restricted health data analyses to 2005 through 2016 and crime data analyses to 1996 through 2017. We modeled outcomes as counts rather than rates, because for crime data, the geographic boundaries of law enforcement agencies do not always correspond to clear populations at risk; for health data, models of counts achieved better preintervention fit. For interpretability, after modeling, we converted results from counts to rates using Census-based denominators for the City of Richmond. We combined fatal and nonfatal outcomes, because fatal outcomes were too infrequent to assess separately. We aggregated counts to the quarterly level to balance capturing short-term variation with ensuring that measures were stable enough for accurate modeling.
We analyzed health data at the Census place level—the named cities and towns in which people reside. We analyzed crime data at the jurisdiction level. In health data, we restricted control cities to those with at least 5000 residents and 1 homicide and assault in the preintervention period. In crime data, we followed the data-cleaning procedures used by the Uniform Crime Reports system22 and restricted to control jurisdictions with complete reporting over the study period. Based on these restrictions, the final pool of control places included 625 and 630 towns and cities for firearm and nonfirearm violence in health data, respectively, and 641 and 753 jurisdictions for firearm and nonfirearm violence in crime data, respectively. Operation Peacemaker began intensive work with its first 21 participants in June 2010. We hypothesized that effects of the program would be immediate and designated the third quarter of 2010 as the intervention start time.
We used a generalization of SCM at the level of the quarter and city or quarter and jurisdiction level to predict postintervention patterns in firearm and nonfirearm violence in Richmond in the absence of Operation Peacemlaker.16,17 SCM has been used in a variety of recent applications to study the results of programs and policies implemented in a single geographic unit with a defined start date.23–25
In traditional SCM, the preintervention outcomes in the treated unit are modeled as a function of the preintervention outcomes in candidate control units, with model coefficients (“weights”) constrained to be nonnegative and sum to 1 with no intercept. Fitted weights are those that optimally predict the treated unit outcomes in the preintervention period. These weights are then used to construct a “synthetic” or predicted outcome series for the treated unit in the pre- and postintervention periods from the optimally weighted combination of control unit outcomes. Good alignment between observed and predicted outcomes for the treated unit in the preintervention period indicates that the weighted combination of control units is effectively predicting outcomes in the treated unit. The key assumption for this approach is that the relationship between intervention and control units in the preintervention period continues in the postintervention period.17
Doudchenko and Imbens generalized SCM by relaxing the constraints of no intercept and weights that are nonnegative and sum to 1.17 This added flexibility can achieve better preintervention fit while retaining the strengths of SCM. Adding flexibility increases the risk of overfitting, because weights can take any positive or negative values to fit the preintervention data. Elastic net combines a linear model with penalties on the number and size of the weights to reduce the likelihood of overfitting. Per Doudchenko and Imbens,17 we used leave-one-out cross-validation, performed strictly on preintervention control units, to select the penalty parameters. On occasion, resulting models were still overfit, as indicated by the large number (≥ 600) of nonzero weights. When this occurred, we selected the penalty parameters that achieved the lowest in-sample pretreatment model error while also being reasonably parsimonious (defined as < 100 nonzero weights).
We summarized associations by comparing average annual counts and rates of homicide and assault observed in the postintervention period to those predicted in the postintervention period in the absence of the program. We conducted statistical inference using placebo tests16,24: we modeled each control unit, in turn, as the intervention unit and quantified the association between Operation Peacemaker and the outcomes. Because Operation Peacemaker did not occur in control units, this procedure provides a distribution of measured associations that are likely owing to chance. The proportion of control units with associations more extreme than that estimated for the intervention unit is a measure of whether the association may be owing to chance (the “Placebo tests” section in the Appendix provides details).
In models of health outcomes, we included sociodemographic covariates predictive of violence, including poverty, education, income inequality, household composition, housing costs, neighborhood characteristics (e.g., civic engagement), job availability, and unemployment (the “Supplemental methods” section in the Appendix provides details). Models of crime data did not include covariates, because the geographic boundaries of law enforcement agencies are not always well defined, making it difficult to assign appropriate place-level covariates. Previous research has also found that excellent preintervention fit is possible without covariates.17,23 As a sensitivity analysis, models of health outcomes without covariates are presented in the Appendix, section “Sensitivity analyses using health data without covariates.” Additionally, because other violence-prevention efforts (e.g., Ceasefire) scaled up in 2012, we tested the sensitivity of our results to restricting the postintervention period to July 2010 through December 2011. Finally, to test the sensitivity of our results to the choice of analytic methods, we tested a differences-in-differences approach26 (Appendix section “Sensitivity analyses in health data using differences-in-differences design” provides details and results).
We conducted the statistical analyses using R version 3.2.4 (R Foundation for Statistical Computing, Vienna, Austria).
Figure 1 presents trends in observed and predicted firearm-related homicides and assaults using health and crime data, by quarter, before and after implementation of Operation Peacemaker. Figure 2 presents the nonfirearm outcomes. In the preintervention period, predicted outcomes constructed from optimally weighted combinations of control cities or jurisdictions aligned with the observed outcomes. The model fit was better for health data than crime data.
FIGURE 1— Observed and Predicted Firearm Homicides and Assaults, by Quarter, Before and After Implementation of the Operation Peacemaker Fellowship, From (a) Health Data and (b) Crime Data: Richmond, CA, 1996–2016
Note. The vertical dotted line indicates the initiation of the Operation Peacemaker Fellowship in June 2010. Health data were available through 2016; crime data were available through 2015.
FIGURE 2— Observed and Predicted Nonfirearm Homicides and Assaults, by Quarter, Before and After Implementation of the Operation Peacemaker Fellowship, From (a) Health Data and (b) Crime Data: Richmond, CA, 1996–2016
Note. The vertical dotted line indicates the initiation of the Operation Peacemaker Fellowship in June 2010. Health data were available through 2016; crime data were available through 2015.
In the postintervention period, comparing observed outcomes to those predicted in the absence of the program indicated that Operation Peacemaker was associated with reductions in firearm homicides and assaults but increases in nonfirearm homicides and assaults. Specifically, after implementation, the program was associated with 55% and 43% fewer firearm homicides and assaults annually in health and crime data, respectively (Table 1). During the same periods, the program was associated with 16% and 3% more nonfirearm homicides and assaults in health and crime data, respectively (Table 1). Placebo tests (Table 1; Appendix Table A; Appendix Figures A–D) indicated that these associations were unlikely to be owing to chance for all outcomes except nonfirearm homicides and assaults in crime data. The program was associated with increases in nonfirearm homicides and assaults in crime data in the first 3.5 postintervention years but reductions thereafter (Figure 2). Post hoc analyses considering 2 separate postintervention periods (July 2010–December 2013 and January 2014–December 2015) to test the associations before and after this reversal still suggested these associations were likely owing to chance (Appendix Table F; cities or jurisdictions assigned nonzero weights in the final analysis varied by outcome and are presented in Appendix Tables B–E).
TABLE 1— Summary of Generalized Synthetic Control Results for the Association of the Operation Peacemaker Fellowship With Firearm and Nonfirearm Homicides and Assaults: Richmond, California, 1996–2016
| Health Data | Crime Data | |||
| Firearm | Nonfirearm | Firearm | Nonfirearm | |
| Richmond (observed) | ||||
| Average annual cases in postintervention period | 85 | 932 | 156 | 900 |
| Rate in postintervention period, per 100 000 | 79 | 859 | 145 | 834 |
| Synthetic Richmonda | ||||
| Average annual cases in postintervention period | 191 | 806 | 273 | 870 |
| Rate in postintervention period (per 100 000) | 176 | 742 | 253 | 806 |
| % difference in cases between Richmond and Synthetic Richmonda | −55 | +16 | −43 | +3 |
| Statistical inferenceb | 0.00 | 0.02 | 0.04 | 0.60 |
Note. The preintervention period was January 2005–June 2010 for health data and January 1996–June 2010 for crime data. The postintervention period was July 2010–December 2016 for health data and July 2010–December 2015 for crime data.
a Predicted in absence of Operation Peacemaker.
b Proportion of control units with outcomes more extreme than Richmond’s.
Restricting the postintervention period to just those quarters before scale-up of other violence-prevention efforts (e.g., Ceasefire) in 2012 did not meaningfully alter the results (Appendix Table G). Removing sociodemographic covariates in analyses of health data made the associations for both firearm and nonfirearm homicides and assaults somewhat weaker but did not substantively change the results (Appendix Figure E; Appendix Table H). Results from the differences-in-differences analysis were also consistent with the main results (Appendix Table I).
To our knowledge, this is the first quasiexperimental study to examine the association of the Operation Peacemaker Fellowship, a novel firearm violence-prevention program, with population-level homicide- and assault-related crimes, deaths, and injuries. We found that the program was associated with significant reductions in firearm violence but possible increases in nonfirearm violence. Alternative explanations for our findings, aside from a causal effect of Operation Peacemaker on firearm and nonfirearm violence, must be considered. Phenomena that affect all units in the analysis equally are controlled by design. For example, to the extent that they affect all places equally, changes in California-wide youth programs, firearm availability, or firearm laws are controlled in health data analysis, and nationwide effects of the Great Recession are controlled in both health and crime analyses.
However, we cannot disentangle the possible effects of Operation Peacemaker from other simultaneous programs or changes specific to Richmond. Firearm violence started declining before Operation Peacemaker, and there are likely many reasons for these declines. For example, ONS opened its doors in 2007 and became fully operational in 2008; the police department changed leadership in 2006 and reorganized their special investigation unit to focus on arresting those responsible for firearm crimes; in 2007, the Rising Youth for Social Equity Center, a key community organization in the city, was founded to provide a safe space for youths affected by firearm and nonfirearm violence in the city; various grassroots antifirearm violence campaigns were initiated by Richmond residents and community leaders beginning in the mid-2000s; and one of these campaigns led to the implementation of Operation Ceasefire, with active call-ins beginning in 2012.27
However, we think the measured associations are most likely attributable to the program, because other major violence-related changes were offset in time from Operation Peacemaker and the nature and intensity of the program was unique. An ongoing ethnographic study of firearm violence in Richmond suggests that between mid-2010 and 2012, Operation Peacemaker appears to have been the only organization providing intensive support services to those actively involved with or most at risk for firearm violence. These individuals were also targeted by police in the years before Operation Peacemaker and by Ceasefire in subsequent years, but the timing of Operation Peacemaker was distinctive, and no other program provided the same level of case management and opportunities (e.g., stipends).
Analyses restricted to this period showed results consistent with those of the main analysis. Furthermore, a previous process evaluation documented that Operation Peacemaker succeeded in deeply engaging and affecting participants in meaningful ways.14,15 By contrast, previous evidence on Ceasefire and community policing has shown less substantial results.2,3,28 However, it is still possible that other less well-documented changes, such as local changes in firearm availability, or the effects of ongoing programs or program enhancements, may have coincided with Operation Peacemaker and contributed to declines. Future research on the timing, content, and funding levels of the various programs and on population subgroups most likely to be affected by particular programs (e.g., youths) may help to disentangle their effects.
In the health data, the program was associated with increases in nonfirearm homicides and assaults. These increases are corroborated by forthcoming qualitative work documenting reports by residents, community leaders, and law enforcement that crimes not involving firearms, such as violent robberies and illicit drug transactions, have persisted or increased during the postintervention period.27 Our crime data provide little insight into this shift. However, a post hoc examination of our health data revealed that, throughout the study period, most nonfirearm violent victimization occurred among Black and Hispanic men aged 15 to 29 years residing in the neighborhoods with the most gang violence27 and that, after Operation Peacemaker, the composition of nonfirearm violence shifted slightly away from deaths and hospitalizations and toward emergency department visits.
We propose several possible post hoc explanations. First, removing key players in firearm violence from active participation may have inadvertently generated violent activity in the face of a power void. One previous study documented similar patterns of increasing violence following drug-related arrests.29 Second, the emphasis on firearm violence over the past decade may have reduced local organization and law enforcement efforts to suppress other types of violence. Third, the program may have induced changes in the nature of violence, such as substitution of firearms for other weapons or bodily force.
Although explicit investigation of firearm-carrying is needed, reports from ONS staff and youths previously involved with gun carrying suggest that as the risk of being shot decreased, the perceived need to carry or use a gun also decreased.27 Thus, altercations or retaliations may have been increasingly pursued, because they were less likely to be fatal. Such altercations may also produce more total injuries, because nonfirearm interactions may occur at closer range and harm more individuals, as opposed to long-range shootings. Some research suggests that the availability of firearms drives the fatality of violent encounters but not the overall amount of violence.30 Substitution effects in the opposite direction—that greater community violence was associated with shifts from nonfirearm to firearm violence—have been documented previously.31 That the trends in firearm and nonfirearm and the composition of control cities for firearm versus nonfirearm violence differed substantially (e.g., Appendix Table B vs C) suggests that firearm and nonfirearm violence may have distinct combinations of determinants. There may also be multiple solutions to optimizing preperiod fit. Further research should examine the dynamics that might underlie these possibilities.
Results from health and crime data showed some differences. Crime data showed larger declines in firearm violence, whereas health data showed larger increases in nonfirearm violence. These differences may be attributable to differences in the types of incidents captured by each data source, the weights assigned to comparison cities used to construct the synthetic control, the quality of the preintervention model fit, or variability in reporting practices across places and times, particularly for crime data.20,21 Reported nonfirearm crimes in Richmond dropped dramatically between 2013 and 2014. This pattern was not observed in control jurisdictions or the synthetic control and may reflect other violence-prevention efforts or unknown changes in reporting. In addition, although the preintervention model fit for crime data was good and consistent with that in the synthetic control literature,16,17,23,24,32 the model fit was exceptionally good for the health data, likely reflecting the large pool of California control cities or towns of varying sizes with irregular trends similar to those of Richmond. By contrast, the nationwide control jurisdictions from crime data were generally larger populations with more stable rates, thus resulting in a less flexible Synthetic Richmond. These differences in control cities or jurisdictions and model fit may imply that results from health data are slightly more accurate.
There were some limitations to our study. First, all nonexperimental studies are at risk for residual confounding. We minimized potential bias by using the program’s well-defined start time and outcomes in comparison cities to control for both unmeasured time-varying violence risk factors and secular trends that are common across cities or jurisdictions. Second, our approach assumes that similar violence-prevention efforts did not happen systematically in control cities simultaneously with Operation Peacemaker. To our knowledge, no other cities implemented similar unique interventions, but other cities may have implemented elements of the program during the postintervention period. If any such cities were weighted in the synthetic control, the measured association would be biased toward the null. Third, health data are indexed to the patient or decedent’s residence not the injury location; some misclassification resulting in bias is possible.
Finally, our analytic approach assumed that the relationship between intervention and control units did not change between the preintervention and postintervention periods and that other factors affecting the outcomes did not change in Richmond concurrent with Operation Peacemaker. We think that these assumptions are reasonable, but violations are possible. For example, shifts in crime reporting or demographics may have occurred in the postintervention period that could have changed the relationship between Richmond and control units.
This study adds to the scant literature on community-based violence-prevention programs and provides more definitive evidence on the effectiveness of Operation Peacemaker in reducing firearm violence. Alternative firearm violence-prevention approaches may include federal-, state-, or local-level regulation.33 Although Operation Peacemaker may have reduced firearm violence, the cooccurring increase in nonfirearm violence raises concerns and should be investigated. Future research should also consider which components of this multifaceted program, or synergies between components, have the most effect. Replications are being conducted in other cities nationally and internationally. Implementers should monitor increases in nonfirearm violence and evaluators have the opportunity to assess this prevention model in other settings.
See also Galea and Vaughan, p.
ACKNOWLEDGMENTS
The authors acknowledge the following funding sources: Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health (NIH), Office of the Director (grant DP2HD080350), University of California Firearm Violence Research Center, and the University of California, Berkeley Committee on Research.
Note. The analyses, interpretations, and conclusions of this article are attributable to the authors and not to the California Department of Public Health or the NIH.
CONFLICTS OF INTEREST
The authors have no conflicts of interest to declare.
HUMAN PARTICIPANT PROTECTION
This study was approved by the California Health and Human Services Agency and University of California, Berkeley Committees for the Protection of Human Subjects.
