Abstract
Public health policy has a profound impact on health status. Missing from the literature is a clear articulation of the definition of evidence-based policy and approaches to move the field forward. Policy-relevant evidence includes both quantitative (e.g., epidemiological) and qualitative information (e.g., narrative accounts).
We describe 3 key domains of evidence-based policy: (1) process, to understand approaches to enhance the likelihood of policy adoption; (2) content, to identify specific policy elements that are likely to be effective; and (3) outcomes, to document the potential impact of policy.
Actions to further evidence-based policy include preparing and communicating data more effectively, using existing analytic tools more effectively, conducting policy surveillance, and tracking outcomes with different types of evidence.
IT HAS LONG BEEN KNOWN that public health policy, in the form of laws, regulations, and guidelines, has a profound effect on health status. For example, in a review of the 10 great public health achievements of the 20th century,1 each of them was influenced by policy change such as seat belt laws or regulations governing permissible workplace exposures. As with any decision-making process in public health practice, formulation of health policies is complex and depends on a variety of scientific, economic, social, and political forces.2
There is a considerable gap between what research shows is effective and the policies that are enacted and enforced. The definition of policy is often broad, including laws, regulations, and judicial decrees as well as agency guidelines and budget priorities.2–4 In a systematic search of “model” public health laws (i.e., a public health law or private policy that is publicly recommended by at least 1 organization for adoption by government bodies or by specified private entities), Hartsfield et al.5 identified 107 model public health laws, covering 16 topics. The most common model laws were for tobacco control, injury prevention, and school health, whereas the least commonly covered topics included hearing, heart disease prevention, public health infrastructure, and rabies control. In only 6.5% of the model laws did the sponsors provide details showing that the law was based on scientific information (e.g., research-based guidelines).
Research is most likely to influence policy development through an extended process of communication and interaction.6 In part, the research–policy interface is made more complex by the nature of scientific information, which is often vast, uneven in quality, and inaccessible to policymakers. Several models for how research influences policymaking have been described,7–9 most of which involve moving beyond a simple linear model to more nuanced and indirect routes of influence, as in gradual “enlightenment.”10 Such nonlinear models of policymaking and decision-making take into consideration that research evidence may hold equal, or even less importance, than other factors that ultimately influence policy, such as policymakers' values and competing sources of information, including anecdotes and personal experience.11 Although not exhaustive, Table 1 highlights several important barriers that should be considered when one is attempting to develop effective policy.12–16
TABLE 1 Barriers to Implementing Effective Public Health Policy
| Barrier | Example |
| Lack of value placed on prevention | Only a small percentage of the annual US health care budget is allocated to population-wide approaches. |
| Insufficient evidence base | The scientific evidence on effectiveness of some interventions is lacking or the evidence is changing over time. |
| Mismatched time horizons | Election cycles, policy processes, and research time often do not match well. |
| Power of vested interests | Certain unhealthy interests (e.g., tobacco, asbestos) hold disproportionate influence. |
| Researchers isolated from the policy process | The lack of personal contact between researchers and policymakers can lead to lack of progress, and researchers do not see it as their responsibility to think through the policy implications of their work. |
| Policymaking process can be complex and messy | Evidence-based policy occurs in complex systems and social psychology suggests that decision-makers often rely on habit, stereotypes, and cultural norms for the vast majority of decisions. |
| Individuals in any one discipline may not understand the policymaking process as a whole | Transdisciplinary approaches are more likely to bring all of the necessary skills to the table. |
| Practitioners lack the skills to influence evidence-based policy | Much of the formal training in public health (e.g., masters of public health training) contains insufficient emphasis on policy-related competencies. |
Although there have been many calls for more systematic and evidence-based approaches to policy development,5,6,17–21 missing from the literature is a clear articulation of the definition of evidence-based policy along with specific approaches that will enhance the use of evidence in policymaking.
Policy change involves both science and art and, therefore, evidence for policymaking can take several forms. The concept of evidence often originates from legal settings in Western soceties. In law, evidence comes in the form of stories, witness accounts, police testimony, expert opinions, and forensic science.22 For policy-relevant evidence, both quantitative data (e.g., epidemiological) and qualitative information (e.g., narrative accounts) are important.
Although the use of research-derived evidence may be a key feature of most policy models,7,9,23 it is not a certainty that scientific evidence will carry as much weight in “real world” policymaking settings as other types of evidence. Policymakers operate on a different hierarchy of evidence than scientists,17 leaving the 2 groups to live in so-called parallel universes.14 According to interviews with policymakers, many respondents reported that they were not trained to distinguish between good and bad data, and were, therefore, prone to the influence of misused “facts” often presented by interest groups.24 Similarly, McDonough reported that in policy debates in state legislatures, data were used as “rhetorical weapons used to bolster competing values.”25(p210) Because numbers exert a powerful and widespread influence on policy debates, they can lose their objective meaning.26
Quantitative evidence for policymaking (i.e., data in numerical quantities) can take many forms, ranging from scientific information in peer-reviewed journals, to data from public health surveillance systems, to evaluations of individual programs or policies.27,28 Many consider the strongest evidence to be that from systematic reviews (e.g., the Guide to Community Preventive Services29 or the Cochrane Reviews30), which sum up the results of primary scientific studies that meet explicit criteria (i.e., decision rules). Using data from reviews of public health laws, Moulton et al. searched the English language literature over the past 5 years. They identified 65 systematic reviews and found that of 52 public health laws, 27 were found effective, 23 had insufficient evidence to judge effectiveness, 1 was harmful, and 1 was found to be ineffective.31 Yet single studies and evaluations are more commonly used to support policy than are systematic reviews,32,33 in part because of the time and expense of conducting a systematic review or the insufficient number or quality of studies on a particular topic.
Other quantitative data can be collected from policymakers themselves to provide general guidance on policy approaches and strategic information on specific public health issues. For example, in a survey of 292 US state policymakers,34 respondents expressed a strong preference for short, easy-to-digest data. Younger respondents were more likely to use electronic information than were older policymakers. The most trusted sources of information were those not having a stake in the outcome and those providing state-by-state comparisons. Surveys of policymakers can provide useful data on priorities and obstacles for specific health issues,35,36 attitudes and voting intentions,37 and perceptions of lobbyists and lobbying.38
Qualitative evidence involves nonnumerical observations, collected by methods such as participant observation, group interviews, or focus groups. Qualitative evidence can make use of the narrative form as a powerful means of influencing policy deliberations, setting priorities, and proposing policy solutions by telling persuasive stories that have an emotional hook and intuitive appeal. This often provides an anchor for statistical evidence, which, in turn, offers the powerful persuasive impact of the law of large numbers, in addition to being verifiable and having high credibility.39 In studying the impact of evidence on policy to address health disparities, qualitative data, such as the effects of policy initiatives on children and families, has been persuasive and powerful in shaping the agenda.40
The incorporation of quantitative evidence within a compelling story can provide a powerful lever in the policy process. Studies from the communication field have examined the effectiveness of using statistical data versus stories for persuasion. These have shown that, although quantitative evidence alone more frequently has a stronger persuasive effect than qualitative evidence alone,41 the combination of the 2 types of evidence appears to have a stronger persuasive impact than either type of evidence alone.42
Governmental policy systems vary widely in their structure and scope, ranging from totalitarian to democratic governments. We focused the descriptions of evidence-based policy on multicentric (democratic) governments. Whether at a local, state, or federal level, the purpose of a representative body is to enact rules, laws, or ordinances that are in turn implemented by executive or administrative agents. We focused primarily on “big P” policies (e.g., formal laws, rules, regulations enacted by elected officials) as contrasted with “small p” policies (e.g., organizational guidelines, internal agency decisions or memoranda, social norms guiding behavior).43,44
Evidence-based policymaking has largely been an incremental progression.45 For example, tobacco control advocates have long sought comprehensive restrictions on tobacco use, access, and secondhand smoke exposure. However, public policies on these topics were developed over decades. In 1987, the US House of Representatives banned smoking on domestic flights of 2 hours or less; in 1992, Congress passed the Synar Amendment46 requiring states to adopt and enforce restrictions on tobacco sales to minors; in 1996, the US Food and Drug Administration published a final rule that restricted youth access to tobacco products (which was later overturned by the US Supreme Court)47,48; and in the past decade and a half, states and municipal governments have been extremely active in developing laws and regulations aimed at smoke-free worksites and public places.49–51
We propose that evidence-based policy can be conceptualized as a continuum spanning 3 domains—process, content, and outcome (Table 2). Furthermore, as discussed earlier, there is no single, “best” type of evidence.
TABLE 2 Domains of Evidence-Based Public Health Policy
| Domain | Objective | Data Sources | Example |
| Process | To understand approaches to enhance the likelihood of policy adoption | Key informant interviews | Understanding the lessons learned from different approaches and key players involved in state health reforms. |
| Case studies | |||
| Surveys of setting-specific political contexts | |||
| Content | To identify specific policy elements that are likely to be effective | Systematic reviews | Developing model laws on tobacco that make use of decades of research on the impacts of policy on tobacco use. |
| Content analyses | |||
| Outcome | To document the potential impact of policy | Surveillance systems | Tracking changes in rates of self-reported seat belt use in relation to the passage of seat belt laws. |
| Natural experiments tracking policy-related endpoints | |||
| Describing the cost-effectiveness of child immunization requirements. |
Recognizing and identifying key factors that inform the policy process is also critical to furthering evidence-based policy. Policymaking is complicated and the factors that inhibit or facilitate the process are equally complex.17 There are very distinct stages or “streams” as Kingdon noted that, when coupled together, increase the odds of a policy being adopted. The first stream is the problem––agenda setting and how certain problems or conditions come to be regarded as problems worthy of governmental intervention. The second stream is policy––the alternative policy approaches that may be taken to address those problems.4 The third stream, politics, recognizes those factors both inside and outside government that influence the policymaking process. Public policies must be not only “technically sound, but also politically and administratively feasible.”52(p311)
Documenting influential political factors can comprise an evidence base for the process domain of evidence-based policy. Factors such as the national mood, organized political forces (e.g., interest groups, lobbyists), changes in governmental participants such as legislative or administrative turnover, jurisdictional boundaries or turf “wars” between governmental agencies, and the necessity of compromise or bargaining all affect the policy process.4,53 According to numerous accounts, one of the reasons that health care reform legislation failed in the early to mid-1990s was a failure to understand the politics involved with policymaking.4,53 Furthermore, the approach taken by the executive branch at the time was very much a rational, comprehensive approach to policymaking that sought to identify all of the possible alternatives, weigh the costs and benefits of each alternative, and choose the best approach among the alternatives. Proponents of that approach failed to learn from the lessons of past health care reform efforts, which indicated the need for an incremental approach to change rather than aiming for an all or nothing strategy.4,53,54
Many factors affect the policymaking process, including successful advocacy. In the study of social movements, progress hinges on the standing of those articulating an issue and the presence of a policy “sparkplug.”55,56 Case studies show that policy entrepreneurs or champions (i.e., leaders from professional, political, or interest groups who effectively advocate policy) have played key roles in policy reforms, including making major reforms in the historically intractable arena of Chicago, Illinois, public schools57 and the rise of managed care as a dominating force in the US health care system.58 From the health policy field, several authors20,59–62 describe successful examples of evidence-based advocacy and provide a guide for those who seek to move from research to advocacy. A strong partnership of researchers and advocates ensures that policies are grounded in science, so messages are tailored to the target audience, and models of persuasive communication (e.g., social marketing) are applied.
Policy content focuses on identifying the specific policy elements that are likely to be effective. As noted earlier, both quantitative and qualitative data can be used by policymakers to determine the appropriate policy intervention. Such information may be gleaned from systematic reviews and other scientific research including content analyses that offer an evidence base to inform decisionmaking.29 For example, the Institute of Medicine has recognized that trans fatty acids (trans fats) do not provide any known health benefit and that they are linked to coronary heart disease.63 One way to reduce or eliminate exposure to trans fats is to regulate their inclusion in food products.64 As of August 2008, 7 US cities or counties had limited or restricted trans fats65,66 and California is the first state to enact a law banning trans fats.66
We need to better understand and describe evidence-based elements within existing or proposed policy. For example, in examining 6.5 years of state legislation on physical education, Eyler67 used systematic reviews and national standards to identify 4 specific bill components that are scientifically supported (i.e., minutes in physical education, physical education activity, teacher certification, and an environmental element including facilities and equipment).68–70 Eyler conducted a content analysis and found that 28% of state laws had at least 1 evidence-based element yet only 0.5% had all 4 evidence-based elements.67
Documenting the effects of implemented policies (policy outcome) is equally important in supporting evidence-based policy. Policy evaluations are critical to the understanding of the impact of policies on community- and individual-level behavior changes. They should include “upstream” (e.g., presence of zoning policies supportive of physical activity), “midstream” (e.g., the enrollment in walking clubs), and “downstream” (e.g., the rate of physical activity) factors.71 By far, the majority of quantitative measures are available for downstream outcomes.71 One evaluation framework, the RE-AIM framework,72 can be applied to evaluations of a policy and its impact.73 RE-AIM has 5 dimensions: (1) reach (who or how many will be affected by the policy), (2) effectiveness (proximal or distal impacts and unintended consequences), (3) adoption (policy diffusion and participation level), (4) implementation (costs as well as enforcement and compliance), and (5) maintenance (institutionalizing the policy or program).
Policy evaluations may employ both qualitative and quantitative methodologies and may make use of “natural experiments” surrounding the adoption and implementation of the policy. These evaluations involve naturally occurring circumstances where different populations are exposed or not exposed to a potentially causal factor (e.g., a new policy) such that it resembles a true experiment in which study participants are assigned to exposed and unexposed groups. For example, scientific evidence led to the change in polio vaccination policy in the United States from that of an entirely oral poliovirus vaccine prior to 1997, to a schedule of inactivated poliovirus vaccine followed by oral poliovirus vaccine in 1997 through 1999, to a schedule of entirely inactivated poliovirus vaccine in 2000. When the policy changed from oral poliovirus vaccine only to inactivated poliovirus vaccine followed by oral poliovirus vaccine, the mean number of cases of vaccine-associated paralytic poliomyelitis declined by 54%. Since the conversion to an entirely inactivated poliovirus vaccine schedule in 2000, no cases of vaccine-associated paralytic poliomyelitis have been reported in the United States.74
Making public policy is a continuous or recursive process that relies heavily upon scientific evidence and other influences (Figure 1).75,76 Scientific evidence that examines the impact of public policies on systems and individual-level behavior change is one possible source of feedback. “Policy reinvention” is another step in the feedback loop—i.e., policies evolve as they diffuse.77 The development and diffusion of smoke-free air laws in states and municipalities in the United States began with restricting smoking to designated areas, followed by restricting smoking to separately ventilated and separately enclosed areas, and, in the past several years, to complete bans on smoking.50,51
FIGURE 1 The interplay of factors influencing evidence-based public health policy.
In addition, how the policy issue is framed is as important as the process and content. As Jewell and Bero recently noted, research-based evidence
must be packaged to incite and persuade, to translate that knowledge into something that is understandable by the average legislator, average citizen.24(p196)
For example, efforts to ban or restrict the content of vending machines in schools did not mobilize until the issue was framed in a way such that vending machines were considered a vector for risk factors and behaviors that may be linked to obesity (such as the relationship between soft drink consumption and childhood overweight and obesity).24 The feedback and framing processes are enhanced by partnerships among researchers, practitioners, and policymakers.
This short review of the nature of evidence-based policy sets the stage for 6 issues that will help in advancing the field, organized across the 3 domains of process, content, and outcome. Although the list that follows is not exhaustive, it involves the themes that we believe are among the highest priorities. Building on our review and past literature,32,33,78 we propose the following working definition: to improve public health outcomes, evidence-based policy is developed through a continuous process that uses the best available quantitative and qualitative evidence.
Seneca, a philosopher in ancient Rome, noted that “luck is what happens when preparation meets opportunity.”79 This notion applies to modern-day policymaking. Luck and timing undoubtedly play important roles in policy success and we know that scientific studies are not always conducted at the right time to influence policy decisions.80,81 For success in the process, one often needs to proactively analyze and assemble data so that evidence is ready when a policy window or opportunity emerges.82,83
Although it is well established that data can be powerful in shaping policy decisions,84 they are sometimes not in the form most useful for policymakers.85 One study of long-term care policymaking indicated “loud and clear, that academic-quality research is not reaching” policymakers.86(p320) For example, surveillance data often provide disease or risk factor data at the national or state level, yet these reports can be lengthy and are often lacking local-level data on health disparities.87 Data need to be in a form that (1) shows public health burden, (2) demonstrates priority of an issue over many others, (3) shows relevance at the local (voting district) level, (4) shows benefits (or sometimes harms88) from an intervention, (5) personalizes an issue by telling a compelling story of how peoples' lives are affected,14 and (6) estimates the cost of intervention. Such data need to be presented in short and concise formats (e.g., issue briefs) that directly address the issue that is being debated.
On the basis of credible evaluations, it is possible to sift apart the “active ingredients” of various policy interventions (i.e., the essential elements that contribute to effectiveness). Thus, the content of legislation can be developed based on the key elements that are likely to have the greatest public health impact, balancing effectiveness and population impact. This is the concept underpinning model legislation, yet even when model language exists, it is often lacking a scientific basis or has not been widely tested.5
A diverse and rich set of tools puts information at the fingertips to shape the content of evidence-based policy. These tools include meta-analysis, decision analysis, cost-effectiveness analysis, and simulation modeling; all are underutilized.89,90 The existence of the tool alone is not enough; often training and technical assistance are needed to enhance uptake among potential users.
To examine the adoption, implementation, and impact of evidence-based policy, we need systems in place to help us monitor patterns and trends in policies.91–93 A few early efforts are underway to develop public health policy surveillance systems. For example, a group of federal and volunteer-based agencies have developed policy surveillance systems for tobacco, alcohol, and, more recently, school-based nutrition and physical education.94–97
Evidence comes in numerous forms. It may be helpful to consider policy evidence in a typology rather than a hierarchy because adherence to a strict hierarchy of study designs may reinforce an “inverse evidence law” by which interventions most likely to influence whole populations (e.g., policy change) are least valued in an evidence matrix emphasizing randomized designs.98,99 In addition, policy outcomes can be monitored with triangulated methods (accumulation of evidence from a variety of sources to gain insight into a particular topic, often combining quantitative and qualitative data) to understand content and track progress. Successful monitoring of outcomes will also require sources beyond the usual public health data sets (e.g., tax revenue data, polling data, marketing information).
Policy has had, and will continue to have, a vast impact on our daily lives and on public health indicators in part because of its long-term effects and relative low cost. Many of the public health programs now being implemented have a significant focus on policy change. To improve these programs and to further evidence-based policy, we need to use the best available evidence and expand the role of researchers and practitioners to communicate evidence packaged appropriately for various policy audiences; to understand and engage all 3 streams (problem, policy, politics) to implement an evidence-based policy process; to develop content based on specific policy elements that are most likely to be effective; and to document outcomes to improve, expand, or terminate policy.
Governments spend significant sums on health-related research (about $30 billion annually in the United States) with the implied obligation that this investment will improve the health of the public. Better application of the tenets of evidence-based policy is likely to accelerate this improvement.
Acknowledgments
This work was funded through the National Institutes of Health (grant NCI 1R01CA124404-01) and the Centers for Disease Control and Prevention, Prevention Research Centers Program (contract U48/DP000060).
Human Participant Protection
No approval was required.
