Funders of programs in public health and community health are increasingly concerned about the sustainability of changes they initiate. Despite a recent increase in sustainability research and evaluation, this literature has not developed a widely used paradigm for conducting research that can accumulate into generalizable findings. We provide guidance for research and evaluation of health program sustainability, including definitions and types of sustainability, specifications and measurements of dependent variables, definitions of independent variables or factors that influence sustainability, and suggestions for designs for research and data collection. We suggest viewing sustainability research as a further stage in the translation or dissemination of research-based interventions into practice. This perspective emphasizes ongoing relationships with earlier stages of a broader diffusion framework, including adoption and implementation processes.

Research and evaluation of health-related interventions usually end along with the external funding, whether the research is testing the efficacy of a new health promotion intervention or assessing the effectiveness of a broader community-oriented program. For several decades, researchers of health, governmental, and education programs have been exploring research questions concerning what happens in adopting organizations and their communities after external funding stops.1–7 Are the tested interventions continued? Are organizational variables—or other variables that facilitate fidelity of implementation—predictive of longer-term sustainability? How do factors influencing sustainability relate to other phases in a broader change process, including the adoption and implementation of efficacious interventions? What are the public policy and ethical implications if funded and effective public health interventions are not sustained?

Despite long-term interest in issues of program sustainability, sustainability research has not coalesced into a widely used set of research questions, operational definitions and procedures, or a research paradigm. A coherent paradigm would enable results from this diverse body of research to accumulate into well-founded research findings and recommendations for public health, contributing to a movement toward creating a broader science of global sustainability.8 In the absence of guidance for conducting program sustainability research and evaluation, individual studies tend to develop anew their own definitions, variables to include, and methods for data collection and analysis. We are frequently reinventing the wheel in this area of research because researchers from diverse areas of specialized content (e.g., heart health, substance abuse prevention, HIV prevention programs) do not know what sustainability research has been done in other topical fields. Their fields of study may be substantively different, but those fields often use similar processes to achieve adoption, implementation, and sustainability. Researchers from diverse fields need a coherent road map describing recommended research methods and suggesting ways to avoid pitfalls identified by previous researchers.

We drew from several sources to suggest an agenda for conducting improved research and evaluation on the sustainability of health care, health promotion, and disease prevention programs. One source of this guidance is the discussion generated by more than 50 participants in a session on sustainability research at the third annual Conference on the Science of Dissemination and Implementation, sponsored by the National Institutes of Health (NIH).9 This session discussed concepts, methods, and measures to understand and assess sustainability in the context of program life cycles, funder priorities, and the realities that face organizational implementers in communities. A second source for this article was our invited participation in a workshop concerning the sustainability of education programs in schools, funded by the National Science Foundation.10 A third input for this paper was our own research on sustainability and planning for the diffusion of health programs, as well as numerous conference sessions with program managers exploring their views about sustainability.7,11–13 Lastly, we drew on our experience with funding organizations that play important roles in setting the health promotion and disease prevention agendas that affect what occurs in communities.

We hope that this article will guide researchers and funders of public health programs in making explicit and transparent decisions about research methods and clearly reporting their methods in publications, thereby enabling accumulation of findings about sustainability. We focus primarily on situations in which a specific project has been funded or an intervention is the focus of a research grant, to encourage follow-up research and evaluation to assess the sustainability of those specific interventions. We use the terms “intervention” and “program” interchangeably. The suggested guidance may also be relevant to broader dissemination initiatives to foster the diffusion of evidence-based health programs within and across communities of practice, but we have not focused on research to tackle those larger issues of broader and sustained systemic change. We envision sustainability in relation to previous stages in an overall programmatic life cycle of intervention development, adoption, implementation (with potential iterative adaptations), and sustainability.7,14–17

Funders of innovative health care, disease prevention, and community health promotion programs want to know: are their investments leading to longer-term beneficial outcomes, or do they fade away after the seed money is spent? Sustainability research may be able to address those questions, as well as to suggest ways to promote and facilitate long-term sustainability. A further reason to study sustainability involves ethical concerns whenever researchers collect data involving nonresearcher partners in both clinical and community contexts. What are community partners left with if funding for serving their clients ends with the timeline of research? Is it ethical for researchers to build up an intervention in situ and then abandon it abruptly when the research funding ends, in light of the accumulated research-to-practice evidence showing that simple hand-offs frequently do not work?18,19 Is it ethical for funders to develop innovative programs but then expect others to sustain them if they prove effective? Finally, sustainability is itself an important research topic within the broader context of translational and dissemination research on the diffusion of effective programs. When we do the work of spreading and supporting effective programs across many public health settings, we need to know whether these programs and their beneficial effects are sustained.

It is important to note here that the term “sustainability” has various meanings in other content domains. For example, sustainability may refer to the economic viability of an income-generating program in a developing country, the growth and maintenance of a “green” economy, or the maintenance of a natural biological community in an ecological steady state.20 Although there might be some parallels among these various types of sustainability, we are confining this article to issues related to research and evaluation concerning the continuation or discontinuation of health programs in organizations, frequently for the benefit of community residents, that received funding from an external agency.

Sustainability is the continued use of program components and activities for the continued achievement of desirable program and population outcomes. Other terms that have been used by previous researchers in this domain include continuation, confirmation, maintenance, durability, continuance, and institutionalization. There are some nuanced differences among these terms, but they all usually refer to the continued use of program components and activities beyond their initial funding period and sometimes to continuation of desired intended outcomes; this is what we mean by sustainability. Generally speaking, the likelihood of sustainability is heightened when there is an alignment, compatibility, or convergence of (1) problem recognition in the external organizational environment or community, (2) the program in question, and (3) internal organizational objectives and capacities.2,6,21,22 This orientation implies a multilevel system of a health program implemented by individuals, embedded in an organization, that operates within a community context or interorganizational network over time.23 Therefore, research on sustainability can require several layers of data collection to capture the multiple components of the systems involved in such continuation.

Sustainability as Outcome Versus Process

Several sustainability researchers24,25 define sustainability as a set of processes that take place during the earlier stages of a life cycle of a project; these may include decisions during initial project planning or adoption, as well as organizational support and financial strategies during implementation. We agree that sustainability outcomes are likely to be affected by these processes during the earlier phases of a project, as well as by the environmental context, but we believe that a process definition of sustainability presents challenges for planning research or evaluation on this topic. Without explicit definitions of outcome variables, along with measures of hypothesized influences on those outcomes, research often cannot accumulate or disconfirm findings about predictors of sustainability.

An important point here concerns the timing of research on sustainability. If data collection does not extend to assessing the actual continuation of program activities or outcomes beyond the initial implementation stage, the researcher cannot know whether those outcomes do, in fact, happen. Therefore, defining the term sustainability in terms of processes that facilitate these outcomes might encourage studies that do not extend far enough in time to assess actual continuation. An example is the common terminology for translational research, “dissemination and implementation research” (as used in the NIH-sponsored conferences by that name), which does not emphasize the critically important end point of sustained use in practice.

Yet as health programs persist in practice, they often change over time. Some program components may be implemented and maintained with fidelity; other components may be modified by staff for multiple reasons. Sustainability, like implementation, is not necessarily a steady state.26 Thus, although sustainability may usefully be considered as a set of outcomes, it is variable and can unfold as a set of processes that can incorporate recursive learning in an organization and community over time.

Sustainability Research and Program Efficacy

Under ideal circumstances, practitioners should adopt and attempt to sustain only those interventions with evidence for efficacy in reaching their stated outcomes, and sustainability research should focus on those evidence-based interventions. Certainly, we do not intend to encourage sustaining interventions that are not effective in producing beneficial outcomes for consumers. Yet in practice, the outcomes of efficacy research in community settings may be ambiguous: some communities or organizations achieve the intended outcomes, but others do not; or some intended changes show statistical significance, and others do not. In addition, if the intervention is a relatively new one undergoing an efficacy trial, often the research results are not available when local program managers need to decide whether to attempt to sustain that program or not. Then, the ethical problems of prematurely abandoning potentially useful community-oriented interventions become salient.

For some interventions, evidence may accumulate over time to show the strength of their outcomes, or the lack thereof, rather than being provided by a definitive efficacy trial. Furthermore, the funding organization or government agency may decide that its continued association with a popular program is reason enough to continue its funding. When decisions are needed about both program continuation and plans for sustainability research, should efforts to support program sustainability or plans for sustainability research be encouraged if the efficacy of the program itself is uncertain? Sustainable outcomes are likely to require processes that start long before the initial funding ends.

It seems that this dilemma concerning when sustainability research should be encouraged has no easy solution, other than for intervention research and sustainability research to proceed in parallel. Preliminary results of community trials should made available to program managers before the end of the funded period, to let them know whether promoting sustainability is worthwhile. Managers of community programs should continuously evaluate whether intended outcomes are achieved for their sustained interventions and should be attuned to new research findings from controlled studies of the comparative effectiveness of those interventions. For both intervention effectiveness and sustainability, research processes should not be considered as linear processes; they should be composed of multiple sources and types of evidence proceeding in parallel toward sustained delivery of evidence-based programs.

A recent review of evidence-based research and evaluations on the sustainability of health programs suggested that about 40% to 60% of such interventions do continue in some form, but the methodology behind this evidence is relatively weak.11 Often, authors did not provide clear definitions for their concepts and variables, research designs were based on relatively weak self-reports, and even basic comparisons between sustained and unsustained projects were lacking. Although some authors explained that their initial research in this relatively uncharted territory was “exploratory,” it is now time to go beyond explorations to build a research-based map for conducting better-quality sustainability research. Because nearly all interventions to be sustained take place within an organizational or community context, it is also vital to use methods appropriate for organizational and community research, including those embedded in sophisticated qualitative and longitudinal designs.

We organized this discussion around several key components for research: dependent variables, independent variables, data-collection methods, and overall research designs.

Dependent Variables

Many previous studies have used only a dichotomous definition of sustainability: did the program continue or not? Nearly half (9 of 19) of the studies in Scheirer's previous review of sustainability research11 had no explicit definition or only a sketchy operational definition of this dependent variable. We suggest a more detailed conceptualization of sustainability outcomes in terms of 6 types of potential dependent variables:

1. Whether benefits or outcomes for consumers, clients, or patients are continued (when the intervention provides services to individuals).

This individual-level dependent variable requires that an information system continues to capture at least the volume of services provided to consumers. Even better would be continued documentation of changed behavioral or clinical outcomes among clients, if a data system is available to provide evidence of the intended continued effects among clients. Continued achievement of client benefit after external funding stops can be realized if stakeholders are aware of the benefits achieved and they address the business-as-usual difficulties of their achievement.27,28

2. Continuing the program activities or components of the original intervention.

This issue has been the focus of much previous research and is a legitimate dependent variable for this line of research. Rather than being phrased as a dichotomy—“Did the program continue or not?”—the research would be more explicit by enumerating the components of the intervention and inquiring about the extent to which each component is continued.12 This type of operationalization would thus build on work in implementation science29,30 by first distinguishing manifest program components (e.g., trained coaches, written intervention protocols, interagency collaborative review) from the theoretical components or constructs that underlie the intervention (e.g., stages of change, self-efficacy, or behavioral reinforcement theories).

Ideally, the program developers and researchers would have tested and differentiated the manifest program components as operationalizations of either core manifest components (those program structures and processes that causally lead to observed desired outcomes) or customizable manifest components that implementers can be encouraged to modify without logical or actual harm to the effectiveness of the intervention. Adaptations of the customizable components may contribute to the host organization's or community's identification with the program; for instance, through modifications to the language for branding the intervention and communicating with its intended beneficiaries, use of images that closely mirror a particular new target population's demographics, or additional steps or resources that a particular adopting organization has at its disposal. This specification and measurement of the activities or components defining the program links sustainability with the previous stages in an overall life cycle of intervention development, adoption, implementation (with potential iterative adaptations), and sustainability.7,14–17

3. Maintaining community-level partnerships or coalitions developed during the funded program.

Programs that came about as a result of formal community coalitions may see the continuance of the coalition after the initial program funding ends, even if the coalition does not continue all the specific program-level activities they implemented during the funded period. The community's readiness and capacity for interagency communication, cooperation, and collaboration may be a valuable sustained outcome that could lead to a new set of activities or benefits for consumers.31 Sometimes coalitions spin off programs as social or policy entrepreneurs by encouraging others to adopt a program or vesting a coalition program with its own scaled-up staff and organization. Maintaining the broader community capacity for change could lead to changes in the social environment that ultimately create population-wide benefits. Participants in our discussions of these issues emphasized that maintaining community-level partnerships is important to longer-term work on the focus issue, even if the activities of a specific program are not continued.

4. Maintaining new organizational practices, procedures, and policies that were started during program implementation.

Sustainability researchers have long recognized that the extent to which a host organization or partner organization changes its practices, procedures, and policies within its operations and structure can be an important reflection of degree of program institutionalization.5 Yet sustained change in organizational policies or procedures could be an important outcome in itself, whether or not other program activities remain. For example, starting up and sustaining a no-smoking policy in a workplace would be an important outcome, even if other components from a broader smoking-cessation program were not sustained in that location.

5. Sustaining attention to the issue or problem.

A valued outcome of program implementation, especially in high-profile efforts, is general heightened issue salience. Social problems that continue to be recognized as public issues through sustained organizational resources and sustained media coverage can lead to public perceptions of increased issue severity, as well as policymakers paying greater policy attention to the issue and allocating more resources to it.32 Researchers studying sustainability can often identify valid longitudinal archival indicators of issue salience in media, public, and policy agendas that reflect the activity and successes of issue proponents, media advocates, and policy entrepreneurs.

6. Program diffusion and replication in other sites.

Another potential longer-term outcome from worthy innovative programs is that the underlying concepts or interventions themselves may spread to other locations.33,34 The innovation, including the ideas or principles upon which it is based, may diffuse into use at other locations even if it is not maintained in the initial location. The extent of dissemination activities by staff or proponents of the initial program is an indirect proxy indicator for this type of potential sustainability outcome, although such dissemination itself does not measure the actual extent of adoption or implementation of a program by other agencies or communities. Dissemination activities are conceptually distinct from and logically antecedent to subsequent diffusion into actual use.35 Further detailed research is needed to assess whether, and under what conditions, dissemination activity by program staff or a sponsoring change agency does in fact lead decision-makers in other organizations or communities to notice, try, adopt, implement, and sustain the same or adapted versions of public health programs.

Additional issues.

Two additional issues about the dependent variables of study in sustainability research warrant attention here. The first issue is adaptation: how much adaptation or change of the program components can occur while still defining the intervention as “sustained”? This question is closely related to assessing the fidelity of implementation, with a need to define the core components and customizable components of an intervention to guide both implementation and sustainability.36–38 Previous research has indicated that modification to a research-based intervention is often desirable to foster implementation and sustainability, especially if changes reflect additions to the intervention rather than subtractions from it.39 But if modifications have been made, is it still the same program?

A second issue is one of threshold: what extent or components of the program must be present for a program to be counted as sustained, especially if the research is conducted across a number of program sites? This is a dosage or induction strength question. Defining levels of sustainability with explicitly described sets of components for each level can reduce the need to dichotomize an outcome variable into sites that did or did not sustain an intervention.40

The sustainability literature has not developed any consensus or guidance on which type of sustainability outcome should receive priority for research. Often, the type of data collected for sustainability studies depends primarily on the funding available for this continued research. Collecting data about continued benefits for individual clients is usually more expensive than is a simple survey of project directors asking whether they continued their program activities. If feasible, the collection and assessment of information about long-term unintended consequences, including negative consequences of a program, could enrich the program sustainability knowledge base. Detecting these additional consequences may require continuing collection of qualitative or epidemiologic data drawn from the broader context surrounding a specific program and its effects on a population.

Independent Variables

Sustainability research is often initiated with a practical motivation: what factors are related to keeping interventions alive after their initial funding? Can funders enhance these factors to increase the probability that programs they fund will be sustained? It is likely that some analysts who view sustainability as an ongoing process are strongly attuned to an advisory role, by emphasizing that processes influencing sustainability begin long before the initial funding ends.24,25 Funders' questions refer to key process factors (or, in research terms, independent variables) to identify predictors of stronger sustainability outcomes. Previous research has not identified a uniform or common set of such factors that appear to influence sustainability across diverse contexts. Different studies tend to examine different factors or rely on informants to suggest what the informants believe influenced sustainability, instead of systematically assessing any common set of predictive factors.

Nevertheless, an overall framework of influences on sustainability (suggested earlier by Shediac-Rizkallah and Bone41) is emerging from a number of studies.11 These influences include:

1. Characteristics of the intervention, specifically.

The intervention is (1) flexible or adaptable from its original form, (2) inexpensive or can be delivered by volunteers, and (3) supported by evidence for its effectiveness.

2. Factors in the organizational setting, specifically.

There is (1) a good fit between the specific intervention and the host organization's mission and operating routines, (2) the presence of an internal champion to advocate for the program, (3) the existing capacity and leadership of the organization, and (4) whether the program's key staff or clients believe it to be beneficial.

3. Factors in the community environment of each intervention site, specifically.

There is (1) the existence of partnerships that lead to nonmonetary support of the focal organization, and (2) whether other funders or funding are potentially available in that environment.

Future research may well identify other key factors that flesh out this framework because previous research has often not included a broad set of hypothesized influences to test which factors are or are not predictors of longer-term sustainability.

These 3 sets of factors are not exclusively related to sustainability; they are also related to initial adoption of interventions and subsequent implementation. Influences on sustainability reflect the overall life cycle of an innovation, in which innovations are adopted that are compatible with an organization's mission, capacities, and operating procedures, and full implementation with appropriate adaptation to the local context is a key factor leading to longer-term sustainability.7,11

Program managers often believe that the availability of external funding is the major factor leading to sustainability (i.e., the major independent variable for sustainability outcomes, in research terms). Research has found that although funding sources may be necessary for sustainability, funding is often not the only influence on these outcomes, which are also related to characteristics of the intervention, organizational setting, and community environment.11,41 Nevertheless, previous sustainability research has identified 2 major types of funding streams: (1) institutionalization of the intervention, when those new activities or services can be incorporated into the agency's ongoing budget and operating procedures; and (2) continued external financial support via grants, volunteer services, or donor support. Institutionalization is also termed “routinization,” especially by Yin's major work to characterize the organizational processes involved in this form of continuation.42,43 Achieving institutionalization is often desirable to ensure long-term sustainability, but other programs have been continued for many years by combinations of grants, local fundraising drives, volunteer work, and other short-term funding sources.44 Programs that are started by smaller community agencies might never have the fee-for-service status or other budget sources that support institutionalization within a larger health service agency.

The term “institutionalization” can also reflect a larger set of accommodating changes in organizational practice to help implement, and then sustain, a new health program. This lesson is evident from diffusion research on educational innovations and new business technologies: the greater the accommodating changes within an organization in response to the implementation of an innovation, the higher the likelihood that the innovation will persist.45,46 When organizations make accommodative changes to ease program implementation, sustainability of that program is heightened. Yet, maintenance of organizational conditions that supported an intervention while it was externally funded may be difficult because of new organizational demands and factors such as staff turnover and wavering commitment.47 Researchers should recognize that institutionalization is not the only path to achieving some forms of sustainability. We do not advise that sustainability be equated with institutionalization.

Our sources for this article support the proposition that characteristics of the intervention, the organizational setting, and the community environment are all relevant to sustainability, confirming the importance of moving beyond an exclusive emphasis on funding sources when designing sustainability research. A stronger emphasis on the political and economic environment as the context for long-term sustainability may be necessary in an era of continual change—such as is now occurring in the US health care system—by using a systems approach to this research topic.48,49

A Generic Conceptual Framework for Sustainability

The hypothesized relationships among the relevant dependent and independent variables, and the degree to which they are embedded in a policy and financial environment, are summarized in the conceptual diagram shown in Figure 1. Although this framework does not include all the factors that may come into play in a particular context, it indicates that financial sources are hypothesized as intervening variables between the diverse set of factors influencing sustainability and potential longer-term sustainability outcomes. In this conceptualization, continued financial support is not synonymous with sustainability, but the availability of resources is hypothesized as a key influence on sustainability outcomes.

All of these processes take place in an encompassing broader context: the social, policy, and financial environments. By “financial environment,” we mean the underlying potential sources for funding in support of interventions of a specific type, such as whether foundations emphasize smoking cessation or obesity prevention, or whether government agencies provide funding for specific types of interventions. In some circumstances, the contextual factors may come to the foreground of the picture as the major influences on sustaining public health programs, especially when major changes in these environmental forces overwhelm the internal organizational and community supports that might normally underpin sustainability. For these circumstances, the researcher may need to adopt an ecological or nested systems research approach35,38 rather than attempting to confirm or disconfirm specific hypothesized influences on sustainability outcomes.

How much does sustainability of a specific program depend on its organizational context as opposed to its environmental context? This is an important question that should be addressed in the research we call for in this article. Sustainability is sometimes viewed as the product of ongoing alignments between the program and the implementing organization in response to drivers in the external organizational environment, such as policy changes and media attention.50 In this perspective, sustainability outcomes resulting from adaptive processes may not be predictable if they are not characterized by rational (evidence-based) decision-making and instead result from a broad range of political and opportunistic factors that are known to affect decision-making.51,52 Organizations and their community and larger contexts exhibit changing interests, capacities, and characteristics over time; programs may be sustained in part because their characteristics fit well with these contextual attributes and can adapt to them as different attributes become more or less important over time. This is an important perspective for future research, to determine whether sustainability outcomes can be related to a predictable set of processes characterized as independent variables.

Previous research has reported data collected primarily via self-reports from project managers about the continued status of their projects, often obtained from surveys conducted via telephone, mail, or Internet.11,12 Such surveys can include detailed questions about the specific activities or program components that were sustained, whether client outcomes are still measured, or other details about partnerships or dissemination to go beyond simply asking whether the program was maintained after the target funding ended. Other potentially useful data-collection methods include information technology systems that routinely collect data on individual clients,53 trained observer observations of the fidelity of continued service delivery, site visits to interview multiple informants about the target intervention and supporting organizational processes, agency or coalition documents (such as minutes of coalition meetings), and case studies that combine multiple sources of data. Little information is available about the potential extent of bias in self-reported data, such as whether project directors report accurately as survey respondents. Comparative research using multiple methods to assess the same sustainability outcome(s) and predictive factors is needed so that the accuracy of self-reports can be better understood.

Data-collection methods are often constrained by relatively limited funding for follow-up research. Future research on sustainability could be greatly enhanced if funders matched their financial support to the complexity of the issues involved. Ideally, data-collection systems for obtaining information about client outcomes would be continued into the period following the initial research or intervention, especially if a client-oriented information technology system was developed or used during the project, so that data could be extracted from electronic health records or other archival sources. Data collection within implementing organizations could be cross-validated by using multiple respondents or informants per implementing organization; it is not known whether multiple observers tend to report similar perceptions of organizational processes or actions after termination of the initial funding.

Another unresolved question is timing: when should data be collected to assess whether a project is sustained? Research has been conducted at a variety of time spans after focus funding ended, from 6 months to 6 or more years. We recommend that sustainability be assessed no sooner than 1 year after a specific funding source ends. When a public health or clinical data system was started up or used during the initial research or evaluation, that data system should continue to be used indefinitely to assess whether new clients continue to benefit from the intervention.

Research on health-related interventions has traditionally focused on the causal efficacy of the intervention itself, especially by using randomized controlled trial designs. Although evaluation literature has expanded the repertoire of methods for efficacy research to encompass several types of nonexperimental designs,54 the emphasis is still on the outcomes of the intervention itself, rather than on variables affecting the organizational contexts surrounding implementation and sustainability. Such designs to assess the efficacy of an intervention are applicable for sustainability studies only when an intervention is being tested to increase the probability of program sustainability. For example, an intervention to provide training for program managers in planning for sustaining their programs might be assessed for efficacy by a randomized trial. Would such training be strong enough to overcome the co-occurring organizational and environmental factors affecting the sustainability of their health programs?

For most studies of sustainability, a nonexperimental approach must be employed—and such an approach may well be preferable—to examine the relationships and processes that relate the relevant independent variables to 1 or more sustainability outcomes as dependent variables. The research questions for a specific study should be clearly delineated, and the research design chosen should be congruent with those questions. In addition, appropriate unit(s) of analysis must be chosen for the study, usually focusing on the organizations or communities that implemented and may sustain the target interventions. However, a common dilemma in this research field is the “too many variables” problem: there may be too many relevant potential independent variables to include in a quantitative analysis with a limited number of organizations in each study, when organizations are the unit of analysis.

Several research designs could be relevant for use in future sustainability research, as well as in studies of interventions to increase the probability of sustainability. Each design emphasizes different research questions and different aspects of sustainability.

Multivariate Regression Analysis

If data can be collected from a large enough number of organizations, some of which report that they have not sustained the target innovation(s), then statistical models can assess the strength of relationships between the hypothesized set of influencing factors (independent variables) with 1 or more sustainability outcomes (e.g., the number of components or activities sustained within each organization, the number of clients served as a proportion of the number served during the funded period, or simply the presence or absence of the target program), while controlling for underlying differences among the organizations in the study. For example, see the follow-up study by O'Loughlin et al. of 189 heart-health programs in Canada.55 Potential sources of bias in this type of research are that nonsustained programs may be less likely to respond to data-collection requests, and it may be more difficult to locate knowledgeable respondents for programs that were not sustained.

In-Depth Case Studies

Tracing the processes that occurred in sustaining organizations, along with parallel examinations in organizations that did not sustain, can be illuminated by in-depth case studies of what happened and why in each organization.56,57 Ideally, these case studies would collect an initial wave of data before the end of external funding and would then collect 1 or 2 additional waves of data a year or more after the funding ended. Such longitudinal studies are likely to utilize the ethnographic methods needed for in-depth study of the organizational and contextual processes affecting later sustainability. Researcher site visits to interview multiple informants per site would be particularly valuable because then the perspectives of diverse persons involved with the intervention could be compared. Strong methods are now available for cross-comparisons of multiple cases to generate valid conclusions.58–60 A recent example of comparative case study research on program sustainability is the study by Savaya et al.61 of social programs in Israel, which included analysis of program characteristics, host organization factors, and the social and political environment.

System Change Approaches

Previous discussions and literature have emphasized that sustainability may be strongly influenced by unpredictable environmental and financial factors. Research may need to view sustainability as a product of each ecological system in which the focus intervention is embedded.43,48 What are the drivers in that system, and how do they affect the continuation of specific interventions? Is the sustainability of specific funded interventions separable from the more generic context of forces in the constantly changing health care system? Within a systemic change perspective, multiple types of data on policy and financial sources that support or undermine sustainability may be needed, as well as data on the actions undertaken within specific organizations to help maintain a specific program or intervention.

Policy-Oriented Research

Sustainability of specific health programs is very likely to be influenced by the broader policy environment, such as the many changes in the US health care system addressed in the 2010 health insurance reform legislation. A further approach to sustainability research would be to trace the effects of changes in federal funding policies on local agencies' attempts to sustain related programs. Which types of interventions are favored by new funding provisions? For example, are interventions included within fee-for-service reimbursement for health care providers favored over nonreimbursable interventions aimed at broader public health system changes?

We have emphasized the importance of developing a well-defined research paradigm for improving research and evaluation of the sustainability of health programs that affect the health of the public. To better understand how to achieve sustainability, we need stronger research providing evidence concerning which strategies work best for which types of sustainability outcomes and for which types of interventions. This will require policymakers and funders to recognize the important roles that sustainability plays in the broader life cycle of program development, efficacy and effectiveness research, dissemination to many relevant locations, and achievement of organizational support and resources for sustainability. Sustainability is a key outcome—perhaps the key outcome—in this chain of outcomes that allows large-scale improvement in population health to occur.

In a previous article, Scheirer posed the question: “Is sustainability possible?”11 Here, we raise a related and perhaps more fundamental question: is sustainability desirable? We encourage consideration of this question because so many researchers, intervention proponents, service providers, and policymakers are decidedly of 2 minds. When we have an effective public health program that has been implemented and is being sustained, we are fully supportive; but when we have 1 or more new programs for which we seek adoption, implementation, and eventual sustainability, we do all we can to displace yesterday's evidence-based program. Furthermore, sustaining a program within an ongoing organization could become a hollow shell of activities perpetuated for their own sakes, especially if benefits for clients are not achieved. One public health researcher has questioned whether institutionalization ought to be a goal of the life cycle of program development and delivery.62 He suggested instead that the capacity building and innovativeness generated by the development of new programs are the more important outcomes that should be sustained. But if efficacious programs are not sustained into widespread practice, substantial developmental resources may be wasted, along with the loss of potential benefits for improving human health.

If the guidance in our article were used to improve our research-based knowledge about sustainability, several desirable consequences could follow. First, program managers and funders would have more effective guidance about how to increase the likelihood of sustained effective programs, to ensure a deeper evidence base for the “how to do it” literature on sustainability. Second, program management and coordination could be improved, as administrators became more aware of coordinating strategies needed to achieve long-term beneficial outcomes from multiple programs serving multiple needs. Third, there could be better linkage between short-term funding for developing new programs and innovations and longer-term support for sustained programs to improve population health. Finally, this in-depth knowledge about sustainability and its influences would contribute to the broader research agendas for translation and dissemination of effective practices into widespread use.

Our 2010 NIH conference session about health program sustainability and measurement emphasized the importance of how actions taken early in a program's life cycle could heighten the likelihood of program sustainability. For example, programs to be disseminated to community decision-makers could be selected partly on the basis of which program is most likely to be sustained for which host community. Standardized “readiness-to-sustain” criteria that are acknowledged as important in program sustainability could be the basis for Web-based decision-making tools that could be developed and validated across and within topical fields. Example cases of long-sustained programs, in which researchers selected the programs on the basis of having been sustained for a long time, are one source for the measures that could anchor such tools.56 Long-sustained interventions26 may hold important keys for the design of formative-stage tools. The result, when applied, could be the improvement of effective programs so that their likelihood of reaching sustained use is heightened through iterative tailored feedback for program redesign and readiness rankings of communities and organizations as program hosts.

Sustainability research and measurement should ultimately become key components and research foci within the comparative effectiveness research agenda as it pertains to public health.63 How can we responsibly claim to assess effectiveness if we have no data on which interventions are most likely to be sustained in practice? How can we influence widespread practice if we do not incorporate a better understanding of the organizational and environmental contexts that affect sustained practice? Why bother with what is effective, if it is also fleeting?

Acknowledgments

J. W. Dearing's time to work on this article was supported by the National Cancer Institute (award CA-P20-137219).

This article grew out of discussion at a Think Tank session on “Improving Methods for Research on Health Program Sustainability” at the Third Annual Conference on the Science of Dissemination and Implementation, sponsored by the National Institutes of Health.

Note. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute or the National Institutes of Health.

Human Participant Protection

No protocol approval was required because no human research participants were involved.

References

1. Bamberger M, Cheema S. Case Studies of Project Sustainability: Implications for Policy and Operations from Asian Experience. Washington, DC: World Bank; 1990. CrossrefGoogle Scholar
2. Yin RK, Heald KA, Vogel ME. Tinkering With the System. Lexington, MA: Lexington Books; 1977. Google Scholar
3. Weiss CH, Bucuvalas MJ. Social Science Research and Decision-Making. New York, NY: Columbia University Press; 1980. Google Scholar
4. Kirkhart K, Conner RF. Evaluating and maintaining planned change. In: Glaser EM, Abelson HH, Garrison KN, eds. Putting Knowledge to Use. San Francisco, CA: Jossey-Bass; 1983:221251. Google Scholar
5. Goodman RM, McLeroy KR, Steckler AB, Hoyle RH. Development of level of institutionalization scales for health promotion programs. Health Educ Q. 1993;20(2):161178. Crossref, MedlineGoogle Scholar
6. Altman DG. Sustaining interventions in community systems: on the relationship between researchers and communities. Health Psychol. 1995;14(6):526536. Crossref, MedlineGoogle Scholar
7. Scheirer MA. The life cycle of an innovation: adoption versus discontinuation of the fluoride mouth rinse program in schools. J Health Soc Behav. 1990;31(2):203215. Crossref, MedlineGoogle Scholar
8. Clark WC, Dickson NM. Sustainability science: the emerging research program. Proc Natl Acad Sci USA. 2003;100(14):80598061. Crossref, MedlineGoogle Scholar
9. Scheirer MA, Dearing JW. Improving methods for research on health program sustainability. Paper presented at: Third Annual NIH Conference on the Science of Dissemination and Implementation: Methods and Measures; March 15–16, 2010; Bethesda, MD. Available at: http://conferences.thehillgroup.com/obssr/DI2010/documents/ConcurrentSession2/2F_Scheirer-Dearing_ImprovingMethods.pdf. Accessed June 27, 2011. Google Scholar
10. Center for Elementary Mathematics and Science Education, University of Chicago. Sustaining Change in Education: Finding Shared Language and Common Ground. Chicago, IL: Researchers Without Borders; 2009. Google Scholar
11. Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am J Eval. 2005;26(3):320347. CrossrefGoogle Scholar
12. Scheirer MA, Hartling G, Hagerman D. Defining sustainability outcomes of health programs: illustrations from an online survey. Eval Program Plann. 2008;31(4):335346. Crossref, MedlineGoogle Scholar
13. Dearing JW, Maibach E, Buller DB. A convergent diffusion and social marketing approach for disseminating proven approaches to physical activity promotion. Am J Prev Med. 2006;31(suppl 4):S11S23. Crossref, MedlineGoogle Scholar
14. Rovniak LS, Hovell MF, Wojcik JR, Winett RA, Martinez-Donate AP. Enhancing theoretical fidelity: an e-mail-based walking program demonstration. Am J Health Promot. 2005;20(2):8595. Crossref, MedlineGoogle Scholar
15. Ory MG, Mier N, Sharkey JR, Anderson LA. Translating science into public health practice: lessons from physical activity interventions. Alzheimers Dement. 2007;3(suppl 2):S52S57. Crossref, MedlineGoogle Scholar
16. Glasgow RE. Critical measurement issues in translational research. Res Soc Work Pract. 2009;19(5):560568. CrossrefGoogle Scholar
17. Dearing JW. Applying diffusion of innovation theory to intervention development. Res Soc Work Pract. 2009;19(5):503518. Crossref, MedlineGoogle Scholar
18. Backer TE, David SL, Saucy G, eds. Reviewing the Behavioral Science Knowledge Base on Technology Transfer. Bethesda, MD: National Institute on Drug Abuse; 1995. NIH publication954035. Google Scholar
19. Gustafson DH, Flatley Brennan P, Hawkins RP. Investing in E-Health: What It Takes to Sustain Consumer Health Informatics. New York, NY: Springer; 2007. CrossrefGoogle Scholar
20. Strange T, Bayley A. Sustainable Development: Linking Economy, Society, Environment. Paris, France: Organization for Economic Co-operation and Development; 2008. CrossrefGoogle Scholar
21. Gruen RL, Elliott JH, Nolan ML, et al. Sustainability science: an integrated approach for health-programme planning. Lancet. 2008;372(9649):15791589. Crossref, MedlineGoogle Scholar
22. Katz E. The characteristics of innovations and the concept of compatibility. Paper presented at: Rehovoth Conference on Comprehensive Planning of Agriculture in Developing Countries; August 19–29, 1963; Rehovoth, Israel. Google Scholar
23. Schensul JJ. Community, culture and sustainability in multilevel dynamic systems intervention science. Am J Community Psychol. 2009;43(3–4):241256. Crossref, MedlineGoogle Scholar
24. Johnson K, Hays C, Center H, Daley C. Building capacity and sustainable prevention innovations: a sustainability planning model. Eval Program Plann. 2004;27(2):135149. CrossrefGoogle Scholar
25. Pluye P, Potvin L, Denis JL. Making public health programs last: conceptualizing sustainability. Eval Program Plann. 2004;27(2):121133. CrossrefGoogle Scholar
26. Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network; 2005. Google Scholar
27. Steadman HJ, Cocozza JJ, Dennis DL, et al. Successful program maintenance when federal demonstration dollars stop: the ACCESS program for homeless mentally ill persons. Adm Policy Ment Health. 2002;29(6):481493. Crossref, MedlineGoogle Scholar
28. Blasinsky M, Goldman HH, Unutzer J. Project IMPACT: a report on barriers and facilitators to sustainability. Adm Policy Ment Health. 2006;33(6):718729. Crossref, MedlineGoogle Scholar
29. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. Crossref, MedlineGoogle Scholar
30. Century J, Rudnick M, Freeman C. A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval. 2010;31(2):199218. CrossrefGoogle Scholar
31. Dearing JW. The state of the art and the state of the science of community organizing. In: Thompson T, Dorsey A, Miller K, Parrott R, eds. Handbook of Health Communication. Mahwah, NJ: Lawrence Erlbaum; 2003:207220. Google Scholar
32. Dearing JW, Rogers EM. Agenda-Setting. Thousand Oaks, CA: Sage Publishing; 1996. Google Scholar
33. Weiss CH. Theory-based evaluation: why aren't we doing it?New Dir Program Eval. 1997;76:4155. CrossrefGoogle Scholar
34. Rogers EM. Diffusion of Innovations. 5th ed. New York, NY: Free Press; 2003. Google Scholar
35. Dearing JW, Kreuter MW. Designing for diffusion of cancer communication innovations. Patient Educ Couns. 2010;81(suppl 1):S100S110. Crossref, MedlineGoogle Scholar
36. Brekke JS, Phillips E, Pancake L, O A, Lewis J, Duke J. Implementation practice and implementation research: a report from the field. Res Soc Work Pract. 2009;19(5):592601. CrossrefGoogle Scholar
37. Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18(2):237256. Crossref, MedlineGoogle Scholar
38. Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41(3–4):327350. Crossref, MedlineGoogle Scholar
39. Blakely CH, Mayer JP, Gottschalk RG, et al. The fidelity-adaptation debate: implications for the implementation of public sector social programs. Am J Community Psychol. 1987;15(3):253268. CrossrefGoogle Scholar
40. LaPelle NR, Zapka J, Ockene JK. Sustainability of public health programs: the example of tobacco treatment services in Massachusetts. Am J Public Health. 2006;96(8):13631369. LinkGoogle Scholar
41. Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13(1):87108. Crossref, MedlineGoogle Scholar
42. Yin RK. Changing Urban Bureaucracies: How New Practices Become Routinized. Lexington, MA: Lexington Books; 1979. Google Scholar
43. Yin RK. Life histories of innovations: how new practices become routinized. Public Adm Rev. 1981;41(1):2128. CrossrefGoogle Scholar
44. Stevens B, Peikes D. When the funding stops: do grantees of the Local Initiative Funding Partners Program sustain themselves?Eval Program Plann. 2006;29(2):153161. CrossrefGoogle Scholar
45. Hutchinson J, Huberman M. Knowledge Dissemination and Use in Science and Mathematics Education: A Literature Review. Arlington, VA: National Science Foundation; 1993. Report NSF 97-75. Google Scholar
46. Leonard-Barton D. Implementation as mutual adaptation of technology and organization. Res Policy. 1988;17(5):251267. CrossrefGoogle Scholar
47. August GJ, Bloomquist ML, Lee SS, Realmuto GM, Hektner JM. Can evidence-based prevention programs be sustained in community practice settings? The Early Risers' Advanced-Stage Effectiveness Trial. Prev Sci. 2006;7(2):151165. Crossref, MedlineGoogle Scholar
48. Best A, Holmes B. Systems thinking, knowledge and action: towards better models and methods. Evid Policy. 2010;6(2):145159. CrossrefGoogle Scholar
49. Leischow SJ, Milstein B. Systems thinking and modeling for public health practice. Am J Public Health. 2006;96(3):403405. LinkGoogle Scholar
50. Kingdon JW. Agendas, Alternatives, and Public Policies. New York, NY: Longman; 2003. Google Scholar
51. Cohen M, March J, Olsen J. A garbage can model of organization choice. Adm Sci Q. 1972;17(1):125. CrossrefGoogle Scholar
52. Weiss CH. The experimenting society in a political world. In: Bickman L, ed. Validity and Social Experimentation. Thousand Oaks, CA: Sage; 2000:283302. Google Scholar
53. Jaen CR, Ferrer RL, Miller WL, et al. Patient outcomes at 26 months in the patient-centered medical home National Demonstration Project. Ann Fam Med. 2010;8(suppl 1):S57S67. Crossref, MedlineGoogle Scholar
54. Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. New York, NY: Houghton Mifflin; 2002. Google Scholar
55. O'Loughlin J, Renaud L, Richard L, Gomez LS, Paradis G. Correlates of the sustainability of community-based heart health promotion interventions. Prev Med. 1998;27(5 pt 1):702712. Crossref, MedlineGoogle Scholar
56. Wright DB. Care in the country: a historical case study of long-term sustainability in 4 rural health centers. Am J Public Health. 2009;99(9):16121618. LinkGoogle Scholar
57. Stetler CB, Ritchie JA, Rycroft-Malone J, Schultz AA, Charns MP. Institutionalizing evidence-based practice: an organizational case study using a model of strategic change. Implement Sci. 2009;4:78. Crossref, MedlineGoogle Scholar
58. Yin RK. Case Study Research: Design and Methods. Beverly Hills, CA: Sage; 1984. Google Scholar
59. Miles MB, Huberman AM. Qualitative Data Analysis. 2nd ed. Thousand Oaks, CA: Sage; 1994. Google Scholar
60. Goodman RM, Steckler A. A framework for assessing program institutionalization. Knowledge Technol Policy. 1989;2(1):5771. CrossrefGoogle Scholar
61. Savaya R, Spiro S, Elran-Barak R. Sustainability of social programs: a comparative case study analysis. Am J Eval. 2008;29(4):478493. CrossrefGoogle Scholar
62. Green LW. Is institutionalization the proper goal of grantmaking [comment]?Am J Health Promot. 1989;3:44. Crossref, MedlineGoogle Scholar
63. Lauer MS, Collins FS. Using science to improve the nation's health system: NIH's commitment to comparative effectiveness research. JAMA. 2010;303(21):21822183. Crossref, MedlineGoogle Scholar

Related

No related items

TOOLS

SHARE

ARTICLE CITATION

Mary Ann Scheirer, PhD, and James W. Dearing, PhDMary Ann Scheirer is with Scheirer Consulting, Princeton, NJ. James W. Dearing is with the Center for Health Dissemination and Implementation Research, Kaiser Permanente Institute for Health Research, Denver, CO. “An Agenda for Research on the Sustainability of Public Health Programs”, American Journal of Public Health 101, no. 11 (November 1, 2011): pp. 2059-2067.

https://doi.org/10.2105/AJPH.2011.300193

PMID: 21940916