© 2004 American Public Health Association
At the time of the study, Glen P. Mays was with Mathematica Policy Research, Washington, DC and Paul K. Halverson was with the Public Health Practice Program Office, Centers for Disease Control and Prevention, Atlanta, Ga. Edward L. Baker and Rachel Stevens are with the North Carolina Institute for Public Health, School of Public Health, University of North Carolina, Chapel Hill. Julie J. Vann is with the School of Medicine, University of North Carolina. Correspondence: Requests for reprints should be sent to Glen P. Mays, PhD, MPH, Department of Health Policy and Management, College of Public Health, University of Arkansas for Medical Sciences, 4301 West Markham, #820, Little Rock, AR 72205-7199 (e-mail: MaysGlenP{at}uams.edu).
Objectives. We examined the availability and perceived effectiveness of 20 basic public health activities in the communities where most Americans reside. Methods. A self-administered questionnaire was mailed to the 497 directors of US local health departments serving at least 100 000 residents. Results. On average, two thirds of the 20 public health activities were performed in the local jurisdictions surveyed, and the perceived effectiveness rating averaged 35% of the maximum possible. In multivariate models, availability of public health activities varied significantly according to population size, socioeconomic measures, local health department spending, and presence of local boards of health. Conclusions. Local public health capacity varies widely across the nations most populous communities, highlighting the need for targeted improvement efforts.
A strong public health infrastructure is essential for preventing, preparing for, and responding to health threats on a populationwide basis.1,2 This infrastructure includes systems for identifying health risks in the population, preventing and controlling communicable disease outbreaks; educating the public and health professionals about health risks and prevention practices; ensuring access to needed health services; and protecting the safety of water, food, and other environmental health resources.3 Although the importance of the public health infrastructure is widely recognized, efforts to measure and improve performance in public health have lagged behind comparable activities in medical practice.411 In the present study, we sought to aid in addressing this gap by examining the performance of core public health activities in the nations most populous communities. The performance of the nations public health system has received growing attention in recent years as this system has been challenged by emerging health threats and by trends in health policy and the health care marketplace. Fifteen years ago, a study commission convened by the National Academy of Sciences Institute of Medicine (IOM) found that an array of factorsincluding stagnant public funding, new and resurgent diseases, a leadership deficit, and a persistent indigent care burdenhad left the nations public health system in disarray.4 The IOM report and related studies helped to mobilize public health improvement initiatives across the nation, many of which focused on local health departments as the essential public health providers in most communities.1216 Federal health objectives for the year 2000 established the target that at least 90% of the US population be served by a local health department that effectively carries out core public health functions.1719 More recently, growing public concern about health threats such as new and resurgent infectious diseases, community violence, and bioterrorism has given emphasis and urgency to the task of improving the nations public health infrastructure.2024 Recognizing this fact, Congress passed the Public Health Improvement Act in November 2000 and, more recently, committed new federal funds for bioterrorism preparedness to help strengthen the public health infrastructure at the local, state, and national levels. Previous efforts to measure the availability and adequacy of essential public health services at the community level have produced evidence of substantial gaps and wide variation in terms of performance. A 1993 study based on a national sample of local public health agencies showed that, on average, only 50% of 10 activities regarded as important elements of public health practice were performed by these agencies.25 A similar study of local public health agencies in 6 states showed that in 1993 only 56% of 26 activities regarded as essential public health practices were available within the jurisdictions served by the departments.26 A third survey involving a national sample of local agencies in 1995 revealed that only 56% of 20 activities deemed important to public health practice were performed by the average agency.27 Because local public health agencies carry much of the responsibility for implementing state and federal public health programs, local gaps in basic public health activities may compromise the effectiveness of the nations public health system in preventing, detecting, and controlling potential health threats.4,5,13,16 The adequacy of the nations public health infrastructure cannot be determined fully without examining the contributions made by organizations other than official governmental public health agencies.1,2830 Studies in selected communities have suggested that medical care providers, community-based organizations, and even managed care plans are contributing to public health activities with increasing frequency and intensity.3139 Nonetheless, little systematic evidence exists regarding the roles that these organizations play within the nations public health system. This study revisited the question of local public health performance, seeking systematic evidence about who contributes to basic public health activities at the community level. We focused attention on the most populous local public health jurisdictionsthose with 100 000 or more residents. We chose to examine these jurisdictions because they serve approximately 70% of the total US population and because they exhibit greater homogeneity in regard to public health resources than do smaller jurisdictions.40 The goals of this study were 3-fold: (1) to examine variation in the availability and perceived effectiveness of essential public health services in the nations most populated areas, (2) to examine the types of organizations that contribute to these services, and (3) to examine the organizational and community characteristics associated with local public health performance.
Measuring Local Public Health Performance We measured local public health performance using an instrument developed through a series of research projects sponsored by the Public Health Practice Program Office of the US Centers for Disease Control and Prevention.25,28,41 These projects identified activities regarded as important for maintaining and improving public health at the community level, and each activity was linked to 1 of the 3 core public health functions of assessment, policy development, and assurance as articulated by the IOM.4 These activities were identified through expert panel meetings, literature reviews, and local health department case studies and surveys.28,4246 In 1995, findings from the projects were reviewed and consolidated such that a combined set of 20 public health activities were identified to serve as indicators of local public health performance.27,47
Activities were selected on the basis of expert opinions of their importance in improving public health and their statistical association with other summary measures of public health performance.43,48,49 Each activity was measured with a simple yes/no question asked of the local health department director concerning whether a specific public health activity is performed in the departments jurisdiction (Table 1
The 20 public health activities were included on a survey instrument administered to local health department directors. All of the performance measures based on these activities were self-reported by local health department directors and therefore reflected the perceptions and perspectives of the respondents. No evidence of systematic overreporting or underreporting was found for any of the measures during extensive in-person site visits conducted in the jurisdictions of 8 departments that responded to the survey during 1995.30 Four types of performance measures were examined in this study, all of which were developed and tested in earlier studies of local public health performance.28,48,49 First, availability measures were computed from responses to each of the 20 questions asking whether the activity is performed in the jurisdiction. A second set of measures reflected the perceived effectiveness of each activity based on the local health directors rating on a 5-point Likert scale ranging from "meets no needs" to "fully meets needs." A third set of measures reflected the local health department contribution to each public health activity based on the directors rating, on a 5-point scale ranging from "none" to "all," of the level of total community effort contributed by the department.
Finally, we asked directors to indicate the types of organizations other than the local health department that participate in performing each activity, using a defined checklist of 11 organizational categories and an open-ended response option for "other." These responses were used to compute participation measures for each organization type and each public health activity. In the case of activities that explicitly referenced tasks performed by the local public health agency (e.g., activity 2 in Table 1
We computed the 4 types of measures for each of the 20 public health activities individually. In addition, we combined the activity-specific measures to compute average measures of availability and perceived effectiveness for each of the 3 public health functions identified by the IOM: assessment (activities 1 through 6 in Table 1
Sample Selection and Survey Methodology
Data Analysis
The independent variables used in each model are summarized in Table 2
To assess the possibility of estimation bias owing to outlier values, we reestimated models while excluding the largest 5% of observations based on population size and spending variables, and we found results consistent with the full-sample estimates. In the case of all dichotomous independent variables, we used the method suggested by Kennedy to transform semilogarithmic regression coefficients into measures of each variables relative effect on the dependent variable.50 We computed the standard errors for all regression coefficients using the HuberWhite covariance matrix to correct for correlations among observations taken from the same state.51,52
We received usable responses from 356 departments out of 497 departments, representing a 71% response rate. Analyses (t tests) confirmed that no statistically significant differences (at the P < .05 threshold) existed between responding and nonresponding departments according to population size of jurisdiction, ethnic composition of jurisdiction, departmental full-time-equivalent staffing, or annual departmental expenditures. Chi-square tests indicated that response rates did not vary significantly (at the P < 0.05 threshold) by type of jurisdiction (city, county, or multicounty) or by educational background of the respondent (physician, nonphysician graduate degree, other). We therefore concluded that our 29% nonresponse rate was unlikely to have been a substantial source of bias in our results.
Descriptive characteristics for the responding public health agencies are presented in Table 2 One fifth of the departments functioned as centralized units of a state health agency, while another 22% operated as fully decentralized agencies under local governmental control. In the remainder of the departments, administrative control was either shared by state and local governments through intergovernmental agreements or divided between state and local governments on the basis of functional areas/geographic regions (a "mixed" authority relationship).53 Fewer than one third of the departments were responsible for directly providing comprehensive primary care services such as routine medical care for underserved children and adults, while more than half of the departments were responsible for providing a full range of environmental health services, including monitoring of water quality, food safety, and environmental toxins.
Availability and Perceived Effectiveness Measures
In contrast, the activities least likely to be performed included evaluations of the effects of public health services on community health status (35%), analyses of participation in preventive and screening services (28%), and resource allocation planning based on priority health needs (26%). When we grouped public health activities into 1 of the 3 core functions identified by the IOM, we found, on average, only slight differences in availability.
Perceptions regarding the effectiveness of public health activities varied widely across communities and appeared to cluster at lower ranges of the distribution. On average, local health directors rated the effectiveness of their jurisdictions public health activities at 35% of the maximum possible score that would be obtained if all activities were performed at levels fully meeting community needs (Table 3 Other activities had relatively low ratings in terms of perceived effectiveness despite being available in more than three quarters of the jurisdictions surveyed. These activities included providing health information to the public (33%); addressing health needs through service provision or linkage to services (36%); and developing support and communication networks among healthrelated organizations, the media, and the public (42%). Effectiveness ratings appeared to be somewhat higher for assessment activities and assurance activities than for policy development activities (P < .05).
Perceived Contribution and Participation Measures In most jurisdictions, a mix of state and local governmental agencies, medical care providers, and nonprofit community organizations contributed to performing public health activities along with the local public health agency. Participation by federal agencies, managed care plans, and community health centers appeared to be considerably less common (taking place in 44%, 45%, and 47% of jurisdictions, respectively), although community health center participation was much higher in the subset of jurisdictions that had centers located within their boundaries (75% of jurisdictions). Hospitals and state government agencies appeared to participate in the largest scope of public health activities on average (37% of the 20 activities), followed by local government agencies and community nonprofit organizations (32%). The most limited scope of participation was reported for federal agencies and managed care plans. In the case of most types of organizations, participation in policy development activities was reported to be somewhat more frequent than participation in assessment and assurance activities.
Correlates of Performance
Results also indicated that approximately 13% fewer public health activities were performed in jurisdictions with centralized statelocal health agency relationships than in jurisdictions with shared or mixed statelocal relationships. In addition, approximately 10% more activities were performed in jurisdictions with policymaking boards of health than in jurisdictions without these bodies (P < .05).
The perceived effectiveness of public health activities varied significantly according to community poverty rates, racial composition, and presence of local boards of health (Table 4 Perceived effectiveness also appeared to be positively associated with local health department spending, but this finding was statistically significant only at the P < .10 level. Other community and institutional characteristicsincluding measures of hospital and physician resources, type of governmental jurisdiction, and types of categorical services offered by the local health departmentwere not significant predictors of the availability or perceived effectiveness of public health activities after other model variables were taken into account.
The availability and perceived effectiveness of public health activities appear far from ideal within the communities in which most Americans reside. On average, one third of the 20 activities considered to be basic elements of local public health practice were not performed in the jurisdictions surveyed. Of the activities that were performed, most were rated only partially effective in meeting the existing community need. Among the activities least likely to be available in local jurisdictions were those that have been argued as essential features of a responsive and effective public health system, such as allocating public health resources in ways that are consistent with priority health needs, analyzing the implementation of recommended preventive and screening services, and evaluating the effects of public health services on community health status.1 Together, these findings suggest that many of the nations largest local public health systems have relatively limited capacities for ensuring that available public health resources are being used most effectively and efficiently to improve community health. Two public health activities that are essential for addressing emerging public health threats such as bioterrorism and new and resurgent infectious diseasesinvestigation of adverse health events and provision of laboratory serviceswere found to be widely available and were rated as highly effective in the jurisdictions studied. These results suggest that most local health officials believe their jurisdictions are relatively well positioned for detecting new public health threats. However, it is important to recognize that these responses were obtained before the discovery and investigation of anthrax exposure cases in October 2001, events that may have led many health officials to reconsider the adequacy of their investigative capacities. Moreover, health officials expressed much less confidence about the effectiveness of other activities likely to be essential for responding to emerging public health threats. These activities include support and communication networks among health-related organizations to enable a coordinated response to new threats, the capacity to deploy and redirect public health resources as necessary to address newly emerging health needs, and the capacity to inform the public about health issues and risks. Gaps in the performance of these activities raise troubling questions about the ability of the nations local public health infrastructure to respond swiftly and effectively to emerging health threats such as bioterrorism. Evidence of wide variation and substantial gaps in public health capacity within the nations largest population centers lends urgency to the call for improvements in the public health system. Findings from this study suggest several promising approaches for local public health improvement. First, we found that many organizations other than the local health department make significant contributions to the local public health effort, suggesting that successful improvement strategies should target the full complement of organizations that currently contribute to, or can potentially contribute to, public health activities.54 We found that in most communities these organizations participate in a relatively narrow scope of activities, suggesting that there are untapped opportunities for collaborationespecially in regard to activities that go beyond planning and policy development efforts. At the same time, we found evidence that local health departments provide large and often dominant shares of their communitys total public health effort, indicating that improvement efforts must also focus on building and maintaining a strong and effective local governmental presence within public health delivery systems. Second, findings from multivariate analyses suggested that strategies for improving public health infrastructure should consider the economic and institutional structures that shape local public health endeavors. In particular, we found evidence that local public health spending levels, governing boards, and statelocal administrative relationships are all associated with the availability or perceived effectiveness of public health activities. These findings suggest a need for policymakers to examine the adequacy of funding streams and administrative structures for local public health systems, especially now that new federal funds are becoming available for upgrading state and local public health jurisdictions preparedness for bioterrorism and other public health threats.55 In making use of these additional resources, policymakers should consider the types of public health activities that are currently underfunded and underperformed in local jurisdictions, along with the administrative relationships and governance structures needed to support effective public health decisionmaking and response at the local level. Third, our findings suggest that public health improvement strategies should focus special attention on communities at elevated risk of being underserved by public health activities. Similar to the results of studies of medically underserved areas, our results indicate that less populous communities, those with higher poverty rates, and those with larger minority populations are most vulnerable to underservice. Additional work is needed to elucidate the factors contributing to underservice within these communities, which may include a lack of available organizations to contribute to public health activities,56,57 economies of scale in the performance of activities, and competing health and social service needs that draw resources away from activities. Because this study did not include public health jurisdictions with fewer than 100 000 residents, our findings may understate the degree to which smaller communities are underserved by public health activities. The results of this study should be interpreted carefully in view of several important limitations. First, our findings cannot be generalized beyond the surveyed population of large local health department jurisdictions with at least 100 000 residents. Although these jurisdictions contain approximately 70% of the total US population, they represent only about 20% of the total number of local public health jurisdictions.58 Second, although the 20 public health activities examined were identified by experts as important basic elements of local public health practice, they do not represent a comprehensive and exhaustive set of activities required for effective local public health systems. As a result, our findings cannot be generalized beyond the 20 public health activities studied, and some of our evidence may overemphasize or underemphasize selected types of public health activities. In particular, the aggregate measures of availability and perceived effectiveness examined in this study assigned considerable weight to planning and policy development functions while omitting other potentially important tasks such as enforcement of public health laws and regulations and assurance of a competent public health workforce. Third, the public health performance measures included in our study did not capture possible variations in public health performance levels within local jurisdictions. Consequently, we may have overstated the availability of public health activities if some of the activities reported as taking place were performed only in certain parts of a jurisdiction. Because this study took place before the terrorist events of 2001, it provides a useful baseline for future research examining whether and how the nations public health infrastructure has evolved since that time. The studys findings and its limitations highlight the need for additional efforts to define, measure, and improve core elements of public health practice, such as the activities now under way as part of the Centers for Disease Control and Preventions National Public Health Performance Standards Program.59 This and related efforts promise to stimulate additional research aimed at identifying gaps in the nations public health infrastructure and providing insight into how best to strengthen this infrastructure. Only through such activities can we develop the evidence base needed by local public health organizations to fulfill their role as the nations first line of defense against and response to emerging health threats.
This research was supported through a cooperative agreement between the Centers for Disease Control and Prevention and the Association of Schools of Public Health (CA# S230-15/16). Assistance with sampling and data collection was provided by William Kalsbeek, PhD, and Ashley Bowers, MA, of the Survey Research Unit, Department of Biostatistics, School of Public Health, University of North Carolina at Chapel Hill. C. Arden Miller, MD, of the Department of Maternal and Child Health, School of Public Health, University of North Carolina at Chapel Hill, provided valuable advice on the design and conduct of the study.
Human Participant Protection
Contributors G. P. Mays designed and conducted the data analysis and led the interpretation of findings and the writing of the article. P. K. Halverson conceived the study and participated in interpreting findings and writing the article. E. L. Baker, R. Stevens, and J. J. Vann participated in interpreting findings and in revising the article. Accepted for publication May 2, 2003.
1. Baker EL, Melton RJ, Stange PV, et al. Health reform and the health of the public: forging community health partnerships. JAMA.1994;272:12761282.
2. Fielding J, Halfon N. Where is the health in health system reform? JAMA.1994;272:12921296. 3. Public Health in America. Washington, DC: Office of Public Health and Science, US Dept of Health and Human Services; 1994. 4. Institute of Medicine. The Future of Public Health. Washington, DC: National Academy Press; 1988. 5. Roper WL, Baker EL Jr, Dyal WW, Nicola RM. Strengthening the public health system. Public Health Rep.1992;107:609615.[Web of Science][Medline] 6. Roper WL, Koplan JP, Stinnett AA. Public health in the new American health system. Front Health Serv Manage.1994;10(4):3236.[Medline] 7. Halverson PK, Nicola RM, Baker EL. Performance measurement and accreditation of public health organizations: a call to action. J Public Health Manage Pract. 1998;4(4):57.[Medline] 8. Palmer H. Measuring clinical performance to provide information for quality improvement. Qual Manage Health Care. 1996;4(2):16. 9. McGlynn EA. Choosing and evaluating clinical performance measures. Joint Commission J Qual Improvement. 1998;24:470479. 10. Mays GP, Halverson PK, Scutchfield DF. Behind the curve? What we know and need to learn from public health systems research. J Public Health Manage Pract.2003;9(3):179182.[Medline] 11. McGlynn EA, Asch SM. Developing a clinical performance measure. Am J Prev Med.1998;14(suppl 3):1421.[Web of Science][Medline] 12. Mays GP, Hatzell TA, Halverson PK, Kaluzny AD. Continuous quality improvement in public health organizations. In: McLaughlin C P, Kaluzny A D, eds. Continuous Quality Improvement in Health Care: Theory, Implementation, and Applications. 2nd ed. Gaithersburg, Md: Aspen Publishers; 1999:360403. 13. Mays GP, Halverson PK, Miller CA. Assessing the performance of local public health systems: a survey of state health agency efforts. J Public Health Manage Pract.1998;4(4):6378.[Medline] 14. Scutchfield FD, Beversdof CA, Hiltabiddle SE, Violante T. A survey of state health department compliance with the recommendations of the Institute of Medicine report. The future of public health. J Public Health Policy.1997;18:1329.[Web of Science][Medline] 15. Miller CA, Halverson P, Mays G. Flexibility in measurement of public health performance. J Public Health Manage Pract.1997;3(5):78. 16. Koplin AN. A national program to restructure local public health agencies in the United States. J Public Health Policy.1993;14:393402.[Medline] 17. Healthy People 2000: National Health Promotion and Disease Prevention Objectives. Washington, DC: US Dept of Health and Human Services; 1991. 18. Oberle MW, Baker EL, Magenheim MJ. Healthy People 2000 and community health planning. Annu Rev Public Health.1994;15:259275.[Web of Science][Medline] 19. Healthy People 2010 Draft Objectives. Washington, DC: US Dept of Health and Human Services; 1999.
20. Henderson DA. The looming threat of bioterrorism. Science.1999;283:12791282. 21. McDade JE, Franz D. Bioterrorism as a public health threat. Emerg Infect Dis.1998;4:493494.[Web of Science][Medline] 22. Winett LB. Constructing violence as a public health problem. Public Health Rep.1998;113:498507.[Web of Science][Medline]
23. Holden C. Violence seen as public health issue. Science. 1985;230:1257. 24. Public Healths Infrastructure: A Status Report. Atlanta, Ga: Centers for Disease Control and Prevention; 2001. 25. Turnock BJ, Handler AS, Hall W, Potsic S, Nalluri R, Vaughn EH. Local health department effectiveness in addressing the core functions of public health. Public Health Rep.1994;109:653658.[Web of Science][Medline] 26. Richards TB, Rogers JJ, Christenson GM, Miller CA, Taylor MS, Cooper AD. Evaluating local public health performance at a community level on a statewide basis. J Public Health Manage Pract.1995;1(4):7083.[Medline] 27. Turnock BJ, Handler AS, Miller CA. Core function-related local public health practice effectiveness. J Public Health Manage Pract. 1998;4(5):2632.[Medline]
28. Miller CA, Moore KS, Richards TB, McKaig CA. A proposed method for assessing public health functions and practices. Am J Public Health. 1994;84:17431749. 29. Halverson PK, Miller CA, Kaluzny AD, et al. Performing public health functions: the perceived contribution of public health and other community agencies. J Health Hum Serv Adm.1996;18:288303.[Medline] 30. Mays GP, Miller CA, Halverson PK, eds. Local Public Health Practice: Trends and Models. Washington, DC: American Public Health Association; 2000. 31. Lasker R. Medicine and Public Health: The Power of Collaboration. New York, NY: New York Academy of Medicine; 1997. 32. Mays GP, Halverson PK, Kaluzny AD. Collaboration to improve community health: trends and alternative models. Joint Commission J Qual Improvement Health Care.1998;24:518540.
33. Reiser SJ. Topics for our times: the medicine/public health initiative. Am J Public Health.1997;87:10981099. 34. Institute of Medicine. Healthy Communities: The Future of Public Health. Washington, DC: National Academy Press; 1997. 35. Gordon RL, Baker EL, Roper WL, Omenn GS. Prevention and the reforming US health system: changing roles and responsibilities for public health. Annu Rev Public Health.1996;17:489509.[Web of Science][Medline] 36. Centers for Disease Control and Prevention. Prevention and managed care: opportunities for managed care organizations, purchasers of health care, and public health agencies. MMWR Morb Mortal Wkly Rep.1995;44(RR-14):112.[Medline] 37. Mays GP, Halverson PK, Kaluzny AD. Health promotion and disease prevention in managed care: examining strategies for coverage, delivery, and use. Compensation Benefits Manage.1999;15(3):3745. 38. Halverson PK, Mays GP, Kaluzny AD, Richards TB. Not-so-strange bedfellows: models of interaction between managed care plans and public health agencies. Milbank Q.1997;75:113138.[Web of Science][Medline]
39. Roper WL, Mays GP. The changing managed carepublic health interface. JAMA.1998;280:17391740. 40. 1997 National Profile of Local Health Departments. Washington, DC: National Association of County and City Health Officials; 1998. 41. Handler AS, Turnock BJ, Hall W, et al. A strategy for measuring local public health practice. Am J Prev Med.1995;11(suppl 2):2935.[Web of Science][Medline] 42. Miller CA, Moos M-K. Local Health Departments: Fifteen Case Studies. Washington, DC: American Public Health Association; 1981. 43. Miller CA, Richards TB, Christenson GM, Koch GG. Creating and validating practical measures for assessing public health practices in local communities. Am J Prev Med.1995;11(suppl 6):2428.[Web of Science][Medline] 44. Miller CA, Moore KS, Richards TB, Kotelchuck M, Kaluzny AD. Longitudinal observations on a selected group of local health departments: a preliminary report. J Public Health Policy.1993;14:3450.[Medline] 45. Dyal WW. Ten organizational practices of public health: a historical perspective. Am J Prev Med.1995;11(suppl 6):68.[Web of Science][Medline] 46. Turnock BJ, Handler A, Dyal WW, et al. Implementing and assessing organizational practices in local health departments. Public Health Rep.1994;109:478484.[Web of Science][Medline] 47. Turnock BJ. Public Health: What It Is and How It Works. Gaithersburg, Md: Aspen Publishers; 1997. 48. Miller CA, Moore KS, Richards TB, McKaig C. A screening survey to assess local public health performance. Public Health Rep.1994;109:659664.[Web of Science][Medline] 49. Miller CA, Richards TB, Davis SM, et al. Validation of a screening survey to assess local public health performance. J Public Health Manage Pract.1995;1(1):6371.[Medline] 50. Kennedy PE. Estimation with correctly interpreted dummy variables in semilogarithmic equations. Am Econ Rev.1990;71:801. 51. Norton EC, Bieler GS, Ennett ST, Zarkin GA. Analysis of prevention program effectiveness with clustered data using generalized estimating equations. J Consult Clin Psychol. 1996;64:919926.[Web of Science][Medline] 52. White H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Am Econ Rev.1980;48:817838.
53. DeFriese GH, Hetherington JS, Brooks EF, et al. The program implications of administrative relationships between local health departments and state and local government. Am J Public Health.1981;71:11091115. 54. Halverson PK. Embracing the strength of the public health system: why strong government public health agencies are vitally necessary but insufficient. J Public Health Manage Pract.2002;8(1):98100.[Medline] 55. Guidance for Fiscal Year 2002 Supplemental Funds for Public Health Preparedness and Response for Bioterrorism. Atlanta, Ga: Centers for Disease Control and Prevention; 2002. Announcement 99051.
56. Halverson PK, Mays GP, Kaluzny AD. Working together? Organizational and market determinants of collaboration between public health and medical care providers. Am J Public Health. 2000;90:19131916. 57. Mays GP, Halverson PK, Stevens R. The contributions of managed care plans to public health practice: evidence from the nations largest local health departments. Public Health Rep.2001;116(suppl 1):5067.[Web of Science][Medline] 58. Brown C, Rawding N. An overview of the nations health departments. In: Mays GP, Miller CA, Halverson PK, eds. Local Public Health Practice: Trends and Models. Washington, DC: American Public Health Association; 2000:1322. 59. Corso LC, Wiesner PJ, Halverson PK, Brown CK. Using the essential services as a foundation for performance measurement and assessment of local public health systems. J Public Health Manage Pract.2000;6(5):118.[Medline] This article has been cited by other articles:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||