Abstract
Popular magazines often rank cities in terms of various aspects of quality of life. Such ranking studies can motivate people to visit or relocate to a particular city or increase the frequency with which they engage in healthy behaviors.
With careful consideration of study design and data limitations, these efforts also can assist policymakers in identifying local public health issues. We discuss considerations in interpreting ranking studies that use environmental measures of a city population’s public health related to physical activity, nutrition, and obesity.
Ranking studies such as those commonly publicized are constrained by statistical methodology issues and a lack of a scientific basis in regard to design.
FOR CENTURIES, PLACES TO live have been ranked on the basis of factors that contribute to quality of life, such as friendliness, wealth, crime, and health; in a 17th-century ranking, for example, areas with more plentiful game, heavier livestock, and lower mortality from Indian attacks were promoted as more “livable.”1 Further, recent examples are numerous, such as the Places Rated Almanac, a book that rates and ranks 354 metropolitan areas in terms of cost of living, job outlooks, transportation, education, health care, crime, the arts, recreation, and climate.1 Popular magazines often publish rankings as well. For instance, Natural Health magazine ranked “America’s Healthiest Cities” in 2001 (in terms of 37 criteria in the areas of amenities, physical health, environment, and happiness),2 and Men’s Fitness magazine has ranked “America’s Fattest Cities” annually since 1999 (in terms of 16 categories including number of fitness centers and fast-food restaurants, measures of the natural environment and climate, and number of parks and recreational areas).3 “Best places” are also proclaimed on the Internet, examples being Money Magazine’s “Best Places to Live” (factors considered are climate, crime, housing, education, economy, health, arts and leisure, and transportation)4 and Fast Forward’s “Sperling’s Best Places” (criteria are housing, cost of living, crime, education, economy, health, and climate).5
Ranking studies can garner considerable press coverage, can influence local public health and environmental policies, and motivate populations to work toward healthier lifestyles. In Philadelphia, after the release of “America’s Fattest Cities 2000,” the mayor implemented a new public health program in which he challenged the city’s population to lose 76 tons of weight in 76 days.6 In such ways, rankings of cities can effectively raise awareness of the factors influencing quality of life. In addition, local governments may use the findings to attract new residents, businesses, or tourists. For example, the Web site of the Visitors Association of Portland, Ore, touts the city as a great place to visit and live,7 in part as a result of the high ranking it achieved in the “America’s Fattest Cities 2001” article.
Nevertheless, controversies exist about whether ratings accurately reflect the “livability” of cities and the extent to which such reports can be misleading. A city’s ranking varies depending on the quality of life criteria used in a particular study. Furthermore, these criteria typically include public health prevalence data and environmental measures with multiple sources of variability that are ignored when ranking studies are done. To date, there has not, to our knowledge, been a systematic analysis of ranking studies attempting to determine the extent to which their findings are methodologically sound. Editors of studies published in popular magazines and on the Internet are not bound by criteria imposed by peer-reviewed journals such as requirements regarding complete source citations and discussion of study limitations.
We provide an analysis designed to help policymakers interpret ranking studies that appear in the popular press. We discuss considerations in developing ranking studies that use environmental measures of a city population’s public health related to physical activity, nutrition, and obesity in the hopes of stimulating greater interaction between policymakers and those who publish such studies.
Ranking studies can compare cities according to disease outcomes, behavior prevalence, correlates of health measures, or a combination of these indicators. Studies limited to behavior prevalence are the simplest, because national health surveys report prevalence rates at local levels.8 More commonly, studies compare cities primarily on the basis of environmental correlates of health measures. Ideally, the scientific community would publish a list of environmental and behavioral measures derived from multilevel ecological modeling studies, and these measures would be weighted in regard to their relative importance in determining disease outcomes. Such measures would involve the use of timely, readily available sources of comparable data for relevant geographic units, and study designers would construct indices with appropriate weights based on scientific theory and empirical evidence. However, this ideal scenario does not yet exist.
Theoretical frameworks for environment–disease relationships are still in their infancy owing to the shift in public health paradigms in the mid-1990s to encompass multilevel causes of disease.9 For example, current frameworks for obesity-related research distinguish between physical and social environments, behaviors, and disease outcomes.10 No clear evidence exists as yet to quantify relationships between environment, physical activity/nutrition, obesity, and disease,10 and ecological-level studies are lacking that include a wide range of readily available indicators such as those used in city rankings. However, a recent study11 revealed that several economic nutrition indicators (e.g., number of full-service and fast-food restaurants12 and average cost of a meal prepared at home13) exhibited significant associations with obesity prevalence rates. “Walkability” (e.g., presence of sidewalks, enjoyable scenery)14 and number of locations available for exercise (e.g., walking trails, parks, indoor gyms)15 also have been correlated with physical activity. Future scientific research will provide additional empirical evidence on which city rankings can be based.
Meanwhile, ranking studies are popular and will continue to be published in part as a result of the plausibility of relationships between environmental factors, behavior, and health outcomes. Editors are left to do their best with limited resources, and many appropriately choose a combination of available statistics on health behaviors and environmental factors. Cities are complex systems with multiple causal pathways between environment, population dynamics, behavior, and health conditions. Ranking studies may oversimplify these complex systems. Furthermore, combining environmental, behavioral, and disease outcome measures without clarifying the differences between them may confuse and mislead readers. Public health policymakers can benefit from ranking studies while recognizing that their findings may need to be reinterpreted once more sound hierarchical linear models are developed.
Other than lack of scientific basis, limitations posed by available data are often the greatest weakness associated with a ranking study. The primary issue is the paucity of comparable, timely data collected at the city level through the use of stable and reliable procedures and representative samples of the population of interest.
In comparisons of cities, information is required at the city level or the level of the metropolitan statistical area (MSA), which comprises the central city and the suburbs and surrounding counties economically tied to the central city. Most cities and MSAs, however, do not routinely collect all of the data required to create the desired indices for ranking studies. Until recently, state-level averages had been reported for most of the available health data collected nationally, because these data were derived from probability samples and required a minimum sample size to achieve acceptable statistical confidence.8 Yet, state-level averages may not adequately represent the health situation in any of the state’s cities, and this is particularly the case in states covering large geographic areas.
Another complication is that some environmental indicators are measured for central cities but not the entire MSA. Faced with this dilemma, editors may resort to combining city-, MSA-, and state-level data or may impute data from the mean without documenting their decisions in published reports. Policymakers are left to investigate the data sources themselves.
Similarly, ranking studies based on data from different years or outdated sources should be interpreted with caution. Older sources can be misleading when the phenomenon under consideration is changing at a rapid pace. For example, published summary MSA-level health statistics on obesity trends occurring in the 1990s were based on data gathered during the early part of the decade, after which obesity prevalence rates increased rapidly. Studies involving the use of older data for indicators that tend to be more stable over time, such as acres of parks per 10 000 people, do not face the same problem.
Measures used to rank cities may not be calculated and defined according to currently accepted standards and should be interpreted accordingly. For example, public health definitions of “overweight,” “obesity,”16 and “physical activity”17 have been revised in recent years, but statistics including both current and past definitions are widely used. Furthermore, beginning in 2001, household, transportation, and leisure-time physical activity were measured in national health surveys to concur with current public health recommendations; previous measures reported only leisure-time exercise, one type of all possible physical activity.8
Data gathered from some of the sources used to rank cities, including federal sources (e.g., the US Bureau of the Census [http://www.census.gov], the US Environmental Protection Agency [http://www.epa.gov], and the National Oceanic and Atmospheric Association [http://www.noaa.gov]), are collected according to scientific methods and are well documented. The Centers for Disease Control and Prevention’s Behavioral Risk Factor Surveillance System8 also is considered a reliable data source that is particularly well documented and based on a scientifically selected sample.
Other sources are more problematic and should be interpreted with caution. For example, in-house surveys are often derived from convenience samples that may systematically exclude certain groups by focusing on a nonrepresentative sample within a city or MSA. Likewise, only about half of all businesses are listed in business address databases, whereas the Census of Retail Trade12 uses a representative sample.
In ranking studies, data are often transformed to normal distributions, although statistical ranking methods were developed to analyze data that are not normally distributed. Rankings based on normal distributions identify the best and worst cities but misrepresent the relative positions of the many cities in between, the reason being that most of the scores cluster near the mean. The highest and lowest scores are easy to identify, but the remaining values could be statistically indistinguishable. Meaningful measures of statistical uncertainty regarding city scores are difficult to derive in city ranking studies, because such scores are sums or averages of point estimates for differing constructs. Therefore, the numerical scores on which these rankings are based will be more useful to policymakers.
Rankings may be based on indices in which each construct represents a combination of related measures (e.g., a climate index calculated via data on temperature, precipitation, snowfall, and sunshine) that are given numerical scores. Policymakers can group cities with similar numerical scores by, for example, assigning letter grades (A+, A, A–, etc.), a method that has intuitive appeal and commands attention in a society accustomed to being graded or evaluated. Indices and letter grades reduce the effects of imperfect data by giving individual measures low weight in the overall score and by creating categorical measures. Despite its appeal, however, the technique of creating indices via simple averaging assumes that all data sources are of equal quality and appropriateness, which may not be true.
Ranking studies published in magazines and books and on Web sites often attract media and government attention and are taken seriously, regardless of their designs and limitations. When cities refer to their rankings in public health programs,6 on the Internet, and in local chamber of commerce publications,7 it would be useful to include a brief discussion of study limitations along with rankings and scores. As demonstrated here, all ranking studies involve limitations that affect data interpretation. Topics to be touched on include, but are not limited to, lack of a scientific basis for linking study indicators and lack of effective measures focusing on important aspects of the environment, physical activity, and nutrition; the latter issue is a consequence of the difficulty of measuring these factors, poor data quality (e.g., inconsistent geographic level of analysis, outdated statistics, poor coverage), or both.
Another limitation that should be noted is the use of multiple geographic units of analysis for statistical data; if possible, such data should be reanalyzed in hierarchical linear models as they become available. Ranking studies can be valuable tools for the public health field and for local governments if methodological limitations are assessed and taken into consideration.
Rankings of cities can play an important role in raising awareness of public health issues and illuminating what policymakers can do to address these issues. In addition, they provide policymakers and the public more information about the health challenges they face and allow progress to be monitored over time on a range of indicators. If their city’s comparisons with higher ranked cities suggest an environmental issue that might be health related (e.g., poor air quality, higher number of fast food restaurants, lower number of recreational areas), public health policymakers can use published ranking studies to justify a local investigation into the problem. Such data also can be used to raise awareness about lifestyle choices among residents; to market “healthy cities” and “active cities” as attractive places to visit, live, and do business; and to hold government and the private sector accountable for doing what is necessary to keep residents healthy.
Richard R. Andrews’ residency was funded through a grant from the Centers for Disease Control and Prevention.
