Abstract
Objectives. We compared the effectiveness and cost-effectiveness of population-based recall (Pop-recall) versus practice-based recall (PCP-recall) at increasing immunizations among preschool children.
Methods. This cluster-randomized trial involved children aged 19 to 35 months needing immunizations in 8 rural and 6 urban Colorado counties. In Pop-recall counties, recall was conducted centrally using the Colorado Immunization Information System (CIIS). In PCP-recall counties, practices were invited to attend webinar training using CIIS and offered financial support for mailings. The percentage of up-to-date (UTD) and vaccine documentation were compared 6 months after recall. A mixed-effects model assessed the association between intervention and whether a child became UTD.
Results. Ten of 195 practices (5%) implemented recall in PCP-recall counties. Among children needing immunizations, 18.7% became UTD in Pop-recall versus 12.8% in PCP-recall counties (P < .001); 31.8% had documented receipt of 1 or more vaccines in Pop-recall versus 22.6% in PCP-recall counties (P < .001). Relative risk estimates from multivariable modeling were 1.23 (95% confidence interval [CI] = 1.10, 1.37) for becoming UTD and 1.26 (95% CI = 1.15, 1.38) for receipt of any vaccine. Costs for Pop-recall versus PCP-recall were $215 versus $1981 per practice and $17 versus $62 per child brought UTD.
Conclusions. Population-based recall conducted centrally was more effective and cost-effective at increasing immunization rates in preschool children.
Vaccination is recognized as one of the greatest public health achievements of the 20th century.1 Childhood vaccines developed in the previous century were associated with declines in the incidence of major childhood infectious diseases by 98% or more compared with baseline 20th century annual morbidity rates.2,3 Despite this, only 44.3% of children aged 19 to 35 months received all recommended vaccines in 2009.4 Because of the importance of timely vaccination in young children, one of the nation’s top health goals as outlined in Healthy People 2020, is to increase the proportion of children aged 19 to 35 months who receive all recommended doses of childhood vaccines to 80%.4
Based on strong evidence of effectiveness, the Community Preventive Services Task Force5,6 recommends the use of reminder/recall for increasing immunization rates, including notification for upcoming immunizations (reminders) or recall notices for overdue immunizations (recall). The use of regional or state immunization information systems (IISs) can greatly facilitate reminder/recall because such systems cannot only identify children who need immunizations but often can also generate reminder postcards or electronic data that can be used to produce autodialer messages. Current national data suggest that despite strong national recommendations, few providers are doing any type of reminder/recall for immunizations.7 Because of this, there has been interest in determining whether reminder/recall efforts might be more feasible and less costly to conduct centrally by health departments using a regional or state IIS.
We conducted a population-based, cluster-randomized pragmatic trial comparing the effectiveness of practice-based recall versus population-based recall conducted by the state health department using the Colorado Immunization Information System (CIIS) in increasing immunization rates among children aged 19 to 35 months as well as the cost-effectiveness of each method.
This study was a stratified cluster-randomized pragmatic trial in 14 counties (8 rural, 6 urban) in Colorado with randomization at the county level within rural and urban strata. We followed the criteria established for National Institutes of Health–funded pragmatic trials.8,9
CIIS receives client and vaccine event data through live data entry into the Web-enabled application and through electronic transfers from data sources maintained by providers and insurers throughout the state. CIIS also includes historical data about immunizations given outside of the state if entered by a Colorado provider. The percentage of children younger than 6 years in Colorado with at least 2 immunization records in CIIS was 82% in 2009. Because it is populated with names and demographic information from the Colorado Electronic Birth Certificate database each week, CIIS has the capacity to assess population-based immunization rates based on county of residence.
We selected 14 of the 64 counties in Colorado to participate because they were relatively uniform with respect to prespecified criteria (Table 1) and did not have characteristics that could confound the trial. Exclusion criteria, the rationale for the criteria, and the numbers of counties excluded on the basis of each included (1) ongoing, existing county-wide reminder/recall efforts because we could not determine if patients responded to our recall or to existing recall efforts (6 rural and 4 urban counties excluded); (2) low CIIS saturation levels in the county, defined as less than 80% of 2-year-old children in the county with at least 2 immunizations in CIIS, because we were not able to adequately assess the results of the intervention (10 rural and 4 urban counties excluded); and (3) frontier counties with populations of less than 10 000 because the health care delivery systems in these counties were different from those in larger counties (23 counties excluded). An additional 2 urban counties were excluded because their populations were substantially smaller than the other urban counties and 1 other county was excluded because its vaccine refusal rate was much higher than all other Colorado counties.
TABLE 1— Baseline County Demographics Population-Based Counties vs Practice-Based Counties: Colorado, 2009
| CIIS Information | Race | Ethnicity | Income | Providers | ||||||
| Counties | % CIISa | Children Aged 19–35 Mo,b No. | Up-to-Date Rate,c % | White, % | Non-White and ≥ 2 Races,d % | Hispanic/ Latino, % | Median Income,e $ | Ratio of Pediatric to Family Medicine Practicesf | Community Health Clinicsg | Total Providers |
| Population-based | ||||||||||
| Urban 1 | 93 | 10 008 | 50 | 88 | 15 | 14 | 66 059 | 18/53 | 6 | 77 |
| Urban 2 | 84 | 5343 | 36 | 85 | 18 | 11 | 56 281 | 7/38 | 4 | 49 |
| Urban 3 | 82 | 3143 | 38 | 79 | 25 | 41 | 39 016 | 6/22 | 1 | 29 |
| Rural 1 | 93 | 366 | 37 | 75 | 29 | 46 | 35 960 | 1/1 | 1 | 3 |
| Rural 2 | 87 | 1274 | 39 | 83 | 19 | 30 | 74 220 | 1/12 | – | 13 |
| Rural 3 | 84 | 614 | 42 | 90 | 12 | 12 | 39 714 | 1/3 | 3 | 7 |
| Rural 4 | 93 | 1720 | 39 | 82 | 21 | 28 | 62 716 | 2/6 | 2 | 10 |
| Overallh | 88 | 3210 | 40 | 83 | 20 | 26 | 53 424 | 36:135i | 17 | 188 |
| Practice-based | ||||||||||
| Urban 4 | 88 | 11 807 | 51 | 73 | 31 | 18 | 56 099 | 21/47 | 6 | 74 |
| Urban 5 | 70 | 12 354 | 29 | 80 | 25 | 15 | 55 621 | 14/53 | 5 | 72 |
| Urban 6 | 82 | 6056 | 43 | 83 | 20 | 28 | 54 578 | 2/20 | 4 | 26 |
| Rural 5 | 83 | 242 | 31 | 94 | 8 | 8 | 58 209 | 1/5 | – | 6 |
| Rural 6 | 79 | 350 | 27 | 89 | 13 | 16 | 39 953 | –/3 | 3 | 6 |
| Rural 7 | 97 | 401 | 49 | 79 | 25 | 40 | 30 373 | –/5 | 3 | 8 |
| Rural 8 | 100 | 234 | 37 | 78 | 25 | 42 | 37 350 | –/2 | 1 | 3 |
| Overallh | 86 | 4492 | 38 | 82 | 21 | 24 | 47 455 | 38:135i | 22 | 195 |
| Pj | .56 | .7 | .61 | .7 | .65 | .95 | .37 | .16 | .56 | .65 |
Note. CIIS = Colorado Immunization Information System.
aNumerator based on number of children aged 0–3 years, 11 months with ≥ 2 immunization encounters in CIIS; denominator based on 2010 Population Forecasts Source of Population Data: State Demography Office—numbers as before start of intervention.
bNumber of children aged 19–35 months per county as of June 2010 according to CIIS.
cPercent up-to-date rate is based on CIIS data for children aged 19–35 months who were up-to-date according to Advisory Committee on Immunization Practices guidelines.
dAll other races and those reporting ≥ 2 races (total is > 100%)
eData taken from the 2010 US Census.
fData compiled via telephone survey conducted by CIIS in 2009.
gCommunity Health Centers data compiled and updated by Colorado Department of Public Health and the Environment.
hThese values reflect the mean of each category mean.
iRatio of pediatric practices to family medicine practices. This is the sum of all providers.
jP values are between the overall values for population-based counties and practice-based counties.
We identified existing practices in all counties based on a state survey that had been conducted by CIIS in 2009; sites were included if they offered primary care visits and immunizations to children aged 19 to 35 months. Within rural and urban strata, counties were randomized to practice-based or population-based recall using a random number generator.
The target population included children who were aged 19 to 35 months as of June 2010 with a CIIS address in one of the selected counties. This included children born in Colorado and those who moved into a study county before the study if they were entered into CIIS. Because CIIS is populated from vital statistics, it also included children who were born in the county but whose provider did not enter information into CIIS, thereby providing the best population-based estimate of children within each county. The study population was fixed from this point in time, and children who either moved into or out of the counties after this time were not added to or subtracted from the study population. Although this meant that children who moved might have been inappropriately assigned during the study period, the potential bias was likely balanced between the 2 study arms.
The population-based intervention included an initial letter from CIIS and the local county health department in June 2010, notifying parents of children who were not up-to-date (UTD) and suggesting they have their child vaccinated at their primary care site or, if they had no usual source of care, at the county health department. Six weeks later, a postcard was mailed to those still not UTD, and 4 weeks after that, a final postcard was sent to those still not UTD. If any mailings were returned without a forwarding address, researchers contacted the provider of last service in CIIS to obtain new addresses. Children who moved out of the state were removed from the mailing lists, although not from the study population denominator. All practices in the population-based counties were sent a letter from their local Public Health Department, informing them about the centralized recall intervention.
In the practice-based intervention counties, all primary care practice sites were invited 5 times to attend 1 of 6 training webinars by CIIS on how to conduct practice-based recall. In the trainings, it was recommended that the practices use the mail method for recall with the same number of recalls as in the population-based counties. Practices were offered financial support for mailings if they did recall for 19- to 35-month-old children throughout the summer of 2010.
The definition of UTD was based on the national Advisory Committee on Immunization Practices recommended series of antigens (4:3:1:3:3:1:3) to be received by the age of 19 to 20 months, including 4 DtaP (diphtheria, tetanus, and acellular pertussis), 3 polio, 1 MMR (measles, mumps, and rubella), 3 Hib (Hemophilus influenza B), 3 Hepatitis B, 1 varicella (chickenpox), and 3 pneumococcal conjugate vaccines.10 Children could be brought UTD in CIIS by receiving needed vaccines or updating immunization records, as both might be attributed to the recall effort. The “reach” of each intervention was estimated based on the Re-Aim framework11; in the population-based counties, we estimated that children were reached by the intervention if no mailed notices were returned after attempts to update addresses. In the practice-based recall counties, we did not have information about how many notices were returned to practices; therefore, we conservatively estimated that all children sent a recall notice were reached.
The primary outcome measures were (1) UTD status, (2) documentation of any new vaccine 6 months after the intervention among 19- to 35-month-old children who were not UTD, and (3) cost–benefit comparison of the 2 interventions.
The study was powered to detect a 4% difference in immunization rates among children who were not UTD at baseline between the 2 arms. The final eligible cohort was established by matching child identification numbers from the follow-up CIIS database obtained in December 2010 to the baseline cohort database from June 2010; 98.3% of children initially identified had a record in the follow-up database. Site of last service was defined as the site where the most recent immunization was given, or if that information was missing, the site where the most recent immunization data were entered.
The major analyses were intent-to-treat. We first compared percentages and then modeled the 2 immunization outcomes, using generalized linear mixed-effects models (PROC GLIMMIX) with a logit link function and unstructured covariance matrix. Mixed-effects models are needed when the assumption of independence is violated, such as in the case where patients are nested within practices.12,13 Because practices tend to be relatively homogeneous with respect to their patients, and because patients sometimes cross county lines for services, site of last service was included as a random effect to account for the clustering of patients within practices. County baseline UTD rates and rural or urban location were included as fixed effects in all models.
A secondary analytic cohort was created for the practice-based intervention, focusing only on the primary care practice sites that conducted recall to determine the impact of recall on rates for only those practices that conducted recall. All comparisons were set at a priori level of significance at P < .05. All analyses were performed using SAS version 9.2 (SAS Institute Inc, Cary, NC).
Because the population-based recall was a centralized effort, all practices and their representative children who were not UTD were included in the cost-effectiveness analysis. In the practice-based recall intervention counties, only practices that conducted recall and their representative children were included. The costs for conducting recall included (1) staff education and training in conducting recall, (2) staff time related to conducting recall, (3) mailing and printing costs associated with recall, and (4) time spent correcting mailing addresses. Time spent by personnel was reported by staff involved in recall at the state health department and by practice staff identified as having conducted recall through reports generated by CIIS on a weekly basis. Nonpersonnel costs related to mailings and postage were identified through contractual invoices. Personnel costs were estimated using the position title linked to median salaries listed by the Bureau of Labor Statistics for the State of Colorado. Benefits were estimated at 25% of median salary.
Of the 55 173 children identified as residing in the 14 study counties as of June 2010, 58.2% (32 125) needed 1 or more immunizations according to CIIS records. Figure 1 shows a consort diagram for the trial. Table 1 compares the characteristics of the patient populations, providers, and CIIS penetration in counties randomized to the 2 intervention arms, demonstrating no significant differences in any of the prespecified criteria at baseline.
FIGURE 1— CONSORT Diagram.
Note. CIIS = Colorado Immunization Information System; UTD = up-to-date.
The intervention in the population-based counties involved all 188 practice sites and 12 832 children (Figure 2). Approximately 40% of initial mailings were returned initially, and after attempts to update these addresses, the final “bad address” rate was 15%. Thus, we estimated that approximately 85% of the children in the population-based counties (n = 10 907) were reached. Of the 195 practices in the practice-based counties, 26 practices participated in a webinar and 10 practice sites conducted recall for 19- to 35-month-old children. Five practices chose to mail a single recall postcard and 5 chose to make a single personal phone call to families. This resulted in 5% of patients (n = 887) who might have been reached by the intervention, assuming no bad addresses or telephone numbers.
FIGURE 2— Comparison of reach of intervention in counties that were (a) population-based and (b) practice-based.
Note. Population-based counties included 188 practice sites and 12 832 children eligible for recall. Practice based counties included 195 practice sites and 18 735 children eligible for recall.
aAssuming 100% receipt of reminder/recall.
bAssuming 85% receipt of reminder/recall.
As shown in Figure 3, a significantly higher percentage of children needing immunizations became UTD in population-based counties compared with practice-based counties (F[1,31007] = 13.48; P < .001). The findings were similar for the outcome of receipt of at least 1 vaccine (F[1,31006] = 23.36; P < .001). Estimates of relative risk from multivariable modeling were 1.23 (95% confidence interval [CI] = 1.10, 1.37) for becoming UTD and 1.26 (95% CI = 1.15, 1.38) for documented receipt of any vaccine in population-based versus practice-based counties, respectively. All models were adjusted for baseline county UTD rates and rural location (all P > .05).
FIGURE 3— Percentage brought up-to-date (UTD) of those needing vaccines at baseline.
Among the 10 practices that actually conducted recall, 23.9% of children needing vaccines became UTD and 39.2% were documented to have received 1 or more vaccines. These were significantly higher than the rates achieved in the population-based counties for these 2 outcomes (both P < .001). Of note, children in the 10 practices that chose to do recall were significantly more likely to be UTD (59%) compared with children in the practices that chose not to do recall in the practice-based counties (39%; P < .001). None of the practices that conducted recall chose to be compensated by the study for their costs.
As shown in Table 2, the cost per practice for the practice-based recall method was more than 6 times the cost of the population-based method. For those practices that conducted recall, the cost per UTD child was $62 compared with $17 for the population-based method.
TABLE 2— Cost Analysis, Population-based versus Practice-based Recall for Childhood Immunizations, 2009
| Variable | Population-Based Counties | Practice-Based Counties |
| Total no. of children not UTD at baseline (eligible) | 12 832 | 18 735 |
| No. of practice sites conducing recall efforts | 188 | 10 |
| No. of eligible children in recall practices | 12 832 | 887 |
| Total costs (staff time, supplies), $ | 40 367 | 13 153 |
| Cost per practice, $ | 215 | 1315 |
| Cost per eligible child (total cost/total eligible), $ | 3 | 15 |
| No. of children who became UTD | 2394 | 212 |
| Cost per child who became UTD, $ | 17 | 62 |
| No. of children with ≥ 1 documented immunization | 4083 | 348 |
| Cost per ≥ 1 documented immunization, $ | 10 | 38 |
Note. UTD = up-to-date.
Although strong evidence and national recommendations support the use of reminder/recall for increasing immunization rates in young children, only a small minority of primary care private practices are currently conducting reminders/recalls.7 The present study demonstrated that recall conducted centrally using data from the state IIS populated by birth vital statistics data with direct mailings to families from state and county public health departments was significantly more effective at increasing vaccination rates than were intensive efforts to support recall efforts by primary care practices. The increased effectiveness of the population-based approach was because of the fact that few practices did recall, even when provided with technical and financial assistance, thereby severely limiting the reach of practice-based recall efforts at the population level. When practices did conduct recall, they were slightly more effective at raising rates than was the population-based approach, although at much greater expense. Overall, recall conducted centrally was much more cost-effective.
Numerous previous studies demonstrated the effectiveness of reminder/recall methods when implemented within practice settings.14–17 However, review of the 47 trials included in a recent Cochrane review of the effectiveness of reminder/recall demonstrated that at least 35 were done with the assistance of an outside research study team, and none directly measured the sustainability of reminder/recall once the trial was completed.17 Current national data suggest that only 16% of providers are doing any type of reminder/recall,7 and it is unknown how many are using an IIS to do so. Barriers cited by providers included insufficient staff time and competing demands of primary care, staff turnover, costs related to mailings or telephone call reminders, and the lack of computerized systems to readily identify patients who needed immunizations.7,18 However, a recent study demonstrated that even when practices that are currently using a state IIS are offered training and technical assistance with reminder/recall using the IIS, few practices actually followed through with conducting reminder/recall.18 The present study not only offered training and technical assistance, but offered to pay for mailings, eliminating many previously identified barriers. Yet, despite these efforts, only 13% of practices attended the training offered, and 5% of practices conducted a recall during the study period. The practices that chose to do recall were not typical, in that they had higher baseline immunization rates before the trial. Improving technological interfaces between electronic medical records or billing systems and state IISs might increase provider involvement, but given available evidence in practices that already have the technical capabilities to conduct reminder/recall using an IIS and the many competing demands of primary care practice, advances in information technology might not make a substantial difference.
Given the findings of the present study, centralized recall by public health departments should be considered as a more effective and much less expensive alternative, if it is found to be acceptable to providers and parents. Population-based recall efforts such as the one described will need to rely on statewide IISs. According to the Centers for Disease Control and Prevention annual survey of IIS programs, as of December 31, 2009, 49 states plus Washington, DC, were operating a state IIS.19 Overall, in the United States, approximately 77% of children younger than 6 years were estimated to have participated in an IIS in 2009.19 In addition, in 2009, 42 of 47 (89%) states reported that their IIS was populated with birth data, usually from vital statistics but less commonly from birth hospital records.20 Based on these data, centralized, population-based recall of young children based on birth records is a realistic alternative at the present time in most states.
One challenge of using IIS data that has been populated with birth data is the amount of incorrect addresses in these data. Although we were able to decrease the number of incorrect addresses from 40% to 15% by contacting the site of last service in CIIS, these efforts increased the cost substantially. Contact information could be improved and cost reduced by at least 2 methods. First, if providers routinely updated demographic information in the state IIS, this could substantially increase the quality of contact data and permit a more accurate denominator in the IIS. This has been difficult for practices to do when it required data entry on a case-by-case basis, but could be greatly facilitated by increased use of Health Level Seven (HL7) messaging that permits automatic uploads from electronic medical records to IIS databases.21 In addition, further iterations of population-based recall could include a collaborative approach, whereby recall is conducted centrally by the public health department, but with the active involvement of primary care providers. Given the increased effectiveness of practice-based recall among those who conducted recall in our study, the optimal method might be one in which providers assist in uploading new addresses and the public health department generates notices that appear to come from both the health department and the individual practice. In this way, costs are reduced by sending to more valid addresses, the practice is credited by families for caring for their children, and the recall is more effective and less intrusive because it is endorsed by the child’s provider. A more collaborative approach could also be helpful in improving the accuracy of CIIS data, and therefore, in decreasing the amount of unnecessary recall.
There were important strengths and limitations to the data presented here. As much as feasible, this trial was population-based and, therefore, not subject to the types of bias introduced by studying subpopulations only. However, we were unable to examine differences in outcomes within socioeconomic or racial/ethnic subgroups that might have been of interest. We were also able to measure both effectiveness and cost-effectiveness for entire counties. The estimates of the denominator for the counties were not exact, although the degree of uncertainty should be similar between the 2 study arms. We were not able to control for other efforts in the area of immunization delivery in the counties of which we were unaware. Because this was a comparative effectiveness trial of 2 county-based interventions, there was no true control group in which to assess the rate of becoming UTD without any interventions. In addition, the number of practices that chose to participate in recall activities could have varied by geographic location, although national data suggested that our results about provider willingness to conduct recall were not unusual. The cost-effectiveness analysis was conducted from the perspective of the parties that might be responsible for conducting recall (either the health department or the practice) and did not take into account the costs and benefits from the perspective of the family or the much broader, theoretical perspective of society as a whole. The social benefits and costs to increasing immunization rates are expected to be the same in both types of interventions, and only vary by the rate itself. For the purpose of this study, it was important to focus on the costs to the potential implementers if reminder/recall methods are to be more widely adopted. Finally, because practice participation in CIIS is voluntary, incomplete entry of data into the CIIS undoubtedly resulted in artificially low immunization rates, both at baseline and after the interventions in all counties, although this would not be expected to affect the comparative trial.
This trial supports consideration of a new paradigm in the way reminder/recall for immunizations should be conducted in this country. Before this approach can be embraced, however, several issues need to be addressed. First, more data are needed about the acceptability of centralized recall to families and providers. Preliminary data from a survey conducted in the counties involved in this trial suggested only 15% of physicians would not want the public health department to conduct recalls, and 51% reported they thought population-based recall conducted by the health department was the optimal approach.22 Data from parents are lacking at present. Second, although more cost-effective than the practice-based approach, the implementation of population-based reminder/recall would still require additional resources either from state or federal funds or shared resources between practices and the public sector. Overwhelming evidence demonstrated that at a societal level, raising immunization UTD rates before entry into preschool would not only be cost-effective but cost saving,23–40 but we, as a society, would need to be willing to make the initial investment. If a population-based approach were used to remind parents about the need for preventive care before they were overdue for immunizations rather than recalling them after the fact, it could be more cost-effective than we demonstrated in this study because it could increase preventive care without inappropriately recalling children who were already immunized. Data from this trial supported a national dialogue to address options for more effective, less costly reminder/recall approaches to increase immunization levels in young children.
Acknowledgments
This study was supported by the National Library of Medicine of the National Institutes of Health (Award RC1LM010513). Clinical Trials Registration Number: NCT01296906 (https://register.clinicaltrials.gov/prs/app/action/SelectProtocol?sid=S0002V4A&selectaction=View&uid=U0000872&ts=5&cx=-ie3log).
Note. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Library of Medicine or the National Institutes of Health.
Human Participant Protection
The Colorado institutional review board (COMIRB) first approved the study (#09-0725) as an Exempt protocol September 2009. After some amendments to the protocol, it was found to fall under the Expedited category in August 2010. The current COMIRB protocol number is #10-0823.
