Abstract
Objectives. We evaluated an intervention to support health workers after training in Integrated Management of Childhood Illness (IMCI), a strategy that can improve outcomes for children in developing countries by encouraging workers' use of evidence-based guidelines for managing the leading causes of child mortality.
Methods. We conducted a randomized trial in Benin. We administered a survey in 1999 to assess health care quality before IMCI training. Health workers then received training plus either study supports (job aids, nonfinancial incentives, and supervision of workers and supervisors) or usual supports. Follow-up surveys conducted in 2001 to 2004 assessed recommended treatment, recommended or adequate treatment, and an index of overall guideline adherence.
Results. We analyzed 1244 consultations. Performance improved in both intervention and control groups, with no significant differences between groups. However, training proceeded slowly, and low-quality care from health workers without IMCI training diluted intervention effects. Per-protocol analyses revealed that workers with IMCI training plus study supports provided better care than did those with training plus usual supports (27.3 percentage-point difference for recommended treatment; P < .05), and both groups outperformed untrained workers.
Conclusions. IMCI training was useful but insufficient. Relatively inexpensive supports can lead to additional improvements.
To reduce the enormous burden of child mortality in developing countries, the World Health Organization (WHO) and other partners developed the Integrated Management of Childhood Illness (IMCI) strategy.1 IMCI has 3 components: improving case management practices of health workers (especially in outpatient health facilities), strengthening health systems, and promoting community and family health practices. To improve case management practices, IMCI encourages the use of evidence-based guidelines for identifying and treating the leading causes of child deaths (e.g., pneumonia, diarrhea, and malaria2) in first-level health facilities that lack sophisticated diagnostic equipment and treatments. WHO recommends implementing the guidelines through an 11-day in-service training course, a follow-up visit to health workers' facilities in 4 to 6 weeks to reinforce new practices, and job aids (a flipchart and wall chart of clinical algorithms, a pictorial counseling guide, and a 1-page form for recording a patient's assessments, illness classifications, and treatments). For brevity, we describe this implementation process as IMCI training.
More than 110 countries are implementing IMCI, and studies have demonstrated that the strategy can improve quality of care at health facilities3–6 and seems to reduce mortality.7 However, these studies also revealed substantial room for improvement in adherence to IMCI guidelines. For example, IMCI-trained health workers correctly treated only 58% to 73% of children needing an oral antimicrobial.5,6,8,9 To improve adherence, health workers need support after IMCI training.9
In 1999, Benin adopted the IMCI strategy and began planning its introduction. Assistance was provided through a US-funded malaria control project, the Africa Integrated Malaria Initiative. During planning, concerns were raised about WHO's implementation approach: the training might not lead to long-term changes in health worker practices, and printing an IMCI recording form for each patient would be unaffordable. Therefore, we designed a package of supports to follow IMCI training and conducted a trial to evaluate them. We characterized the effectiveness and cost of the posttraining supports (primary objective) and IMCI training (secondary objective) on health care quality for all illnesses combined.
The study area, the Ouémé and Plateau departments in southeastern Benin (Figure A, available as a supplement to the online version of this article at http://www.ajph.org), typified West Africa, with widespread poverty, weak infrastructure, endemic malaria, and high child mortality.10,11 The trial design was a pre–post study with randomized controls. The Ouémé and Plateau departments were divided into 2 areas (i.e., 2 units of randomization), each comprising 8 communes (small districts). We randomly chose a slip of paper from a bag to assign 1 area to receive IMCI training plus study supports and the other to receive IMCI training plus “usual supports” (Figure 1). WHO has not specified exactly what follow-up supports to IMCI training should be; usual supports were the control-area supports we assumed would have been provided in the absence of our study and generally reflected supports outside our setting. Because supervision was a key intervention and communes were each supervised by 1 person, communes were not divided. The 2 study areas were designed to be similar: each included relatively inaccessible communes, and each included part of the largest city, Porto-Novo. Communes within each study area were geographically grouped to minimize contamination. The study was unblinded because it was impossible to conceal the study supports from health workers and surveyors.
FIGURE 1 Timeline of Integrated Management of Childhood Illness (IMCI) training and the study to evaluate posttraining supports: Benin, 1999–2004.
aZones 1 and 2 were randomized as a group to be the intervention area; zones 3 and 4 were randomized as a group to be the control area. Zone 1 was the trial intervention area in which IMCI was piloted: training began in 2001, and IMCI-trained health workers received study supports. Zone 2 was the trial intervention area in which IMCI began later: training began in 2003, and IMCI-trained health workers received study supports. Zone 3 was the trial control area in which IMCI was piloted: training began in 2001, and IMCI-trained health workers received usual supports. Zone 4 was the trial control area in which IMCI began later: training began in 2003, and IMCI-trained health workers received usual supports.
We conducted 4 health facility surveys: a baseline (before IMCI training) survey in 1999 and 3 follow-up surveys after IMCI implementation began (2001, 2002, and 2004). Public and private licensed health facilities were eligible for inclusion if they provided outpatient services to children and their level of care was appropriate for IMCI (1 referral hospital and 1 subspecialty hospital were excluded). Consultations were eligible if they involved children aged 1 week to 59 months being seen for any illness during regular working hours (typically 8 am–6 pm) on weekdays.
Surveys used cluster sampling, with a cluster defined as all ill-child consultations at a health facility on 1 day. Sampling for each survey was done independently. We used systematic sampling12 with a computer-generated random starting point to select health facilities and then select 1 cluster (i.e., the survey visit date) for each facility. We did not give advance notice of our visit to health facility staff. The unit of observation was an ill-child consultation. Three surveys (1999, 2001, and 2004) produced equal probability samples of health facilities and ill-child consultations in the entire study area, and 1 survey (2002) covered half the study area (i.e., the 8 nonrandomly selected pilot communes in which the Ministry of Health initially decided to implement IMCI [Figure 1]).
IMCI was implemented with WHO's approach. IMCI training courses had 24 to 25 participants and a participant-to-trainer ratio of approximately 4 to 1, were conducted off site, were taught by experienced IMCI-trained clinicians who had received an additional 5-day course on teaching IMCI, and used multiple educational methods (i.e., lectures and reading, interactive discussions, role play, videotaped examples, and clinical practice). Nearly all IMCI-trained health workers received the recommended follow-up visit 4 to 8 weeks after the IMCI course. In accordance with governmental policy, IMCI training was not offered to nursing aides (i.e., health workers with no formal medical training) because the course was considered too complex. Although we intended implementation to take approximately 1 year (in 2001), because of funding and logistical problems, it took 4 (5 courses in 2001, 2 in 2002, 3 in 2003, and 1 in 2004).
In the intervention area, we implemented a multifaceted strategy to support IMCI-trained health workers (Table A, available as a supplement to the online version of this article at http://www.ajph.org). First, we conducted a 5-day workshop with 3 facilitators (A. K. R, F. O., and M. L.), 8 supervisors (physicians), and 3 other health officials to develop, practice, and encourage use of a protocol for supportive IMCI supervision. Our protocol recommended 2 supervision visits every 3 months, alternating between the health worker's clinic and supervisor's health facility (i.e., a hospital in which clinical supervision could include seeing severely ill patients); a checklist to aid supervisors as they observed consultations, provided constructive feedback, and helped health workers solve problems; and supervision of supervisors, in which a senior pediatrician with extensive IMCI experience observed supervision visits and provided constructive feedback to supervisors on their performance. Second, we printed and distributed 2 job aids to each IMCI-trained health worker: patient registers that replaced the IMCI recording form (Figure B, available as a supplement to the online version of this article at http://www.ajph.org) and a counseling guide. Third, just after the IMCI course, we organized half-day training sessions for groups of 5 to 20 health workers on use of the job aids and on the supervision checklist, so they would know what to expect during supervision. Fourth, we implemented nonfinancial incentives (supplemental Table A).
All components were implemented together. No financial support was provided for supervision, because government and community-based funding was supposed to cover supervision field costs. Notably, almost no supervision occurred in the intervention's first 6 months (July–December 2001), so we added a fifth component: 1- to 3-day workshops every 3 months in which supervisors presented their supervision results (and if not all visits were completed, supervisors explained why), engaged in problem solving related to IMCI implementation and supervision, and planned the next round of supervision; some of the workshops included clinical practice at a hospital. However, despite this additional support, supervision records and checklists revealed that only 29% (348/1186) of planned supervision visits actually occurred (A. K. Rowe et al., unpublished data, May 2008).
The following influenced the design of our study supports: (1) the opinion that multifaceted interventions targeting multiple determinants of health worker practices were more likely to be effective than single interventions13–17; (2) our view that key categories of strategies to improve health worker performance in low-resource settings (aside from training) included supervision (to model ideal practices, motivate health workers, and provide focused training), job aids (as reminders and to make work easier), and incentives (for motivation); (3) our knowledge that supervision was weak in Benin (hence supervision for supervisors); and (4) the price of IMCI patient recording forms, which were considered unaffordable (hence our provision of a job aid that could replace the IMCI form).
IMCI-trained health workers in the control area received usual supports: job aids (i.e., packets of IMCI recording forms) and some IMCI-specific supervision (supplemental Table A). In addition, health workers in all areas potentially benefited from 5 vehicles for supervision donated in 2002, decentralization throughout Benin (commune supervisors were given some control over budgets), and results of our surveys (shared at least annually).
Methods for evaluating health care quality are described elsewhere.12,18 Briefly, after obtaining consent from health workers and children's caretakers (usually the mother), we collected data with 5 methods: (1) silent observation of consultations, with observations recorded on a checklist; (2) interviews with caretakers as they left the facility to ascertain prescribed medications and caretakers' understanding of treatment instructions; (3) child reexamination by a study clinician, out of the health worker's view, to obtain an expert, independent determination of the child's IMCI illness classifications; (4) health facility assessments to evaluate supplies and equipment; and (5) health worker interviews to obtain information on demographics, training, supervision, opinions, and knowledge. After reexamination, inadequately treated children were given appropriate medications without charge. Surveyors were trained until the agreement of practice results of surveyors and study investigators was greater than 90%.
Definitions of illness classifications (e.g., malaria, pneumonia) were based on Benin's adaptation19 of WHO's generic IMCI guidelines.1 (Potentially life-threatening illnesses, essential treatments, and treatment quality categories are defined in Box A, available as a supplement to the online version of this article at http://www.ajph.org.)
Data were double-entered and verified with Epi Info, version 6 (Centers for Disease Control and Prevention, Atlanta, GA). Analyses were restricted to initial consultations. Analyses were performed with SAS version 9.1 (SAS Institute Inc, Cary, NC). For hypothesis testing and confidence interval (CI) estimation, α was set at .05.
We analyzed 3 quality-of-care outcomes. Two were dichotomous treatment outcomes: recommended treatment and recommended or adequate treatment (supplemental Box A). The latter outcome was the complement of inadequate treatment (i.e., the percentage receiving recommended or adequate treatment equaled 100% minus the percentage receiving inadequate treatment). The third outcome was a continuous, child-level IMCI adherence index from 0% to 100%, defined as the percentage of needed IMCI tasks that were performed (see Box B, available as a supplement to the online version of this article at http://www.ajph.org).
We first performed intention-to-treat analyses. For each outcome, a regression model was constructed with dummy variables for time (2001 and 2004, with 1999 as the referent), study area (intervention or control), and 2 study area–time interactions (see Box C, available as a supplement to the online version of this article at http://www.ajph.org). The interactions, which compared intervention-area time trends (1999–2001 and 1999–2004) with control-area trends, were the main effects. The 2002 survey was omitted because it did not cover the entire study area.
For the dichotomous treatment outcomes, logistic regression modeling was performed with the SAS GENMOD procedure. For the continuous adherence outcome, linear regression modeling was performed with the REGRESS procedure in SUDAAN version 8.0.0 (Research Triangle Institute, Research Triangle Park, NC). Both procedures use generalized estimating equations to account for correlation (i.e., similarity of health care quality for children in the same cluster). We used an exchangeable working correlation structure. For all outcomes, we evaluated 20 factors (e.g., medicine availability, supervision, caseload, demographic factors, and case complexity) as potential confounders of the study area–outcome association by entering factors into models 1 at a time. Factors that changed main effects by more than 20% were considered confounders and retained in the final model.
Effect sizes were defined as absolute percentage point difference of differences (e.g., [follow-up – baseline]intervention – [follow-up – baseline]control) derived from adjusted outcome values (Table 1 and supplemental Box C).14 For dichotomous outcomes, effect sizes were estimated with predicted probabilities (i.e., adjusted outcome values) from the logistic regression models at baseline and follow-up time points for the intervention and control areas, with confounders held constant; 95% CIs were estimated with bootstrapping.20 For the continuous outcome, effect sizes and 95% CIs were obtained directly from the model.
TABLE 1 Per-Protocol Analysis of the Effect of Study Supports and Integrated Management of Childhood Illness (IMCI) Training on Case Management Quality for Ill Children During Initial Consultations at Outpatient Health Facilities in Ouémé and Plateau Departments: Benin, 1999–2004
| Outcomes for the 3 Health Worker Exposure Groups | Recommended Treatment | Recommended or Adequate Treatment | Percentage of Needed Tasks Performed per Child |
| IMCI trained with study supports | |||
| Baseline | |||
| Consultations, no. | 102 | 102 | 123 |
| Unadjusted outcome value, %a | 15.7 | 22.6 | 21.3 |
| Adjusted outcome value, %a | 15.8 | 22.0 | 21.5 |
| Follow-up | |||
| Consultations, no. | 127 | 127 | 146 |
| Unadjusted outcome value, %a | 54.3 | 55.1 | 76.6 |
| Adjusted outcome value, %a | 62.2 | 62.1 | 77.4 |
| IMCI trained with usual supports | |||
| Baseline | |||
| Consultations, no. | 106 | 106 | 119 |
| Unadjusted outcome value, %a | 21.7 | 25.5 | 24.4 |
| Adjusted outcome value, %a | 21.3 | 25.5 | 24.9 |
| Follow-up | |||
| Consultations, no. | 238 | 238 | 265 |
| Unadjusted outcome value, %a | 37.4 | 45.8 | 64.0 |
| Adjusted outcome value, %a | 40.4 | 50.3 | 65.2 |
| Not IMCI trained | |||
| Baseline | |||
| Consultations, no. | 164 | 164 | 188 |
| Unadjusted outcome value, %a | 19.5 | 27.4 | 25.6 |
| Adjusted outcome value, %a | 17.0 | 24.8 | 26.2 |
| Follow-up | |||
| Consultations, no. | 364 | 364 | 403 |
| Unadjusted outcome value, %a | 17.0 | 22.0 | 29.9 |
| Adjusted outcome value, %a | 17.1 | 22.8 | 31.8 |
| Effect sizesbc | |||
| Effect of study supports,d percentage-point change (95% CI) | 27.3* (10.8, 44.5) | 15.3 (–2.3, 33.5) | 15.5** (9.0, 22.0) |
| Effect of IMCI training,e percentage-point change (95% CI) | 19.1* (4.2, 33.5) | 26.8** (12.9, 41.3) | 34.7** (28.5, 40.9) |
| Effect of IMCI training + study supports,f percentage-point change (95% CI) | 46.4** (35.5, 62.1) | 42.1** (27.2, 59.5) | 50.2** (45.8, 54.7) |
Note. CI = confidence interval. The baseline period was 1999. The follow-up period was 2001–2004.
a Columns 2 and 3 show percentages; column 4 shows mean percentages.
b Based on quality of care adjusted for case complexity. In the model used to estimate the effect size for recommended treatment, the intraclass correlation coefficient was 0.094; the mean cluster size was 4.00 (1101 consultations/275 clusters); and the design effect was 1.28. In the model for recommended or adequate treatment, the intraclass correlation was 0.071; the mean cluster size was 4.00; and the design effect was 1.21. For percentage of needed tasks performed per child, the intraclass correlation was 0.388; the mean cluster size was 4.44 (1244 consultations/280 clusters); the design effect was 2.33; and the multiple R2 was 0.731.
c For example, the value 27.3 for the effect of study supports equals the improvement in treatment quality provided by health workers in the IMCI-trained group receiving study supports from baseline to follow-up (i.e., 62.2%–15.8%, or 46.4 percentage points) minus the improvement in the IMCI-trained group receiving usual supports from baseline to follow-up (i.e., 40.4%–21.3%, or 19.1 percentage points). The time × study supports interaction term from the multivariate model was statistically significant, and the 95% CI of the 27.3 percentage-point effect size excluded 0 (i.e., was statistically significant).
d Study supports versus usual supports for IMCI-trained health workers.
e IMCI-trained health workers with usual supports versus health workers with no IMCI training.
f IMCI-trained health workers with study supports versus health workers with no IMCI training. This is the sum of effects from the IMCI group receiving study supports versus the IMCI group receiving usual supports and the IMCI group receiving usual supports versus the group receiving no IMCI training.
* P value of the interaction term from the multivariable regression model is between .01 and .049.
** P value of the interaction term from the multivariable regression model is <.01.
Unfortunately, because IMCI training proceeded slowly, many consultations in the follow-up surveys were performed by health workers who had not received IMCI training and who provided low-quality care. Thus, in the intention-to-treat analyses, effects of the study supports were diluted because both study areas contained a mix of IMCI-trained (and better performing) health workers and nontrained health workers.
To account for slow IMCI implementation, we also performed per-protocol analyses with a pre–post study design with nonrandomized controls.21 Three health worker exposure groups were compared: IMCI trained in intervention areas (all received study supports), IMCI trained in control areas (all received usual supports), and non–IMCI trained in any geographic area (per-protocol analysis controls). We used the same methods as in the intention-to-treat analysis (supplemental Box C), except for 3 points. First, we included the 2002 survey. Second, because sample sizes for individual years were sometimes small and effect sizes were generally similar, we combined all follow-up surveys. Third, to estimate baseline (pre-IMCI) outcome values for the 3 health worker exposure groups, we divided the 1999 survey into 3 parts: the 4 IMCI pilot communes in the intervention area (zone 1 [supplemental Figure A], as baseline for the IMCI study supports group), the 4 IMCI pilot communes in the control area (zone 3, as baseline for the IMCI usual-supports group), and the 8 communes that were not IMCI pilots (zones 2 and 4, as baseline for the non–IMCI group).
In per-protocol analyses of dichotomous treatment outcomes, case complexity (mean-centered number of IMCI tasks needed per child [supplemental Box B], which helped adjust for case mix) was identified as a confounder and retained in all models (supplemental Box C, last term in model). For comparability, case complexity was added to models for the continuous-adherence index.
For the treatment outcomes, incremental cost-effectiveness analyses were conducted for 3 interventions: study supports, IMCI training plus usual supports, and IMCI training plus study supports (Table B, available as a supplement to the online version of this article at http://www.ajph.org). We took the provider perspective (i.e., the perspective of an organization that funded quality-improvement strategies, such as a governmental or donor organization). The time horizon was the follow-up period (December 2001–October 2004). We did not use discounting because the follow-up period was relatively short. Incremental costs were the variable costs for IMCI training ($850 per health worker) and study and usual supports (supplemental Table A). Incremental effects were estimated by multiplying effect sizes (Table 1) by the estimated number of children with potentially life-threatening illnesses seen at all eligible health facilities over the follow-up period. Because caseloads differed between the group receiving study supports and the one receiving usual supports, incremental costs and effects were scaled to a constant 100 000 consultations. For the incremental effect, we only considered children seen on weekdays during regular working hours (to match our survey methods), a conservative decision that probably led to an underestimation of the true effects.
We observed 1577 ill-child consultations, including 1244 initial consultations for any illness (Table 2). Caretakers of an additional 25 children either refused to participate or withdrew (participation rate = 1577/1602, or 98.4%). Among initial consultations, according to the study's expert reexaminations, the most common IMCI illness classifications were uncomplicated malaria (69.9%), mild anemia (44.0%), uncomplicated pneumonia (24.0%), and diarrhea (15.0%); 1101 (88.5%) children had a potentially life-threatening illness, and 196 (15.8%) had a severe classification. Dysentery, measles, and malnutrition (other than anemia) were uncommon ( < 3%). Initial consultations were observed during 301 health facility visits to 114 health facilities and performed by 267 health workers. In the follow-up period, even though the plan called for all children to be seen by IMCI-trained health workers, only half were (411 of 814 initial consultations; Table 2).
TABLE 2 Survey Sampling, Enrollment of Study Participants, and Scale-Up of Integrated Management of Childhood Illness (IMCI) Guidelines: Benin, 1999–2004
| Baseline Survey (Before IMCI Training) | Follow-Up Survey 1 | Follow-Up Survey 2 | Follow-Up Survey 3 | ||
| Data collection | July 28–Oct 29, 1999 | Dec 3, 2001–Jan 25, 2002 | Sept 16–Oct 30, 2002 | July 19–Oct 6, 2004 | |
| Zones sampleda | 1, 2, 3, 4 | 1, 2, 3, 4 | 1, 3 | 1, 2, 3, 4 | |
| Surveys in intervention area | |||||
| Health facilities sampled/total eligible, no. | 48/48 | 55/60 | 22/22 | 54/72 | |
| Health facilities open and with ≥ 1 ill child consultation | 41 | 47 | 18 | 40 | |
| IMCI-trained health workers/total health workers who performed ≥ 1 ill child consultation, no. (%) | 0/50 (0) | 21/69 (30.4) | 15/28 (53.6) | 34/58 (58.6) | |
| Among IMCI-trained health workers, median time since IMCI training, mo (range) | … | 2 (1–4) | 12 (1–16) | 20 (1–37) | |
| Ill children enrolled,b no. | 288 | 225 | 77 | 156 | |
| Initial consultations performed by IMCI-trained health workers/total initial consultations, no. (%) | 0/225 (0) | 52/186 (30.0) | 39/65 (60.0) | 63/126 (50.0) | |
| Surveys in control area | |||||
| Health facilities sampled/total eligible, no. | 39/39 | 45/51 | 33/33 | 46/58 | |
| Health facilities open and with ≥ 1 ill child consultation | 38 | 40 | 30 | 40 | |
| IMCI-trained health workers/total health workers who performed ≥ 1 ill child consultation, no. (%) | 0/51 (0) | 23/47 (48.9) | 17/37 (45.9) | 31/53 (58.5) | |
| Among IMCI-trained health workers, median time since IMCI training, mo (range) | … | 4 (1–6) | 12 (2–15) | 20 (1–37) | |
| Ill children enrolled,b no. | 295 | 168 | 154 | 214 | |
| Initial consultations performed by IMCI-trained health workers/total initial consultations, no. (%) | 0/205 (0) | 71/127 (55.9) | 61/138 (44.2) | 125/172 (72.7) | |
Note. Ellipses indicate that the data are not applicable.
a Zone 1 was the trial intervention area in which IMCI was piloted in 2001, zone 2 was the trial intervention area in which IMCI began in 2003, zone 3 was the trial control area in which IMCI was piloted in 2001, and zone 4 was the trial control area in which IMCI began in 2003.
b Excluded 1 child in 1999 (consultation accidentally not observed), 4 in 2001 (all withdrew after initially agreeing), 10 in 2002 (6 refused, 4 withdrew), and 10 in 2004 (8 refused, 2 withdrew).
Characteristics of study groups were similar (Table C, available as a supplement to the online version of this article at http://www.ajph.org), except that children in control areas were significantly more often seen by IMCI-trained health workers in the follow-up period (P = .003). For all outcomes, health care quality improved over time (Figure 2); however, differences in improvements between intervention and control areas were close to zero and not statistically significant. The intention-to-treat analysis, which was of a randomized controlled study that was flawed by slow IMCI implementation, yielded no evidence that the study supports improved treatment quality.
FIGURE 2 Intention-to-treat analysis of the effect of posttraining supports for health workers trained to use Integrated Management of Childhood Illness (IMCI) guidelines on (a) the percentage of children receiving recommended care, (b) the percentage of children receiving recommended or adequate care, and (c) the mean percentage of needed tasks performed per child: Benin, 1999–2004.
Note. 1999 was the baseline period (pre-IMCI); 2001–2004 was the follow-up period.
Characteristics of exposure groups were generally similar, and small differences were an unlikely source of bias (supplemental Table C).
IMCI-trained health workers frequently used job aids, which reflected use of IMCI guidelines. In 91.0% (374 of 411) of consultations, health workers used the IMCI flipchart of clinical algorithms. The IMCI group receiving usual supports used IMCI recording forms in 84.9% (225 of 265) of consultations. By contrast, the IMCI group receiving study supports rarely (10 of 146, or 6.8%) used IMCI recording forms; instead, as intended, study registers were almost always used (144 of 146, or 98.6%). Surprisingly, the IMCI group receiving study supports rarely used the study counseling guide (14 of 146, or 10.2% of consultations).
Recommended treatment improved over time in both IMCI groups (Table 1, Figure 3a; see also Tables D and E and Box D, available as supplements to the online version of this article at http://www.ajph.org). Improvements in the IMCI group receiving study supports were 27.3 percentage points (95% CI = 10.8, 44.5) greater than in the IMCI group receiving usual supports, which reflected the effect of the study supports. Improvements in the IMCI group receiving usual supports were 19.1 percentage points (95% CI = 4.2, 33.5) greater than in the group without IMCI training, which reflected the effect of IMCI training. Summing these effects reveals the large improvement attributable to IMCI training plus study supports (46.4 percentage points). Percentages for individual years are shown in supplemental Table D, and effect sizes for individual years (i.e., 1999 vs 2001, 1999 vs 2002, and 1999 vs 2004) are shown in supplemental Table E.
FIGURE 3 Per-protocol analysis of the effect of training on Integrated Management of Childhood Illness (IMCI) guidelines and posttraining supports, by exposure to interventions on (a) the percentage of children receiving recommended care, (b) the percentage of children receiving recommended or adequate care, and (c) the mean percentage of needed tasks performed per child: Benin, 1999–2004.
Note. 1999 was the baseline period (pre-IMCI); 2001–2004 was the follow-up period.
aAll outcomes were adjusted for case complexity.
bResults for the IMCI group receiving study supports were significantly greater than those for the IMCI group receiving usual supports only for the comparison of 1999 versus 2004. Results for the IMCI group receiving usual supports were significantly greater than were those for the group without IMCI training only for the comparison of 1999 versus 2001. Results for the IMCI group receiving study supports were significantly greater than those for the group without IMCI training for all years.
cResults for the IMCI group receiving study supports were never significantly greater than those for the IMCI group receiving usual supports. Results for the IMCI group receiving usual supports were significantly greater than those for the group without IMCI training for the comparisons of 1999 versus 2001 and of 1999 versus 2004. Results for the IMCI group receiving study supports were significantly greater than those for the group without IMCI training for all years.
dResults for the IMCI group receiving study supports were significantly greater than those for the IMCI group receiving usual supports for the comparisons of 1999 versus 2002 and of 1999 versus 2004. Results for the IMCI group receiving usual supports were significantly greater than those for the group without IMCI training for all years. Results for the IMCI group receiving study supports were significantly greater than those for the group without IMCI training for all years.
For recommended or adequate treatment and the adherence index (Figure 3), effects were roughly similar to those of recommended treatment, except that the effect of study supports did not reach statistical significance for recommended or adequate treatment (15.3 percentage points; 95% CI = –2.3, 33.5). In a sensitivity analysis for recommended or adequate treatment, which assumed dosage quality was adequate when prescriptions were incomplete, results were generally similar to the main analysis, except that effect sizes were smaller (results not shown).
Costs per 100 000 consultations for health worker supports ($11 232–$27 046) were considerably less than for IMCI training ($132 617–$218 961), and study supports were only somewhat more expensive than usual supports (supplemental Table A). Compared with usual supports, study supports cost $0.58 (95% CI = $0.36, $1.46) per additional child receiving recommended treatment (i.e., the cost-effectiveness ratio; supplemental Table B). The cost-effectiveness ratio for recommended or adequate treatment was $1.03, although study supports could have had a small negative effect. Cost-effectiveness ratios comparing no IMCI to IMCI plus usual or study supports ranged from $5 to $8, although CIs were sometimes wide.
Although in-service training is often used to implement clinical guidelines, numerous studies show that training alone is insufficient for achieving high levels of adherence.22 Thus, we addressed the question, “What should follow training?” Our study focused on IMCI, but the results (and challenges) are relevant to the broader issue of increasing health worker adherence to any clinical guideline.
Per-protocol analyses showed that study supports were significantly associated with an increase of 27 percentage points in recommended treatment and an increase of 16 percentage points in the proportion of IMCI tasks performed—increases that were in addition to improvements attributed to IMCI training with usual supports. A nonsignificant trend was noted toward improved recommended or adequate treatment (and thus reduced inadequate treatment). Compared with usual supports, the extra cost of study supports was relatively low: approximately $16 000 per 100 000 children with a potentially life-threatening illness, or less than $1 per additional child receiving recommended treatment. We are not aware of other published studies that evaluated supports after IMCI training; however, a review of strategies to improve health workers' use of pharmaceuticals in developing countries found a median maximum effect size of 27 percentage points (range: 16–56) among 6 studies of administrative or managerial interventions such as supervision.14
Regarding IMCI effectiveness, we found that IMCI training with usual supports was significantly associated with improvements of 19 to 35 percentage points for all 3 outcomes compared with no IMCI training. These results fall on the low end of the wide range of results from other studies that evaluated IMCI training without additional health worker or health system supports (effect sizes for correct oral antimicrobial use from 3 studies ranged from 28.6 to 94.0 percentage points [A. K. Rowe et al., unpublished data, October 2008]), although varying outcome definitions and study designs complicate such comparisons. Compared with no IMCI, effect sizes for IMCI training with study supports were greater, ranging from 42 to 50 percentage points. These results compare favorably to those of other studies that evaluated IMCI training with additional health worker or health system supports (effect sizes for correct oral antimicrobial use from 7 studies ranged from 3.9 to 75.7 percentage points [A. K. Rowe et al., unpublished data, October 2008]).
Our study had 2 main complications: the intervention and the study design changed. These issues illustrate the challenges of health systems research but also exemplify an important lesson on changing health systems. The intervention changed because little supervision occurred initially. We reasoned that doing nothing risked having intervention areas receive even less support than control areas, which was ethically unacceptable; we also believed that the timing was early enough that a new support (i.e., quarterly supervision workshops) could be introduced and still grouped with the original study supports.
The second complication arose because IMCI-trained health workers performed fewer than expected consultations. Initially, the explanation was training delays, but we later learned of other reasons, such as health workers without IMCI training being assigned busy weekday shifts and some health workers resisting IMCI guidelines (A. K. Rowe, unpublished data, May 2008). We shared these findings with government officials on several occasions, which resulted in only modest improvement. Regardless of the cause, we thought that changing from a randomized design (intention-to-treat analysis) to a pre–post study with nonrandomized controls (per-protocol analysis) would give a more accurate picture of our intervention's effect.21
Whenever an intention-to-treat analysis is substituted with one comparing participants who were or were not exposed to interventions, the relation between determinants of exposure and outcomes must be examined (e.g., in clinical trials, patients who are too ill to complete experimental treatment are more likely to die). During our study's follow-up period, whether a child was seen by a health worker with no IMCI training or an IMCI-trained health worker receiving study or usual supports was essentially by chance, because it seemed highly unlikely that caretakers knew the training and support background of health workers. For health workers, IMCI exposure depended on government selection criteria for training: nursing aides were ineligible, all facilities needed at least 1 IMCI-trained health worker, and preference was given to health facility directors (to promote acceptance of IMCI) and health workers with high caseloads of ill children. Thus, although the per-protocol analysis might have introduced bias in comparisons of IMCI-trained and nontrained health workers, there were no obvious sources of bias in comparisons of study supports and usual supports.
Our study had other potential limitations. First, observation of consultations could have influenced health worker practices, perhaps overestimating quality somewhat23; this influence would likely have affected all study groups similarly, however, and thus probably did not substantially bias effect sizes. Second, prescriptions were sometimes incomplete, which probably led to the underestimation of quality for some consultations. Our sensitivity analysis suggested some bias in effect sizes, but overall conclusions were not affected. Third, it is difficult to assess the validity of the assumption that control-area supports generally reflected supports elsewhere. However, we described our study area and usual supports; readers can evaluate how applicable this assumption is to other settings.
From a programmatic perspective, despite our generally positive results, we do not necessarily recommend our supports for implementing IMCI or other guidelines. The experience and complications of this study underscored the importance of continuous monitoring so that managers can identify problems and evaluate whether solutions are working. For example, although we did not formally implement a monitoring system, our occasional observations revealed in late 2001 that almost no supervision was occurring and later that supervision increased after we implemented supervision workshops. This experience also highlighted the importance of being flexible and modifying interventions when they fail. Therefore, programs should consider implementing specific interventions (e.g., the workshops, job aids, and incentives we studied) in the context of a quality-improvement process with ongoing plan–do–study–act cycles (i.e., develop and implement quality-improvement activities, monitor effects, and either continue or modify activities).24
Although our supervision protocol included some quality-improvement elements, we initially decided not to implement a formal quality-improvement process because it seemed complicated, and scant evidence supported its use in developing countries (some successes have since been described25–27). We now believe that such a process can be useful, or at least that not using it risks failure. Before the quality-improvement process is implemented widely, however, further evaluations are needed to better characterize its effectiveness and cost, how it should be operationalized in large health systems with weak infrastructure, and how it responds to common, intractable problems that might only be solved at the ministerial or international level (e.g., insufficient budgets). Given the size of new, exciting disease-control initiatives, such as those for AIDS and malaria, in which the provision of expensive medicines and commodities is being scaled up, implementation research on the quality-improvement process should be an urgent priority.
From a methodologic perspective, when analyzing results of dichotomous performance outcomes, we found it useful to construct logistic regression models that adjusted for confounders and then to estimate difference-of-differences effect sizes and CIs on an arithmetic scale with predicted probabilities from the models (supplemental Box C). To the best of our knowledge, ours was the first study of health worker performance to use this method. Such effect sizes are easy to understand and can be directly fed into cost-effectiveness analyses. This approach can be used with simpler study designs, such as controlled postintervention–only studies.
Our study showed that training can be useful for implementing clinical guidelines but that it is not enough. Relatively inexpensive posttraining supports can lead to additional improvements. Perhaps better still, programs should consider implementing supports in the context of a quality-improvement process. In other words, program managers should pay attention, act on results, and be flexible. Implementation research on the quality-improvement process should be an urgent priority. Finally, wherever IMCI training is implemented, program managers should ensure that IMCI-trained health workers perform consultations.
Acknowledgments
Funding was provided by the United States Agency for International Development's Africa Integrated Malaria Initiative (project 936-3100).
We are indebted to the many community members, health workers, supervisors, surveyors, drivers, and support staff who gave their time and energy to make this project possible. In particular, we thank Loukmane Agbo-Ola and Paul Kple-Faget for their support of our research activities and François Cokou for his assistance with data management. We acknowledge the support of Africare, the managing partner responsible for the implementation of the Africa Integrated Malaria Initiative in Benin with the Benin Ministry of Health.
Human Participant Protection
The study protocol was approved by the ethics committee of the Benin Ministry of Health and the Centers for Disease Control and Prevention's human subjects review board. The 1999 survey was considered program evaluation and written consent was not required; verbal consent, however, was requested from all participants (health workers and children's caretakers). Surveys from 2001 to 2004 were considered research, and written informed consent was requested from all participants.
