Adjusting for multiple testing when reporting research results: the Bonferroni vs Holm methods.
M Aickin and
H Gensler
Arizona Cancer Center, University of Arizona, Tucson 85724, USA.
Public health researchers are sometimes required to make adjustmentsfor multiple testing in reporting their results, which reducesthe apparent significance of effects and thus reduces statisticalpower. The Bonferroni procedure is the most widely recommendedway of doing this, but another procedure, that of Holm, is uniformlybetter. Researchers may have neglected Holm's procedure becauseit has been framed in terms of hypothesis test rejection ratherthan in terms of P values. An adjustment to P values based onHolm's method is presented in order to promote the method'suse in public health research.
Related articles in AJPH:
On the Holm, Simes, and Hochberg multiple test procedures.
M. J. Yelland, K. R. Sweeting, J. A. Lyftogt, S. K. Ng, P. A. Scuffham, and K. A. Evans Prolotherapy injections and eccentric loading exercises for painful Achilles tendinosis: a randomised trial
Br. J. Sports Med.,
July 6, 2010;
(2010)
bjsm.2009.057968v3.
[Abstract][Full Text]
D. P. Belobrajdic, J. Frystyk, N. Jeyaratnaganthan, U. Espelund, A. Flyvbjerg, P. M. Clifton, and M. Noakes Moderate energy restriction-induced weight loss affects circulating IGF levels independent of dietary composition
Eur. J. Endocrinol.,
June 1, 2010;
162(6):
1075 - 1082.
[Abstract][Full Text][PDF]
G. Michel, C. E. Rebholz, N. X. von der Weid, E. Bergstraesser, and C. E. Kuehni Psychological Distress in Adult Survivors of Childhood Cancer: The Swiss Childhood Cancer Survivor Study
J. Clin. Oncol.,
April 1, 2010;
28(10):
1740 - 1748.
[Abstract][Full Text][PDF]
A. Green, A. Unaldi, and C. Weir Empiricism versus connoisseurship: Establishing the appropriacy of texts in tests of academic reading
Language Testing,
April 1, 2010;
27(2):
191 - 211.
[Abstract][PDF]
G. B. McAnulty, S. C. Butler, J. H. Bernstein, H. Als, F. H. Duffy, and D. Zurakowski Effects of the Newborn Individualized Developmental Care and Assessment Program (NIDCAP) at Age 8 Years: Preliminary Data
Clinical Pediatrics,
March 1, 2010;
49(3):
258 - 270.
[Abstract][PDF]
C. L. Craig, A. Bauman, and B. Reger-Nash Testing the hierarchy of effects model: ParticipACTION's serial mass communication campaigns on physical activity in Canada
Health Promot. Int.,
March 1, 2010;
25(1):
14 - 23.
[Abstract][Full Text][PDF]
A. A. Tarnutzer, C. J. Bockisch, and D. Straumann Roll-Dependent Modulation of the Subjective Visual Vertical: Contributions of Head- and Trunk-Based Signals
J Neurophysiol,
February 1, 2010;
103(2):
934 - 941.
[Abstract][Full Text][PDF]
J.S. Bell, S.E. Aaltonen, M.S. Airaksinen, D. Volmer, M.S. Gharat, R. Muceniece, A. Vitola, V. Foulon, F.A. Desplenter, and T.F. Chen Determinants of Mental Health Stigma Among Pharmacy Students in Australia, Belgium, Estonia, Finland, India and Latvia
International Journal of Social Psychiatry,
January 1, 2010;
56(1):
3 - 14.
[Abstract][PDF]
K. D. Ahuja, I. K Robertson, and M. J Ball Acute effects of food on postprandial blood pressure and measures of arterial stiffness in healthy humans
Am. J. Clinical Nutrition,
August 1, 2009;
90(2):
298 - 303.
[Abstract][Full Text][PDF]
C. L. Kidner, T. G. Mayer, and R. J. Gatchel Higher Opioid Doses Predict Poorer Functional Outcome in Patients with Chronic Disabling Occupational Musculoskeletal Disorders
J. Bone Joint Surg. Am.,
April 1, 2009;
91(4):
919 - 927.
[Abstract][Full Text][PDF]
G. Thomalla, H. R. Siebner, M. Jonas, T. Baumer, K. Biermann-Ruben, F. Hummel, C. Gerloff, K. Muller-Vahl, A. Schnitzler, M. Orth, et al. Structural changes in the somatosensory system correlate with tic severity in Gilles de la Tourette syndrome
Brain,
March 1, 2009;
132(3):
765 - 777.
[Abstract][Full Text][PDF]
A. Robinson, A. Manzi, and J. Triesch Object perception is selectively slowed by a visually similar working memory load
J Vis,
December 1, 2008;
8(16):
7 - 7.
[Abstract][Full Text][PDF]
F. Calabrese, S. Baraldo, E. Bazzan, F. Lunardi, F. Rea, P. Maestrelli, G. Turato, K. Lokar-Oliani, A. Papi, R. Zuin, et al. IL-32, a Novel Proinflammatory Cytokine in Chronic Obstructive Pulmonary Disease
Am. J. Respir. Crit. Care Med.,
November 1, 2008;
178(9):
894 - 901.
[Abstract][Full Text][PDF]
M. D. Ris, D. W. Beebe, F. D. Armstrong, J. Fontanesi, E. Holmes, R. A. Sanford, and J. H. Wisoff Cognitive and Adaptive Outcome in Extracerebellar Low-Grade Brain Tumors in Children: A Report From the Children's Oncology Group
J. Clin. Oncol.,
October 10, 2008;
26(29):
4765 - 4770.
[Abstract][Full Text][PDF]
P. Brantley, L. Appel, J. Hollis, V. Stevens, J. Ard, C. Champagne, P. Elmer, D. Harsha, V. Myers, M. Proschan, et al. Design considerations and rationale of a multi-center trial to sustain weight loss: the weight loss maintenance trial
Clinical Trials,
October 1, 2008;
5(5):
546 - 556.
[Abstract][PDF]
U. Yamaguchi, R. Nakayama, K. Honda, H. Ichikawa, T. Hasegawa, M. Shitashige, M. Ono, A. Shoji, T. Sakuma, H. Kuwabara, et al. Distinct Gene Expression-Defined Classes of Gastrointestinal Stromal Tumor
J. Clin. Oncol.,
September 1, 2008;
26(25):
4100 - 4108.
[Abstract][Full Text][PDF]
S. Ish-Shalom, E. Segal, T. Salganik, B. Raz, I. L. Bromberg, and R. Vieth Comparison of Daily, Weekly, and Monthly Vitamin D3 in Ethanol Dosing Protocols for Two Months in Elderly Hip Fracture Patients
J. Clin. Endocrinol. Metab.,
September 1, 2008;
93(9):
3430 - 3435.
[Abstract][Full Text][PDF]
G. Turato, A. Barbato, S. Baraldo, M. E. Zanin, E. Bazzan, K. Lokar-Oliani, F. Calabrese, C. Panizzolo, D. Snijders, P. Maestrelli, et al. Nonatopic Children with Multitrigger Wheezing Have Airway Pathology Comparable to Atopic Asthma
Am. J. Respir. Crit. Care Med.,
September 1, 2008;
178(5):
476 - 482.
[Abstract][Full Text][PDF]
L. Daniels, R. A Gibson, K. Simmer, P. Van Dael, and M. Makrides Selenium status of term infants fed selenium-supplemented formula in a randomized dose-response trial
Am. J. Clinical Nutrition,
July 1, 2008;
88(1):
70 - 76.
[Abstract][Full Text][PDF]
M. M. Madeleine, L. G. Johnson, A. G. Smith, J. A. Hansen, B. B. Nisperos, S. Li, L.-P. Zhao, J. R. Daling, S. M. Schwartz, and D. A. Galloway Comprehensive Analysis of HLA-A, HLA-B, HLA-C, HLA-DRB1, and HLA-DQB1 Loci and Squamous Cell Cervical Cancer Risk
Cancer Res.,
May 1, 2008;
68(9):
3532 - 3539.
[Abstract][Full Text][PDF]
L. P. Svetkey, V. J. Stevens, P. J. Brantley, L. J. Appel, J. F. Hollis, C. M. Loria, W. M. Vollmer, C. M. Gullion, K. Funk, P. Smith, et al. Comparison of Strategies for Sustaining Weight Loss: The Weight Loss Maintenance Randomized Controlled Trial
JAMA,
March 12, 2008;
299(10):
1139 - 1148.
[Abstract][Full Text][PDF]
J.-P. Rospars, P. Lansky, M. Chaput, and P. Duchamp-Viret Competitive and Noncompetitive Odorant Interactions in the Early Neural Coding of Odorant Mixtures
J. Neurosci.,
March 5, 2008;
28(10):
2659 - 2666.
[Abstract][Full Text][PDF]
M. M. Lell, C. Gmelin, C. Panknin, K. T. Eckel, M. Schmid, W. A. Bautz, and H. Greess Thin-Slice MDCT of the Neck: Impact on Cancer Staging
Am. J. Roentgenol.,
March 1, 2008;
190(3):
785 - 789.
[Abstract][Full Text][PDF]
M. Azizi, J. Menard, A. Bissery, T.-T. Guyene, and A. Bura-Riviere Hormonal and Hemodynamic Effects of Aliskiren and Valsartan and Their Combination in Sodium-Replete Normotensive Individuals
Clin. J. Am. Soc. Nephrol.,
September 1, 2007;
2(5):
947 - 955.
[Abstract][Full Text][PDF]
J. F Morgan p Value fetishism and use of the Bonferroni adjustment
Evid. Based Ment. Health,
May 1, 2007;
10(2):
34 - 35.
[Full Text][PDF]
J. M. Saucier, J. Yu, A. Gaikwad, R. L. Coleman, J. K. Wolf, and J. A. Smith Determination of the optimal combination chemotherapy regimen for treatment of platinum-resistant ovarian cancer in nude mouse model
Journal of Oncology Pharmacy Practice,
March 1, 2007;
13(1):
39 - 45.
[Abstract][PDF]
A. A. Tarnutzer, S. Ramat, D. Straumann, and D. S. Zee Pursuit Responses to Target Steps During Ongoing Tracking
J Neurophysiol,
February 1, 2007;
97(2):
1266 - 1279.
[Abstract][Full Text][PDF]
D. W. Beebe, D. Lewin, M. Zeller, M. McCabe, K. MacLeod, S. R. Daniels, and R. Amin Sleep in Overweight Adolescents: Shorter Sleep, Poorer Sleep Quality, Sleepiness, and Sleep-Disordered Breathing
J. Pediatr. Psychol.,
January 1, 2007;
32(1):
69 - 79.
[Abstract][Full Text][PDF]
J S. Bell, R. Johns, G. Rose, and T. F Chen A Comparative Study of Consumer Participation in Mental Health Pharmacy Education
Ann. Pharmacother.,
October 1, 2006;
40(10):
1759 - 1765.
[Abstract][Full Text][PDF]
K. D. Ahuja, I. K Robertson, D. P Geraghty, and M. J Ball Effects of chili consumption on postprandial glucose, insulin, and energy metabolism
Am. J. Clinical Nutrition,
July 1, 2006;
84(1):
63 - 69.
[Abstract][Full Text][PDF]
L. Kruse, A. M. Mitchell, C. A. Camargo Jr, J. Hernandez, and J. A. Kline Frequency of Thrombophilia-Related Genetic Variations in Patients with Idiopathic Pulmonary Embolism in an Urban Emergency Department
Clin. Chem.,
June 1, 2006;
52(6):
1026 - 1032.
[Abstract][Full Text][PDF]
M. T. Sandinha, M. A. Farquharson, I. C. McKay, and F. Roberts Monosomy 3 Predicts Death but Not Time until Death in Choroidal Melanoma
Invest. Ophthalmol. Vis. Sci.,
October 1, 2005;
46(10):
3497 - 3501.
[Abstract][Full Text][PDF]
J. E. Aikens, R. Bingham, and J. D. Piette Patient-Provider Communication and Self-care Behavior Among Type 2 Diabetes Patients
The Diabetes Educator,
September 1, 2005;
31(5):
681 - 690.
[Abstract][Full Text][PDF]
D. J. English, R. Thompson, J. C. Graham, and E. C. Briggs Toward a Definition of Neglect in Young Children
Child Maltreat,
May 1, 2005;
10(2):
190 - 206.
[Abstract][PDF]
S. Kuhlmann, M. Piel, and O. T. Wolf Impaired Memory Retrieval after Psychosocial Stress in Healthy Young Men
J. Neurosci.,
March 16, 2005;
25(11):
2977 - 2982.
[Abstract][Full Text][PDF]
J. G. Ray Neural tube defects
Can. Med. Assoc. J.,
January 18, 2005;
172(2):
160 - 160.
[Full Text][PDF]
J.A. C. Delaney Neural tube defects
Can. Med. Assoc. J.,
January 18, 2005;
172(2):
157 - 158.
[Full Text][PDF]
K Wickens, D Barry, A Friezema, R Rhodius, N Bone, G Purdie, and J Crane Obesity and asthma in 11-12 year old New Zealand children in 1989 and 2000
Thorax,
January 1, 2005;
60(1):
7 - 12.
[Abstract][Full Text][PDF]
M. Azizi, J. Menard, A. Bissery, T.-T. Guyenne, A. Bura-Riviere, S. Vaidyanathan, and R. P. Camisasca Pharmacologic Demonstration of the Synergistic Effects of a Combination of the Renin Inhibitor Aliskiren and the AT1 Receptor Antagonist Valsartan on the Angiotensin II-Renin Feedback Interruption
J. Am. Soc. Nephrol.,
December 1, 2004;
15(12):
3126 - 3133.
[Abstract][Full Text][PDF]
T. A. King, M. L. Gemignani, W. Li, D. D. Giri, K. S. Panageas, F. Bogomolniy, C. Arroyo, N. Olvera, M. E. Robson, K. Offit, et al. Increased Progesterone Receptor Expression in Benign Epithelium of BRCA1-Related Breast Cancers
Cancer Res.,
August 1, 2004;
64(15):
5051 - 5053.
[Abstract][Full Text][PDF]
H. Als, F. H. Duffy, G. B. McAnulty, M. J. Rivkin, S. Vajapeyam, R. V. Mulkern, S. K. Warfield, P. S. Huppi, S. C. Butler, N. Conneman, et al. Early Experience Alters Brain Function and Structure
Pediatrics,
April 1, 2004;
113(4):
846 - 857.
[Abstract][Full Text][PDF]
T. A Strand, R. K Adhikari, R. K Chandyo, P. R Sharma, and H. Sommerfelt Predictors of plasma zinc concentrations in children with acute diarrhea
Am. J. Clinical Nutrition,
March 1, 2004;
79(3):
451 - 456.
[Abstract][Full Text][PDF]
R. H. Aseltine Jr and R. DeMartino An Outcome Evaluation of the SOS Suicide Prevention Program
Am J Public Health,
March 1, 2004;
94(3):
446 - 451.
[Abstract][Full Text][PDF]
S. Jacquemont, R. J. Hagerman, M. A. Leehey, D. A. Hall, R. A. Levine, J. A. Brunberg, L. Zhang, T. Jardini, L. W. Gane, S. W. Harris, et al. Penetrance of the Fragile X-Associated Tremor/Ataxia Syndrome in a Premutation Carrier Population
JAMA,
January 28, 2004;
291(4):
460 - 469.
[Abstract][Full Text][PDF]
Q.-G. V. Nguyen, J. N. Buskin, C. L. Himeda, M. A. Shield, and S. D. Hauschka Differences in the Function of Three Conserved E-boxes of the Muscle Creatine Kinase Gene in Cultured Myocytes and in Transgenic Mouse Skeletal and Cardiac Muscle
J. Biol. Chem.,
November 21, 2003;
278(47):
46494 - 46505.
[Abstract][Full Text][PDF]
M. Cargo, L. Levesque, A. C. Macaulay, A. McComber, S. Desrosiers, T. Delormier, and L. Potvin Community governance of the Kahnawake Schools Diabetes Prevention Project, Kahnawake Territory, Mohawk Nation, Canada
Health Promot. Int.,
September 1, 2003;
18(3):
177 - 187.
[Abstract][Full Text][PDF]
Writing Group of the PREMIER Collaborative Researc Effects of Comprehensive Lifestyle Modification on Blood Pressure Control: Main Results of the PREMIER Clinical Trial
JAMA,
April 23, 2003;
289(16):
2083 - 2093.
[Abstract][Full Text][PDF]
F A Medeiros and R Susanna Jr Comparison of algorithms for detection of localised nerve fibre layer defects using scanning laser polarimetry
Br J Ophthalmol,
April 1, 2003;
87(4):
413 - 419.
[Abstract][Full Text][PDF]
R. C. Hermida, C. Calvo, D. E. Ayala, J. R. Fernandez, L. M. Ruilope, and J. E. Lopez Evaluation of the extent and duration of the "ABPM effect" in hypertensive patients
J. Am. Coll. Cardiol.,
August 21, 2002;
40(4):
710 - 717.
[Abstract][Full Text][PDF]
J. Tang, S. Tang, E. Lobashevsky, A. D. Myracle, U. Fideli, G. Aldrovandi, S. Allen, R. Musonda, and R. A. Kaslow Favorable and Unfavorable HLA Class I Alleles and Haplotypes in Zambians Predominantly Infected with Clade C Human Immunodeficiency Virus Type 1
J. Virol.,
August 15, 2002;
76(16):
8276 - 8284.
[Abstract][Full Text][PDF]
F. M. Sacks, L. P. Svetkey, W. M. Vollmer, L. J. Appel, G. A. Bray, D. Harsha, E. Obarzanek, P. R. Conlin, E. R. Miller, D. G. Simons-Morton, et al. Effects on Blood Pressure of Reduced Dietary Sodium and the Dietary Approaches to Stop Hypertension (DASH) Diet
N. Engl. J. Med.,
January 4, 2001;
344(1):
3 - 10.
[Abstract][Full Text][PDF]
D. L. Mattey, A. B. Hassell, M. J. Plant, N. T. Cheung, P. T. Dawes, P. W. Jones, W. Thomson, K. V. Poulton, A. H. Hajeer, and W. E. R. Ollier The influence of HLA-DRB1 alleles encoding the DERAA amino acid motif on radiological outcome in rheumatoid arthritis
Rheumatology,
December 1, 1999;
38(12):
1221 - 1227.
[Abstract][Full Text][PDF]
T. G. M. Vrijkotte, L. J. P. van Doornen, and E. J. C. de Geus Work Stress and Metabolic and Hemostatic Risk Factors
Psychosom Med,
November 1, 1999;
61(6):
796 - 805.
[Abstract][Full Text][PDF]
C. A. Schneider, E. Voth, D. Moka, F. M. Baer, J. Melin, A. Bol, R. Wagner, H. Schicha, E. Erdmann, and U. Sechtem Improvement of myocardial blood flow to ischemic regions by angiotensin- converting enzyme inhibition with quinaprilat IV: A study using [15O] water dobutamine stress positron emission tomography
J. Am. Coll. Cardiol.,
October 1, 1999;
34(4):
1005 - 1011.
[Abstract][Full Text][PDF]
E. Mulder, W. Linssen, J Passchier, J. Orlebeke, and E. de Geus Interictal and Postictal Cognitive Changes in Migraine
Cephalalgia,
July 1, 1999;
19(6):
557 - 565.
[Abstract][PDF]
T. V Perneger Adjusting for multiple testing in studies is less important than other concerns
BMJ,
May 8, 1999;
318(7193):
1288a - 1288.
[Full Text]
D. L Mattey, A. B Hassell, M. Plant, P. T Dawes, W. R Ollier, P. W Jones, A. A Fryer, J. E Alldersea, and R. C Strange Association of polymorphism in glutathione S-transferase loci with susceptibility and outcome in rheumatoid arthritis: comparison with the shared epitope
Ann Rheum Dis,
March 1, 1999;
58(3):
164 - 168.
[Abstract][Full Text]
M. Aickin Other method for adjustment of multiple testing exists
BMJ,
January 9, 1999;
318(7176):
127a - 127.
[Full Text]
J. Lynch, N. Krause, G. A. Kaplan, R. Salonen, and J. T. Salonen Workplace Demands, Economic Reward, and Progression of Carotid Atherosclerosis
Circulation,
July 1, 1997;
96(1):
302 - 307.
[Abstract][Full Text]
J. Lynch, G. A. Kaplan, R. Salonen, and J. T. Salonen Socioeconomic Status and Progression of Carotid Atherosclerosis : Prospective Evidence From the Kuopio Ischemic Heart Disease Risk Factor Study
Arterioscler Thromb Vasc Biol,
March 1, 1997;
17(3):
513 - 519.
[Abstract][Full Text]