Abstract
The statistical power and Type I error rate of several homogeneity tests, usually applied in meta-analysis, are compared using Monte Carlo simulation: (1) The chi-square test applied to standardized mean differences, correlation coefficients, and Fisher's r-to-Z transformations, and (2) S&H-75 (and 90 percent) procedure applied to standardized mean differences and correlation coefficients. Chi-square tests adjusted correctly Type I error rates to the nominal significance level while the S&H procedures showed higher rates; consequently, the S&H procedures presented greater statistical power. In all conditions, the statistical power was very low, particularly when the sample had few studies, small sample sizes, and presented short differences between the parametric effect sizes. Finally, the criteria for selecting homogeneity tests are discussed.
Similar content being viewed by others
References
Alexander, R. A., Scozzaro, M. J. & Borodkin, L. J. (1989). Statistical and empirical examination of the chi-square test for homogeneity of correlations in meta-analysis, Psychological Bulletin 106: 329-331.
Bangert-Drowns, R. L. (1986). Review of developments in meta-analytic method, Psychological Bulletin 99: 388-399.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd edn.) Hillsdale, NJ: Erlbaum.
Cooper, H. M. (1989). Integrating Research: A Guide for Literature Reviews (2nd edn.) Beverly Hills, CA: Russell Sage Foundation.
Cooper, H. M. & Hedges, L. V. (eds) (1994). The Handbook of Research Synthesis. New York: Russell Sage Foundation.
Cornwell, J. M. (1993). Monte Carlo comparisons of three tests for homogeneity of independent correlations, Educational and Psychological Measurement 53: 605-618.
Cornwell, J. M. & Ladd, R. T. (1993). Power and accuracy of the Schmidt and Hunter meta-analytic procedures, Educational and Psychological Measurement 53: 877-895.
GAUSS (1992). The GAUSS System (Vers. 3.0). Washington: Aptech Systems, Inc.
Glass, G. V., McGaw, B. & Smith, M. L. (1981). Meta-Analysis in Social Research. Beverly Hills, CA: Russell Sage Foundation.
Hall, J. A. & Rosenthal, R. (1991). Testing for moderator variables in meta-analysis: issues and methods, Communication Monographs 58: 437-448.
Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators, Journal of Educational Statistics 6: 107-128.
Hedges, L. V. (1982). Fitting categorial models to effect sizes from a series of experiments, Journal of Educational Statistics 7: 119-137.
Hedges, L. V. (1994). Fixed effects models, pp. 285-299 in H. M. Cooper & L. V. Hedges (eds), The Handbook of Research Synthesis. New York: Russell Sage Foundation.
Hedges, L. V. & Olkin, I. (1985). Statistical Methods for Meta-Analysis. New York: Academic Press.
Hunter, J. E. & Schmidt, F. L. (1990). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Beverly Hills, CA: Russell Sage Foundation.
Johnson, B. T. (1993). DSTAT 1.10: Software for the Meta-Analytic Review of Research Literatures [manual]. Hillsdale, NJ: Erlbaum.
Johnson, B. T. & Turco, R. M. (1993). The value of goodness-of-fit indices in meta-analysis: a comment on Hall and Rosenthal, Communication Monographs 59: 388-396.
Johnson, B. T. Mullen, B. & Salas, E. (1995). Comparison of three major meta-analytic approaches, Journal of Applied Psychology 80: 94-106.
Koslowsky, M. & Sagie, A. (1994). Components of artifactual variance in meta-analytic research, Personnel Psychology 47: 561-574.
Osburn, H. G., Callender, J. C., Greener, J. M. & Ashworth, S. (1983). Statistical power of tests of the situational specificity hypothesis in validity generalization studies: a cautionary note, Journal of Applied Psychology 68: 115-122.
Rosenthal, R. (1991). Meta-Analytic Procedures for Social Research (revised edn). Newbury Park, CA: Russell Sage Foundation.
Rosenthal, R. (1994). Parametric measures of effect size, pp. 231-244 in H. M. Cooper & L. V. Hedges (eds), The Handbook of Research Synthesis. New York: Russell Sage Foundation.
Sackett, P. R., Harris, M. M. & Orr, J. M. (1986). On seeking moderator variables in the meta-analysis of correlational data: a Monte Carlo investigation of statistical power and resistance to Type I error. Journal of Applied Psychology 71: 302-310.
Sagie, A. & Koslowsky, M. (1993). Detecting moderators with meta-analysis: an evaluation and comparison of techniques, Personnel Psychology 46: 629-640.
Sánchez-Meca, J. & Ato, M. (1989). Meta-análisis: una alternativa metodológica a las revisiones tradicionales de la investigación [Meta-analysis: a methodological alternative to traditional research reviews], pp. 617-669 in J. Arnau & H. Carpintero (eds), Tratado de Psicología General. I: Historia, Teoría y Método. Madrid: Alhambra.
Shadish, W. R. & Haddock, C. K. (1994). Combining estimates of effect size, pp. 261-281 in H. M. Cooper & L. V. Hedges (eds), The Handbook of Research Synthesis. New York: Russell Sage Foundation.
Spector, P. E. & Levine, E. L. (1987). Meta-analysis for integrating study outcomes: a Monte Carlo study of its susceptibility to Type I and Type II errors. Journal of Applied Psychology 72: 3-9.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Sánchez-Meca, J., Marín-Martínez, F. Homogeneity tests in meta-analysis: a Monte Carlo comparison of statistical power and Type I error. Quality & Quantity 31, 385–399 (1997). https://doi.org/10.1023/A:1004298118485
Issue Date:
DOI: https://doi.org/10.1023/A:1004298118485