Skip to main content
Log in

Homogeneity tests in meta-analysis: a Monte Carlo comparison of statistical power and Type I error

  • Published:
Quality and Quantity Aims and scope Submit manuscript

Abstract

The statistical power and Type I error rate of several homogeneity tests, usually applied in meta-analysis, are compared using Monte Carlo simulation: (1) The chi-square test applied to standardized mean differences, correlation coefficients, and Fisher's r-to-Z transformations, and (2) S&H-75 (and 90 percent) procedure applied to standardized mean differences and correlation coefficients. Chi-square tests adjusted correctly Type I error rates to the nominal significance level while the S&H procedures showed higher rates; consequently, the S&H procedures presented greater statistical power. In all conditions, the statistical power was very low, particularly when the sample had few studies, small sample sizes, and presented short differences between the parametric effect sizes. Finally, the criteria for selecting homogeneity tests are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alexander, R. A., Scozzaro, M. J. & Borodkin, L. J. (1989). Statistical and empirical examination of the chi-square test for homogeneity of correlations in meta-analysis, Psychological Bulletin 106: 329-331.

    Google Scholar 

  • Bangert-Drowns, R. L. (1986). Review of developments in meta-analytic method, Psychological Bulletin 99: 388-399.

    Google Scholar 

  • Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd edn.) Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Cooper, H. M. (1989). Integrating Research: A Guide for Literature Reviews (2nd edn.) Beverly Hills, CA: Russell Sage Foundation.

    Google Scholar 

  • Cooper, H. M. & Hedges, L. V. (eds) (1994). The Handbook of Research Synthesis. New York: Russell Sage Foundation.

    Google Scholar 

  • Cornwell, J. M. (1993). Monte Carlo comparisons of three tests for homogeneity of independent correlations, Educational and Psychological Measurement 53: 605-618.

    Google Scholar 

  • Cornwell, J. M. & Ladd, R. T. (1993). Power and accuracy of the Schmidt and Hunter meta-analytic procedures, Educational and Psychological Measurement 53: 877-895.

    Google Scholar 

  • GAUSS (1992). The GAUSS System (Vers. 3.0). Washington: Aptech Systems, Inc.

    Google Scholar 

  • Glass, G. V., McGaw, B. & Smith, M. L. (1981). Meta-Analysis in Social Research. Beverly Hills, CA: Russell Sage Foundation.

    Google Scholar 

  • Hall, J. A. & Rosenthal, R. (1991). Testing for moderator variables in meta-analysis: issues and methods, Communication Monographs 58: 437-448.

    Google Scholar 

  • Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators, Journal of Educational Statistics 6: 107-128.

    Google Scholar 

  • Hedges, L. V. (1982). Fitting categorial models to effect sizes from a series of experiments, Journal of Educational Statistics 7: 119-137.

    Google Scholar 

  • Hedges, L. V. (1994). Fixed effects models, pp. 285-299 in H. M. Cooper & L. V. Hedges (eds), The Handbook of Research Synthesis. New York: Russell Sage Foundation.

    Google Scholar 

  • Hedges, L. V. & Olkin, I. (1985). Statistical Methods for Meta-Analysis. New York: Academic Press.

    Google Scholar 

  • Hunter, J. E. & Schmidt, F. L. (1990). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Beverly Hills, CA: Russell Sage Foundation.

    Google Scholar 

  • Johnson, B. T. (1993). DSTAT 1.10: Software for the Meta-Analytic Review of Research Literatures [manual]. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Johnson, B. T. & Turco, R. M. (1993). The value of goodness-of-fit indices in meta-analysis: a comment on Hall and Rosenthal, Communication Monographs 59: 388-396.

    Google Scholar 

  • Johnson, B. T. Mullen, B. & Salas, E. (1995). Comparison of three major meta-analytic approaches, Journal of Applied Psychology 80: 94-106.

    Google Scholar 

  • Koslowsky, M. & Sagie, A. (1994). Components of artifactual variance in meta-analytic research, Personnel Psychology 47: 561-574.

    Google Scholar 

  • Osburn, H. G., Callender, J. C., Greener, J. M. & Ashworth, S. (1983). Statistical power of tests of the situational specificity hypothesis in validity generalization studies: a cautionary note, Journal of Applied Psychology 68: 115-122.

    Google Scholar 

  • Rosenthal, R. (1991). Meta-Analytic Procedures for Social Research (revised edn). Newbury Park, CA: Russell Sage Foundation.

    Google Scholar 

  • Rosenthal, R. (1994). Parametric measures of effect size, pp. 231-244 in H. M. Cooper & L. V. Hedges (eds), The Handbook of Research Synthesis. New York: Russell Sage Foundation.

    Google Scholar 

  • Sackett, P. R., Harris, M. M. & Orr, J. M. (1986). On seeking moderator variables in the meta-analysis of correlational data: a Monte Carlo investigation of statistical power and resistance to Type I error. Journal of Applied Psychology 71: 302-310.

    Google Scholar 

  • Sagie, A. & Koslowsky, M. (1993). Detecting moderators with meta-analysis: an evaluation and comparison of techniques, Personnel Psychology 46: 629-640.

    Google Scholar 

  • Sánchez-Meca, J. & Ato, M. (1989). Meta-análisis: una alternativa metodológica a las revisiones tradicionales de la investigación [Meta-analysis: a methodological alternative to traditional research reviews], pp. 617-669 in J. Arnau & H. Carpintero (eds), Tratado de Psicología General. I: Historia, Teoría y Método. Madrid: Alhambra.

    Google Scholar 

  • Shadish, W. R. & Haddock, C. K. (1994). Combining estimates of effect size, pp. 261-281 in H. M. Cooper & L. V. Hedges (eds), The Handbook of Research Synthesis. New York: Russell Sage Foundation.

    Google Scholar 

  • Spector, P. E. & Levine, E. L. (1987). Meta-analysis for integrating study outcomes: a Monte Carlo study of its susceptibility to Type I and Type II errors. Journal of Applied Psychology 72: 3-9.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sánchez-Meca, J., Marín-Martínez, F. Homogeneity tests in meta-analysis: a Monte Carlo comparison of statistical power and Type I error. Quality & Quantity 31, 385–399 (1997). https://doi.org/10.1023/A:1004298118485

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1004298118485

Navigation