Reliability of scores from the National Exam of Upper Secondary Education (Enem)

Authors

DOI:

https://doi.org/10.15448/1980-8623.2020.2.31145

Keywords:

national exam of upper secondary education (Enem), reliability, composite reliability, composite reliabilitybifactor model

Abstract

The National Exam of Upper Secondary Education (ENEM) generates a score for each domains it assess: mathematics, languages, natural sciences, and humanities. Considering the relevance of the Exam in the access to higher education and in other practical aspects of Brazilian students’ life, the present study investigates the reliability of the scores from the four domains. We used as a sample the scores of the students who participated in the 2011 edition of the Exam. The analyzes involved the estimation of the parameters of an oblique four-factor model and a bifactor model using confirmatory factor analysis, as well as the estimation of composite and omega reliability of the four domains and the general performance factor, in the case of the bifactor model. We used the 30 competences of each domain as observable variables. The results indicated high reliability only for the scores from the general factor.

Downloads

Download data is not yet available.

Author Biographies

Cristiano Mauro Assis Gomes, Universidade Federal de Minas Gerais (UFMG), Belo Horizonte, MG.

Doutor em Educação pela Universidade Federal de Minas Gerais (UFMG, Belo Horizonte, MG, Brasil), professor do Departamento de Psicologia da Universidade Federal de Minas Gerais em Belo Horizonte, MG.

Hudson Fernandes Golino, University of Virginia (UVA), Charlottesville, VA.

Doutor em Neurociências pela Universidade Federal de Minas Gerais (UFMG, Belo Horizonte, MG, Brasil), professor do Departamento de Psicologia da University of Virginia em Charlottesville, VA, Estados Unidos da América.

Alexandre José de Souza Peres, Universidade Federal de Mato Grosso do Sul (UFMS), Paranaíba, MS.

Doutor em Psicologia Social, do Trabalho e das Organizações pela Universidade de Brasília (UNB, Brasília, DF, Brasil). Professor do Câmpus de Paranaíba da Universidade Federal de Mato Grosso do Sul em Paranaíba, MS.

References

Beaujean, A. A. (2014). Latent variable modeling using R: A step-by-step guide. Routledge.

Cangur, S., & Ercan, I. (2015). Comparison of model fit indices used in structural equation modeling under multivariate normality. Journal of Modern Applied Statistical Methods, 14(1), 152-167. 10.22237/jmasm/1288584240

Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. London, UK: Cambridge University Press. 10.1017/CBO9780511571312

Carroll, J. B. (2003). The higher-stratum structure of cognitive abilities: Current evidence supports g and about ten broad factors. In H. Nyborg (Ed.), The scientific study of general intelligence: tribute to Arthur R. Jensen (pp. 1-20). Oxford, UK: Pergamon Press. 10.1016/B978-008043793-4/50036-2

Golino, H. F. (2014). ENEM: an implementation of functions to help automatic downloading, importing, cleaning and scoring of the Brazilian’s National High School Exam (ENEM). Software não-publicado. (DOI INEXISTENTE)

Golino, H. F., Gomes, C. M. A., Amantes, A., & Coelho, G. (Eds.). (2015). Psicometria contemporânea. Compreendendo os modelos Rasch. Casa do Psicólogo.

Gomes, C. M. A. (2010). Avaliando a avaliação escolar: notas escolares e inteligência fluida. Psicologia em Estudo, 15(4), 841-849. Recuperado de http://www.redalyc.org/articulo.oa?id=287123084020

Gomes, C. M. A., & Borges, O. (2008). Limite da validade de um instrumento de avaliação docente. Avaliação Psicológica, 7(3), 391-401. Recuperado de http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712008000300011&lng=pt&tlng=pt.

Gomes, C. M. A. & Borges, O. N. (2009). O ENEM é uma avaliação educacional construtivista? Um estudo de validade de construto. Estudos em Avaliação Educacional, 20(42), 73-88. 10.18222/eae204220092060

Gomes, C. M. A., Golino, H. F., & Peres. (2016). Investigando a validade estrutural das competências do ENEM: quatro domínios correlacionados ou um modelo bifatorial. Boletim na Medida (INEP-Ministério da Educação), 5(10), 33-30. Recuperado de http://portal.inep.gov.br/documents/186968/494037/BOLETIM+NA+MEDIDA+-+N%C2%BA+10/4b8e3d73-d95d-4815-866c-ac2298dff0bd?version=1.1

Gomes, C. M. A., Golino, H. F., & Peres, A. J. S. (2018). Análise da fidedignidade composta dos escores do ENEM por meio da análise fatorial de itens. European Journal of Education Studies, 5(8), 331-344. 10.5281/zenodo.2527904

Gomes, C. M. A., & Jelihovschi, E. (2019). Presenting the regression tree method and its application in a large-scale educational dataset. International Journal of Research & Method in Education. 10.1080/1743727X.2019.1654992

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2013). Multivariate data analysis. Pearson new international edition (7th ed.). Pearson Education Limited.

Hogan, T. P. (2013). Psychological testing: A practical introduction (3rd ed.). John Wiley & Sons.

Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (2012a). Microdados do ENEM – 2011.

Exame Nacional do Ensino Médio: Manual do Usuário. Recuperado de http://portal.inep.gov.br/web/guest/microdados

Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. (2012b). Matriz de Referência do ENEM. Recuperado de http://download.inep.gov.br/educacao_basica/enem/downloads/2012/matriz_referencia_enem.pdf

Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. (2018). Edital nº 16, de 20 de Março de

Exame Nacional Do Ensino Médio - Enem 2018. Recuperado de http://download.inep.gov.br/educacao_basica/enem/edital/2018/edital_enem_2018.pdf

Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2018). semTools: Useful tools for structural equation modeling (version 0.5-1). [Computer software]. Recuperado de https://CRAN.R-project.org/package=semTools

Kline, R. B. (2016). Principles and practice of Structural Equation Modeling (4th ed.). New York, USA: The Guilford Press.

McDonald, R. P. (1999). Test theory: A unified treatment. L. Erlbaum Associates

Pires, A. A. M., & Gomes, C. M. A. (2017). Three mistaken procedures in the elaboration of school exams: explicitness and discussion. PONTE International Scientific Researches Journal, 73(3), 1-14. 10.21506/j.ponte.2017.3.1

Pires, A. A. M., & Gomes, C. M. A. (2018). Proposing a method to create metacognitive school exams. European Journal of Education Studies, 5(8), 119-142. 10.5281/zenodo.2313538

R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Recuperado de http://www.R-project.org/

Raykov, T. (2001). Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints. British Journal of Mathematical and Statistical Psychology, 54, 315-323. 10.1348/000711001159582

Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47 (5), 667-696. 10.1080/00273171.2012.715555

Revelle, W., & Condon, D. M. (2018). Reliability. In P. Irwing, T. Booth, & D. J. Hughes (Eds.). The Wiley Handbook of Psychometric Testing: A multidisciplinary reference on survey, scale and test development (Vols. 1-2) (pp. 709-749). Hoboken, NJ, EUA: John Wiley & Sons.

Rios, J., & Wells, C. (2014). Validity evidence based on internal structure. Psicothema, 26 (1), 108-116. 10.7334/psicothema2013.260

Rosseel, Y., Jorgensen, T. D., Oberski, D., Vanbrabant, J. B. L., Savalei, V., Hallquist, E. M., M., Rhemtulla, M., Katsikatsou, M., Barendse, M., Scharf, F. (2019). lavaan: Latent Variable Analysis (version 0.6-5). [Computer software]. Recuperado de https://cran.r-project.org/web/packages/lavaan/index.html

Valentini, F., Gomes, C. M. A., Muniz, M., Mecca, T. P., Laros, J. A., & Andrade, J. M. (2015). Confiabilidade dos índices fatoriais da Wais-III adaptada para a população brasileira. Psicologia: Teoria e Prática, 17(2), 123-139. 10.15348/1980-6906/psicologia.v17n2p123-139

Downloads

Published

2020-02-12

How to Cite

Gomes, C. M. A., Golino, H. F., & Peres, A. J. de S. (2020). Reliability of scores from the National Exam of Upper Secondary Education (Enem). Psico, 51(2), e31145. https://doi.org/10.15448/1980-8623.2020.2.31145

Issue

Vol. 51 No. 2 (2020)

Section

Articles

License

Copyright

The submission of originals to Psico implies the transfer by the authors of the right for publication. Authors retain copyright and grant the journal right of first publication. If the authors wish to include the same data into another publication, they must cite Psico as the site of original publication.

Creative Commons License

Except where otherwise specified, material published in this journal is licensed under a Creative Commons Attribution 4.0 International license, which allows unrestricted use, distribution and reproduction in any medium, provided the original publication is correctly cited.