Keywords
Scale
Studies based on observational
methodology
Validity evidence
Reliability Calidad metodológica
Escala
Estudios basados en metodología
observacional
Evidencias de validez
Fiabilidad
How to Cite
Abstract
Background: To date, no studies have shown validity evidence of a scale that measures the methodological quality
of studies based on observational methodology (SBOM). This study presents validity evidence of the Methodological
Quality Scale for Studies based on Observational Methodology (MQSOM) based on its internal structure. Method:
MQSOM was applied to 650 journal articles that used observational methodology. The Intraclass Correlation
Coefficient (ICC) was calculated to obtain evidence of inter- and intracoder reliability. Parallel analysis was done using
optimal implementation to study the dimensionality of the scale before conducting an exploratory factor analysis with
a randomly-selected half of the sample, followed by a confirmatory factor analysis with the remaining half. Results:
Both inter- and intra-coder reliability were adequate, ICC > .73. Parallel analysis suggested a lack of unidimensionality,
UniCo = .41; ECV = .31. A second-order factor structure (general factor: Methodological quality) with two first-order
factors (F1 Design, F2 Measurement and Analysis) was obtained, RMSEA = 0.000, NNFI = 1, GFI = .98, AGFI = .97,
with adequate reliability and discrimination results. Conclusions: MQSOM is a short (11 items), useful instrument for
professionals, researchers or assessment commissions when designing, implementing, or evaluating SBOM
References
Anguera, M. T. (2003). Observational methods (General). In R. FernándezBallesteros (Ed.), Encyclopedia of psychological assessment, Vol. 2 (pp. 632-637). Sage.
Anguera, M. T., Blanco-Villaseñor, A., Losada, J. L., & Portell, M.(2018). Guidelines for designing and conducting a study that applies observational methodology. Anuario de Psicología, 48(1), 9-17. https://doi.org/10.1016/j.anpsic.2018.02.001
Anguera, M. T., Blanco-Villaseñor, A., Losada, J. L., & Sánchez-Algarra, P. (2020). Integración de elementos cualitativos y cuantitativos en metodología observacional [Integration of qualitative and quantitative elements in observational methodology]. Ámbitos: Revista Internacional de Comunicación, 49, 49-70. https://doi.org/10.12795/Ambitos.2020. i49.04
Anguera, M. T., Camerino, O., & Castañer, M. (2012). Mixed methods procedures and designs for research on sport, physical education and dance. In O. Camerino, M. Castañer, & M. T. Anguera (Ed.), Mixed methods research in the movement sciences: Case studies in sport, physical education and dance (pp. 3-27). Routledge.
Argudo-Iturriaga, F. M., Garcí-Marín, P., Borges-Hernández, P. J., & Ruíz-Lara, E. (2021). Effects of rules changes on water polo shooting performance according to the final classification: High, medium, and worst level. Journal of Physical Education and Sport, 21(1), 188-200. https://doi.org/10.7752/jpes.2021.01026
Bakeman, R. (1978). Untangling streams of behavior: Sequential analysis of observation data. In G. P. Sackett (Ed.), Observing behavior: Vol. 2. Data collection and analysis methods (pp. 63-78). University of Park Press.
Barbero-García, M. I. (1993). Psicometría II. Métodos de elaboración de escalas [Psychometrics II. Methods to develop scales]. UNED.
Barbero-García, M. I., Vila-Abad, E., & Holgado-Tello, F. P. (2015). Psicometría [Psychometrics]. Sanz y Torres.
Bartlett, M. S. (1954). A note on multiplying factors for various chi-squared approximations. Journal of the Royal Statistical Society, Series B, 16, 296-298. https://www.jstor.org/stable/2984057
Blanco-Villaseñor, A., Losada, J. L., & Anguera, M. T. (2003). Data analysis techniques in observational designs applied to the environment-behavior relation. Medio Ambiente y Comportamiento Humano, 4(2), 111-126. http://mach.webs.ull.es/PDFS/VOL4_2/VOL_4_2_d.pd
Chacón-Moscoso, S., Anguera, M. T., Sanduvete-Chaves, S., Losada, J. L., & Portell, M. (2019). Methodological quality checklist for studies based on observational methodology (MQCOM). Psicothema, 31(4), 458-464. https://doi.org/10.7334/psicothema2019.116
Chacón-Moscoso, S., Anguera, M. T., Sanduvete-Chaves, S., & SánchezMartín, M. (2014). Methodological convergence of program evaluation designs. Psicothema, 26(1), 91-96. https://doi.org/10.7334/7psicothema2013.144
Chacón-Moscoso, S., Sanduvete-Chaves, S., Anguera, M. T., Losada, J.L., Portell, M., & Lozano-Lozano, J. A. (2018). Preliminary checklist for reporting observational studies in sports areas: Content validity. Frontiers in Psychology, 9, Article 291. https://doi.org/10.3389/fpsyg.2018.00291
Chacón-Moscoso, S., Sanduvete-Chaves, S., Lozano-Lozano, J. A., Portell, M., & Anguera, M. T. (2021). From randomized control trial to mixed methods: A practical framework for program evaluation based on methodological quality. Anales de Psicología, 37(3), 599-608. https://doi.org/10.6018/analesps.470021
Chacón-Moscoso, S., Sanduvete-Chaves, S., Portell, M., & Anguera, M. T. (2013). Reporting a program evaluation: Needs, program plan, intervention, and decisions. International Journal of Clinical and Health Psychology, 13(1), 58-66. https://doi.org/10.1016/S1697-2600(13)70008-5
Cochran, W. G., & Chambers, S. P. (1965). The planning of observational studies of human populations. Journal of the Royal Statistical Society. Series A (General), 128(2), 234-266. https://doi.org/10.2307/2344179
Ferrando, P. J., & Lorenzo-Seva, U. (2017). Program FACTOR at 10: Origins, development, and future directions. Psicothema, 29(2), 236-241. https://doi.org/10.7334/psicothema2016.304
Ferrando, P. J., & Lorenzo-Seva, U. (2018). Assessing the quality and appropriateness of factor solutions and factor score estimates in exploratory item factor analysis. Educational and Psychological Measurement, 78, 762-780. https://doi.org/10.1177/0013164417719308
Field, A. (2018). Discovering statistics using IBM SPSS statistics. Sage. Harrison, R. L., Reilly, T. M., & Creswell, J. W. (2020). Methodological rigor in mixed methods: An application in management studies. Journal of Mixed Methods Research, 14(4), 4773-495. https://doi.org/10.1177/1558689819900585
Holgado-Tello, F. P., Chacón-Moscoso, S., Barbero-García, M. I., & VilaAbad, E. (2010). Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Quality & Quantity, 44, 153–166. https://doi.org/10.1007/s11135-008-9190-y
Holgado-Tello, F. P., Suárez-Falcón, J. C., & Morata-Ramírez, M. A. (2019). Modelos de ecuaciones estructurales, desde el “path analysis” al análisis multigrupo: una guía práctica con LISREL [Structural equation models, from path analysis to multigroup analysis: A practical guide with LISREL]. Sanz y Torres.
Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6, 56–60. https://academic-publishing.org/index.php/ejbrm/article/view/1224
Kaiser, H., & Rice, J. (1974). Little Jiffy, Mark Iv. Educational and Psychological Measurement, 34(1), 111–117. https://doi.org/10.1177/001316447403400115
Kalkbrenner, M. T. (2023). Alpha, omega, and h internal consistency reliability estimates: Reviewing these options and when to use them. Counseling Outcome Research and Evaluation, 14(1), 77-88. https://doi.org/10.1080/21501378.2021.1940118
Lappi, O., Rinkkala, P., & Pekkanen, J. (2017). Systematic observation of an expert driver’s gaze strategy-an on-road case study. Frontiers in Psychology, 8, Article 620. https://doi.org/10.3389/fpsyg.2017.00620
Lapresa, D., Santesteban, G., Arana, J., Anguera, M. T., & Aragón, S. (2017). Observation system for analyzing individual boccia BC3. Journal of Developmental and Physical Disabilities, 29, 721-734. https://doi.org/10.1007/s10882-017-9552-2
Leech, N. L., & Onwuegbuzie, A. J. (2010). Guidelines for conducting and reporting mixed research in the field of counseling and beyond. Journal of Counseling & Development, 88(1), 61-66. https://doi.org/10.1002/j.1556-6678.2010.tb00151.x
Maciá, M. J., Sánchez-Sánchez, J., García-Córdoba, J. A., & Gallardo, A. M. (2021). Analysis of the safety and accessibility of sports facilities in secondary school. Cuadernos de Psicología del Deporte, 21(1), 242-257. https://revistas.um.es/cpd/article/view/395671/294531
Muñiz, J., & Fonseca-Pedrero, E. (2019). Ten steps for test development. Psicothema, 31(1), 7-16. https://doi.org/10.7334/psicothema2018.291
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T.C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu,M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., ... & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery, 88, Article 105906. https://doi.org/10.1016/j.ijsu.2021.105906.
Portell, M., Anguera, M. T., Chacón-Moscoso, S., & Sanduvete-Chaves,S. (2015). Guidelines for reporting evaluations based on observational methodology. Psicothema, 27(3), 283-289. https://doi.org/10.7334/psicothema2014.276
Portney, L., & Watkins, M. (2000). Foundations of clinical research: Applications to practice. Prentice Hall.
Sánchez-Meca, J. (2022). Revisiones sistemáticas y metaanálisis en educación: un tutorial [Systematic reviews and meta-analysis in education: A tutorial]. RiiTE Revista Interuniversitaria de Investigación en Tecnología Educativa, 13, 5-40. https://doi.org/10.6018/riite.545451
Sanduvete-Chaves, S., Lozano-Lozano, J. A., Chacón-Moscoso, S., & Holgado-Tello, F. P. (2018). Development of a work climate scale in emergency health services. Frontiers in Psychology, 9, Article 10. https://doi.org/10.3389/fpsyg.2018.00010
Serna, J., Muñoz, V., Hileno, R., Solsona, E., & Sáez de Ocáriz, U. (2017). T-Patterns initiated with ball screen or one-on-one in basketball. Journal of Sport Psychology 26(Suppl. 1), 81-86. https://www.redalyc.org/pdf/2351/235150578014.pdf
von Elm, E., Altman, D. G., Effer, M., Pocock, S. J., Gotzsche, P. C., & Vandenbroucke, J. P. (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. Bulletin of World Health Organization, 85(11), 867-872. https://doi.org/10.2471/blt.07.045120.