Keywords
PROVAL
Validation protocol
Sources of validity evidence Evidencias de validez
PROVAL
Protocolo de validación
Fuentes de evidencia de validez
How to Cite
Abstract
Introduction: Validity is a fundamental aspect of psychological assessment because it examines the extent to which interpretations of the scores produced by an instrument support its intended purpose. While validation studies are frequently conducted, they often emphasize specific phases or validation methods, such as pilot testing or expert judgments, while neglecting current additional sources of validity evidence that could improve an instrument’s psychometric properties. This limitation is partly due to a lack of detailed easy-to-use guidelines, which make validation processes inaccessible to non-expert researchers and practitioners. Additionally, the traditional focus on quantitative methods often overlooks the potential benefits of complementary approaches to validation that will help people design more comprehensive validation studies. Method: To address these challenges, the present study introduces PROVAL, a nine-step protocol designed to guide researchers in conducting validation studies. Results: PROVAL promotes a reflective approach to help researchers and practitioners identify and prioritize the most important sources of validity evidence for their specific validation requirements. Conclusions: An illustrative application of PROVAL is presented, demonstrating its usefulness in optimizing resources and ensuring instruments effectively fulfill their intended purposes.
References
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. American Educational Research Association.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (Eds.). (2014). Standards for educational and psychological testing. American Educational Research Association.
Chen, M. Y., & Zumbo, B. D. (2017). Ecological framework of item responding as validity evidence: An application of multilevel DIF modeling using PISA data. In B.D. Zumbo & A.M. Hubley (Eds.), Understanding and investigating response processes in validation research (pp. 53-68). Springer. https://doi.org/10.1007/978-3-319-56129-5_4
Cizek, G. J., Rosenberg, S. L., & Koons, H. H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68(3), 397-412. https://doi.org/10.1177/0013164407310130
Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kane’s framework. Medical Education, 49(6), 560-575. https://doi.org/10.1111/medu.12678
Creswell, J. W., & Plano-Clark, V. L. (2011). Designing and conducting mixed methods research. Sage Publications.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. ERIC.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957
Dellinger, A. B., & Leech, N. L. (2007). Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research, 1(4), 309-332. https://doi.org/10.1177/1558689807306147
European Federation of Psychologists’ Associations AISBL (EFPA). (2025). Model for the review, description and evaluation of psychological and educational tests (Test review model). https://www.efpa.eu/
Engelhardt, L., & Goldhammer, F. (2019). Validating test score interpretations using time information. Frontiers in Psychology, 10, Article 1131. https://doi.org/10.3389/fpsyg.2019.01131
Harkness, J., & Schoua-Glusberg, A. (1998). Questionnaires in translation. In J. Harkness (Ed.), Cross-cultural survey equivalence (pp. 87-126). Zentrum für Umfragen, Methoden und Analysen -ZUMA-. https://www.ssoar.info/ssoar/handle/document/49733
Iliescu, D., & Greiff, S. (2021). On consequential validity. European Journal of Psychological Assessment, 37(3), 163-166. https://doi.org/10.1027/1015-5759/a000664
International Test Commission. (2017). The ITC guidelines for translating and adapting tests (Second edition). [www.InTestCom.org]
Johnson, R. B., & Christensen, L. (2019). Educational research: Quantitative, qualitative, and mixed approaches. Sage publications.
Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educational Researcher, 33(7), 14-26. https://doi.org/10.3102/0013189X033007014
Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535. https://doi.org/10.1037/0033-2909.112.3.527
Kane, M. T. (2006). Validation In R.L. Breenan (Ed.), Educational Measurement (pp. 17-64). American Council on Education and Praeger.
Kane, M. T. (2013a). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1-73. https://doi.org/10.1111/jedm.12000
Kane, M. (2013b). Validation as a pragmatic, scientific activity. Journal of Educational Measurement, 50, 1115–1122. https://doi.org/10.2307/23353803
Lane, S. (2014). Validity evidence based on testing consequences. Psicothema, 26(1), 127-135. https://doi.org/10.7334/psicothema2013.258
Lane, S., & Marion, S. F. (2025). Validity and validation. In L. L. Cook & M. J. Pitoniak (Eds.), Educational measurement (5th ed., pp. 191–275). Oxford University Press. https://doi.org/10.1093/oso/9780197654965.003.0004
Lissitz, R. W., & Samuelsen, K. (2007). A suggested change in terminology and emphasis regarding validity and education. Educational Researcher, 36(8), 437-448. https://doi.org/10.3102/0013189X07311286
Lorenzo Moledo, M., Quiroga-Carrillo, A., & Ferraces Otero, M. J. (2023). Estudio psicométrico de una escala sobre actitudes hacia la igualdad de género con mujeres reclusas. Revista Española de Investigación Criminológica, 20(2), Article e657. https://doi.org/10.46381/reic.v20i2.657
Messick, S. (1989). Validity. In R. Linn (Ed), Educational Measurement (pp. 13-103). American Council on Education and Macmillan.
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter LM, & de Vet, H. C. (2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology, 63(7), 737-745. https://doi.org/10.1016/j.jclinepi.2010.02.006
Neuert, C. E., & Lenzner, T. (2016). Incorporating eye tracking into cognitive interviewing to pretest survey questions. International Journal of Social Research Methodology, 19(5), 501-519. https://doi.org/10.1080/13645579.2015.1049448
Oren, C., Kennet-Cohen, T., & Turval, E. (2014). Demonstrating the validity of three general scores of PET in predicting higher education achievement in Israel. Psicothema, 26(1), 117-126. https://doi.org/10.7334/psicothema2013.257
Padilla, J.L., & Benítez, I. (2014). Validity evidence based on response processes. Psicothema, 26(1), 136-144. https://doi.org/10.7334/psicothema2013.259
Parks, J. B., & Roberton, M. A. (1998). Influence of age, gender, and context on attitudes toward sexist/nonsexist language: Is sport a special case? Sex Roles, 38(5), 477-494. https://doi.org/10.1023/A:1018766023667
Parks, J. B., & Roberton, M. A. (2000). Development and validation of an instrument to measure attitudes toward sexist/nonsexist language. Sex Roles, 42, 415-438. https://doi.org/10.1023/A:1007002422225
Rios, J., & Wells, C. (2014). Validity evidence based on internal structure. Psicothema, 26(1), 108-116. https://doi.org/10.7334/psicothema2013.260
Rushton, P. W., Routhier, F., Miller, W. C., Auger, C., & Lavoie, M.-P. (2015). French-Canadian translation of the WheelCon-M (WheelCon-M-F) and evaluation of its validity evidence using telephone administration. Disability and Rehabilitation, 37(9), 812-819. https://doi.org/10.3109/09638288.2014.941019
Russell, M. (2022). Clarifying the terminology of validity and the investigative stages of validation. Educational Measurement: Issues and Practice, 41(2), 25-35. https://doi.org/10.1111/emip.12453
Sireci, S., & Benítez, I. (2023). Evidence for test validation: A guide for practitioners. Psicothema, 35(3), 217-226. https://doi.org/10.7334/psicothema2022.477
Sireci, S., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26(1), 100-107. https://doi.org/10.7334/psicothema2013.256
Sireci, S. G. (1998). The construct of content validity. Social indicators research, 45(1), 83-117.
Sireci, S.G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 19-37). Information Age Publishing Inc.
Sireci, S. G. (2013). Agreeing on validity arguments. Journal of Educational Measurement, 50(1), 99-104. https://doi.org/10.2307/23353800
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches (Vol. 46). Sage.
Tsang, S., Royse, C. F., & Terkawi, A. S. (2017). Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi Journal of Anaesthesia, 11(1), 80-89. https://doi.org/10.4103/sja.SJA_203_17
Yasir, A. S. (2016). Cross cultural adaptation & psychometric validation of instruments: Step-wise description. International Journal of Psychiatry, 1(1), 1-4. https://doi.org/10.33140/IJP/01/01/00001
Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.) The concept of validity: Revisions, new directions and applications (pp. 65–82). Information Age Publishing.
Zumbo, B. D. (2025). Looking through the lens of DIF: Embracing the many ways of being human. The Score, 47(4), 2-5.
Zumbo, B. D., & Hubley, A. M. (Eds.). (2017). Understanding and investigating response processes in validation research (Vol. 69). Springer International Publishing. https://doi.org/10.1007/978-3-319-56129-5