Keywords
Technology
Test
Assessment
Measurement TIC
Tecnología
Test
Evaluación
Medición
How to Cite
Abstract
Background: The emergence of digital technology in the field of psychological and educational measurement and assessment broadens the traditional concept of pencil and paper tests. New assessment models built on the proliferation of smartphones, social networks and software developments are opening up new horizons in the field. Method: This study is divided into four sections, each discussing the benefits and limitations of a specific type of technology-based assessment: ambulatory assessment, social networks, gamification and forced-choice testing. Results: The latest developments are clearly relevant in the field of psychological and educational measurement and assessment. Among other benefits, they bring greater ecological validity to the assessment process and eliminate the bias associated with retrospective assessment. Conclusions: Some of these new approaches point to a multidisciplinary scenario with a tradition which has yet to be created. Psychometrics must secure a place in this new world by contributing sound expertise in the measurement of psychological variables. The challenges and debates facing the field of psychology as it incorporates these new approaches are also discussed.
References
Adjerid, I., & Kelley, K. (2018). Big data in psychology: A framework for research advancement. The American Psychologist, 73(7), 899-917.
https://doi.org/10.1037/amp0000190
Aguado, D., Andrés, J. C., García Izquierdo, A. L., & Rodríguez, J. (2019). LinkedIn” Big Four”: Job performance validation in the ICT sector. Journal of Work and Organizational Psychology, 35, 53-64. https://doi.org/10.5093/jwop2019a7
Aguado, D., Rubio, V. J., & Lucía, B. (2011). The Risk Propensity Task (PTR): A proposal for a behavioral performance-based computer test for assessing risk propensity. Anales de Psicología, 27(3), 862-870.
Andrés, J. C., Aguado, D., & De Miguel, J. (2022). What’s behind LinkedIn? Measuring the LinkedIn big four dimensions through rubrics. Psychologist Papers, 43(1), 12-20. https://doi.org/10.23923/pap.psicol.2979
Armstrong, M. B., Landers, R. N., & Collmus, A. B. (2016). Gamifying recruitment, selection, training, and performance management:
Game-thinking in human resource management. In D. Davis & H. Gangadharbatla (Eds.), Emerging research and trends in gamification.
(pp.140-165). IGI Global.
Bangerter, A., Roulin, N., & König, C. J. (2012). Personnel selection as a signaling game. Journal of Applied Psychology, 97, 719-738.
https://doi.org/10.1037/a0026078
Bell, I. H., Lim, M. H., Rossell, S. L., & Thomas, N. (2017). Ecological momentary assessment and intervention in the treatment of psychotic
disorders: A systematic review. Psychiatric Services, 68(11), 1172-1181. https://doi.org/10.1176/appi.ps.201600523
Bennett, R. E. (2015). The changing nature of educational assessment. Review of Research in Education, 39(1), 370-407.
https://doi.org/10.3102/0091732X14554179
Bogarín, A,. Cerezo, R., & Romero, C. (2018). Discovering learning processes using Inductive Miner: A case study with Learning Management Systems (LMSs). Psicothema, 30(3), 322-329. https://doi.org/10.7334/psicothema2018.116.
Boyd, D., & Ellison N. (2007). Social network sites: Definition, history, and scholarship. Journal of Computer-Mediated Communication, 13(1), 210-230. https://doi.org/10.1111/j.1083-6101.2007.00393.x
Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced choice questionnaires. Educational and Psychological Measurement, 71(3), 460-502. https://doi.org/10.1177/0013164410375112
Caballero-Hernández, J. A., Palomo-Duarte, M., & Dodero, J. M. (2017). Skill assessment in learning experiences based on serious games: A systematic mapping study. Computers & Education, 113, 42-60. https://doi.org/10.1016/j.compedu.2017.05.008
Cattell, R. B. (1966). The data box: Its ordering of total resources in terms of possible relational systems. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (pp. 67-128). Rand-McNally.
Cattell, R. B., & Warburton, F. W. (1967). Objective personality and motivation tests. University of Illinois Press.
Collmus, A. B., Armstrong, M. B., & Landers, R. N. (2016). Game-thinking within social media to recruit and select job candidates. In N. R. Landers & B. G. Schmidt (Eds.), Social media in employee selection and recruitment: Theory, practice, and current challenges (pp. 103-124). Springer International Publishing.
Delgado-Gómez, D., Carmona-Vázquez, C., Bayona, S., Ardoy-Cuadros, J., Aguado, D., Baca-García, E., & Lopez-Castroman, J. (2016). Improving impulsivity assessment using movement recognition: a pilot study. Behavior Research Methods, 48(4), 1575-1579. https://doi.org/10.3758/s13428-015-0668-y
Delgado-Gómez, D., Sújar, A., Ardoy-Cuadros, J., Bejarano-Gómez, A., Aguado, D., Miguelez-Fernandez, C., Blasco-Fontecilla, H., & Peñuelas-Calvo, I. (2020). Objective assessment of attention-deficit hyperactivity disorder (ADHD) using an infinite runner-based computer game: a pilot study. Brain Sciences, 10(10), Article 716. https://doi.org/10.3390/brainsci10100716
Donath, J. (2008). Signals in social supernets. Journal of Computer Mediated Communication, 13(1), 231-251. https://doi.org/10.1111/j.1083-6101.2007.00394.x
Ebner-Priemer, U. W., & Trull, T. J. (2009). Ambulatory assessment: An innovative and.promising approach for clinical psychology. European Psychologist, 14(2), 109-119. https://doi.org/10.1027/1016-9040.14.2.109
Elliot, S., Lawty-Jones, M., & Jackson, C. (1996). Effects of dissimulation on self-report and objective measures of personality. Personality and Individual Differences, 21, 335- 343. https://doi.org/10.1016/0191-8869(96)00080-3
Elosua, P. (2022). Impact of ICT on the assessment environment. Innovations through continuous improvement. Psychologist Papers, 43(1), 3-11. https://doi.org/10.23923/pap.psicol.2985
European Commission (2021). Proposal for a Regulation laying down harmonised rules on artificial intelligence. https://op.europa.eu/en/publication-detail/-/publication/e0649735-a372-11eb-9585-01aa75ed71a1/language-en/format-PDF/source-205836026
Espinoza-Cuadros, F., Fernández-Pozo, R., Toledano, D. T., Alcázar-Ramírez, J. D., López-Gonzalo, E., & Hernández-Gómez, L. A. (2016). Reviewing the connection between speech and obstructive sleep apnea. Biomedical engineering online, 15(1), 20. https://doi.org/10.1186/s12938-016-0138-5
Fernández, S., Stöcklin, M., Terrier, L., & Kim, S. (2021). Using available signals on LinkedIn for personality assessment. Journal of Research in Personality, 93, Article 104122. https://doi.org/10.1016/j.jrp.2021.104122
Fokkema, M., Iliescu, D., Greiff, S., & Ziegler, M. (2022). Machine Learning and Prediction in Psychological Assessment: Some Promises and Pitfalls. European Journal of Psychological Assessment, 38(3), 165-175. https://doi.org/10.1027/1015-5759/a000714
Fonseca Pedrero, E., Pérez-Álvarez, M., Al-Halabí, S., Inchausti, F., Muñiz, J., López-Navarro, E., Pérez de Albéniz, A., Lucas Molina, B., Debbané, M., Bobes-Bascarán, M. T., Gimeno-Peón, A., Prado-Abril, J., FernándezÁlvarez, J., Rodríguez-Testal, J. F., González Pando, D., Díez-Gómez, A., García Montes, J. M., García-Cerdán, L., Osma, J., Peris Baquero, Ó., …Marrero, R. J. (2021). Tratamientos psicológicos empíricamente apoyados para adultos: Una revisión selectiva [Evidence-based psychological treatments for adults: A selective review]. Psicothema, 33(2), 188-197. https://doi.org/10.7334/psicothema2020.426
Fonseca-Pedrero, E., Ródenas-Perea, G., Pérez-Albéniz, A., Al-Halabí, S., Pérez, M., & Muñiz, J. (2022). The time of ambulatory assessment.
Psychologist Papers, 43, 21-28. https://doi.org/10.23923/pap.psicol.2983
Frasca, K. J., & Edwards, M. R. (2017). Web‐based corporate, social and video recruitment media: Effects of media richness and source credibility on organizational attraction. International Journal of Selection and Assessment, 25(2), 125-137. https://dx.doi.org/10.1111/ijsa.12165
Funder, D. C. (2012). Accurate personality judgment. Current Directions in Psychological Science, 21, 177-182. https://doi.org/10.1177/0963721412445309
García-Garzón, E., Nieto, M. D., Garrido, L. E., & Abad, F. J. (2020). Bi-factor exploratory structural equation modeling done right: using the SLiDapp application. Psicothema, 32, 607-614. https://doi.org/10.7334/psicothema2020.179
Hall, M., Scherner, P. V., Kreidel, Y., & Rubel, J. A. (2021). A systematic review of momentary assessment designs for mood and anxiety symptoms. Frontiers in Psychology, 12, Article 642044. https://doi.org/10.3389/FPSYG.2021.642044
Heron, K. E., Everhart, R. S., McHale, S. M., & Smyth, J. M. (2017). Using mobile-technology-based ecological momentary assessment (EMA) methods with youth: A systematic review and recommendations. Journal of Pediatric Psychology, 42(10), 1087–1107. https://doi.org/10.1093/JPEPSY/JSX078
Hicks, L. E. (1970). Some properties of ipsative, normative, and forced choice normative measures. Psychological Bulletin, 74(3), 167-184.
https://doi.org/10.1037/h0029780
Hontangas, P. M., Leenen, I., & de la Torre, J. (2016). Traditional scores versus IRT estimates on forced choice tests based on a dominance model. Psicothema, 28(1), 76-82. https://doi.org/10.7334/psicothema2015.204
Jasper, F., & Ortner, T. M. (2014). The tendency to fall for distracting information while making judgments: development and validation of the Objective Heuristic Thinking Test. European Journal of Psychological Assessment, 30, 193-207. https://doi.org/10.1027/1015-5759/a000214
Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1, 389-399.
https://doi.org/10.1038/s42256-019-0088-2
Judge, T. A., Rodell, J. B., Klinger, R. L., Simon, L. S., & Crawford, E. R. (2013). Hierarchical representations of the five-factor model of personality in predicting job performance: Integrating three organizing frameworks with two theoretical perspectives. Journal of Applied Psychology, 98(6), 875-925. https://doi.org/10.1037/a0033901
Kato, P. M., & de Klerk, S. (2017). Serious games for assessment: Welcome to the jungle. Journal of Applied Testing Technology, 18(S1), 1-6.
Kluemper, D. H., Rosen, P. A., & Mossholder, K. W. (2012). Social networking websites, personality ratings, and the organizational context: More than meets the eye? Journal of Applied Social Psychology, 42(1), 1143-1172. https://doi.org/10.1111/j.1559-1816.2011.00881.x
Knight, R. G., & Titov, N. (2009). Use of virtual reality tasks to assess prospective memory: Applicability and evidence. Brain Impairment, 10, 3-13. https://doi.org/10.1375/brim.10.1.3
König, C., Demetriou, A., Glock, P., Hiemstra, A., Iliescu, D., Ionescu, C., Langer, M., Liem, C., Linnenbürger, A., Siegel, R., & Vartholomaios, I. (2020). Some advice for psychologists who want to work with computer scientists on big data. Personnel Assessment and Decisions, 6, 17-23. https://doi.org/10.25035/pad.2020.01.002
Kreitchmann, R. S., Abad, F. J., & Sorrel, M. A. (2021). A genetic algorithm for optimal assembly of pairwise forced choice questionnaires. Behavior Research Methods, 54, 1476-1492. https://doi.org/10.3758/s13428-021-01677-4
Kreitchmann, R. S., Abad, F. J., Ponsoda, V., Nieto, M. D., & Morillo, D. (2019). Controlling for response biases in self-report scales: Forced choice vs. psychometric modeling of Likert items. Frontiers in Psychology, 10. Article 2309. https://doi.org/10.3389/fpsyg.2019.02309
Kreitchmann, R. S., Sorrel, M. A., & Abad, F. J. (2022). On bank assembly and block selection in multidimensional forced choice adaptive assessments. Educational and Psychological Measurement. Advance online publication. https://doi.org/10.1177/00131644221087986
Langenfeld, T., Burstein, J., & von Davier, A. A. (2022). Digital-First Learning and Assessment Systems for the 21st Century. Frontiers in Education, 7, Article 857604. https://doi.org/10.3389/feduc.2022.857604
Landers, R. N (2015). An introduction to game-based assessment: Frameworks for the measurement of knowledge, skills, abilities and other human characteristics using behaviors observed within videogames. International Journal of Gaming and Computer-Mediated Simulations, 7, 4–8.
Law, A. S., Logie, R. H., & Pearson, D. G. (2006). The impact of secondary tasks on multitasking in a virtual environment. Acta Psychologica, 122, 27–44. https://doi.org/10.1016/j.actpsy.2005.09.002
López-Mora, C., Carlo, G., Roos, J., Maiya, S., & González-Hernández, J. (2021). Perceived Attachment and Problematic Smartphone Use in Young People: Mediating Effects of Self-Regulation and Prosociality. Psicothema, 33(4), 564-570. https://doi.org/10.7334/psicothema2021.60
Lejuez, C. W., Read, J. P., Kahler, C., Richards, J. B., Ramsey, S. E., Stuart, G. L., Strong, D.R., & Brown, R. A. (2002). Evaluation of a behavioral
measure of risk taking: The Balloon Analogue Risk Task (BART). Journal of Experimental Psychology: Applied, 8, 75-84. https://doi.org/10.1037//1076-898x.8.2.75
Lumsden, J., Skinner, A., Woods, A. T., Lawrence, N. S., Munafò, M. (2016). The effects of gamelike features and test location on cognitive test performance and participant enjoyment. PeerJ, 4. Article e2184. https://doi.org/10.7717/peerj.2184
Maass, W., Parsons, J., Purao, S., Storey, V. C., & Woo, C. (2018). Data-driven meets theory-driven research in the era of big data: Opportunities and challenges for information systems research. Journal of the Association for Information Systems, 19, Article 12, https://doi.org/10.17705/1jais.00526
Mairesse, F., Walker, M. A., Mehl, M. R., & Moore, R. K. (2007). Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of artificial intelligence research, 30, 457-500. https://www.aaai.org/Papers/JAIR/Vol30/JAIR-3012.pdf
Martínez, A., & Salgado, J. F. (2021). A meta-analysis of the faking resistance of forced choice personality inventories. Frontiers in Psychology, 12, Article 732241. https://doi.org/10.3389/fpsyg.2021.732241
Myin-Germeys, I., & Kuppens, P. (2021). The Open Handbook of Experience Sampling Methodology: A step-by-step guide to designing, conducting, and analyzing ESM studies. The center for Research on Experience sampling and Ambulatory methods Leuven (REAL).
Myin-Germeys, I., Kasanova, Z., Vaessen, T., Vachon, H., Kirtley, O., Viechtbauer, W., & Reininghaus, U. (2018). Experience sampling
methodology in mental health research: New insights and technical developments. World Psychiatry, 17(2), 123-132. https://doi.org/10.1002/wps.20513
Nieto, M. D., Abad, F. J., & Olea, J. (2018). Assessing the Big Five with bifactor computerized adaptive testing. Psychological Assessment, 30(12), 1678-1690. https://doi.org/10.1037/pas0000631
Nikolaou, I. (2014). Social networking web sites in job search and employee recruitment. International Journal of Selection and Assessment, 22(2), 179-189. https://doi.org/10.1111/ijsa.12067
Ortner, T. M., & Proyer, R. T. (2015). Objective personality tests. In T. M. Ortner & F. J. R. Van de Vijver (Eds.), Behavior-based assessment in psychology (pp. 133-149). Hogrefe.
Oswald, F. L. (2020). Future research directions for big data in psychology. In S. E. Woo, L. Tay, & R. W. Proctor (Eds.), Big data in psychological research. (pp. 427-441). American Psychological Association.
Palmier-Claus, J. E., Myin-Germeys, I., Barkus, E., Bentley, L., Udachina, A., Delespaul, P. A., Lewis, S. W., & Dunn, G. (2011). Experience sampling research in individuals with mental illness: reflections and guidance. Acta Psychiatrica Scandinavica, 123(1), 12-20. https://doi.org/10.1111/j.1600-0447.2010.01596.x
Parsons, T. D. (2012). Virtual simulations and the second life metaverse: paradigm shift in neuropsychological assessment. In N. Zagalo, L. Morgado, & Boa-Ventura, A. (Eds.), Virtual worlds and metaverse platforms: New communication and identity paradigms (pp. 234-250). IGI Global.
Pavlov, G., Shi, D., Maydeu-Olivares, A., & Fairchild, A. (2021). Item desirability matching in forced choice test construction. Personality and
Individual Differences, 183, Article 111114. https://doi.org/10.1016/j.paid.2021.111114
Powers, M. B., & Emmelkamp, P. M. (2008). Virtual reality exposure therapy for anxiety disorders: A meta-analysis. Journal of Anxiety Disorders, 22, 561-569. https://doi.org/10.1016/j.janxdis.2007.04.006
Roulin, N., & Bangerter, A. (2013). Social networking websites in personnel selection. Journal of Personnel Psychology 12(1), 143-151.
https://doi.org/10.1027/1866-5888/a000094
Roulin, N., & Levashina, J. (2019). LinkedIn as a new selection method: Psychometric properties and assessment approach. Personnel Psychology, 72(2), 187-211. https://doi.org/10.1111/peps.12296
Russell, S., & Norvig, P. (2021). Artificial intelligence: a modern approach. Pearson.
Ryan, A. M., & Derous, E. (2019). The unrealized potential of technology in selection assessment. Journal of Work and Organizational Psychology, 35(2), 85-92. https://doi.org/10.5093/jwop2019a10
Ryan, A. M., & Ployhart, R. E (2014). A century of selection. Annual Review of Psychology. 65, 693-717. https://doi.org/10.1146/annurev-psych-010213-115134
Salgado, J. F. (2016). A theoretical model of psychometric effects of faking on assessment procedures: Empirical findings and implications for personality at work: A theoretical model of faking psychometric effects. International Journal of Selection and Assessment, 24(3), 209-228. https://doi.org/10.1111/ijsa.12142
Salgado, J. F., Anderson, N., & Tauriz, G. (2015). The validity of ipsative and quasi-ipsative forced choice personality inventories for different occupational groups: A comprehensive meta-analysis. Journal of Occupational and Organizational Psychology, 88(4), 797-834.
https://doi.org/10.1111/joop.12098
Santamaría, P., & Sánchez-Sánchez, F. (2022). Open questions in the use of new technologies in psychological assessment. Psychologist Papers 43(1), 48-54. https://doi.org/10.23923/pap.psicol.2984
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., ... & Ungar, L. H. (2013). Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8(9), Article 73791. https://doi.org/10.1371/journal.pone.0073791
Schwab, K. (2017). The fourth industrial revolution. Crown Publishing Group. Sedano-Capdevila, A., Porras-Segovia, A., Bello, H. J., Baca-García, E., & Barrigon, M. L. (2021). Use of ecological momentary assessment to study suicidal thoughts and behavior: a systematic review. Current Psychiatry
Reports, 23(7), Article 41. https://doi.org/10.1007/s11920-021-01255-7 Spence, M. (1973). Job market signalling. Quarterly Journal of Economics, 87(3), 355-374.
Stark, S., Chernyshenko, O. S., & Drasgow, F. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184-203. https://doi.org/10.1177/0146621604273988
Stone, A. A., & Shiffman, S. (1994). Ecological momentary assessment (EMA) in behavioral medicine. Annals of Behavioral Medicine, 16, 199–202. https://doi.org/10.1093/abm/16.3.199
Stoughton, J.W. (2016). Applicant reactions to social media in selection: Early returns and future directions. In N. R. Landers & B. G. Schmidt (Eds.), Social media in employee selection and recruitment: Theory, practice, and current challenges (pp. 249–263). Springer International Publishing.
Stoughton, J.W., Thompson, L. F., & Meade, A.W. (2015). Examining applicant reactions to the use of social networking websites in pre-employment screening. Journal of Business and Psychology, 30, 73-88. https://doi.org/10.1007/s10869-013-9333-6
Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology, 29(1), 24-54. Trull, T. J., & Ebner-Priemer, U. (2013). Ambulatory Assessment. Annual
Review of Clinical Psychology, 9, 151-176. https://doi.org/10.1146/ANNUREV-CLINPSY-050212-185510
Trull, T. J., & Ebner-Priemer, U. W. (2020). Ambulatory assessment in psychopathology research: A review of recommended reporting guidelines and current practices. Journal of Abnormal Psychology, 129(1), 56–63. https://doi.org/10.1037/ABN0000473
Van Iddekinge, C. H., Lanivich, S. E., Roth, P. L., & Junco, E. (2016). Social media for selection? Validity and adverse impact potential of a Facebookbased assessment. Journal of Management, 42(7), 1811-1835. https://dx.doi.org/10.1177/0149206313515524
van Roekel, E., Keijsers, L., & Chung, J. M. (2019). A review of current ambulatory assessment studies in adolescent samples and practical
recommendations. Journal of Research on Adolescence, 29(3), 560-577. https://doi.org/10.1111/JORA.12471
von Davier, A. A., Mislevy, R. J., & Hao, J. (Ed.) (2021). Computational psychometrics: New methodologies for a new generation of digital learning and assessment into psychometrics. Springer. https://doi.org/10.1007/978-3-030-74394-9
Weiner, J., Herff, C., & Schultz, T. (2016). Speech-based detection of Alzheimer’s disease in conversational German. In International Speech Communications Association (Eds.), Understanding speech processing in human and machines (pp. 1938-1942). Curran Associates, Inc.
Woods, S. A., Ahmed, S., Nikolaou, I., Costa, A. C., & Anderson, N. R. (2020). Personnel selection in the digital age: A review of validity and applicant reactions, and future research challenges. European Journal of Work and Organizational Psychology, 29(1), 64-77.
https://doi.org/10.1080/1359432X.2019.1681401
Youyou, W., Kosinski, M., & Stillwell, D. (2015). Computer-based personality judgments are more accurate than those made by humans. Proceedings of the National Academy of Sciences, 112(4), 1036-1040. https://doi.org/10.1073/pnas.1418680112
Zide, J., Elman, B., & Shahani-Denning, C. (2014). LinkedIn and recruitment: How profiles differ across occupations. Employee relations, 36(5), 583-604. https://doi.org/10.1108/ER-07-2013-0086
Ziegler, M., Schmukle, S., Egloff, B., & Bühner, M. (2010). Investigating measures of achievement motivation(s). Journal of Individual Differences, 31, 15-21. https://doi.org/10.1027/1614-0001/a000002