VTechWorks staff will be away for the Thanksgiving holiday beginning at noon on Wednesday, November 27, through Friday, November 29. We will resume normal operations on Monday, December 2. Thank you for your patience.
 

Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines

Simple item page
dc.contributor.authorHickman, Louisen
dc.contributor.authorHerde, Christoph N.en
dc.contributor.authorLievens, Filipen
dc.contributor.authorTay, Louisen
dc.date.accessioned2023-02-24T14:20:35Zen
dc.date.available2023-02-24T14:20:35Zen
dc.date.issued2023-02en
dc.date.updated2023-02-24T01:58:47Zen
dc.description.abstractAssessment center (AC) exercises such as role-plays have established themselves as valuable approaches for obtaining insights into interpersonal behavior, but they are often considered the “Rolls Royce” of personnel assessment due to their high costs. The observation and rating process comprises a substantial part of these costs. In an exploratory case study, we capitalize on recent advances in natural language processing (NLP) by developing NLP-based machine learning (ML) models to investigate the possibility of automatically scoring AC exercises. First, we compared the convergent-related validity and contamination with word count of ML scores based on models that used different NLP methods to operationalize verbal behavior. Second, for the model that maximized convergence while minimizing contamination with word count (i.e., a model that used both n-grams and Universal Sentence Encoder embeddings as predictors), we investigated the criterion-related validity of its scores. Third, we examined how the interrater reliability of the AC role-play scores affects ML model convergence. To do so, we applied seven NLP methods to 96 assessees' transcriptions and trained 10 sets of ML models across 18 speeded AC role-plays to automatically score assessee performance. Results suggest that ML scores recovered most of the original variance in the overall assessment ratings, and replacing one or more human assessors with ML scores maintained criterion-related validity. Additionally, ML models seemed to exhibit higher convergence when assessors consistently detected and utilized observable behaviors to make ratings (i.e., when interrater reliability was higher). Finally, we provide a step-by-step guide for practitioners seeking to implement ML scoring in ACs.en
dc.description.versionPublished versionen
dc.format.mimetypeapplication/pdfen
dc.identifier.doihttps://doi.org/10.1111/ijsa.12418en
dc.identifier.eissn1468-2389en
dc.identifier.issn0965-075Xen
dc.identifier.orcidHickman, Louis [0000-0002-2752-7705]en
dc.identifier.urihttp://hdl.handle.net/10919/113932en
dc.language.isoenen
dc.publisherWileyen
dc.rightsCreative Commons Attribution-NonCommercial-NoDerivatives 4.0 Internationalen
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/4.0/en
dc.titleAutomatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelinesen
dc.title.serialInternational Journal of Selection and Assessmenten
dc.typeArticle - Refereeden
dc.type.dcmitypeTexten
dc.type.otherJournal Articleen
pubs.organisational-group/Virginia Techen
pubs.organisational-group/Virginia Tech/Scienceen
pubs.organisational-group/Virginia Tech/Science/Psychologyen
pubs.organisational-group/Virginia Tech/All T&R Facultyen
pubs.organisational-group/Virginia Tech/Science/COS T&R Facultyen

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Hickman Herde Lievens Tay-2023-Automatic scoring of speeded interpersonal assessment center exercises via machine learning.pdf
Size:
1.38 MB
Format:
Adobe Portable Document Format
Description:
Published version
Download

Collections

All Faculty Deposits
Scholarly Works, Psychology