Efficiency and Equity in Screening for Participants in Representative Cybersecurity Studies

Abstract

Human factors research in cybersecurity confronts the practical challenge of how to recruit and screen participants with the skills necessary to perform demanding, ecologically valid tasks. Credible results from hacking studies rely on participants possessing sufficient technical skills to maneuver inside a representative network in a cyber range. Screening via self-reports can be unreliable, while rigorous skills testing is resource intensive. This paper presents empirical research investigating the effectiveness of cybersecurity certifications and cyber range skills tests for screening domain-specific competence. The Guarding Against Malicious Biased Threats (GAMBiT) project recruited participants with a two-stage screening process. The project screened 269 candidates for a final sample of 61 qualified participants. We compared the capability between the Offensive Security Certified Professional (OSCP) certification and a custom Capture-the-Flag (CTF) assessment (“Cedar Bunny”) at predicting actual performance in a two-day cyber range experiment. Results indicate that OSCP holders scored significantly higher on the skills test (p < 0.001) and progressed further in the hacking experiment (p = 0.002) than non-holders. Skills test scores correlated moderately and positively with hacking progress (ρ = 0.344). Furthermore, hierarchical regression analysis demonstrated that the skills test provided incremental predictive validity (ΔR2 = 5.2%) beyond certification alone. These findings suggest that while certifications are efficient proxies and skills tests can be effective supplements for identifying viable, albeit non-certified, participants. We propose a mixed screening approach to balance recruitment efficiency and participant diversity.