Toward Cybersecurity Evaluation-by-Design: Implications for Evaluation in Complex Sociotechnical Systems

Abstract

Cybersecurity is a multifaceted, sociotechnical phenomenon shaped by the dynamic interaction of people, processes, and technologies within increasingly complex organizational environments. Evaluation plays a critical role in understanding how well organizations safeguard the confidentiality, integrity, and availability of critical assets, yet prevailing approaches continue to privilege technical performance while under-examining human and organizational dynamics. This study advances the evaluation of cybersecurity practice as a sociotechnical endeavor through an exploratory inquiry that connects extant literature with practitioner-informed framework development. A scoping review of 34 studies drawn from 1,944 records examined how people, process, and technology (PPT) are conceptualized and evaluated in contemporary cybersecurity literature. The synthesis identified three dominant interaction patterns (people-process-technology, people-technology, and people-process) and four cross-cutting themes: human-centric sociotechnical synergy; the need to re-conceptualize evaluation for cybersecurity; the opportunities and ethical tensions associated with artificial intelligence; and technology as a complement rather than a substitute for human judgment. The review revealed a paradox: while cybersecurity discourse increasingly adopts sociotechnical language, evaluation practice remains fragmented and disproportionately focused on technical performance metrics. Building on these insights, the study develops the Cybersecurity Evaluation-by-Design Framework (CEDF) and used concept mapping with practitioners from critical agriculture and life sciences organizations to operationalize it. Guided by a critical realist retroduction approach, findings reveal that practitioners view cybersecurity as a complex adaptive system requiring continuous learning, evaluative thinking, and anticipatory capacity. Practitioner-generated clusters and importance-feasibility ratings informed a dynamic baseline—akin to progress markers—for assessing evolving organizational capabilities and developmental maturity. These findings offer foundational guidance for integrated, context-sensitive cybersecurity evaluation in complex and rapidly evolving environments as well as potential applications beyond cybersecurity, such as health emergency preparedness, climate adaptation, and disaster risk management contexts.