Designing Human-AI Collaborative Systems for Historical Photo Identification

Simple item page
dc.contributor.authorMohanty, Vikramen
dc.contributor.committeechairLuther, Kurten
dc.contributor.committeememberJohnson, Sylvester A.en
dc.contributor.committeememberLee, Sang Wonen
dc.contributor.committeememberLiao, Qingzien
dc.contributor.committeememberNorth, Christopher L.en
dc.contributor.departmentComputer Science and Applicationsen
dc.date.accessioned2023-08-31T08:00:22Zen
dc.date.available2023-08-31T08:00:22Zen
dc.date.issued2023-08-30en
dc.description.abstractIdentifying individuals in historical photographs is important for preserving material culture, correcting historical records, and adding economic value. Historians, antiques dealers, and collectors often rely on manual, time-consuming approaches. While Artificial Intelligence (AI) offers potential solutions, it's not widely adopted due to a lack of specialized tools and inherent inaccuracies and biases. In my dissertation, I address this gap by combining the complementary strengths of human intelligence and AI. I introduce Photo Sleuth, a novel person identification pipeline that combines crowdsourced expertise with facial recognition, supporting users in identifying unknown portraits from the American Civil War era (1861--65). Despite successfully identifying numerous unknown photos, users often face the `last-mile problem' --- selecting the correct match(es) from a shortlist of high-confidence facial recognition candidates while avoiding false positives. To assist experts, I developed Second Opinion, an online tool that employs a novel crowdsourcing workflow, inspired by cognitive psychology, effectively filtering out up to 75% of facial recognition's false positives. Yet, as AI models continually evolve, changes in the underlying model can potentially impact user experience in such crowd--expert--AI workflows. I conducted an online study to understand user perceptions of changes in facial recognition models, especially in the context of historical person identification. Our findings showed that while human-AI collaborations were effective in identifying photos, they also introduced false positives. To reduce these misidentifications, I built Photo Steward, an information stewardship architecture that employs a deliberative workflow for validating historical photo identifications. Building on this foundation, I introduced DoubleCheck, a quality assessment framework that combines community stewardship and comprehensive provenance information, for helping users accurately assess photo identification quality. Through my dissertation, I explore the design and deployment of human-AI collaborative tools, emphasizing the creation of sustainable online communities and workflows that foster accurate decision-making in the context of historical photo identification.en
dc.description.abstractgeneralIdentifying historical photos offers significant cultural and economic value; however, the identification process can be complex and challenging due to factors like poor source material and limited research resources. In my dissertation, I address this problem by leveraging the complementary strengths of human intelligence and Artificial Intelligence (AI). I built Photo Sleuth, an online platform, that helps users in identifying unknown portraits from the American Civil War era. This platform employs a novel person identification workflow that combines crowdsourced human expertise and facial recognition. While AI-based facial recognition is effective at quickly scanning thousands of photos, it can sometimes present challenges. Specifically, it provides the human expert with a shortlist of highly similar-looking candidates from which the expert must discern the correct matches; I call this as the `last-mile problem' of person identification. To assist experts in navigating this challenge, I developed Second Opinion, a tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, named seed-gather-analyze. Further, I conducted an online study to understand the influence of changes in the underlying facial recognition models on the downstream person identification tasks. While these tools enabled numerous successful identifications, they also occasionally led to misidentifications. To address this issue, I introduced Photo Steward, an information stewardship architecture that encourages deliberative decision-making while identifying photos. Building upon the principles of information stewardship and provenance, I then developed DoubleCheck, a quality assessment framework that presents pertinent information, aiding users in accurately evaluating the quality of historical photo IDs. Through my dissertation, I explore the design and deployment of human-AI collaborative tools, emphasizing the creation of sustainable online communities and workflows that encourage accurate decision-making in the context of historical photo identification.en
dc.description.degreeDoctor of Philosophyen
dc.format.mediumETDen
dc.identifier.othervt_gsexam:38416en
dc.identifier.urihttp://hdl.handle.net/10919/116169en
dc.language.isoenen
dc.publisherVirginia Techen
dc.rightsCreative Commons Attribution-NonCommercial-NoDerivatives 4.0 Internationalen
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/4.0/en
dc.subjectHuman-Computer Interactionen
dc.subjectCrowdsourcingen
dc.subjectHuman-AI Collaborationen
dc.subjectFacial Recognitionen
dc.subjectHistorical Photo Identificationen
dc.subjectOnline Communitiesen
dc.subjectDigital Humanitiesen
dc.titleDesigning Human-AI Collaborative Systems for Historical Photo Identificationen
dc.typeDissertationen
thesis.degree.disciplineComputer Science and Applicationsen
thesis.degree.grantorVirginia Polytechnic Institute and State Universityen
thesis.degree.leveldoctoralen
thesis.degree.nameDoctor of Philosophyen

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Mohanty_V_D_2023.pdf
Size:
22.72 MB
Format:
Adobe Portable Document Format
Download

Collections

Doctoral Dissertations