ScreenTrack: Using Visual History for Self-tracking Computer Activities and Retrieving Working Context

People spend a significant amount of time using computers at work, at home, or school. Given users switch tasks and are frequently interrupted or distracted while working, reconstructing working context is inevitable. For example, users sometimes need to revisit an arbitrary task from the past to retrieve necessary information (e.g., webpages, files). In this scenario, retrieving working context can be time-consuming or even impossible; users may rely on their memory and may not be able to retrieve the relevant documents that they used before. Sometimes application provides a chronological history of recently opened documents (files, websites). However, it can be challenging to find the right information they need as there are many and users may not recognize from the text-based data (e.g. web page titles, document file name). Therefore, helping them reconstruct mental context and retrieving relevant applications and files can enhance overall productivity. To that end, the concept of self-tracking, which is widely used in health and fitness, is applied to the context of computer usage. In particular, the idea of using a history of a computer screen can provide visuals that users can associate with existing meta-data (file location, web page URL, time). A user can reconstruct working context from the screen visual that they recognize. The idea of using a visual history of a computer screen activities is tested through the development of ScreenTrack, a program that captures a computer screen regularly and let a user watch a time-lapse video made of computer screenshots, and retrieve applications, files, and web pages from a snapshot of a screen. I hypothesize that the chronological history of computer screen activities can effectively help users navigate visual working context and retrieve information that is associated with a snapshot. Through a controlled user study, it was found that participants were able to retrieve information that they were asked more quickly with ScreenTrack than the control condition with statistical significance (p<0.005). Besides, participants gave positive feedback on the software that they would like to use such software in their computers in various context, but expressed potential concerns of using such software for privacy and computer storage problems. In this thesis, I motivate the need of such software, review the related work, share the design consideration, and introduce design and implementation process, validate the effects of ScreenTrack with a controlled user study.