FlickrIDR: A web-based multimodal search interface based on SuperIDR

TR Number
Date
2013-09-28
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract

In many educational fields, it is important to be able to recognize by sight, such as identifying a fish’s species in wildlife sciences. To develop these types of skills, it is important to be able to look at databases of images pertaining to the topic of study and to be able to examine relationships of images. For example, if one is doing an assignment on identifying a fish’s species by its picture, it may be useful to be able to search for images similar to the one given, taking note of certain unique aspects. FlickrIDR allows a user to specify a Flickr Image Group pertaining to their field of interest and to search through that group for images, for those similar to a given image and/or text. The goal of this project is to develop a web-based interface and system to enable multi-modal (image and annotation) search on Flickr image collections. The project will build upon the SuperIDR prototype and will provide SuperIDR’s functionality, focusing on search, in a web-based environment.
The project was separated into phases. The first phase consisted of doing research regarding the Flickr API and Flickr groups. In this phase, the Flickr API was used to gather information about Flickr groups including total number of images, number of images with annotations, and average number of annotations per image. The second phase involved creating a website as well as enabling searching of the Flickr SuperIDR image group. This was the most time and effort-intensive portion, as the website had to be designed and implemented as well as the ability to search a Flickr image group. In the third phase, the Flickr image group search capability was to be generalized. Whereas the website in phase 2 could search only the SuperIDR image group, the goal of phase 3 was to be able to indicate any Flickr image group for indexing so that any indexed group would be used in searches. As stretch goals, the project was to be released as open source, and with any extra time, research would be done on improving search algorithms. The team was able to complete effectively the 3 primary phases, with some work still left to be done. Phase 1 was completely finished on-schedule after only a few weeks. Phase 2 was completed, though later than anticipated, and also the UI designed and developed in this stage was inadequate for the functionality to be provided in phase 3. The web development proved to be quite difficult due to the need to use javascript to implement subimage selection capabilities, which are still not fully functional. Phase 3 further required of the UI that the user be able to specify a Flickr image group for indexing. This change to the UI was not implemented. The functionality to accomplish this task exists in the code base, however it will need to be integrated with the UI to allow the user to dynamically select groups for indexing. The general functionality of indexing a Flickr image group beyond the SuperIDR image group is complete. However, the ability for the user to specify that image group was not completely provided. The client’s goal was sufficiently fulfilled; however there is still a significant amount of work left to be completed in the future. For example, due to unexpected difficulties in learning new technologies during phase 2, the schedule had to be altered, leaving no time to create a sufficient automated test suite. The subimage selection in the user interface requires further work, as does image selection via URL. The UI and backend functionality need to be updated to enable the user to specify Flickr groups for indexing, though the backend code does have the capability to index Flickr groups. It also may be interesting to consider updating indexed Flickr groups and recognizing groups which have already been indexed to save time, as the indexing process is very time and bandwidth intensive. This project serves as a proof of concept and base for further development in creating a tool which allows users the kind of multi-modal search capabilities available in the SuperIDR in an easy-to-use and generic web interface.

Description
The goal of this project is to develop a web-based interface and system to enable multi-modal (image and annotation) search on Flickr image collections. The project will build upon the SuperIDR prototype and will provide SuperIDR’s functionality, focusing on search, in a web-based environment.
Keywords
Flickr, multi-modal, multimodal, search, image, annotation, SuperIDR
Citation