Semantic Vector Search using an HNSW Index for Twitter Data

Raines, Nicholas; Samarth, Mehta; Lax, Kyada; Justin, Vita; Jonah, Bishop

Semantic Vector Search using an HNSW Index for Twitter Data

dc.contributor.author	Raines, Nicholas	en
dc.contributor.author	Samarth, Mehta	en
dc.contributor.author	Lax, Kyada	en
dc.contributor.author	Justin, Vita	en
dc.contributor.author	Jonah, Bishop	en
dc.date.accessioned	2023-05-09T14:24:10Z	en
dc.date.available	2023-05-09T14:24:10Z	en
dc.date.issued	2023-05-08	en
dc.description.abstract	Semantic Vector Search is a promising alternative to keyword search for information retrieval systems. It allows for semantic meaning to be extracted from multiple different types of documents and doesn't rely on the underlying tokens of text-only documents in order to perform searches. One of the outstanding issues with vector search is that the K Nearest Neighbors (KNN) algorithm is O(N) time complexity, which does not scale well to enterprise applications. Hierarchical Navigable Small World (HNSW) retrieval systems attempt to solve this by implementing an Approximate Nearest Neighbors (ANN) in O(log(N)). This project implements both KNN and HNSW backends for an example use case with Twitter data. We found that the retrieval accuracy of the Vector Search systems was superior to a traditional keyword search system, and that the retrieval time of an HNSW backend not only greatly improves upon KNN, but is even comparable with keyword search.	en
dc.identifier.uri	http://hdl.handle.net/10919/114987	en
dc.language.iso	en	en
dc.rights	Attribution-NonCommercial 4.0 International	en
dc.rights.uri	http://creativecommons.org/licenses/by-nc/4.0/	en
dc.subject	KNN	en
dc.subject	HNSW	en
dc.subject	Vector Search	en
dc.subject	Twitter	en
dc.title	Semantic Vector Search using an HNSW Index for Twitter Data	en
dc.type	Master's project	en