Real-time monitoring of volatile, retail investor-driven price fluctuations in the equity market

There are some failing companies in the market, and people on Reddit look for stocks with high short interest because they are trying a short squeeze. For example, the netizens moved the Meta Materials (MMTLP) price from $1 to $12 before the bubble burst. Another example is the 2021 GameStop (GME) stock. The stock surge was a volatile and unprecedented event driven by a social media frenzy and short squeeze, leading to both gains and losses for investors. We hope that our project can shed more light on these stocks. This project shows the information of targeted tickers, and also applies sentiment analysis on the newest Reddit posts fetched by data streaming as a reference for investors. Apache Kafka is used to ingest the latest data for Spark, which is responsible for real-time data processing. Elasticsearch provides full-text search so users can filter posts by time and keywords. In this project, we proposed a method to delegate the offset values of Kafka brokers to Redis, so the system is resilient even all the machine are down. Our platform presents all the data like a BI tool. Users can find related news and latest Reddit posts from different sources at the same time, and therefore make more informed decisions.