Many of us have experienced long, uninterrupted sessions of scrolling through social media feeds. But has anyone ever paused to ask: what exactly are all these users discussing?

This project seeks to answer that question. Its primary objective is to provide an unbiased, comprehensive analysis of public Mastodon servers by leveraging the capabilities of the BERTopic model. The project is designed with an extensible architecture, allowing for flexible and scalable insights into public discourse across these platforms.

Miro board here

Obtain the project

git clone https://github.com/AndrewDarnall/The-Observer.git
cd The-Observer

Run a setup shell script (builds some container images such as Apache Kafka)

bash ./Scripts/project_setup.sh

Build the project (this might take a while)

docker-compose build

Run the project

docker-compose up

Open your web browser of choice and enter:

localhost:5601/

1. Go to > Saved Objects > Import > The-Observer/Data_Visualization/saved-objects/dashboard_export.ndjson > click import
2. Reload the page as is
3. Go to dashboard and select the 'the_observer' dashboard