Realtime analysis using the twitter stream API

twitter analysis application

The code for the application is on github. I develop and test this application on OS X, but I chose all the dependencies to make it likely to work on other platforms with minimum effort. On OS X I was able to brew install some dependencies and use CMake to install others.

rxcpp

I started working on rxcpp because I want a better way to write gui apps so that they do not block the main thread. I am now able to use rxcpp to build applications that do not block the main thread. I can build apps that use multi-core without writing any calls to thread, mutex or condition_variable. I can easily move work from one thread to another and back with small code changes that are much easier to reason about. I can adopt patterns like the Flux architecture easily. I can easily test code that is working over values-distributed-in-time. I can use algorithms from rxcpp to collect values-distributed-in-time into a model of values-distributed-in-space and then use algorithms from Range-v3 to process the model into a View.

why twitter analysis

I watched a presentation by @nconnaughton - github, twitter recently that inspired me. I am currently working in the Azure Machine Learning team and the live stream analysis using Rx was very attractive.

other c++ libraries

My first task was to research C++ libraries for http requests, oauth, json and a gui. Historically, this has been a big source of pain. C++ libraries have impedence mismatches by differing on error handling, async and resource management patterns. This time I found a set of libraries that did not require a lot of glue work. I have used nlohmann json before, so that was an easy pick. I have been watching dear, imgui and have really enjoyed using it. I wanted to try out beast and I have tried c++ net lib and c++ rest sdk before, but I searched and found this c++ gist that calls the twitter stream api. The gist used the curl and oauth libraries, so I used those for this project.

other resources

I used the twitter stream etiquette docs to code the error handling and this list of stop words to omit common english words.

up next