Mocking a Robust Streaming Twitter Bot

After playing with Twitter's streaming API this year I started to think of all the cool possibilities out there. If you could depend on a bot for a real-time response to any query… Well, there's a lot you could do. Targeted results, calculations, even two-way interactions, all from the pervasive Twitter interface. However, there is some infrastructure issues to tackle first.

The real-time API is cool and relatively easy to work with but it is not dependable. Network outages, timeouts, or even predefined cutoffs from Twitter's end can all interfere with the data streaming across. I've had a few streams working in development for over eight hours without any issue and I assume that one could run for days, it's just that this system (PHP + network streaming) is not meant to run for weeks or months. There are two things that I've thought of to deal with this issue.

First, job queueing. This is a solution that I don't have a ton of experience in, especially setting up the queueing system itself (I've only really worked with an existing Resque messaging application at work). However, this seems like the perfect application. By separating out the collection, validation, and response tasks necessary for an interactive bots you drastically cut down the chance of runtime-crashing bugs and open up some room for redundancy.

That's the second thing: redundancy. Collecting information realtime is great, but what if it fails? You lose out on potential input during that downtime, input that should be recorded and acted upon. That's where a fallback comes in, a periodic script that grabs all the input within a time period and re-processes them. Sure, there might be a delay here, but at least the message will still get through the system.

With the idea of redundancy you can start to add more endpoints for the script to chew through. A single core codebase could be checking for multiple factors (aka multiple twitter handle mentions) and choosing the appropriate items to process and respond to. Developing new functionality would be as simple as adding a new factor to follow and building the response logic. I could spawn new realtime interactive twitter bots with ease. Heh.

First, as with any new area, I want to get my feet wet. I've started working on a simple pushup-time reminder bot that will send out randomized reminders to followers to do pushups. Eventually I'd like the bot to accept input (like count of pushups done, simple report requests, etc) in realtime, making it a nice multi-faceted project. I'm hoping this project will give me an idea of how to build out these things, what sort of surprises I can expect from Twitter, and force me to do some pushups on the side. After this, who knows - I'd love to build out a full text-based game off of this platform one day.