Lazyweb: Automated Crowdsourcing of Website Uptime/Downtime Tracking

This entry was published at least two years ago (originally posted on August 13, 2010). Since that time the information may have become outdated or my beliefs may have changed (in general, assume a more open and liberal current viewpoint). A fuller disclaimer is available.

Last night, Prairie and I were watching Bones on Netflix’s streaming service when Netflix suddenly stopped responding. In order to find out if there were service-wide problems, my first step was to turn to Twitter to see if there were any other people reporting problems — and as it turns out, there were. Reassured that it was a Netflix issue, and not something going wrong with my setup, we popped in a DVD until people on Twitter started reporting that things were working again.

It seems that using Twitter is becoming a more and more common way to get a quick handle on whether a particular website is having issues. This started me thinking about a website that could act as a simple, centralized tracker of uptime/downtime reports, gathered from real-time scanning of the Tweetstream. I don’t have the coding chops to do this, so I’m tossing the idea out to the Lazyweb in case anyone else wants to run with it.

The basic idea seems simple enough: scan the tweetstream for variations on the types of posts people make when a service is showing signs of problems. Basic search strings would be something along the lines of “* is [down|broken]” and “is * [down|broken]“. Anytime a hit is made on the search string, an entry is made in the database with the reported problem site and whatever might be considered relevant data from the source tweet (the tweet text, time/datestamp, perhaps even geolocation data for those tweets that are carrying it). Tracking reports of websites coming back online could be integrated as well, by watching strings such as “* is [back|up|back up|working]“.

The website would display a regularly updating display of downtime/uptime reports, one line per target website, with a series of stats indicating things like how recently the last problem or resolution tweet was recorded, the number of problem or resolution tweets found within the last 10, 30, or 60 minutes, perhaps a map showing geolocation markers that could indicate if downtime is widespread (indicating downtime at the website itself) or geographically targeted (indicating problems with a particular network, carrier, or ISP between the website and the Twitter users reporting problems), and whatever other data might be useful. It might be possible to use CSS to color-code lines depending upon variable such as the rate of problem tweets being found, too.

Anyway, that’s about as formed as the idea is in my head. If this sounds interesting to anyone else, feel free to grab the idea and run. If someone does build this based on this post, though, some mention or credit would be nice. ;)

3 thoughts on “Lazyweb: Automated Crowdsourcing of Website Uptime/Downtime Tracking”

  1. I’m not a coder, but I am a Googler, so I tried forming a simple query (netflix (down OR broken)) with Google Updates for the current day. The query that I formed – http://www.google.com/#hl=en&prmdo=1&tbs=mbl%3A1&q=netflix+%28down+OR+broken%29&aq=f&aqi=&aql=&oq=&gs_rfai=&fp=721ecd4f1146a469 – illustrates some issues with the idea, since it hit phrases such as “thumbs-down,” references to the stock being on the way down, and the like. However, if this were an actual emergency, some good information could flow out of it.

  2. I just now used this idea to verify that the Android Market was down, by searching Twitter. Worked like a charm. I may have to try to put together a quick app that automates this. Good idea!

Comments are closed.