Twimpact Work In Progress

Twimpact has been running smoothly in its small niche of the internet, and we're currently trying to improve the way retweets are crawled and analyzed. The problem is that people often add some comment to the end of the original message, and also edit the original message such that it's not that straightforward to really know whether you have a new tweet or not.

There are also some more bugs, which will be fixed soon. For example, apparently, we weren't handling underscores in user names correctly such that "RT @nfl_games" became a retweet of the user "nfl" with the message "games", which has been retweeted more than 1800 times.

We currently also don't filter out users who retweet a tweet repeatedly or who retweet their own tweets, leading to all kinds of retweet bots and retweet-spam networks being high up in our retweet trends. While that may not be so informative, it is still interesting to see what kind of business ideas people come up with around the twitter platform.

For example, dannywhitehouse apparently has a service called twitter-bomb which I guess does all kinds of nasty things which are certainly not covered by twitter's Terms of Service, but who still managed to amass more than thirteen thousand followers.

In any, case we'll be rolling out the improvements soon, maybe this week, so stay tuned! The only problem we'll run into is that we have to reprocess all the tweets already in our database 8-O

Twimpact!

For the last one and a half months, Matthias Jugel and I have been working on a site which computes impact scores for twitter users based on how often their tweets become retweeted.

The project was really lots of fun so far. The first time we got the thing up and running was around the time of the Iranian elections and suddenly seeing all those tweets in real-time gave a feeling of directly tapping into the twitterverse.

The winner twimpact wise is clearly mashable with a twimpact score of 89 right now and over ten thousand retweeted messages. Other top users include: news cites like breakingnews, cnnbrk, and smashingmag, or celebrities with many, many followers like aplusk (Ashton Kutcher), or iamdiddy (Puff Diddy).

On the entry page you can see a live view of what has been retweeted most in the last hour. It's quite interesting to see what is popping up there. For example, surprisingly, there are many competitions of the form "retweet this and win a laptop" like this one which has been retweeted over 1300 times. Another kind of retweet is the inspirational message from users like deepak_chopra which people like to pass on. But apart from that you have of course current news, interesting links and so on. These are mostly technology and web related, which reflects the user base of twitter quite well, I think.

So go ahead and compute your twimpact score, or just sit back and look at what people are currently retweeting.

Threads, Exceptions, Net::HTTP, Timeouts and JRuby

I recently had some fun debugging a little application of mine written in JRuby which crawls the twitter search API looking for specific tweets. The problem was, every now and again, the crawler would hang even if I set appropriate timeouts. The crawler consisted of two threads, one periodically issuing a search to twitter, and another one writing the results into the database.

The first surprise to me was that in Ruby, by default, if there is an exception in a thread, the thread silently dies, not even issuing an error message. Only after you joined the thread, you get the error message. You can set Thread.abort_on_exception = true which completely kills your application, however. This meant that what appeared to be missed timeouts could just as well be uncaught exceptions.

So when working with multithreaded applications, enclosing the whole thread in a begin .. rescue Exception => e ... end is important if you want to get noticed about errors at all.

But still, the thread would misteriously die without properly handling the timeout. Some digging deeper revealed an old bug report and an interesting article about Ruby and timeouts in general which seemed to imply that timeouts might not always work, in particular if system calls are involved. It was unsure, though, whether the situation is the same for JRuby which uses native threads instead of Green threads (simulated threads by doing explicit time-slicing in the interpreter).

So I started to read the JRuby sources, to understand where and how timeouts are implemented in Net::HTTP. Which lead to my next surprise: Net::HTTP completely handles timeouts through the Timeout module, not on a socket level. It does not use the possibilities to set timeouts on reads or writes but encapsulates all the significant portions of code in Timeout::timeout { ... } calls. I also found a nice old post by Charles Nutter (a.k.a. headius) explaining the implementation in depth.

I guess based on that post, JRuby has started to implement Timeout again in Java (source). And some first tests revealed that this timeout plays well within Net::HTTP, but my crawler was still hanging every once and again.

Finally, I found the last missing piece of information: From the sources, it seems that JRuby's implementation of the Timeout module raises the Timeout::ExitException when the timeout happened. However, that is a ruby 1.9 feature, in ruby 1.8, the exception was named Timeout::Error. So basically, I was catching the wrong exception, in fact not catching it at all, thereby killing the thread (silently). Interestingly, some more testing showed that if JRuby raises a Timeout::Error if you use the Timeout module yourself, but raises a Timeout::ExitException when you're using Net::HTTP.

In the end, I just enclose the whole HTTP request section with a Timeout::timeout of my own, catching both Timeout::Error and Timeout::ExitException and finally everything was running robustly.

So in summary

• Uncaught exceptions silently kill threads in ruby.
  
• Net::HTTP does it's own timeouts through the Timeout module.
  
• Somtimes, JRuby raises Timeout::ExitException, not Timeout::Error. Will be fixed in JRuby 1.4 (see below)
  

I guess I should post a bug report on the latter... .

Update: I submitted a bug report, and it's already fixed! Those jruby people are really incredibly fast!