Easy comet with ruby and redis

Today we want to try something new and funny. Let’s do some Comet“) aka HTTP server push using Ajax and long pulling requests as used in Flash-less chat services such as Campfire or Talker.

Sikwamic was the closest we could find from what we wish to do. Moreover when Redis is in the project, speed is usually there.

Why not take a few minute and install ruby 1.9.2 and sinatra 1.0 unless it’s already what you use.

Install redis

If you’re on Mac OS X, chances are that you’ll want to install redis using homebrew and we cannot blame you because that’s exactly what we did:

brew install redis

You should then be able to start the redis server with something like this:

redis-server `brew --prefix`/etc/redis.conf

You can stop the redis server with:

PREFIX=`brew --prefix` && kill `cat $PREFIX/var/run/redis.pid`

Another solution is to set “daemonize no” in redis.conf so that you can simply stop it with CTRL-C.

Now you also need to install the redis gem to allow ruby to talk to your redis server:

gem install redis

If you’re not on Mac OS X or don’t use homebrew, another solution is to install redis through redis-rb.

Try redis

You can test if it did work by opening an irb console and trying these commands taken from Programmer’s Paradox guide:

require 'rubygems'
require 'redis'
r = Redis.new
r.delete('first_key') #clear it out, if it happens to be set
puts 'Set the key {first_key} to {hello world}'
r['first_key'] = 'hello world'
puts 'The value of {first_key} is:'
puts r['first_key']

If the last command returns “hello world” then you have a working redis server, if you have Errno::ECONNREFUSED errors then the redis server is probably not running.

Try Sikwamic

In the unlickely event where you haven’t installed git yet, you can install it with homebrew using the command:

brew install git

Then clone Sikwamic git repository:

git clone git://github.com/dorkalev/Sikwamic.git
cd Sikwamic

Sikwamic requires mongrel so you’ll have to install the gem:

gem install mongrel --source http://gems.rubyinstaller.org

Then run Sikwamic:

ruby sikwamic.rb

Open your browser.

• http://localhost:9291/set/user_55_name/john to set the current value in redis
• http://localhost:9291/get_if_modified/user_55_name/jeff should show john immediately
• http://localhost:9291/get_if_modified/user_55_name/john should wait 5 seconds for a change before giving up and finally showing john again

It’s easy to see that this behaviour can be used to return the latest messages in a chatroom for example, so it’s exactly what we wanted. But how does it scale?

Benchmark Sikwamic using ab

We used ab aka Apache Benchmark because it’s included in Xcode, so we have it anyway. If you’re not using Mac OS X, you’ll have to install it.

ab -n 2 -c 1 http://127.0.0.1:9291/get_if_modified/user_55_name/john

Simple: 2 connections, no concurrency. It logically takes 10 seconds. Now:

ab -n 2 -c 2 http://127.0.0.1:9291/get_if_modified/user_55_name/john

If you look at where Sikwamic is running, you’ll see the console is littered with errors. That’s really bad. At this point, the application can only handle one client at a time, not exactly what you would have expected. Following the author comment at the bottom of the article, we changed the redis connection to be thread-safe. Restart the server, set user_55_name to john again, now try again:

ab -n 2 -c 2 http://127.0.0.1:9291/get_if_modified/user_55_name/john

This time we have 2 completed requests in 5014ms, so it did actually work. The server was able to take both requests at the same time and keep them open (long pulling) while waiting for the key to change. Let’s try something bigger:

ab -n 100 -c 100 http://127.0.0.1:9291/get_if_modified/user_55_name/john

This time we’re doing 100 concurrent waiting queries. Time taken for tests: 5.146 seconds. It scales pretty good actually. Let’s continue:

ab -n 500 -c 500 http://127.0.0.1:9291/get_if_modified/user_55_name/john

Unluckily, all we get as a result is error “socket: Too many open files”. We must change the command to:

ulimit -n 10100 && ab -n 500 -c 500 http://127.0.0.1:9291/get_if_modified/user_55_name/john

Not much better, this time it tells: “apr_socket_connect(): Connection reset by peer”. The error seems to show a limitation of ab under Mac OS X more than a limitation of our Ruby application itself.

Benchmark Sikwamic using a custom libevent-based http client

We briefly tried httperf but hit the same limitations as when using ab. This post describes how they were able to open 10k concurrent connections with a Mac OS C client to test their application, thanks to a custom libevent-based http client written in C. Let’s try it out.

First, we’ll need to install libevent. Using homebrew, it couldn’t be easier:

brew install libevent

Then download the C source file using your browser or in terminal:

curl http://gist.github.com/raw/201450/a831b040f446fa5a4baea8a703956b75e74b0f07/c10k-test-client.c -o c10k-test-client.c

Compile instructions are at the end of the file. As we use homebrew, the compile command turns to:

gcc -o floodtest -levent -I`brew --prefix`/include -L`brew --prefix`/lib c10k-test-client.c

To show usage, run:

./floodtest --help

Good, so let’s try it against Sikwamic:

ulimit -n 10100 && ./floodtest 10000 127.0.0.1 9291 /get_if_modified/user_55_name/john

ulimit is required to avoid a dreaded “too many open files” error, and for the same reason we must kill the server and start it again with:

ulimit -n 10100 && ruby sikwamic.rb

Running the floodtest again, this time we were able to complete all 10k queries with a maximum concurrency of 494:

Not bad, but maximum concurrency isn’t achieved because the server starts responding after 5 seconds and it takes some time to create those 10k connections. So let’s ramp it to 2 minutes. At this time, sikwamic.rb looks like this:

require "rubygems"
require "rack"
require "redis"

puts  ">> #{REDIS = Redis.new(:thread_safe => true)} "

builder = Rack::Builder.new do
  use Rack::CommonLogger

  map "/set" do
    run Proc.new {|env|
      xxx, action, key, value = env[ "REQUEST_PATH "].split( "/ ")
      REDIS[key.to_s] = value.to_s
      [200, { "Content-Type " =>  "text/html "}, value.to_s ]
    }
  end

  map "/get" do
    run Proc.new {|env|
      xxx, action, key = env[ "REQUEST_PATH "].split( "/ ")
      [200, { "Content-Type " =>  "text/html "}, REDIS[key.to_s].to_s ]
    }
  end

  map "/get_if_modified" do
    run Proc.new {|env|
      xxx, action, key, value = env[ "REQUEST_PATH "].split( "/ ")
      i = 0
      while ((redis_value = REDIS[key.to_s]) == value.to_s) && (i < 480)
        i += 1
        sleep(0.25)
      end
      [200, { "Content-Type " =>  "text/html "}, redis_value ]

    }
  end
  puts ">> Sikwamic loaded!"
end

Rack::Handler::Mongrel.run builder, :Port => 9291

We restart the server and run the same floodtest again. However, mongrel creates a thread per waiting connection and cannot handle more than 950 thread, thus the following error message:

Reaping 950 threads for slow workers because of 'max processors'
Server overloaded with 950 processors (950 max). Dropping connection.

That’s why the floodtest crashes near 1000 connections, just when it starts to be funny.

Time to use sinatra, thin and async

At this point, we want to go beyond Sikwamic limits. Sinatra default backend is thin which can handle asynchronous responses. Thanks to the async_sinatra gem it’s even easy. Let’s install it:

gem install async_sinatra

Let’s modify our app.rb to use async_sinatra:

require  "rubygems "
require  "sinatra/base "
require  "sinatra/async "
require  "redis "

module CometTest
  class App < Sinatra::Base
    register Sinatra::Async
    puts  ">> #{REDIS = Redis.new(:thread_safe => true)} "

    get  "/set/:key/:value " do |key, value|
      REDIS[key] = value
    end

    get  "/get/:key " do |key|
      REDIS[key].to_s
    end

    aget "/get_if_modified/:key/:value" do |key, value|
      redis_value = REDIS[key]
      if redis_value==value
        n, timer = 0, EM::PeriodicTimer.new(0.25) do
          timer.cancel and body redis_value.to_s if (redis_value = REDIS[key])!=value || (n+=1)>480
        end
      else
        body redis_value
      end
    end
  end
end

We’ve added the following config.ru:

Process.setrlimit(Process::RLIMIT_NOFILE, 4096, 65536)
require File.join(File.dirname(__FILE__), "app")
run CometTest::App

With both files in current directory, we start thin this way:

ulimit -n 10100 && thin -p 9291 -e production -R config.ru start

Using your browser (same urls), you can check that the application is behaving like Sikwamic. Let’s flood this one:

ulimit -n 10000 && ./floodtest 10000 127.0.0.1 9291 /get_if_modified/user_55_name/john

This time our ruby server doesn’t throw out errors, however the floodtest itself stops with a segmentation fault. We decided to separate the client running floodtest from the server running thin. On both, we allowed more ports and files to be opened at the same time:

sudo sysctl -w net.inet.ip.portrange.first=32768
sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=32768

Running the floodtest again,