Today I wanted to try the same benchmark against various HTTP server libraries:

• Misultin (Erlang)
• Mochiweb (Erlang)
• Cowboy (Erlang)
• NodeJS (V8)
• Tornadoweb (Python)

I’ve chosen these libraries because they are the ones which currently interest me the most. Misultin, obviously since I wrote it; Mochiweb, since it’s a very solid library widely used in production (afaik it has been used or is still used to empower the Facebook Chat, amongst other things); Cowboy, a newly born lib whose programmer is very active in the Erlang community; NodeJS, since bringing javascript to the backend has opened up a new whole world of possibilities (code reusable in frontend, ease of access to various programmers,…); and finally, Tornadoweb, since Python still remains one of my favourites languages out there, and Tornadoweb has been excelling in loads of benchmarks and in production, empowering FriendFeed.

Two main ideas are behind this benchmark. First, I did not want to do a “Hello World” kind of test: we have static servers such as Nginx that wonderfully perform in such tasks. This benchmark needed to address dynamic servers. Second, I wanted sockets to get periodically closed down, since having all the load on a few sockets scarcely correspond to real life situations.

For the latter reason, I decided to use a patched version of HttPerf. It’s a widely known and used benchmark tool from HP, which basically tries to send a desired number of requests out to a server and reports how many of these actually got replied, and how many errors were experienced in the process (together with a variety of other pieces of information). A great thing about HttPerf is that you can set a parameter, called –num-calls, which sets the amount of calls per session (i.e. socket connection) before the socket gets closed by the client. The command issued in these tests was:

httperf --timeout=5 --client=0/1 --server= --port=8080 --uri=/?value=benchmarks --rate= --send-buffer=4096
        --recv-buffer=16384 --num-conns=5000 --num-calls=10

The value of rate has been set incrementally between 100 and 1,200. Since the number of requests/sec = rate * num-calls, the tests were conducted for a desired number of responses/sec incrementing from 1,000 to 12,000. The total number of requests = num-conns * rate, which has therefore been a fixed value of 50,000 along every test iteration.

The test basically asks servers to:

• check if a GET variable is set
• if the variable is not set, reply with an XML stating the error
• if the variable is set, echo it inside an XML

Therefore, what is being tested is:

• headers parsing
• querystring parsing
• string concatenation
• sockets implementation

The server is a virtualized up-to-date Ubuntu 10.04 LTS with 2 CPU and 1.5GB of RAM. Its /etc/sysctl.conf file has been tuned with these parameters:

# Maximum TCP Receive Window
net.core.rmem_max = 33554432
# Maximum TCP Send Window
net.core.wmem_max = 33554432
# others
net.ipv4.tcp_rmem = 4096 16384 33554432
net.ipv4.tcp_wmem = 4096 16384 33554432
net.ipv4.tcp_syncookies = 1
# this gives the kernel more memory for tcp which you need with many (100k+) open socket connections
net.ipv4.tcp_mem = 786432 1048576 26777216
net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 2500
vm.min_free_kbytes = 65536
vm.swappiness = 0
net.ipv4.ip_local_port_range = 1024 65535
net.core.somaxconn = 65535

The /etc/security/limits.conf file has been tuned so that ulimit -n is set to 65535 for both hard and soft limits.

Here is the code for the different servers.

Misultin

-module(misultin_bench).
-export([start/1, stop/0, handle_http/1]).

start(Port) ->
    misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]).

stop() ->
    misultin:stop().

handle_http(Req) ->
    % get value parameter
    Args = Req:parse_qs(),
    Value = misultin_utility:get_key_value("value", Args),
    case Value of
        undefined ->
            Req:ok([{"Content-Type", "text/xml"}], ["<http_test><error>no value specified</error></http_test>"]);
        _ ->
            Req:ok([{"Content-Type", "text/xml"}], ["<http_test><value>", Value, "</value></http_test>"])
    end.

Mochiweb

-module(mochi_bench).
-export([start/1, stop/0, handle_http/1]).

start(Port) ->
    mochiweb_http:start([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]).

stop() ->
    mochiweb_http:stop().

handle_http(Req) ->
    % get value parameter
    Args = Req:parse_qs(),
    Value = misultin_utility:get_key_value("value", Args),
    case Value of
        undefined ->
            Req:respond({200, [{"Content-Type", "text/xml"}], ["<http_test><error>no value specified</error></http_test>"]});
        _ ->
            Req:respond({200, [{"Content-Type", "text/xml"}], ["<http_test><value>", Value, "</value></http_test>"]})
    end.

Note: i’m using misultin_utility:get_key_value/2 function inside this code since proplists:get_value/2 is much slower.

Cowboy

-module(cowboy_bench).
-export([start/1, stop/0]).

start(Port) ->
	application:start(cowboy),
	Dispatch = [
		%% {Host, list({Path, Handler, Opts})}
		{'_', [{'_', cowboy_bench_handler, []}]}
	],
	%% Name, NbAcceptors, Transport, TransOpts, Protocol, ProtoOpts
	cowboy:start_listener(http, 100,
		cowboy_tcp_transport, [{port, Port}],
		cowboy_http_protocol, [{dispatch, Dispatch}]
	).

stop() ->
	application:stop(cowboy).

-module(cowboy_bench_handler).
-behaviour(cowboy_http_handler).
-export([init/3, handle/2, terminate/2]).

init({tcp, http}, Req, _Opts) ->
    {ok, Req, undefined_state}.

handle(Req, State) ->
    {ok, Req2} = case cowboy_http_req:qs_val(<<"value">>, Req) of
        {undefined, _} ->
			cowboy_http_req:reply(200, [{<<"Content-Type">>, <<"text/xml">>}], <<"<http_test><error>no value specified</error></http_test>">>, Req);
        {Value, _} ->
			cowboy_http_req:reply(200, [{<<"Content-Type">>, <<"text/xml">>}], ["<http_test><value>", Value, "</value></http_test>"], Req)
    end,
    {ok, Req2, State}.

terminate(_Req, _State) ->
    ok.

NodeJS

var http = require('http'), url = require('url');
http.createServer(function(request, response) {
	response.writeHead(200, {"Content-Type":"text/xml"});
	var urlObj = url.parse(request.url, true);
	var value = urlObj.query["value"];
	if (value == ''){
		response.end("<http_test><error>no value specified</error></http_test>");
	} else {
		response.end("<http_test><value>" + value + "</value></http_test>");
	}
}).listen(8080);

Tornadoweb

import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
	def get(self):
		value = self.get_argument('value', '')
		self.set_header('Content-Type', 'text/xml')
		if value == '':
			self.write("<http_test><error>no value specified</error></http_test>")
		else:
			self.write("<http_test><value>" + value + "</value></http_test>")

application = tornado.web.Application([
	(r"/", MainHandler),
])

if __name__ == "__main__":
	application.listen(8080)
	tornado.ioloop.IOLoop.instance().start()

I took this code and run it against:

• Misultin 0.7.1 (Erlang R14B02)
• Mochiweb 1.5.2 (Erlang R14B02)
• Cowboy master 420f5ba (Erlang R14B02)
• NodeJS 0.4.7
• Tornadoweb 1.2.1 (Python 2.6.5)

All the libraries have been run with the standard settings. Erlang was launched with Kernel Polling enabled, and with SMP disabled so that a single CPU was used by all the libraries.

Test results

The raw printout of HttPerf results that I got can be downloaded from here.

Note: the above graph has a logarithmic Y scale.

According to this, we see that Tornadoweb tops at around 1,500 responses/seconds, NodeJS at 3,000, Mochiweb at 4,850, Cowboy at 8,600 and Misultin at 9,700. While Misultin and Cowboy experience very little or no error at all, the other servers seem to funnel under the load. Please note that “Errors” are timeout errors (over 5 seconds without a reply). Total responses and response times speak for themselves.

I have to say that I’m surprised on these results, to the point I’d like to have feedback on code and methodology, with alternate tests that can be performed. Any input is welcome, and I’m available to update this post and correct eventual errors I’ve made, as an ongoing discussion with whomever wants to contribute.

However, please do refrain from flame wars which are not welcomed here. I have published this post exactly because I was surprised on the results I got.

What is your opinion on all this?

—————————————————–

UPDATE (May 16th, 2011)

Due to the success of these benchmarks I want to stress an important point when you read any of these (including mines).

Benchmarks often are misleading interpreted as “the higher you are on a graph, the best that *lib-of-the-moment-name-here* is at doing everything”. This is absolutely the wrongest way to look at those. I cannot stress this point enough.

‘Fast’ is only 1 of the ‘n’ features you desire from a webserver library: you definitely want to consider stability, features, ease of maintenance, low standard deviation, code usability, community, developments speed, and many other factors whenever choosing the best suited library for your own application. There is no such thing as generic benchmarks. These ones are related to a very specific situation: fast application computational times, loads of connections, and small data transfer.

Therefore, please use this with a grain of salt and do not jump to generic conclusions regarding any of the cited libraries, which as I’ve clearly stated in the beginning of my post I all find interesting and valuable. And I still am very open in being criticized for the described methodology or other things I might have missed.

Thank you,

r.

Intrigued, I decided to perform additional testing, by performing a small benchmark of a random key retrieval on lists of different sizes, using those two different functions.

Here’s the code I used:

-module(pvsl).
-define(LIST_SIZES, [10000, 100000, 1000000]).
-define(RETRIES, 1000).
-compile(export_all).

start() ->
	% test for different list sizes
	lists:foreach(fun(N) -> test_list(N) end, ?LIST_SIZES).

test_list(ListSize) ->
	% generate a list of size ListSize of {Key, Val} entries
	KeyList = [{K, K} || K <- lists:seq(1, ListSize)],
	% test this list against both functions
	lists:foreach(fun(Type) -> get_val(Type, now(), KeyList, ListSize, ?RETRIES) end,
		[proplists, lists]).

% test getting values, compute necessary time and output print results
get_val(Type, Start, _KeyList, ListSize, 0) ->
	T = timer:now_diff(now(), Start),
	io:format("computed ~p random key searches on a ~p-sized list in ~p ms using ~p~n",
		[?RETRIES, ListSize, T/1000, Type]);
get_val(proplists, Start, KeyList, ListSize, Tries) ->
	proplists:get_value(random:uniform(ListSize), KeyList),
	get_val(proplists, Start, KeyList, ListSize, Tries - 1);
get_val(lists, Start, KeyList, ListSize, Tries) ->
	lists:keyfind(random:uniform(ListSize), 1, KeyList),
	get_val(lists, Start, KeyList, ListSize, Tries - 1).

I ran this test on my MacBook Pro, Intel Core i5 2.4GHz with 4GB Memory, and Erlang R13B04, with Kernel Polling enabled. These are the results.

roberto$ erl +K true +P 1000000
Erlang R13B04 (erts-5.7.5) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:true]

Eshell V5.7.5  (abort with ^G)
1> c(pvsl).
{ok,pvsl}
2> pvsl:start().
computed 1000 random key searches on a 10000-sized list in 323.373 ms using proplists
computed 1000 random key searches on a 10000-sized list in 12.897 ms using lists
computed 1000 random key searches on a 100000-sized list in 3273.973 ms using proplists
computed 1000 random key searches on a 100000-sized list in 130.592 ms using lists
computed 1000 random key searches on a 1000000-sized list in 34131.905 ms using proplists
computed 1000 random key searches on a 1000000-sized list in 2050.627 ms using lists
ok
3>

Of course, this is far from being a complete benchmark, however it does show that proplists:get_value/2 is considerably slower [up to 25 times on smaller lists on this benchmark run] than lists:keyfind.

Not sure why this is happening, though I would recommend using lists:keyfind/3 on all speed-critical parts of your code.

Basically, websockets allow a two-way asynchronous communication between browser and servers, filling the gap that some technologies such as ajax and comet have tried to fulfill in these recent years. If you want to try this out yourself, you will first need to grab a browser which implements websockets, such as Google Chrome.

The typical html page with javascript code to use websockets is as follows:

<html>
   <head>
     <script type="text/javascript">
         function addStatus(text){
            var date = new Date();
            document.getElementById('status').innerHTML = document.getElementById('status').innerHTML
               + date + ": " + text + "<br>";
         }
         function ready(){
            if ("WebSocket" in window) {
               // browser supports websockets
               var ws = new WebSocket("ws://localhost:8080/service");
               ws.onopen = function() {
                  // websocket is connected
                  addStatus("websocket connected!");
                  // send hello data to server.
                  ws.send("hello server!");
                  addStatus("sent message to server: 'hello server'!");
               };
               ws.onmessage = function (evt) {
                  var receivedMsg = evt.data;
                  addStatus("server sent the following: '" + receivedMsg + "'");
               };
               ws.onclose = function() {
                  // websocket was closed
                  addStatus("websocket was closed");
               };
            } else {
               // browser does not support websockets
               addStatus("sorry, your browser does not support websockets.");
            }
         }
      </script>
   </head>
   <body onload="ready();">
      <div id="status"></div>
   </body>
</html>

Here’s the code to use misultin to handle the requests of this script:

-module(misultin_websocket_example).
-export([start/1, stop/0]).

% start misultin http server
start(Port) ->
	misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req, Port) end},
	{ws_loop, fun(Ws) -> handle_websocket(Ws) end}]).

% stop misultin
stop() ->
	misultin:stop().

% callback on request received
handle_http(Req, Port) ->
	% output
	Req:ok([]).

% callback on received websockets data
handle_websocket(Ws) ->
	receive
		{browser, Data} ->
			Ws:send(["received '", Data, "'"]),
			handle_websocket(Ws);
		_Ignore ->
			handle_websocket(Ws)
	after 5000 ->
		Ws:send("pushing!"),
		handle_websocket(Ws)
	end.

handle_websocket/1 is spawned by misultin to handle the connected websockets. Data coming from a browser will be sent to this process and will have the message format {browser, Data}, where Data is a string(). If you need to send data to the browser, you may do so by using the parametrized function Ws:send(Data), Data being a string() or an iolist().

Compile and run the example here above with misultin_websocket_example:start(8080). Then, open up your Chrome (or other websocket compliant browser) and point it to an .html file containing the above code.

You should normally see this being gradually printed on your browser:

Wed Jan 20 2010 15:18:52 GMT+0100 (CET): websocket connected!
Wed Jan 20 2010 15:18:52 GMT+0100 (CET): sent message to server: 'hello server'!
Wed Jan 20 2010 15:18:52 GMT+0100 (CET): server sent the following: 'received 'hello server!''
Wed Jan 20 2010 15:18:57 GMT+0100 (CET): server sent the following: 'pushing!'
Wed Jan 20 2010 15:19:02 GMT+0100 (CET): server sent the following: 'pushing!'
Wed Jan 20 2010 15:19:07 GMT+0100 (CET): server sent the following: 'pushing!'

In normal environments you may consider serving the .html page from misultin directly. You may do so with the following and complete misultin module:

-module(misultin_websocket_example).
-export([start/1, stop/0]).

% start misultin http server
start(Port) ->
   misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req, Port) end}, {ws_loop, fun(Ws) -> handle_websocket(Ws) end}]).

% stop misultin
stop() ->
   misultin:stop().

% callback on request received
handle_http(Req, Port) ->
   % output
   Req:ok([{"Content-Type", "text/html"}],
   ["
   <html>
      <head>
         <script type=\"text/javascript\">
            function addStatus(text){
               var date = new Date();
               document.getElementById('status').innerHTML = document.getElementById('status').innerHTML
                  + date + \": \" + text + \"<br>\";
            }
            function ready(){
               if (\"WebSocket\" in window) {
                  // browser supports websockets
                  var ws = new WebSocket(\"ws://localhost:", integer_to_list(Port) ,"/service\");
                  ws.onopen = function() {
                     // websocket is connected
                     addStatus(\"websocket connected!\");
                     // send hello data to server.
                     ws.send(\"hello server!\");
                     addStatus(\"sent message to server: 'hello server'!\");
                  };
                  ws.onmessage = function (evt) {
                     var receivedMsg = evt.data;
                     addStatus(\"server sent the following: '\" + receivedMsg + \"'\");
                  };
                  ws.onclose = function() {
                     // websocket was closed
                     addStatus(\"websocket was closed\");
                  };
               } else {
                  // browser does not support websockets
                  addStatus(\"sorry, your browser does not support websockets.\");
               }
            }
         </script>
      </head>
      <body onload=\"ready();\">
         <div id=\"status\"></div>
      </body>
   </html>"]).

% callback on received websockets data
handle_websocket(Ws) ->
   receive
      {browser, Data} ->
         Ws:send(["received '", Data, "'"]),
         handle_websocket(Ws);
      _Ignore ->
         handle_websocket(Ws)
   after 5000 ->
      Ws:send("pushing!"),
      handle_websocket(Ws)
   end.

Please note that the Websocket Protocol still is draft. use with caution.

Here is the simple code for Misultin’s Hello World.

-module(misultin_hello_world).
-vsn('0.1').
-export([start/1, stop/0, handle_http/1]).

% start misultin http server
start(Port) ->
    misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]).

% stop misultin
stop() ->
    misultin:stop().

% callback on request received
handle_http(Req) ->
    Req:ok("Hello World.").

Here’s the code to echo a GET variable in a XML form.

-module(misultin_get_variable).
-vsn('0.1').
-export([start/1, stop/0, handle_http/1]).

% start misultin http server
start(Port) ->
    misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]).

% stop misultin
stop() ->
    misultin:stop().

% callback on request received
handle_http(Req) ->
    % get params
    Args = Req:parse_qs(),
    Value = proplists:get_value("value", Args),
    case Value of
        undefined ->
            Req:ok([{"Content-Type", "text/xml"}], "no value specified");
        _ ->
            Req:ok([{"Content-Type", "text/xml"}], "~s", [Value])
    end.

Available also are additional code examples, and the full list of exports.

You may find Misultin, released under the New BSD License, on its project page on Google code.

In the systems I’m building I need to get as fast as possible in message passing between nodes. Who doesn’t? :) I was getting frustrated when a server could achieve amazing performance over a single node, performance which got much lower once the server got distributed over two Erlang nodes. I therefore tried a small benchmark test to somehow measure the difference in message passing speed within/across Erlang nodes.

The machine used in this benchmark is an apple macbook running Leopard, with 2.0GHz Intel Core 2 Duo and 2GB of DDR3 RAM memory.

This benchmark is a very simple ping request/pong response test. A process ‘A’ sends a ping message to process ‘B’, which replies with a pong message.

Code for the pong process:

% starts pong
start_pong() ->
	register(flood_pong, spawn(fun() -> pong_loop() end)).

pong_loop() ->
	receive
		{{Sender, SenderNode}, Any} ->
			% pong back
			{Sender, SenderNode} ! {pong, Any},
			pong_loop();
		shutdown ->
			io:format("pong shutdown~n",[]);
		_Ignore ->
			pong_loop()
	after 60000 ->
		io:format("pong timeout, shutdown~n",[])
	end.

This very simple code basically starts a process and registers it with the name flood_pong. You can see that to any request with format:

{{Sender, SenderNode}, Any}

flood_pong will reply with a message with format:

{pong, Any}

The code for the ping process, i.e. the one that starts the ping requests, is the following:

% start ping
start_ping(PongNode, Num) ->
	register(flood_ping, spawn(fun() -> ping_loop(Num, now(), Num) end)),
	send(PongNode, Num).

send(_PongNode, 0) ->
	ok;
send(PongNode, Num) ->
	% send a spawned ping
	spawn(fun() -> {flood_pong, PongNode} ! {{flood_ping, node()}, ping_request} end),
	send(PongNode, Num - 1).

ping_loop(Num, Start, 0) ->
	T = timer:now_diff(now(), Start),
	io:format("RECEIVED ALL ~p in ~p ms [~p/min]~n",[Num, T, (Num*60000000/T)]);
ping_loop(Num, Start, Count) ->
	receive
		{pong, _PingBack} ->
			ping_loop(Num, Start, Count-1);
		_Received ->
			ping_loop(Num, Start, Count)
	after 10000 ->
		io:format("ping timeout, missing ~p pong, shutdown~n",[Count])
	end.

This code does two things:

• Starts and registers a flood_ping process, which will be responsible to await and count the incoming pong replies. This allows it to compute the time needed for all the pong replies to get back to the node where the ping requests where first sent.
• Spawns the ping requests [one process per ping request].

Let’s give it a try. First, we have to start up two Erlang nodes, so I fire up terminal and enter:

:~ roberto$ erl +K true +P 500000 -name 'one@rob.loc' -setcookie asd

and in another terminal window:

:~ roberto$ erl +K true +P 500000 -name 'two@rob.loc' -setcookie asd

This initializes two erlang nodes, ‘one@rob.loc’ and ‘two@rob.loc’, both with kernel polling enabled and a maximum number of processes per node set to 500,000.

I need to ensure that the nodes can see each other, so from ‘one@rob.loc’ I ping ‘two@rob.loc’:

(one@rob.loc)1>net_adm:ping('two@rob.loc').
pong
(one@rob.loc)2>nodes().
['two@rob.loc']

Now let’s perform a message passing speed benchmark for messages sent and received within the same Erlang node. First, I start up the pong process:

(one@rob.loc)3> flood1:start_pong().
true

I then need to start the ping requests, and I do so by issuing the start_ping/2 function, which has per arguments the name of the recipient node of the ping requests, and the number of requests to be performed. I therefore set the first parameter to node() and the second to 200,000 [we'll always try out 200,000 ping requests].

(one@rob.loc)4> flood1:start_ping(node(), 200000).
ok
RECEIVED ALL 200000 in 2257727 ms [5315080.166911234/min]

I see therefore that within the same Erlang node, this benchmark gives a result of an average of 5.3 million ping/pong messages per minute. Quite a rush.

Let’s now try the same thing with pong running on ‘two@rob.loc’, and ping requests coming from ‘one@rob.loc’. I start the pong process on ‘two@rob.loc’:

(two@rob.loc)1> flood1:start_pong().

Then I issue the ping request from ‘one@rob.loc’:

(one@rob.loc)5> flood1:start_ping('two@rob.loc', 200000).
ok
RECEIVED ALL 200000 in 16727643 ms [717375.4246189974/min]

I see therefore that on two different Erlang nodes, this benchmark gives a result of an average of 700 thousands ping/pong messages per minute.

This is around 7 times a loss in performance when compared to the same benchmark run on a single Erlang node. It is simply unacceptable: it does mean that a system heavily dependant on message passing and running on a single Erlang node would be considerably faster than one running on distributed nodes [one of the things Erlang is great at doing].

At this point, I tried reducing the overhead in message passing, by using UDP instead of Erlang’s native functionalities. This was no success at all, UDP having all the troubles it’s known for (packet loss, ordering, double packet sending, …) without me getting any interesting speed improvement.

It’s only here that I got this thought: what if Erlang’s message passing overhead is so demanding, due to TCP/IP, that actually queuing messages sent to processes running on the same node and sending them altogether would increase the overall passing speed? TCP/IP already does provide such mechanisms, but what if performing this queuing at Erlang application level could bring significant improvements to infra-nodes message passing?

This is the picture. On both Erlang nodes I run a gen_server [which I called qr] which performs queuing and routing of messages sent from one node to the other. Instead of sending messages directly from a process A on node ‘one@rob.loc’ to a process B on node ‘two@rob.loc’, process A sends a routing request to the qr running on ‘one@rob.loc’, which will then send it to the qr running on ‘two@rob.loc’, which will finally forward it to B. Fact is, the sending qr will queue messages until a certain number of them for the same node have piled up, or a timeout period has expired.

I start by defining the gen_server API to route a message. Here it is:

% Function() -> void()
% Description: Gets a routing request
route({ToPid, ToNode}, Message) ->
	case ToNode =:= node() of
		true ->
			% send directly
			ToPid ! Message;
		false ->
			% queue
			gen_server:cast(?SERVER, {{queue, ToNode}, {ToPid, Message}})
	end.

You can see that we need to pass the Pid of the destination process, as well as the name of the Node where this process is running, together with the Message that we need to send.

In case the node is remote, this will cast a message to qr, which will handle it:

% add incoming routing request to queue
handle_cast({{queue, DestNode}, {ToPid, Message}}, Queue) ->
	% to pid
	Msg = {route, ToPid, Message},
	% get if node exists in queue
	case lists:keysearch(DestNode, 1, Queue) of
		false ->
			% add node
			NewQueue = [{DestNode, {now(), [Msg]}}|Queue];
		{value, {DestNode, {CreationTime, MsgList}}} ->
			% check if queue is long enough
			case length(MsgList) >= ?QUEUELENGTH of
				true ->
					% queue of a node is of maximum lenght, send routing message
					{?QR, DestNode} ! {queue_route, [Msg|MsgList]},
					% empty queue for node
					NewQueue = lists:keydelete(DestNode, 1, Queue);
				false ->
					% add message to queue list and replace node
					NewQueue = lists:keyreplace(DestNode, 1, Queue, {DestNode, {CreationTime, [Msg|MsgList]}})
			end
	end,
	% purge timeout
	PurgedQueue = purge_queue_selective(NewQueue),
	{noreply, PurgedQueue, 200};

[...]

% Function -> PurgedQueue
% Description: Loop queue and purge only nodes on timeout.
purge_queue_selective(Queue) ->
	% check timeout, send and remove element if needed
	FilterFun = fun({DestNode, {CreationTime, MsgList}}) ->
		case timer:now_diff(now(), CreationTime) > ?QUEUETIMEOUT of
			true ->
				% timeout for a node, send routing message
				% io:format("send selective: {?QR, ~p} ! {queue_route, ~p}~n",[DestNode, MsgList]),
				{?QR, DestNode} ! {queue_route, MsgList},
				% delete node from queue
				false;
			false ->
				true
		end
	end,
	% return cleaned queue
	lists:filter(FilterFun, Queue).

The mecanism is quite simple. When a message to be sent to a remote node arrives to qr, we build a list of key tuples [which will be passed along as gen_server State variable] of format:

{DestNode, {CreationTime, MsgList}}

DestNode, i.e. the destination node, is the tuple key. CreationTime is a parameter which allows to know when a timeout for the messages to DestNode has passed. MsgList is a list of tuples of the messages to be sent to DestNode, in format:

{route, ToPid, Message}

We handle this key list to check when to send all the queued messages to the other node, which we do:

• when a certain timeout has passed for that node [computed using CreationTime];
• when the number of messages to be sent has reached a certain size.

We see that the gen_server cast also establishes an overall timeout of 200 ms, which will fire when no other messages are received by qr and thus we need to send the remaining messages out. This timeout is handled with a handle_info call:

% timeout on a cast message, purge queue
handle_info(timeout, Queue) ->
	% purge timeout
	PurgedQueue = purge_queue(Queue),
	% return
	{noreply, PurgedQueue};

[...]

% Function -> PurgedQueue
% Description: Loop queue and purge all nodes.
purge_queue(Queue) ->
	% sent to all remaining
	FilterFun = fun({DestNode, {_CreationTime, MsgList}}) ->
		% send routing message
		{?QR, DestNode} ! {queue_route, MsgList}
	end,
	lists:foreach(FilterFun, Queue),
	% return an empty queue
	[].

As we can see from here over, a message sent from a qr on one node to a qr of another node has format:

{queue_route, MsgList}

With MsgList being a list of the individual messages in format:

{route, ToPid, Message}

When a qr of a node receives such a message, it handles it:

handle_info({queue_route, MsgList}, State) ->
	RouteFun = fun({route, ToPid, Message}) ->
		% local send to node
		ToPid ! Message
	end,
	lists:foreach(RouteFun, MsgList),
	% return
	{noreply, State, 200};

Ok, this basically is it. Let’s now see if this does improve message passing between nodes.

For this, we need to redifine the pong_loop so that it does not send messages directly to the process, but routes its requests to qr instead.

New code for the pong process:

% starts pong
start_pong() ->
	register(flood_pong, spawn(fun() -> pong_loop() end)).

pong_loop() ->
	receive
		{{Sender, SenderNode}, Any} ->
			% pong back
			qr:route({Sender, SenderNode}, {pong, Any}),    % < === ONLY LINE CHANGED ===
			pong_loop();
		shutdown ->
			io:format("pong shutdown~n",[]);
		_Ignore ->
			pong_loop()
	after 30000 ->
		io:format("pong timeout, shutdown~n",[])
	end.

The same goes for the ping process:

% start ping
start_ping(PongNode, Num) ->
	register(flood_ping, spawn(fun() -> ping_loop(Num, now(), Num) end)),
	send(PongNode, Num).

send(_PongNode, 0) ->
	ok;
send(PongNode, Num) ->
	% send a spawned ping                               % === \/ ONLY LINE CHANGED ===
	spawn(fun() -> qr:route({flood_pong, PongNode}, {{flood_ping, node()}, ping_request}) end),
	send(PongNode, Num - 1).

ping_loop(Num, Start, 0) ->
	T = timer:now_diff(now(), Start),
	io:format("RECEIVED ALL ~p in ~p ms [~p/min]~n",[Num, T, (Num*60000000/T)]);
ping_loop(Num, Start, Count) ->
	receive
		{pong, _PingBack} ->
			ping_loop(Num, Start, Count-1);
		_Received ->
			ping_loop(Num, Start, Count)
	after 10000 ->
		io:format("ping timeout, missing ~p pong, shutdown~n",[Count])
	end.

We are all set. Let’s try to immediately perform a test on two different Erlang nodes, pong running on ‘two@rob.loc’, and ping requests coming from ‘one@rob.loc’. I start qr and the pong process on ‘two@rob.loc’:

(two@rob.loc)55> qr:start_link().
{ok,<0.38.0>}
(two@rob.loc)56> flood2:start_pong().
true

Then I start qr and issue the ping request from ‘one@rob.loc’:

(one@rob.loc)44> qr:start_link().
{ok,<0.114.0>}
(one@rob.loc)45> flood2:start_ping('two@rob.loc', 200000).
ok
RECEIVED ALL 200000 in 5690225 ms [2108879.7015935224/min]

I see therefore that on two different Erlang nodes, this benchmark gives a result of an average of 2.1 million ping/pong messages per minute, 3 times as much as without message queuing!

Just to be on the safe side, I try the same benchmark for messages sent via qr to processes running on the same Erlang node.

{ok,<0.3562.6>}
(two@rob.loc)50> flood2:start_pong().
true
(two@rob.loc)51> flood2:start_ping(node(), 200000).
ok
RECEIVED ALL 200000 in 2264539 ms [5299091.779828035/min]

I see therefore that within the same Erlang node, this benchmark gives a result of an average of 5.3 million ping/pong messages per minute, the same that we had without the queuing mechanism.

To summarize:

• without queuing mechanism:
  • same Erlang node: 5.3 million messages/min;
  • different Erlang nodes: 700 K messages/min.
• with queuing mechanism:
  • same Erlang node: 5.3 million messages/min;
  • different Erlang nodes: 2.1 million messages/min.

The complete code to run this on your machine is available here. This whole ‘queuing idea’ is still an experiment, and I’d be more than delighted to hear your feedback, to see whether you are getting the same results, you know how to improve the concept or the code, or you have any considerations at all you would like to share.

UPDATE [April 17th, 2009]

Due to a comment here below from Adam Bregenzer, who performed a test on a Linux box and had very different results, I’ve decided to perform some testing on 3 different OS. Here is a summary of these tests, but please bear in mind that these results are definitely non exhaustive.

1. macbook running Leopard OSX, with 2.0GHz Intel Core 2 Duo and 2GB of DDR3 RAM memory.

• on the same Erlang node: 5.3 million messages/min;
• on different Erlang nodes:
  • without queuing mechanism: 700 K messages/min.
  • with queuing mechanism: 2.1 million messages/min.

2. Ubuntu 8.10 Linux, 64-bit system with 2.5GHz Intel Core 2 Duo [Adam's test]

• on the same Erlang node: 11.7 million messages/min;
• on different Erlang nodes:
  • without queuing mechanism: 4.8 million messages/min.
  • with queuing mechanism: 3.6 million messages/min.

3. Ubuntu 9.04 beta Linux, 32-bit system on a VM with 1 CPU 2.0GHz and 512MB of RAM memory.

• on the same Erlang node: 4.6 million messages/min;
• on different Erlang nodes:
  • without queuing mechanism: 1.6 million messages/min.
  • with queuing mechanism: 960K messages/min.

4. Windows XP, with 3.0GHz Intel Core 2 Duo E8400 and 2.5GB of DDR3 RAM memory.

• on the same Erlang node: 24.7 million messages/min;
• on different Erlang nodes:
  • without queuing mechanism: 2.2 million messages/min.
  • with queuing mechanism: 5.1 million messages/min.

As it is normal with Erlang, results are extremely different depending on many factor, one of which seems to be the OS. On the non-exhaustive tests here above, the qr mechanism does:

• improve message passing performance between Erlang nodes in the OSX and Windows environment;
• decrease message passing performance between Erlang nodes in the Ubuntu Linux environment.

I do not have the insights to understand which characteristics of each OS is switching results in this manner: TCP/IP flow, threads, I/O. Though, I’m currently deploying a similar qr mechanism in my applications, which can be activated if necessary based on individual benchmarking tests.

UPDATE [June 9th, 2009]

I have had the availability of two HP ProLiant DL380, running Ubuntu 8.04 Server 32 bit, therefore I decided to take a shot also to test the same bench on different machines.

Here is a summary of the results of these tests. As usual, please bear in mind that these results are definitely non exhaustive.

• on the same Erlang node: 5.3 million messages/min [on both machines];
• on two different Erlang nodes, both running on the same machine:
  • without queuing mechanism: 1.7 K million messages/min.
  • with queuing mechanism: 3.1 million messages/min.
• on two different Erlang nodes, each running on a different machine:
  • without queuing mechanism: 1.7 K million messages/min.
  • with queuing mechanism: 3.2 million messages/min.

This seems to state that the qr mechanism does improve message passing performance between Erlang nodes, also on the Ubuntu 8.04 environment.

I’ve also updated the original code, so that the destination node identifier is now extracted from the recipient Pid and thus it is not necessary to pass it as variable. This code can be found here.

Here are some attempts at stone balancing.

A Technical Introduction to OpenSpime

OpenSpime Buzz

The whole buzz started when the following video was presented. One of the first persons to blog about this was Bruce Sterling in his blog on Wired, Beyond the Beyond, but OpenSpime has had a worldwide coverage since then (IT Conversations, Cisco’s blogs, Lunch Over IP, and Ugotrade amongst others).

OpenSpime Developer Network

Together with Davide ‘Folletto’ Casali, I’ve written a python library called PyOpenSpime which will easily get you started to OpenSpime.

If you are feeling curious, you may consider joining the OpenSpime Developer Network.

Here’s a concept application that was presented by our team at the Savio Event in London, which clarifies a possible use of virtual assistants in a business environment. As you will see, these scarcely qualify as mere ‘talking heads’.

My opinions regarding how to approach the realization of such applications is outlined in the following article, originally published on UXmatters.

The Composite Intelligence of Virtual Assistants

As early as 1968, influenced by visionary movies such as Stanley Kubrick’s 2001: A Space Odyssey, with its fictional computer character HAL 9000, we envisioned omniscient, intelligent machines that could easily contain the whole range of human knowledge. Then, in 1987, the Knowledge Navigator video from Apple Computer struck our imaginations and greatly boosted our expectations of what computers would be able to do for us in a few years’ time.

It is now 20 years later, but technology is still very far from fulfilling such hopes regarding virtual assistants:

• Voice recognition isn’t immediate.
• People adapt to machines’ lexicons rather than the other way around.
• Computers cannot easily transcend the scope for which they have been programmed.
• Self-learning machines are still experimental.
• The generation of high-quality graphics representing emotionally rich, 3-D avatars is far from being real time, cheap, or extensively available.

The technology for fulfilling such high expectations simply does not yet exist. However, while we wait for technology to catch up with our dreams, the gap between our expectations and the reality of what we can now achieve has inspired two important trends in the development of virtual assistants:

• simulation of generic conversations
• application-specific virtual assistants

From the Generic to the Application Specific

Unfortunately, the first approach—a computer that can simulate a generic conversation with a real person—remains an unrealized dream. However, a series of intelligence-simulation experiments has culminated in the Loebner Prize in Artificial Intelligence—“the first formal instantiation of a Turing Test,” which demonstrates a computer’s intelligence and would award a gold medal and a $100,000 prize “for the first computer whose responses were indistinguishable from a human’s.” Every year, Hugh Loebner awards a bronze medal to the computer software that best simulates a generic conversation with a real person. Interestingly, Marvin Minsky, co-founder of MIT’s Artificial Intelligence (AI) Laboratory and author of several texts on AI, is one of the strongest critics of the Loebner Prize. He has even provocatively offered a monetary reward to anyone who can successfully convince Loebner to revoke his prize.

On the other hand, virtual assistant applications are moving toward becoming application specific, with the aim of responding appropriately to specific users’ needs by providing precise and relevant information.

While the Turing Test approach might offer the fastest results in successfully simulating generic human conversations, in reality, these simulations are often nothing more than escamotages, or sleights of hand, and are not able to achieve a real understanding of users’ ideas. Such experiments concentrate on the use of various tricks to make users believe a virtual assistant has understood them, while in reality, it is more about

• turning users’ questions back on them—as a psychiatrist would do, for instance
• suddenly changing the mood of the virtual assistant to surprise—covering the fact that it simply hadn’t understood the user’s input in the first place
• switching the topic of conversation to one the application can handle better—by pretending not to be interested in what a user has said
• building strong characters that tend to impose their will—thus getting control of the flow of a conversation to ensure it remains on safe ground

On the other hand, the second approach—application-specific virtual assistants—is a more slowly evolving process that would require considerable investment and the use of diverse software technologies. However, its aim is to try to build applications that actually can have a real understanding of users’ input. Circumscribing an application’s context simplifies the understanding of users’ needs, thus reducing the amount of intelligence it needs to understand and communicate effectively. This has the double effect of reducing technology requirements to a level we can realistically attempt and increasing the level of specialization each virtual assistant can achieve. We can concentrate all our efforts on an application’s main purpose rather than trying to support the broad knowledge that responding to generic conversations would require. Since, in comparison to the wide open context of the first approach, this approach limits an application’s context much more, it is generally less appealing to users’ imaginations, while at the same time, being much more effective in responding to users’ requests.

While the tricks inherent in the first approach are definitely worth considering, with their goal of increasing the level of human-like interaction virtual assistant applications can offer, I believe only the second approach enables us to build applications that really meet users’ needs. Typically, users rather quickly become accustomed to a virtual assistant, but get annoyed by any useless chatter, so the ability to carry on a generic conversation might not be desirable anyway. Also, the goal of the second approach is to build business applications, so in my opinion, this will be the approach that leads us down the path toward intelligent virtual assistants.

The Intelligence of Software

What we mean by intelligent virtual assistants often points directly to our definition of artificial intelligence. Each of us has some perception of what AI is, but if we try to unambiguously define it, we will rapidly fall into a series of borderless conceptions that bounce from science fiction reminiscences—the ability of the HAL 9000 to perceive the emotions of its interlocutors and understand all of their requests—to pragmatic requirements—machines that can automate tasks requiring complex decisions—to cutting-edge technologies—self-learning machines—or to your impressions the first time you ever saw a computer providing an answer to a query of yours. So, as you can see, it is more a matter of people’s perception of what intelligence really is. When an AI application actually succeeds, it is common to hear comments like “It was not that hard after all,” or “It’s not real AI anyway.” According to AI pioneer Raymond Kurzweil, such responses are inherent in the nature of AI itself. As he said in a c|net interview:

“It turns out to be easy to replicate most adult achievements, like playing chess or guiding cruse missiles, etcetera. At least a lot of them we can replicate in machines. Every time a machine does that, it’s the nature of artificial intelligence to say, “Well, that wasn’t very difficult after all.” The hard things are what children do, recognizing parents or playing with a dog.”

You can easily imagine, therefore, why the definition of artificial intelligence has been cause for discussion for decades now.

However, it is not the purpose of this article to discuss what artificial intelligence is or how we can emulate human intelligence. Rather, my intent is to understand how virtual assistants can approach intelligence—strictly from a pragmatic point of view.

Since software is the primary factor in making a machine act intelligently, I prefer to use the term software intelligence instead of artificial intelligence. This avoids people’s being misled by the expectations surrounding the latter and allows us to concentrate on the practical considerations that are involved in the creation of intelligent virtual assistants.

I have grouped the software intelligence we need to build virtual assistants into five different levels of intelligence, which together compose an intelligence users perceive as a monolithic, composite intelligence, within the context of their experience (thus, the use of the term composite intelligence in this article’s title). The five levels of software intelligence are:

• input intelligence
• emotional intelligence
• logical intelligence
• knowledge intelligence
• output intelligence

Input Intelligence

There are numerous input devices that enable a user to input his requests to a virtual assistant—for instance, keyboards, touch screens, microphones, even motion capture systems. Various types of input intelligence—the first level of software intelligence—enable a virtual assistant to immediately interpret such inputs. Some examples include the following:

• semantic intelligence—A semantic engine can immediately parse the text a user enters via a keyboard and identify the user’s needs by disambiguating the semantic meaning of the user’s words, thus avoiding the misinterpretation of ambiguous requests.
• voice-recognition intelligence—Software can interpret the speed or change of tone in a user’s speech as emotional values that the other levels of intelligence can then use to more properly adapt to the user’s requests. Thus, voice recognition encompasses not only the ability to capture our words as we speak, but the intelligence enabling the software to perceive and interpret changes in speed and tone.
• facial-recognition intelligence—Software can also perceive a user smiling and interpret a smile as an emotional value that the other levels of intelligence can use.

Input intelligence is, therefore, able to add a lot of metadata to the software’s understanding of a user’s intended inputs—the kinds of metadata that typically enhance human conversations.

Emotional Intelligence

Input intelligence can understand both a user’s actual communications and the metadata relating to a user’s requests, which have intrinsic emotional value that emotional intelligence can interpret. For example, if a user angrily shook his arms while pronouncing the sentence “Yeah, you helped me alright,” emotional intelligence would correctly disambiguate the sentence. Rather than perceiving the remark to be of a positive and satisfied nature, which the user’s words alone would suggest, it would understand that, in reality, the user is unsatisfied with his experience.

Moreover, by keeping track of a discussion’s evolution, software with emotional intelligence would gain a better understanding of the overall direction in which the dialogue was going—whether satisfying or dissatisfying to a user. Finally, emotional intelligence can also communicate a virtual assistant’s emotional status, which sets the ground for a more human-like type of interaction.

Thus, emotional intelligence can disambiguate mixed input signals, while providing an emotional dimension to the user experience by tracking the emotional status of both the user and the virtual assistant.

Logical Intelligence

The software that provides logical intelligence evaluates all available information and analyzes a user’s requests. It then continues the flow of the discussion by providing the virtual assistant’s next comment or question, or the answer to the user’s query, if it has already gathered all required information from the user.

Logical intelligence comprises the set of technologies that a virtual assistant uses to perform the necessary troubleshooting operations and manage the flow of discussion—such as rule-based engines, case-based reasoning, and bayesian or neural networks.

Knowledge Intelligence

While logical intelligence performs the core tasks of troubleshooting and managing dialogue, it often needs to rely upon data to successfully perform such tasks. It is not enough to merely analyze a user’s needs and identify the related solution. It is of utmost importance that the data a virtual assistant gathers from its knowledge bases matches the information the logical intelligence requests.

For example, if the logical intelligence has identified that a user wants to know about Polish furniture—that is, furniture manufactured in Poland—if the knowledge base were to reply with information on how to polish furniture—that is, clean it—it would have indeed failed to provide an intelligent response to the user’s query. In this case, the integration of a semantic search solution with the logical intelligence would have instead provided the correct information.

Knowledge intelligence lets a virtual assistant correctly gather data from knowledge bases. Not every interaction requires knowledge intelligence be consulted. For example, a single user input might be just a greeting, which logic intelligence alone could handle very well.

Output Intelligence

Output intelligence determines how a virtual assistant presents its response to a user’s request. It has at its disposal both the information the logical intelligence generated and that which the knowledge intelligence ultimately gathered. This information might be something the virtual assistant should say, but also might relate to documents or videos the user should view. Output intelligence also knows the emotional status of both the user and the virtual assistant.

Therefore, output intelligence integrates every bit of information and metadata that the other levels of software intelligence have computed, in order to build the best possible representation of the information. This includes graphic representations of facial expressions and gestures, voice patterns—which might reflect, for instance, the rush a user is in or the emotional status of the virtual assistant—and all of the interface design issues relating to the optimization of this representation.

Using Software Intelligence

Attentive readers will already have figured out that these five levels of software intelligence occur sequentially. The progressive elaboration on a user’s input becomes enriched through the successive steps, until the software intelligence finds the data that matches the information the user has requested and presents it to the user. All necessary levels of intelligence process every user request, to progressively identify what data the user is asking for. It is a process of input targeting, which relates to identifying and disambiguating a user’s requests, while gradually performing a process of output targeting, or computing the emotional and informational response that the software presents to satisfy a user’s request.

These five levels of software intelligence can, in my opinion, make the dream of virtual assistants a reality. Collectively, they make up the concept of composite intelligence, which comprises various software components—each gifted with some moderate degree of intelligence. Under the appropriate creative and technological conditions, this composite intelligence could provide a coherent solution that enables a degree of intelligence that might even be superior to the sum of its components. To fulfill the potential of such a solution, the individual components require a means of communication—including a common communications protocol—that can optimize and boost their overall performance.

Hopefully, my attempt to classify the software components that together support the perceived intelligence of virtual assistants contributes to the development of an understanding of the diverse professional domains the realization of such applications touches. Most people can readily understand and accept that software that can compute and rationally respond to a user’s requests is a vital part of the composition of a virtual assistant. However, it rapidly becomes apparent that we can achieve this capability only by first deeply understanding the nature of a user’s needs by

• using a combination of input technologies and emotional algorithms
• ensuring the appropriate integration of technology and data sources
• providing a rich user experience that gives users an overall feeling of confidence in virtual assistants

Ultimately, if we can achieve all of this, users will gain the necessary degree of familiarity with virtual assistants that will, in the end, open their minds to letting virtual assistants meet their needs.

Originally published on UXmatters, October 8th, 2007.

Trailer

Here is the audio trailer i did for this project :)

PhotoShakr Demo

Check out the example below, directly from a leeander Flickr set.