[Kepler-Project] Re: Basic benchmark result

Jacques Chester jacques at chester.id.au
Sat Jun 13 02:39:11 GMT+2 2009


Guillame;

Herewith my notes.

You can instruct Lighttpd to launch as many FCGI workers
as you like, but it's a little counter-intuitive. Specify it
with the "max-procs" setting in the fastcgi.server.


I was interested by your results, and so decided to have a little
shootout of my own.

Here are my results for plain HTML, PHP and Lua.

----------------------------------------------------------------------
Plain HTML, default Lighttpd on Ubuntu VM:

ab -c 10 -n 10000 http://10.1.1.6/hello_world.html
...
Requests per second:    3163.00 [#/sec] (mean)
Time per request:       3.162 [ms] (mean)
Time per request:       0.316 [ms] (mean, across all concurrent  
requests)
Transfer rate:          1000.79 [Kbytes/sec] received

----------------------------------------------------------------------
Plain PHP with default Lighttpd settings:
ab -c 10 -n 10000 http://10.1.1.6/hello_world.php
...
Requests per second:    1431.15 [#/sec] (mean)
Time per request:       6.987 [ms] (mean)
Time per request:       0.699 [ms] (mean, across all concurrent  
requests)
Transfer rate:          373.24 [Kbytes/sec] received

----------------------------------------------------------------------
Plain PHP, with PHP_FCGI_CHILDREN increased to 16:

ab -c 10 -n 10000 http://10.1.1.6/hello_world.php
...
Requests per second:    1388.17 [#/sec] (mean)
Time per request:       7.204 [ms] (mean)
Time per request:       0.720 [ms] (mean, across all concurrent  
requests)
Transfer rate:          361.95 [Kbytes/sec] received

----------------------------------------------------------------------
PHP, default settings, with XCache opcode cache installed:
ab -c 10 -n 10000 http://10.1.1.6/hello_world.php
...
Requests per second:    1076.88 [#/sec] (mean)
Time per request:       9.286 [ms] (mean)
Time per request:       0.929 [ms] (mean, across all concurrent  
requests)
Transfer rate:          280.81 [Kbytes/sec] received

----------------------------------------------------------------------
Now the interesting stuff -- using Lua.

This involved a lot of futzing about with ubuntu. 9.04 intrepid
includes promising-looking packages for WSAPI, but they are incomplete.
Apparently 9.10 karmic will have integrated WSAPI packages which get
over the "it just works" hurdle that I am able to vault by myself.

My installation is thus luarocks-based. As has been noted on this  
mailing
list (thanks Google!), this means installing luafilesystem and rings,
both of which are required, neither of which are automatic dependencies.
I consider this a bug.

Anyhow, moving along.

----------------------------------------------------------------------
First I run the microbenchmark with the default 4 wsapi.fcgi instances:

ab -c 10 -n 10000 http://10.1.1.6/hello_world.ws
...
Requests per second:    1664.60 [#/sec] (mean)
Time per request:       6.007 [ms] (mean)
Time per request:       0.601 [ms] (mean, across all concurrent  
requests)
Transfer rate:          374.03 [Kbytes/sec] received


----------------------------------------------------------------------
And now at max-procs = 16:
ab -c 10 -n 10000 http://10.1.1.6/hello_world.ws
...
Requests per second:    1526.44 [#/sec] (mean)
Time per request:       6.551 [ms] (mean)
Time per request:       0.655 [ms] (mean, across all concurrent  
requests)
Transfer rate:          342.92 [Kbytes/sec] received


----------------------------------------------------------------------
And now at max-procs = 32:
ab -c 10 -n 10000 http://10.1.1.6/hello_world.ws
...
Requests per second:    1487.43 [#/sec] (mean)
Time per request:       6.723 [ms] (mean)
Time per request:       0.672 [ms] (mean, across all concurrent  
requests)
Transfer rate:          334.26 [Kbytes/sec] received

In this latter test I looked at load. It never breaks past 0.07.

----------------------------------------------------------------------
Now max-procs = 32 with higher concurrency:
ab -c 50 -n 10000 http://10.1.1.6/hello_world.ws

Requests per second:    1434.58 [#/sec] (mean)
Time per request:       34.854 [ms] (mean)
Time per request:       0.697 [ms] (mean, across all concurrent  
requests)
Transfer rate:          322.32 [Kbytes/sec] received

Not a huge improvement, but then, not much degradation either.

----------------------------------------------------------------------
Last of all, something a bit different. The lighttpd "mod_magnet" module
allows you to embed lua scripts directly into the web server. Let's see
how that performs:

ab -c 10 -n 10000 http://10.1.1.6/hello_world
...
Requests per second:    3005.95 [#/sec] (mean)
Time per request:       3.327 [ms] (mean)
Time per request:       0.333 [ms] (mean, across all concurrent  
requests)
Transfer rate:          675.37 [Kbytes/sec] received

Notice that this is twice as fast as any other FastCGI configuration and
only slightly slower than raw HTML.

----------------------------------------------------------------------
The same again, with much higher concurrency:
ab -c 250 -n 10000 http://10.1.1.6/hello_world
...
Requests per second:    2653.38 [#/sec] (mean)
Time per request:       94.220 [ms] (mean)
Time per request:       0.377 [ms] (mean, across all concurrent  
requests)
Transfer rate:          596.03 [Kbytes/sec] received

Still much better than FCGI.


There is a big "gotcha" for this, however. Lighttpd is single-threaded  
by
design, and the pass through a lua script is a single-threaded pass. In
scenarios where you have a long-running lua script, this will be worse  
than
the fcgi scenario.

----------------------------------------------------------------------
The busy-wait script, run on FCGI with 16 wsapi instances:

ab -c 10 -n 10000 http://10.1.1.6/busy_world.ws
...
Requests per second:    954.08 [#/sec] (mean)
Time per request:       10.481 [ms] (mean)
Time per request:       1.048 [ms] (mean, across all concurrent  
requests)
Transfer rate:          214.32 [Kbytes/sec] received

----------------------------------------------------------------------
The busy-wait script, run inside lighttpd with mod_magnet:

ab -c 10 -n 1000 http://10.1.1.6/busy_world
Requests per second:    5.20 [#/sec] (mean)
Time per request:       1922.162 [ms] (mean)
Time per request:       192.216 [ms] (mean, across all concurrent  
requests)
Transfer rate:          1.17 [Kbytes/sec] received


The results are catastrophically worse for this case -- it was so slow  
that
I killed the original test and ran it again with only 1000 requests  
(10x less).
This is because every request is dealt with by itself. The normal  
model allows
lighty to push a job onto a fastcgi worker, let them run while it  
services the
next HTTP request, then send the reply when it's ready. mod_magnet  
breaks
this model. You have been warned.


----------------------------------------------------------------------
My conclusions:

* Lua is faster than PHP in a FastCGI environment
* Lua probably scales better -- compare the multi-worker examples.
* Embedding lua in lighttpd using mod_magnet can be incredibly fast ...
* ... but does not scale for long-running scripts.
* I should be studying for my exams, instead of spending a morning on
this stuff.


Cheers,

JC







More information about the Kepler-Project mailing list