Yesterday Donovan Preston released new versions of both eventlet and Spawning. What are those, you ask? Eventlet is a networking library written using coroutines instead of normal subroutes, which makes writing networked non-blocking IO applications much much simpler. Spawning is a WSGI server, written using eventlet, which supports all of the things you'd expect of a good WSGI server: multiple processes, multiple threads, etc.
Considering that I sit next to Donovan at work all day, I've overheard him extolling the numerous advantages to using a server such as Spawning--the most obvious of which is completely graceful code reloading. Donovan has given a presentation explaining how all of this works, the slides of which probably explain it better than I could. When he told me that he'd added the ability to easily run Django apps with Spawning, I decided to check it out.
First, I installed spawning:
sudo easy_install Spawning
And away setuptools went and installed all of the prerequisites and the package itself. (I have had problems with this in the past, but grabbing greenlet, eventlet, simplejson, PasteDeploy, and Spawning and installing them individually does the trick).
The next thing to do is go to the directory which holds your settings.py, or at least make setup.py available on the Python path. I tend to find it easier to just go to the directory. Then type the following:
spawn --factory=spawning.django_factory.config_factory settings --port 9090 -s 4 -t 100
This starts up a Spawning server with 4 processes and 100 threads. I chose those numbers almost completely arbitrarily. (Well, that's not entirely true, my Apache setup previously had 4 processes and 100 requests per child. I know that requests per child doesn't map at all to threads, but that's where I got the number.) The next thing to do is visit your site, but instead of visiting the normal port 80 or 8000, visit port 9090. If you're running it on your own box, that should be http://127.0.0.1:9090/.
For me, it worked like a charm. It felt like my server was responding faster than ever, but at that point it was just a feeling. To get some quantitative analysis, I ran apachebench with 20 concurrent requests for a total of 10000 requests. On my Apache + mod_wsgi setup, I got 235.65 requests per second. That was really good, I thought! However, with the Spawning setup, I got 347.20 requests per second. I would need to test this much more in-depth if I were a statistician, but it's good enough for me as it did confirm my qualitative analysis.
If you're viewing this on my website directly, then you've already used Spawning, as I've switched over this blog to use the new server. Let me know what you think! Has my site slowed to a crawl? Is it going faster than ever (because I know everyone remembers the speed at which eflorenzano.com loads)?
In all, it was an extremely easy upgrade. I would recommend that everyone who has an interest in these types of things at least try it--especially if you're looking into other pure-python WSGI servers like CherryPy.
UPDATE: If you were having troubles reaching the site before, it's because I was having problems with my database due to another app on the same server, not due to anything that Spawning was doing wrong.
All Content
So can this be a replacement for apache/mod_wsgi? Can I just proxy django requests from nginx straight through to the spawning server?
Eric, great article, going to try that out now.
Although I think there is a typo in "spawn -factory=.." which needs to be "spawn --factory=.."
And don't you mean "The next thing to do is go to the directory which holds your **settings.py**.." instead?
Good point on both counts! Fixed now. I must have been tired when I wrote this :)
This sounds very very interesting, but for a real test, try bumping the number of cocurrent requests to something fun like 500. A number like that should make your OS grind into a slow and painfull death (assuming you are using mpm_prefork and not running with an external wsgi process group).
Also, you really need to either perform a "warmup" of apache to force it into spawning it's offspring, once the processes are all started in apache (again, assuming you are using mpm_prefork) you should get almost the same speed.
The main benifit of not using os threads is not the speed of which your requests finish, but the scalability!
It actually goes beyond warming up of Apache, but I believe you more or less have identified where the results may be a bit suspect.
As far as I can tell with Spawning is that because it knows it is only hosting the one application, it can preload the application when it first starts all the processes. On the other hand Apache/mod_wsgi by default uses lazy loading and the application code is only loaded when the first request hits a process. You would therefore see a delay for initial requests and with large frameworks like Django, that can be noticeable.
This would be compounded by fact that Apache may also dynamically spawn additional processes if number of concurrent requests greater than initial number of processes.
So, I would suspect that these results for Apache/mod_wsgi may be counting application startup cost which makes them misleading.
With Macports (on leopard) I had to symlink spawn to /opt/local/bin myself, which isn't very hard:
cd /opt/local/bin;
sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.5/bin/spawn;
Then OS X users can try this out :-)
What was the URL you were requesting against actually doing? For example, Django site home page, or something else?
MacOS X often gives quite divergent results to other platforms for some Python network based applications, but for similar type setup as Apache which comes with Leopard, that is, 100 processes with single thread in each, best I could manage with a hello world application for 'ab -c 20' was about 1250 requests/sec for Spawning and yet for mod_wsgi it was around 4000 requests/sec. Spawning produced improved results when threads were introduced, that is 10 processes with 10 threads each, resulting in 1500 requests/sec, but still quite short of mod_wsgi.
Maybe you can do some more tests using just a hello world WSGI application as that should give a better indication of relative raw performance as it takes database and rendering systems out of the equation.
Really great news! It much easy to deploy!
Have you tried out yield ... http://yield.sf.net/ ? It has a WSGI interface but the rest is written in C++. They claim they outperform anybody else. Would be interesting to compare...
Eric,
For me, your site is loading nice and fast - however the same cannot be said for all of the third party sites you call out to.
Al.
Eric, thanks for the link, it appeared really when I've been interesting in software like Spawning. :)
I have found some time ago thing named fapws2 and two days ago my friend found another thing named cogen. They both are wsgi servers, and I decided to have small test between all servers I know (test is not absolutely comprehensive and was made only for my own needs).
So there are some results: http://dumpz.org/1789/
I've written small article about that, but sadly it is in russian: http://piranha.org.ua/blog/2008/07/31/wsgi-servers-short/
I haven't written anything that couldn't be seen on previous link, but if there are some interest, I can translate it to English.
You can't use as little as 500 requests when doing a benchmark as what you get can be greatly affected by operating system performance in making processes active and any server/application startup costs. The way the referenced benchmarks have been done are therefore probably not a good indicator of relative performance.
For Apache/mod_wsgi the slowest configuration was also chosen and for some of the results for tested systems show all of the requests failing which means the application wasn't even being tested properly.
> You can't use as little as 500 requests when doing a benchmark as what you get can be greatly affected by operating system performance in making processes active and any server/application startup costs.
There was few ab runs for every server setup before doing each control test (and btw there was few runs of ab -n 500), so there is no overhead from system. By the way, I was curious and used to run `ab -c 200 -n 2000` for few of setups (I'm not sure why I've chosen -c 50 -n 500, maybe I need to re-do testing?), and results was almost the same (relatively, of course).
> For Apache/mod_wsgi the slowest configuration was also chosen
Huh... How to speed up? I'm curious to see how mod_wsgi will win. :)
> for some of the results for tested systems show all of the requests failing which means the application wasn't even being tested properly.
Failing? Where? I see 'Failed requests: 0' for all of results...
In respect of the errors, all I can say is I must have been drinking too much cheap vodka, otherwise I blame Google, as I clicked through to the ab output from Google translation of original Russian article and so Google screwed up formatting.
... more to come, since only 1000 characters allowed per post.
Anyway, I still believe using only 500 requests in a benchmark is not reliable. Personally I try never to use anything less than 10000, at least when trying to gauge raw performance. Using a high number of concurrent requests can also be problematic depending on the web server and how it is configured.
For example with Apache, if the configuration is prefork and the idle number of processes is a lot less than 50 and in between a prior test and a new test the Apache maintenance tasks has kicked in and shutdown any extra processes it may have created for prior test, then it would need to go and start extra process again to satisfy the transient demand being created. Since it is a new process the application has to be loaded again.
... more again
Graham,
Simply put: you are way smarter than I am, so I'm going to basically take your word for it that mod_wsgi is faster :)
The page that I tested was the index page of this site, which gets loaded from Memcached. It was already loaded into Apache. The only reason that I can see for Spawning to have been faster in my tests is because it actually has a non-negligable amount of data to respond with, leveraging its nonblocking IO/coroutine reactor.
Like I said, I'm no statistician and I only did a VERY rudimentary test. The main point was not to put down mod_wsgi as I really like it (and have put my money where my mouth is--donating through micropledge), but instead to point out another interesting server that people might not know about.
The point if anything in me responding was trying to understand how the test was conducted. I have no problem if something is truly faster than mod_wsgi, just means I'll try and make mod_wsgi faster if possible, and if not, make it more attractive to people in other ways as an overall solution.
Also, even if one underlying hosting mechanism is faster than other, normally when one loads up a large application, the difference as far as how each performs is usually negligible in the end. This is because the hosting mechanism isn't normally where the bottleneck is. This is why I don't understand why the results you were getting were markedly different and why I was speculating whether it was somehow counting time taken to load application for Apache/mod_wsgi.
Consider dropping over to mod_wsgi list on Google groups, as have thread there with some analysis about Spawning going already which might be of interest.
If publishing benchmarks it is important to document full configuration. If this isn't provided then one can't duplicate it nor be sure how it was set up. One also can't rely on one test only, you would need to do it across a range of concurrency values as different servers behave differently for different levels of concurrency.
Doing a test of single concurrency with a hello world application is also important as that gives best indication of underlying performance. Looking at your comparison you are not using a hello world application but some real world application where requests can take up to 2 seconds each. The actual work of the underlying web server in respect of an individual request would be in the order of tens of milliseconds which is a very small percentage of your overall request time. As a result you aren't even testing the underlying web server, you are really testing the application, its rendering system and any database access.
You're simply right, especially talking about application. I'm going to repeat my post (maybe in English), using this application: http://my.piranha.org.ua/perftest.tar.bz2
Using this application and nginx + fapws2 I've got 3900 req/s (this is `ab -c 100 -n 10000`).
Using mod_wsgi I've got 3100 req/s with next config:
WSGIDaemonProcess piranha user=piranha group=www-data threads=4 processes=2
They both are way better than any Python server (fapws2 is primarily C module). Maybe I shouldn't use daemon mode for getting better speed?
I've got 3500 req/s after disabling daemon mode. But this means that every apache child will get python interpreter inside and that I can host only 1 web site in apache (or I'll get even bigger childs).
How are you serving static content? nginx?
Could you perhaps describe the setup in a little more detail(maybe post the settingsfile too?), I would really like to duplicate it.
Martin
yes, that would be very helpful!
In Ubuntu make sure to install setuptools and libssl-dev before you can try the steps in the blog post:
sudo apt-get install python-setuptools
sudo apt-get install libssl-dev
Regards,
Antonio
Lima-Peru
Cool stuff. Thanks a lot for your quick tutorial!
I tried it with a few different thread settings, and with high numbers (>50) Python kept crashing really hard on my Mac (not even an exception, just the plain Python crash).
I didn't do too much testing, but 5 processes / 15 threads performed quite well.
Maybe it's just me, but my apache/mod_python setup consistently outperforms spawning. "ab -c 20 -n 10000" was my test and apache/mod_python beat spawning in every pairing of processes/threads.
performance is spawning red holt is the games copy world i tried it with a few different thread setting, and with higt numbers (>250) python kept crashing really hard on my microsoft windows
In my test Spawning is much slower:
http://pastebin.com/f30eaf404
It feels like quite helpful, but if it can be used on windows platform?
I also did some tests (on debian lenny) and spawning was _a lot_ (3 or 4 times) slower than mod_wsgi....
If you set it to use one process it works quite nicely as a replacement for django's built-in test server. (I found having more than one process caused it to have problems consistantly picking up source changes).
Teste
test²
Although Spawning + Django is working absolutely fine on my desktop PC running Ubuntu, I did notice that CPU usage is upto 100% constantly on both cores, even if the Django project is idle and not serving anything. It doesn't seem to affect my machine performance at all thought, because it took me a while to even realize this.
Just set this up and Munin is showing 4GB of RAM committed...that doesn't sound very good.
Why do they use greenlets instead of Python 2.5 generators?