Yesterday Donovan Preston released new versions of both eventlet and Spawning. What are those, you ask? Eventlet is a networking library written using coroutines instead of normal subroutes, which makes writing networked non-blocking IO applications much much simpler. Spawning is a WSGI server, written using eventlet, which supports all of the things you'd expect of a good WSGI server: multiple processes, multiple threads, etc.
Considering that I sit next to Donovan at work all day, I've overheard him extolling the numerous advantages to using a server such as Spawning--the most obvious of which is completely graceful code reloading. Donovan has given a presentation explaining how all of this works, the slides of which probably explain it better than I could. When he told me that he'd added the ability to easily run Django apps with Spawning, I decided to check it out.
First, I installed spawning:
sudo easy_install Spawning
And away setuptools went and installed all of the prerequisites and the package itself. (I have had problems with this in the past, but grabbing greenlet, eventlet, simplejson, PasteDeploy, and Spawning and installing them individually does the trick).
The next thing to do is go to the directory which holds your settings.py, or at least make setup.py available on the Python path. I tend to find it easier to just go to the directory. Then type the following:
spawn --factory=spawning.django_factory.config_factory settings --port 9090 -s 4 -t 100
This starts up a Spawning server with 4 processes and 100 threads. I chose those numbers almost completely arbitrarily. (Well, that's not entirely true, my Apache setup previously had 4 processes and 100 requests per child. I know that requests per child doesn't map at all to threads, but that's where I got the number.) The next thing to do is visit your site, but instead of visiting the normal port 80 or 8000, visit port 9090. If you're running it on your own box, that should be http://127.0.0.1:9090/.
For me, it worked like a charm. It felt like my server was responding faster than ever, but at that point it was just a feeling. To get some quantitative analysis, I ran apachebench with 20 concurrent requests for a total of 10000 requests. On my Apache + mod_wsgi setup, I got 235.65 requests per second. That was really good, I thought! However, with the Spawning setup, I got 347.20 requests per second. I would need to test this much more in-depth if I were a statistician, but it's good enough for me as it did confirm my qualitative analysis.
If you're viewing this on my website directly, then you've already used Spawning, as I've switched over this blog to use the new server. Let me know what you think! Has my site slowed to a crawl? Is it going faster than ever (because I know everyone remembers the speed at which eflorenzano.com loads)?
In all, it was an extremely easy upgrade. I would recommend that everyone who has an interest in these types of things at least try it--especially if you're looking into other pure-python WSGI servers like CherryPy.
UPDATE: If you were having troubles reaching the site before, it's because I was having problems with my database due to another app on the same server, not due to anything that Spawning was doing wrong.
All Content
By nuggien at 3:17 a.m. on July 31, 2008
So can this be a replacement for apache/mod_wsgi? Can I just proxy django requests from nginx straight through to the spawning server?
By Jannis Leidel at 6 a.m. on July 31, 2008
Eric, great article, going to try that out now.
Although I think there is a typo in "spawn -factory=.." which needs to be "spawn --factory=.."
And don't you mean "The next thing to do is go to the directory which holds your **settings.py**.." instead?
By Eric Florenzano at 10:55 a.m. on July 31, 2008
Good point on both counts! Fixed now. I must have been tired when I wrote this :)
By Mads Sülau Jørgensen at 6:28 a.m. on July 31, 2008
This sounds very very interesting, but for a real test, try bumping the number of cocurrent requests to something fun like 500. A number like that should make your OS grind into a slow and painfull death (assuming you are using mpm_prefork and not running with an external wsgi process group).
Also, you really need to either perform a "warmup" of apache to force it into spawning it's offspring, once the processes are all started in apache (again, assuming you are using mpm_prefork) you should get almost the same speed.
The main benifit of not using os threads is not the speed of which your requests finish, but the scalability!
By Graham Dumpleton at 6:21 p.m. on July 31, 2008
It actually goes beyond warming up of Apache, but I believe you more or less have identified where the results may be a bit suspect.
As far as I can tell with Spawning is that because it knows it is only hosting the one application, it can preload the application when it first starts all the processes. On the other hand Apache/mod_wsgi by default uses lazy loading and the application code is only loaded when the first request hits a process. You would therefore see a delay for initial requests and with large frameworks like Django, that can be noticeable.
This would be compounded by fact that Apache may also dynamically spawn additional processes if number of concurrent requests greater than initial number of processes.
So, I would suspect that these results for Apache/mod_wsgi may be counting application startup cost which makes them misleading.
By Oscar at 6:40 a.m. on July 31, 2008
With Macports (on leopard) I had to symlink spawn to /opt/local/bin myself, which isn't very hard:
cd /opt/local/bin;
sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.5/bin/spawn;
Then OS X users can try this out :-)
By Graham Dumpleton at 7:24 a.m. on July 31, 2008
What was the URL you were requesting against actually doing? For example, Django site home page, or something else?
MacOS X often gives quite divergent results to other platforms for some Python network based applications, but for similar type setup as Apache which comes with Leopard, that is, 100 processes with single thread in each, best I could manage with a hello world application for 'ab -c 20' was about 1250 requests/sec for Spawning and yet for mod_wsgi it was around 4000 requests/sec. Spawning produced improved results when threads were introduced, that is 10 processes with 10 threads each, resulting in 1500 requests/sec, but still quite short of mod_wsgi.
Maybe you can do some more tests using just a hello world WSGI application as that should give a better indication of relative raw performance as it takes database and rendering systems out of the equation.
By ilya at 7:25 a.m. on July 31, 2008
Really great news! It much easy to deploy!
By donkton at 7:56 a.m. on July 31, 2008
Have you tried out yield ... http://yield.sf.net/ ? It has a WSGI interface but the rest is written in C++. They claim they outperform anybody else. Would be interesting to compare...
By Al at 7:58 a.m. on July 31, 2008
Eric,
For me, your site is loading nice and fast - however the same cannot be said for all of the third party sites you call out to.
Al.
By Alexander Solovyov at 8:08 a.m. on July 31, 2008
Eric, thanks for the link, it appeared really when I've been interesting in software like Spawning. :)
I have found some time ago thing named fapws2 and two days ago my friend found another thing named cogen. They both are wsgi servers, and I decided to have small test between all servers I know (test is not absolutely comprehensive and was made only for my own needs).
So there are some results: http://dumpz.org/1789/
I've written small article about that, but sadly it is in russian: http://piranha.org.ua/blog/2008/07/31/wsgi-servers-short/
I haven't written anything that couldn't be seen on previous link, but if there are some interest, I can translate it to English.
By Graham Dumpleton at 6:28 p.m. on July 31, 2008
You can't use as little as 500 requests when doing a benchmark as what you get can be greatly affected by operating system performance in making processes active and any server/application startup costs. The way the referenced benchmarks have been done are therefore probably not a good indicator of relative performance.
For Apache/mod_wsgi the slowest configuration was also chosen and for some of the results for tested systems show all of the requests failing which means the application wasn't even being tested properly.
By Alexander Solovyov at 4:37 a.m. on Aug. 1, 2008
> You can't use as little as 500 requests when doing a benchmark as what you get can be greatly affected by operating system performance in making processes active and any server/application startup costs.
There was few ab runs for every server setup before doing each control test (and btw there was few runs of ab -n 500), so there is no overhead from system. By the way, I was curious and used to run `ab -c 200 -n 2000` for few of setups (I'm not sure why I've chosen -c 50 -n 500, maybe I need to re-do testing?), and results was almost the same (relatively, of course).
> For Apache/mod_wsgi the slowest configuration was also chosen
Huh... How to speed up? I'm curious to see how mod_wsgi will win. :)
> for some of the results for tested systems show all of the requests failing which means the application wasn't even being tested properly.
Failing? Where? I see 'Failed requests: 0' for all of results...
By Graham Dumpleton at 2:13 a.m. on Aug. 2, 2008
In respect of the errors, all I can say is I must have been drinking too much cheap vodka, otherwise I blame Google, as I clicked through to the ab output from Google translation of original Russian article and so Google screwed up formatting.
... more to come, since only 1000 characters allowed per post.
By Graham Dumpleton at 2:14 a.m. on Aug. 2, 2008
Anyway, I still believe using only 500 requests in a benchmark is not reliable. Personally I try never to use anything less than 10000, at least when trying to gauge raw performance. Using a high number of concurrent requests can also be problematic depending on the web server and how it is configured.
For example with Apache, if the configuration is prefork and the idle number of processes is a lot less than 50 and in between a prior test and a new test the Apache maintenance tasks has kicked in and shutdown any extra processes it may have created for prior test, then it would need to go and start extra process again to satisfy the transient demand being created. Since it is a new process the application has to be loaded again.
... more again
By Eric Florenzano at 2:25 a.m. on Aug. 2, 2008
Graham,
Simply put: you are way smarter than I am, so I'm going to basically take your word for it that mod_wsgi is faster :)
The page that I tested was the index page of this site, which gets loaded from Memcached. It was already loaded into Apache. The only reason that I can see for Spawning to have been faster in my tests is because it actually has a non-negligable amount of data to respond with, leveraging its nonblocking IO/coroutine reactor.
Like I said, I'm no statistician and I only did a VERY rudimentary test. The main point was not to put down mod_wsgi as I really like it (and have put my money where my mouth is--donating through micropledge), but instead to point out another interesting server that people might not know about.
By Graham Dumpleton at 4:04 a.m. on Aug. 2, 2008
The point if anything in me responding was trying to understand how the test was conducted. I have no problem if something is truly faster than mod_wsgi, just means I'll try and make mod_wsgi faster if possible, and if not, make it more attractive to people in other ways as an overall solution.
Also, even if one underlying hosting mechanism is faster than other, normally when one loads up a large application, the difference as far as how each performs is usually negligible in the end. This is because the hosting mechanism isn't normally where the bottleneck is. This is why I don't understand why the results you were getting were markedly different and why I was speculating whether it was somehow counting time taken to load application for Apache/mod_wsgi.
Consider dropping over to mod_wsgi list on Google groups, as have thread there with some analysis about Spawning going already which might be of interest.
By Graham Dumpleton at 2:20 a.m. on Aug. 2, 2008
If publishing benchmarks it is important to document full configuration. If this isn't provided then one can't duplicate it nor be sure how it was set up. One also can't rely on one test only, you would need to do it across a range of concurrency values as different servers behave differently for different levels of concurrency.
Doing a test of single concurrency with a hello world application is also important as that gives best indication of underlying performance. Looking at your comparison you are not using a hello world application but some real world application where requests can take up to 2 seconds each. The actual work of the underlying web server in respect of an individual request would be in the order of tens of milliseconds which is a very small percentage of your overall request time. As a result you aren't even testing the underlying web server, you are really testing the application, its rendering system and any database access.
By Alexander Solovyov at 4:10 p.m. on Aug. 5, 2008
You're simply right, especially talking about application. I'm going to repeat my post (maybe in English), using this application: http://my.piranha.org.ua/perftest.tar.bz2
Using this application and nginx + fapws2 I've got 3900 req/s (this is `ab -c 100 -n 10000`).
Using mod_wsgi I've got 3100 req/s with next config:
WSGIDaemonProcess piranha user=piranha group=www-data threads=4 processes=2
They both are way better than any Python server (fapws2 is primarily C module). Maybe I shouldn't use daemon mode for getting better speed?
By Alexander Solovyov at 4:17 p.m. on Aug. 5, 2008
I've got 3500 req/s after disabling daemon mode. But this means that every apache child will get python interpreter inside and that I can host only 1 web site in apache (or I'll get even bigger childs).
By Martin at 8:10 a.m. on July 31, 2008
How are you serving static content? nginx?
Could you perhaps describe the setup in a little more detail(maybe post the settingsfile too?), I would really like to duplicate it.
Martin
By nuggien at 11:14 a.m. on July 31, 2008
yes, that would be very helpful!
By Antonio Ognio at 11:57 a.m. on July 31, 2008
In Ubuntu make sure to install setuptools and libssl-dev before you can try the steps in the blog post:
sudo apt-get install python-setuptools
sudo apt-get install libssl-dev
Regards,
Antonio
Lima-Peru
By Jan Oberst at 11:58 a.m. on July 31, 2008
Cool stuff. Thanks a lot for your quick tutorial!
I tried it with a few different thread settings, and with high numbers (>50) Python kept crashing really hard on my Mac (not even an exception, just the plain Python crash).
I didn't do too much testing, but 5 processes / 15 threads performed quite well.
By nuggien at 4:14 p.m. on July 31, 2008
Maybe it's just me, but my apache/mod_python setup consistently outperforms spawning. "ab -c 20 -n 10000" was my test and apache/mod_python beat spawning in every pairing of processes/threads.
By gotor at 7:20 p.m. on July 31, 2008
performance is spawning red holt is the games copy world i tried it with a few different thread setting, and with higt numbers (>250) python kept crashing really hard on my microsoft windows
By Clodoaldo at 9:12 p.m. on July 31, 2008
In my test Spawning is much slower:
http://pastebin.com/f30eaf404
By sean at 10:16 p.m. on July 31, 2008
It feels like quite helpful, but if it can be used on windows platform?
By wilk at 4:19 a.m. on Aug. 1, 2008
I also did some tests (on debian lenny) and spawning was _a lot_ (3 or 4 times) slower than mod_wsgi....
By Andrew Ingram at 6:30 a.m. on Aug. 1, 2008
If you set it to use one process it works quite nicely as a replacement for django's built-in test server. (I found having more than one process caused it to have problems consistantly picking up source changes).
By Teste at 7:51 p.m. on Aug. 3, 2008
Teste
By Test² at 11:48 a.m. on Aug. 26, 2008
test²
By Rob van der Linde at 2:12 a.m. on Aug. 7, 2008
Although Spawning + Django is working absolutely fine on my desktop PC running Ubuntu, I did notice that CPU usage is upto 100% constantly on both cores, even if the Django project is idle and not serving anything. It doesn't seem to affect my machine performance at all thought, because it took me a while to even realize this.
By Chris at 9:34 p.m. on Aug. 18, 2008
Just set this up and Munin is showing 4GB of RAM committed...that doesn't sound very good.
By web design company at 2:34 p.m. on Aug. 22, 2008
Why do they use greenlets instead of Python 2.5 generators?
By Валерий Дасаев at 6:18 p.m. on April 10, 2009
Я тоже также думал, пока не прочитал пару аналогичных статей по этой же теме.
By wholesale jewelry at 12:30 a.m. on May 4, 2009
Good site,i like ur blog,i will visit it often,thanks.
By ladies golf gloves at 10:33 a.m. on May 16, 2009
Thanks for the help; a friend of mine has been trying to figure this out for weeks!
By 925 silver jewelry at 2:18 a.m. on May 20, 2009
Good site,i like ur blog,i will visit it often,thanks.
By SC2 at 7:26 p.m. on May 27, 2009
performance is spawning red holt is the games copy world i tried it with a few different thread setting, and with higt numbers (>250) python kept crashing really hard on my microsoft windows
By club penguin at 9:24 p.m. on May 31, 2009
I still believe using only 500 requests in a benchmark is not reliable. Personally I try never to use anything less than 10000, at least when trying to gauge raw performance. Using a high number of concurrent requests can also be problematic depending on the web server and how it is configured.
By jewelry wholesale at 3:21 a.m. on June 5, 2009
jewelry
By jiames at 1:11 p.m. on June 8, 2009
Thanks for this, we will find more from this.
By Online High Schools at 1:34 a.m. on June 10, 2009
i will visit it often,thanks.
By Accredited High School Diploma at 1:34 a.m. on June 10, 2009
Good site,i like ur blog
By Online High School Diploma at 1:35 a.m. on June 10, 2009
Thanks for the help
By custom conservatories at 3:15 a.m. on June 10, 2009
"Eventlet" is very easy to use my friend told me that.
By Cheap WoW Gold at 4:48 a.m. on June 13, 2009
You are so kind, you topic is so meaningful for us.
By injectuon molding at 9:21 p.m. on June 13, 2009
thanks for your posts, nice!
By jewelry at 9:50 p.m. on June 15, 2009
jewelry wholesale
By jewelry wholesale at 9:53 p.m. on June 15, 2009
jewelry wholesale
By jewelry at 9:54 p.m. on June 15, 2009
wholesale jewelry
By hitloop at 9:48 a.m. on June 16, 2009
This sounds very interesting, but for a real test, try to reach the number of requests for cocurrent something fun, like 500. Some, like the operating system to grind to a slow and painful death
By Online university credit transfer at 12:19 a.m. on June 17, 2009
It doesn't seem to affect my machine performance at all thought, because it took me a while to even realize this.
By Accredited Online university at 12:20 a.m. on June 17, 2009
Nice & Great Site.
By Writing Legal Document at 7:14 a.m. on June 19, 2009
Nice work you have done, I like it.
By Lighting Fixtures at 4:50 a.m. on June 20, 2009
It is nice and a great work out by you guys.really feel interesting.
By sexy Lingerie at 8 p.m. on June 22, 2009
As the net moves to IPv6 I think this will become more viable and I can see this really taking off. While there are a number of things that need to be worked on. If we start now, maybe by the time edge to edge connectivity is a reality once again, this will be mature enough.
By free games at 2:54 a.m. on June 23, 2009
It is really interesting.It doesn't seem to affect my machine performance at all thought, because it took me a while to even realize this.Thanks for the post.
By Dissertation Advice at 8:11 a.m. on June 24, 2009
That's great of u..well done..nice workk
By Essay Writing Service at 8:15 a.m. on June 24, 2009
Nice post...Gooooood work!!!!!!!!!!thanks for the share..
By jordan shoes at 2:15 a.m. on June 25, 2009
Nice work you have done, I like it.
By jordan shoes at 2:15 a.m. on June 25, 2009
wholesale jewelry
By ugg boots at 2:16 a.m. on June 25, 2009
jewelry wholesale
By nike shoes at 2:17 a.m. on June 25, 2009
It is nice and a great work out by you guys.really feel interesting
By tiffany jewellery at 2:17 a.m. on June 25, 2009
That's great of u..well done..nice workk
By Fat loss program at 1:49 a.m. on June 29, 2009
Well done..nice work done.,.If we start now, maybe by the time edge to edge connectivity is a reality once again, this will be mature enough.
By weight loss at 1:51 a.m. on June 29, 2009
Good post...thanks for the info given...I still believe using only 500 requests in a benchmark is not reliable. Personally I try never to use anything less than 10000, at least when trying to gauge raw performance.
By Carleton at 5:47 a.m. on July 2, 2009
Hello. The first step to getting the things you want out of life is this: Decide what you want.
I am from Montenegro and too poorly know English, give true I wrote the following sentence: "Fleas primarily are annoying biting pests."
Thank :( Carleton.
By Stop Dreaming Start Action at 6:59 a.m. on July 2, 2009
accurate, concise, bold! Are you me. thank you and wish a nice day
By Rusli Zainal Sang Visioner at 7 a.m. on July 2, 2009
hello, this is my first time i visit here. I found so many interesting in your blog especially its discussion. keep up the good work.
By glasgow cosmetic dentistry at 7:50 a.m. on July 2, 2009
Effective way of sharing useful stuff and interesting information.