A few days ago, someone asked me "What's the next big thing?" The context was a discussion of web development tools, as they exist today. At first I laughed and brushed off the question as cliché, because everyone seems to have a different answer and almost nobody ends up being correct. But it is a good question to ask oneself, and I'm noticing that more and more people seem to be searching for that "next big thing".
The standard response du jour seems to be "cloud computing". However with a term like that, you can understand why there are disagreements as to what exactly cloud computing is. I'm not so convinced. I'm not convinced that thick client computing will come back, either, but certainly the thought of distributed computing will continue to resurface--which is quite directly at odds with the concept of cloud computing (depending on how you define cloud computing, of course).
Examining today's problems
Before we can talk about what will emerge as the next big thing, we need to first look at what sucks right now:
- Concurrency. Do I even need to spell out why this sucks right now? It's easy to get wrong, and anyone who thinks that it isn't, is probably getting it wrong.
- Database technology, specifically relational databases. This area of study is a proven and well-understood area of computer science, and for simple tasks, using a database could not get any easier. Especially with the advance of object relational mappers that are starting to generate really good SQL, it's easier than ever to set up some data structures to store and retrieve data from a database. Try to start scaling this solution, as many people are, and you'll see why this technology (as it is today) doesn't hold up.
- Push. With the emergence of data whose value is highly time dilated (think Twitter, FriendFeed, etc.), systems which operate under a "pull"-only interface are growing long in the tooth. In fact, FriendFeed has a post explaining just how hard this is for them (and proposing their own protocol which hasn't seemed to get any traction).
Those next big things
Given the aforementioned problems facing developers today, is the outlook bleak? Is there no hope for the future? Of course not! As always, there are some exciting new technologies that are addressing these very problems.
- Erlang. People who hype it up will tell you that this magically solves all concurrency problems, and they are simply being overzealous. That being said, erlang encourages a style of programming that is extremely conducive to writing highly scalable applications. Not only that, but there are over 20 years worth of experience and effort that have gone into making this language rock solid. We've seen, over the past few years, an uptake in the number of applications that are written in erlang. I expect that trend to continue, and as our need for highly concurrent applications increases, that rate will increase as well. It's a good time to be an erlang programmer!
- CouchDB. This is a system which, from the ground up, was designed for scale. Built upon the aforementioned Erlang programming language, and upon the well tested and trusted OTP methodology, CouchDB takes a completely new approach to databases. It is a document-oriented database, which means that it's best suited towards those documents which can easily be written down on a piece of paper. (A business card is the prototypical example of a type of document that would be great to store in CouchDB.) It's a blog post unto itself, but the forward-looking nature of CouchDB, along with it's adherence to the fundamentals of what makes the web work so well, has gotten a lot of people interested in the project. I'd bet my career on CouchDB becoming a Very Big Deal.
- XMPP. Let's first start with the bad things about it. It's complicated, overly abstract, obtuse, and there doesn't seem to be a succinct definition of what it is. You may disagree with any or all of those points, but one thing is clear: what it does well, it does better than anything else, and that is push communication. Look for laconi.ca and twitter to be early leaders in this field, and for the technology to be simplified and clarified. If you're not subscribed to Jack Moffitt's blog, do so. Now. He has pretty much single handedly rekindled my excitement for this technology, and with a few more people like him, we'll see this technology execute on its vast potential.
- Python and Django. Of course I'm biased due to my closeness to these projects, and these are already big deals really. But I think they will continue to further distinguish themselves from the competition. Both projects have a reputation for not making important design decisions rashly and for sticking to their ideals. As long as this trend continues, I believe the number of people flocking to these technologies will continue to grow.
What do you think?
I suspect I won't meet a lot of resistance on these choices. But I hope I do! What do you think about these choices? Is there something that I'm missing? Please share your thoughts in a blog post or in the comments.
An aside
I'm going to try to post one blog post each day of the month of November, as a challenge to myself and with some of my friends. Others participating are:
Please check out their blogs and see all of the amazing content that they're already creating.
All Content
By g at 1:36 a.m. on Nov. 2, 2008
You missed Message Queuing systems like RabbitMQ - they are likely going to be as common as memcached someday soon.
Django, while a nice piece of software, doesn't feel like it belongs here. Python however has shown to be extremely versatile and will likely smartly integrate functional and concurrent implementations.
By Eric Florenzano at 1:17 a.m. on Nov. 2, 2008
You're right, Message Queueing systems are a sure-fire thing that will be huge going forward. The problem that I had with listing one, is because I don't see a clear winner. Every single one out there has some fatal flaw that makes me cringe. I have to say that I think once beanstalkd adds persistence, they'll be the clear winners in this space.
Regarding Django, I respectfully disagree with you :)
By daaku at 1:51 a.m. on Nov. 2, 2008
Of the 4 items in the technology list, if you consider ejabberd to be one of the best XMPP implementations around - then Erlang gets 3 of the 4 items. Now if we only had <a href="http://wiki.reia-lang.org/wiki/Reia_Programming_Language">python on erlang</a>.
As far as Push is concerned, I'm not sure if that's the answer. Seems like it moves the burden to the wrong player. I guess Push vs Pull is always incentive based.
By Eric Florenzano at 1:23 a.m. on Nov. 2, 2008
I hadn't realized how erlang-centric my post was. Other than the direct callout to erlang, the rest is by chance. If CouchDB were written in C, or Java, or Scala it would have made the list (assuming it kept the same properties).
Regarding Reia, however, I'm not entirely sure I agree with some of their fundamental design decisions. One of the things that I like about Erlang is not necessarily the syntax, but what it does is encourage you to code in such a way that will parallelize well. Creating a Python-like syntax threatens to remove much of that encouragement. (I'm specifically talking about their choice to do away with single assignment).
About push: I'm not convinced that it's the Right Thing Always, but for the instances where pull isn't working out right now, push technologies are the right thing to do instead of pulling more often. You could be right though, I dunno :)
By evgen at 1:28 a.m. on Nov. 2, 2008
Agreed. Reia is a cute exercise, but some of the decisions made for the sake of staying closer to python (e.g. re-binding variables) strike me as choices that will lead to problems down the road. A far more appealing option in my mind would be for someone to dust off the py-interface code (an erlang node that wraps a python interpreter) and get it back in sync with the current python versions.
By Andreas at 1:25 p.m. on Nov. 2, 2008
whatever these things will be the next big thing or not i gladly read more about them on your blogging-spree. Specially couchdb and xmpp and what prolblems they could solve to me as a django developer. good luck with the 1post-a-day!
By Dave K at 3:48 p.m. on Nov. 2, 2008
This reminds me a lot of something about 10 years ago. The truth is, most apps, don't have to scale like Twitter does, and those problems HAVE been solved. It's just the "web 2.0" crowd is largley unaware of what the solutions are and thus keep re-inventing the wheel. The sad truth is people like eBay, Amazon and the other titans of Web 1 laugh at the pewny scaling problems twitter has. What's needed isn't the next big thing, it's better programmers and people who know what the heck they're doing!
By wow powerleveling at 9:03 p.m. on Nov. 6, 2008
Triggers have some downsides too. They do their invisible work and tend to be forgotten over time.
Don't forget that your model is not normalized. score and num_votes could (and should?) be computed at runtime.
If you don't want to do that triggers or updateable views are fine solutions.
By sexy lingerie at 11:10 a.m. on June 4, 2009
Excellent points! I agree with every single one. One way that helped me alot is to hang out in the IRC channel #django. I basically hang out there as much as I can and see what people are having problems with and seeing how I would solve them. This in turn helped me run through the code base and see how the internals work
By WoW Gold at 4:47 a.m. on June 13, 2009
Thx for sharing the exciting info with us.
By plastic injection moulding at 9:30 p.m. on June 13, 2009
Jack is like that!
By Custom Dissertation at 12:23 a.m. on June 17, 2009
This in turn helped me run through the code base and see how the internals work
By Thesis Service at 12:24 a.m. on June 17, 2009
I guess Push vs Pull is always incentive based.
By Coursework Writing at 12:24 a.m. on June 17, 2009
However with a term like that, you can understand why there are disagreements as to what exactly cloud computing is.
By business grants at 11:29 p.m. on June 21, 2009
very nice and interesting article !
By jordan shoes at 1:54 a.m. on June 25, 2009
Triggers have some downsides too. They do their invisible work and tend to be forgotten over time.
By jordan shoes at 1:54 a.m. on June 25, 2009
I guess Push vs Pull is always incentive based
By ugg boots at 1:55 a.m. on June 25, 2009
This in turn helped me run through the code base and see how the internals work
By nike shoes at 1:55 a.m. on June 25, 2009
However with a term like that, you can understand why there are disagreements as to what exactly cloud computing is.
By tiffany jewellery at 1:56 a.m. on June 25, 2009
Agreed. Reia is a cute exercise, but some of the decisions made for the sake of staying closer to python (e.g. re-binding variables) strike me as choices that will lead to problems down the road. A far more appealing option in my mind would be for someone to dust off the py-interface code (an erlang node that wraps a python interpreter) and get it back in sync with the current python versions.
By Stop Dreaming Start Action at 7:04 a.m. on July 2, 2009
Thanks for sharing this great article! That is very interesting Smile I love reading and I am always searching for informative information like this
By Rusli Zainal Sang Visioner at 7:04 a.m. on July 2, 2009
hello, this is my first time i visit here. I found so many interesting in your blog especially its discussion. keep up the good work.