Bret Taylor's blog: The technology behind Tornado, FriendFeed's web server - Bret Taylor's blog
dave
· 3 months ago
This is very interesting!
I've been a big fan of the FF API, esp the realtime stuff.
I have a feeling this will plug nicely into rssCloud.
:-)
Ken Sheppardson
· 3 months ago
Nice! Thank you, Bret and crew. I'm a little curious... when you were getting started, did you consider Twisted at all? I'd be interested to hear your opinion of that framework, if you have one.
Bret Taylor
· 3 months ago
When we started, we did use Twisted. In practice, I found Twisted tedious. The deferred abstraction works, but I didn't love it in practice. Likewise, the HTTP/web support in Twisted is very chaotic (see http://twistedmatrix.com/trac/wiki/WebDevelopme... - even they acknowledge this). In general, it seems like Twisted is full of demo-quality stuff, but most of the protocols have tons of bugs.
Given all those factors, it didn't seem to provide a lot of value. Our core I/O loop is actually pretty small and simple, and I think resulted in fewer bugs than would have come up if we had used Twisted.
Thomas Hervé
· 3 months ago
> In general, it seems like Twisted is full of demo-quality stuff, but most of the protocols have tons of bugs.
It seems like a gratuitous comment. Most protocols implemented in Twisted are the most achieved implementations available in Python. Coming from a project where we can't see any single unittests, I'm a bit skeptical.
The complexity in Twisted core loop also comes from the support of multiple platforms. I can also see the ideas that were picked from the Twisted implementation in your loop code :) I'd like to see what would Tornado loop looks like if it supported OS X and Windows (in an efficient way, select doesn't really count).
Also, did you consider edge-triggered epoll? It would be a really interesting test that could make a real difference from Twisted.
dustin
· 3 months ago
I suppose it's an application-specific thing. inlineDeferreds is very easy, though. I wish some of the twisted community were able to address your issues early on.
The bigger issue is that there are places where tornado could be used today to do great good if it were built on twisted (things like buildbot really need a web framework).
Of course, I don't mean to sound like I don't appreciate the work you guys did and gave away. I was very excited to see it. I just can't use it for anything right now. :(
progrium
· 3 months ago
Yeah, I'm not a fan of the web stuff in Twisted ... however you're doing the same basic thing at the bottom... So how does Tornado compare to Twisted Web performance wise? Roughly the same?
Bret Taylor
· 3 months ago
I can't imagine there is much of a performance difference. The bottom is not that complex in my opinion. See ioloop.py and iostream.py for the details.
roder
· 3 months ago
I posted a performance test of Tornado and Twisted.web here:
Did you file any bug reports? I don't mean this as an implicit criticism -- I honestly wonder if you reported some bugs that might be relevant to me as a Twisted user or as a Twisted developer.
zooko
· 3 months ago
... and give me and others some measurement of your allegation of bugginess.
Since Twisted has extensive unit tests and a hard policy of not changing code without accompanying unit tests, there is a lot of functionality in it that I'm sure is not buggy. However, this might not be the same functionality that you were trying to use. A bug report would give us insight into that.
dustin
· 3 months ago
I've done a first pass implementation of tornado's web framework on top of twisted:
dustin, wow. I see that twisted has a "non-blocking interface to the standardized DB-API 2.0 API" http://twistedmatrix.com/projects/core/document... -- is that a good way to back tornado with a rdbms?
dustin
· 3 months ago
I've used the twisted enterprise stuff in some apps for non-blocking RDBMS action. twitterspy supports both the twisted async interface to sqlite and a couchdb interface (I use the latter). twitterspy as I run it == async couchdb + async memcached + async http streaming + async xmpp interface (and a bit more).
I also saw that someone filed an issue against tornado about running async system commands and displaying their output. If you go about three minutes into this video: http://www.vimeo.com/5998733 you'll see Matt Ingenthron demonstrating a twisted-based web server I wrote that runs system commands and streams replicates of the output (processed through a python filter) to whomever happens to be looking.
This is why Tornado is both interesting and mildly frustrating. I've got a handful of little frameworks like the above that are doing useful realtime web stuff that could all use an easy to use web framework that doesn't block. Hopefully by porting tornado to twisted, I can get some more web attention directed at these projects.
igorgue
· 3 months ago
I second that, I'd like to see twisted.web in that chart
Kenneth Reitz
· 3 months ago
OMG i can't wait to build my next web app with it. I've been a bit frustrated lately with django....
Arek
· 3 months ago
I'm saying it with utmost respect : what ?
I really like this project, but Django and this aren't really comparable and suit totally different projects. Are you using Django for something overly simple that wouldn't require most of it's functionality and are getting stuck on something ?
rrodriguer
· 3 months ago
thanks im looking for the guatemala open source community in orde to do our own miniblogging server
rrodriguer
· 3 months ago
Great news im looking to do a guatemalan server to share with the opensource community thks
traviscooper
· 3 months ago
very very cool
Cleverson Zampieri
· 3 months ago
ohh.... how many lines... ;)
gabriel
· 3 months ago
why not php?¿
liuliu
· 3 months ago
php is more about traditional web. It works for stage-based http which is perfect (memory cleanup after page sent, no garbage at all). when comes to COMET, it is really hard to do this with php in efficient way (obvious solution is "blocking" http server for long polling). Lighttpd wants to implement a "mailbox" model which suppose to be perfect for php on COMET mission, but almost two years passed...
gabriel
· 3 months ago
thanks! very nice explanation, as you may supose i am an php developer (early stage), i have done mostly zend framework develops, for a couple of weeks i had been researching about python with "django", "orange" for data mining develop. And now facebook comes with "tornado", this real-tiem framework, i thinking in learning this amazing language. Maybe starting with tornado :;). thanks again!
fuentesjr
· 3 months ago
Great work! Looking forward to this project and glad to see the crew pushing the web forward.
alecflett
· 3 months ago
Very neat. (I was gonna compare this to Twisted but I see after originally posting that that has been done)
vssaokar
· 3 months ago
This is cool! I am assuming Tornado could easily run on google app engine?
Derek Gathright
· 3 months ago
Long polling on Google App Engine won't be possible because of current limitations.
mckoss
· 3 months ago
It might be nice to port some of the authentication bits to appengine/webapp or appengine/django.
gotwood
· 3 months ago
i think i just got wood.
michaelrhanson
· 3 months ago
Awesome, Bret. Thanks for making this happen.
Paul OFlaherty
· 3 months ago
This is such great news. Thank you. Off to download and scour the documentation....
michele
· 3 months ago
Wow! it seems like I was waiting for something like this to happen... thanks!
Keep up the great work, what I like the most and I really hope you will keep while tornando evolves is the KISS approach, no orm, and (as you said) "à la carte" additional features.
Hameedullah Khan
· 3 months ago
Thank you Bret for releasing this master piece.
kumaresan
· 3 months ago
very interesting.. do this also have supporting developer tools like... eclipse plugin... etc.
darkimmortal
· 3 months ago
Nginx wins yet again.
Mark Essel
· 3 months ago
I have much to glean from these wonderful open source shares. Thanks again Bret and the super team formally known as friendfeed :D
rvijapurapu
· 3 months ago
Do you think we will be able to use this on Google Appengine?
The asynchronous features are not available in Google AppEngine, but a subset of Tornado's features do work in AppEngine. See http://www.tornadoweb.org/documentation#wsgi-an... and the included "appengine" example for a full-featured App Engine example app.
I don't get it. The more I think about it, the more I dislike it. What a colossal wasted opportunity. It would have been so much less effort to have thrown your weight behind using Twisted core and helping twisted.web, and you would have ended up with a better result to boot. It's a bit late to be all community friendly. I don't blame you for doing things your own way, it's just the scale of the missed chance that really gets to me. Why didn't you bother to show up in the Twisted community, ask a few questions, ask for help, suggest some directions, air your grievances with Twisted.web etc? I did all that, and the reaction from the Twisted folk was always superb. No, you had to go off and try to re-invent the whole friggin thing. Jeez.
gpshead
· 3 months ago
Stop being such twisted fanboys people, be thankful for the newly released code and project. Working with an existing project is academically admirable but quite often not practical. If that were all anyone ever did, nothing new and improved would ever be created. Competition is good.
I will try to use this as soon as possiable. It looks great.
Roger Lancefield
· 3 months ago
Very interesting Bret, and kudos to you and your team for opening up this code.
When you say CherryPy was running "standalone" do you mean it was tested running its built-in web server? I'm not being defensive, critical or sceptical, merely interested :)
Satya Prakash
· 3 months ago
Everywhere open source and open web. What Microsoft will do? If MS where the biggest search engine then he may have charge us for searching or shown a popup ads in between .
Eiso Kant
· 3 months ago
I wanted to thank you guys. I am going to be building a web app for which I was dreading to use PHP long-polling. I looked at Twisted.web and other Java alternatives but they all seemed too complicated to use (I know PHP and Ruby, not Java or Python).
After exploring Tornado and trying it out, I am growing incredibly fond of it, especially it's ease of use (and Python is not looking to difficult from here, lots of similarities to Ruby).
Thank you!
Eiso
elenamorgan
· 3 months ago
Selecting a suitable development environment depends a lot on the nature of the site and the project requirements. One needs to carefully thing about the technologies to be deployed as you are likely to live with it for a long time.
So I would like to recommend one of my client who provides solutions in both PHP and .Net technologies.
class LoginHandler(tornado.web.RequestHandler): def post(self): # Process login username and password self.set_secure_cookie("user_id", user["id"]) self.redirect("/home") throws the following exception ... Exception: Cannot redirect after headers have been written
All Comments
I've been a big fan of the FF API, esp the realtime stuff.
I have a feeling this will plug nicely into rssCloud.
:-)
Given all those factors, it didn't seem to provide a lot of value. Our core I/O loop is actually pretty small and simple, and I think resulted in fewer bugs than would have come up if we had used Twisted.
It seems like a gratuitous comment. Most protocols implemented in Twisted are the most achieved implementations available in Python. Coming from a project where we can't see any single unittests, I'm a bit skeptical.
The complexity in Twisted core loop also comes from the support of multiple platforms. I can also see the ideas that were picked from the Twisted implementation in your loop code :) I'd like to see what would Tornado loop looks like if it supported OS X and Windows (in an efficient way, select doesn't really count).
Also, did you consider edge-triggered epoll? It would be a really interesting test that could make a real difference from Twisted.
The bigger issue is that there are places where tornado could be used today to do great good if it were built on twisted (things like buildbot really need a web framework).
Of course, I don't mean to sound like I don't appreciate the work you guys did and gave away. I was very excited to see it. I just can't use it for anything right now. :(
http://www.apparatusproject.org/blog/2009/09/tw...
Since Twisted has extensive unit tests and a hard policy of not changing code without accompanying unit tests, there is a lot of functionality in it that I'm sure is not buggy. However, this might not be the same functionality that you were trying to use. A bug report would give us insight into that.
http://github.com/dustin/tornado
10 files changed, 92 insertions(+), 1389 deletions(-)
I also saw that someone filed an issue against tornado about running async system commands and displaying their output. If you go about three minutes into this video: http://www.vimeo.com/5998733 you'll see Matt Ingenthron demonstrating a twisted-based web server I wrote that runs system commands and streams replicates of the output (processed through a python filter) to whomever happens to be looking.
This is why Tornado is both interesting and mildly frustrating. I've got a handful of little frameworks like the above that are doing useful realtime web stuff that could all use an easy to use web framework that doesn't block. Hopefully by porting tornado to twisted, I can get some more web attention directed at these projects.
I really like this project, but Django and this aren't really comparable and suit totally different projects. Are you using Django for something overly simple that wouldn't require most of it's functionality and are getting stuck on something ?
I am assuming Tornado could easily run on google app engine?
Keep up the great work, what I like the most and I really hope you will keep while tornando evolves is the KISS approach, no orm, and (as you said) "à la carte" additional features.
If so do you think we will be able to maintain the connections? http://groups.google.com/group/google-appengine...
I don't get it. The more I think about it, the more I dislike it. What a colossal wasted opportunity. It would have been so much less effort to have thrown your weight behind using Twisted core and helping twisted.web, and you would have ended up with a better result to boot. It's a bit late to be all community friendly. I don't blame you for doing things your own way, it's just the scale of the missed chance that really gets to me. Why didn't you bother to show up in the Twisted community, ask a few questions, ask for help, suggest some directions, air your grievances with Twisted.web etc? I did all that, and the reaction from the Twisted folk was always superb. No, you had to go off and try to re-invent the whole friggin thing. Jeez.
http://www.bagoscomp.com
When you say CherryPy was running "standalone" do you mean it was tested running its built-in web server? I'm not being defensive, critical or sceptical, merely interested :)
If MS where the biggest search engine then he may have charge us for searching or shown a popup ads in between .
After exploring Tornado and trying it out, I am growing incredibly fond of it, especially it's ease of use (and Python is not looking to difficult from here, lots of similarities to Ruby).
Thank you!
Eiso
So I would like to recommend one of my client who provides solutions in both PHP and .Net technologies.
http://www.redalkemi.com
class LoginHandler(tornado.web.RequestHandler):
def post(self):
# Process login username and password
self.set_secure_cookie("user_id", user["id"])
self.redirect("/home")
throws the following exception ...
Exception: Cannot redirect after headers have been written
Am I missing something ?