One More Blog

Welcome to the Blog of Eric Moritz; wannabe philathropist, geek and over-all swell guy

Your own appengine, maybe not yet

Well, I started looking at both Parallel Python and CouchDB.

CouchDB still seems to be a viable replacement for the BigTable backend to datastore. A GQL parser will have to be written to interface with Couch's views, but that's probably easy enough.

After looking at PP (Parallel Python) a little more. It's definitely not the correct solution. PP only works with a limitted python scope. You couldn't say, load all of Django inside of PP.

The way that appengine works is simple CGI, this makes every request autonomous. It doesn't need to know what machine it's running on.

One possible implementation is to distribute your source code across all nodes. With some kind of virtual hosting router and load balance system, you could make your own appengine on a small scale.

mod_fastcgi on lighttpd allows you to load balance application servers inside of lighttpd. You could distribute fastcgi servers across all 20 app servers all with a pool of min_children/20. So if you need at least 10 children you could have 1 app server per site per machine. I'm not completely sure, but I don't think mod_fastcgi load balancing provides any fault tolerance.

So the idea is to have each http request load balanced to each lighttpd server. each lighttpd server load balances fastcgi among all the nodes. Each fastcgi app server can access the couchdb cloud of all the nodes.

Posted by on April 8, 2008

4 Comments

Tags: appengine, couchdb, diy, parellel, python, tahoe

Comments

I had already been debating a CouchDB and Amazon SimpleDB abstraction layer. Google's Datastore API doesn't seem that far from some of what I had in mind, either. I don't think it would be a huge deal to create a unified library for all three. I just wish the query syntaxes were closer amongst the three to make things a little bit easier. (You could possibly throw in Facebook Data API as a fourth backend for the library...)

WorldMaker on April 9, 2008 at 2:06 a.m. (Suggest removal)

Well, if someone implemented GQL for CouchDB, I'm sure it wouldn't be a stretch to do the same for SimpleDB.

eric on April 9, 2008 at 8:07 a.m. (Suggest removal)

Amazon's SimpleDB query language appears the weakest of the bunch (not much more than basic boolean filtering with very few operators), so I'd probably start with that and it should be extremely simple to convert it to GQL and Javascript, as the basic query interface, and then provide hooks to do more complicated GQL/Javascript/FBQL queries similar to Django ORM's extras().

WorldMaker on April 9, 2008 at 2:58 p.m. (Suggest removal)

personally I see something like hbase as a better option.

ps. your password_reset url is broken
pps. can you enable openID on your site? it's deadsimple -- http://code.google.com/p/django-autho...

kryton on April 22, 2008 at 3:03 a.m. (Suggest removal)

Post a comment


Password: (Forgotten your password?)


Who are you? Log In | Sign up