February 17th, 2008 A 'problem' with the Mongrel/Rails platform A request comes in to the web server, and if it's dynamic, falls down to a waiting mongrel process. Mongrel calls Rails which wraps a big lock around most of the request/response cycle so the mongrel thats serving the request has to wait for a response from Rails to unlock and start serving other requests. This is the Blocked Thread stability anti-pattern that Michael Nygard talks about in Release It!. The problem is that Nginx or Apache doesn't know that a mongrel is blocked and keeps blindly sending requests. Thats means that even a single blocked mongrel will result in some slow responses for requests that come in to that same mongrel. Cut the Request Fat And one way to do that is to use the "Skinny Request, Fat Backend" principle. What that means in practical terms is pretty simple: Do as little as possible inside of the Request/Response cycle.
In the months prior to leaving Heavy, I led an exciting project to build a hosting platform for our online products on top of Amazon’s Elastic Compute Cloud (EC2). We eventually launched our newest product at Heavy using EC2 as the primary hosting platform. I’ve been following a lot of what other people have been doing with EC2 for data processing and handling big encoding or rendering jobs. We set out to build a fairly standard LAMP hosting infrastructure where we could easily and quickly add additional capacity. In fact, we can add new servers to our production pool in under 20 minutes, from the time we call the “run instance” API at EC2, to the time when public traffic begins hitting the new server. This includes machine startup time, adding custom server config files and cron jobs, rolling out application code, running smoke tests, and adding the machine to public DNS. What follows is a general outline of how we do this.