So Xen, EC2, Vmware, and other technologies are uprooting traditional thoughts on system management. At RailsConf we had presentations by Rails Machine, Engine Yard, and Amazon Web Services on the subject. There's little doubt that this is the future.
Thus, I think it's time to start a conversation about how we help further this move by putting together a handful of official Rails stacks. I'm envisioning something like a set of complete images for web, app, and db servers respectively. Running perhaps nginx, mongrel, and mysql for starters.
Over time, the vision is that we can build even more infrastructure around these stacks. And that they will become the convention for Rails development, so Rails and capistrano can assume them in their default setups.
Even further out, I could see something like a XenAdapter or EC2Adapter where you could do something like cap xen:app:setup 123.123.123, be prompted for user name and password, and an app server would be configured on that machine and your application installed on it.
But let's not get ahead of ourselves. First, lets just get this idea going. I'm particularly interested in hearing from people with first- hand experience running Rails under virtualization and from people with experience building stacks.
On May 21, 1:42 pm, DHH <david.heineme...@gmail.com> wrote:
> But let's not get ahead of ourselves. First, lets just get this idea > going. I'm particularly interested in hearing from people with first- > hand experience running Rails under virtualization and from people > with experience building stacks.
ThoughtWorks announced paid support for their RHEL/Centos stack in July. It wouldn't take much to image it for EC2 or other VMs. http://studios.thoughtworks.com/rubyworks
> Over time, the vision is that we can build even more infrastructure > around these stacks. And that they will become the convention for > Rails development, so Rails and capistrano can assume them in their > default setups.
I've always envisioned this problem being solved by a thin layer on the server that provides a set a of deployment tasks callable by Cap, local shell commands, or web interface. For example, the Rails Machine gem provides a cap task for "restart_web" that calls "sudo /etc/init.d/ httpd restart" on the server. With a simple Rakefile installed on the server, you can do 'cap web:restart' and it could call 'rake web:restart' on the server. The web:restart rake task could do the restart for whatever web server was installed/configured. Rake tasks can also be called directly from the shell or even through web interface. I'm definitely moving away from doing putting a lot of complexity in the cap actions.
I've experimented with this kind of stuff, but have never released as I only provide support for Apache/Mongrel/Rails/MySQL. My customers can run other stuff, but they don't get the same level of support. These generic deployment tasks are already in my recipes: setup_server, setup_app, restart_app, etc. (soon to get cap2 namespaces) More info here:
I'm a bit partial to the idea of deployment adapters as people have real requirements for choosing one stack over another.
> But let's not get ahead of ourselves. First, lets just get this idea > going. I'm particularly interested in hearing from people with first- > hand experience running Rails under virtualization and from people > with experience building stacks.
At Rails Machine, we run Xen and build our stacks on top of Centos 4.4 and Ubuntu Feisty Fawn support is in beta. We use Apache 2.2 w/ mod_proxy_balancer, mongrel_cluster, and MySQL. Nginx and Postgres are somewhat supported. I typically deploy servers from pre-built image files, but have also been working on a dependency-based system (inspired by talks with Zed) based on Rake.
On May 21, 10:42 am, DHH <david.heineme...@gmail.com> wrote:
> So Xen, EC2, Vmware, and other technologies are uprooting traditional > thoughts on system management. At RailsConf we had presentations by > Rails Machine, Engine Yard, and Amazon Web Services on the subject. > There's little doubt that this is the future.
Agreed 100%
I have been playing around with EC2, went to the session on RightScale & talked with Ezra from engine yard about their setup at the conference.
The thing I think the community needs and I would like to start is an open-source version of the functionality of RightScale (web app contral pannel for a cluster) taking the ideas of architecture behind Amazon but flexible enough to run on other setups.
Imagine a rails app to control, monitor & scale a cluster. Imagine being able to choose to run your cluster on EC2 (Xen), SQS (messaging) & S3 (storage) --OR-- your own hardware with Xen, Reliable Messaging & GFS (for example).
-Tim Dysinger
PS - Puppet is also interesting for controlling nodes but was not mentioned at the conf. http://swik.net/puppet
On May 21, 4:42 pm, DHH <david.heineme...@gmail.com> wrote:
> Even further out, I could see something like a XenAdapter or > EC2Adapter where you could do something like cap xen:app:setup > 123.123.123, be prompted for user name and password, and an app server > would be configured on that machine and your application installed on > it.
This is *exactly* what I had in mind when I began work on Capazon[1], a cap extension library for EC2. I'll leave the best practices for the stacks to those with more experience in the hosting vertical, but the facility to easily deploy a pre-built stack to EC2 is already here.
It's not a one step process in the public version of Capazon, but my development version does this with a callback the provided Capistrano 'deploy:setup' task in:
task :before_setup do ec2.setup_keypair ec2.run_instance ec2.authorize_ssh_and_web_access ec2.setup_user end
This way, a user could require their choice of 'capazon' or 'capaxen' (fictitious name) in their capistrano recipe then do the normal 'cap setup; cap deploy:cold' and be up and running. I'd love to hear what others think about this pattern.
> On May 21, 4:42 pm, DHH <david.heineme...@gmail.com> wrote: > > Even further out, I could see something like a XenAdapter or > > EC2Adapter where you could do something like cap xen:app:setup > > 123.123.123, be prompted for user name and password, and an app server > > would be configured on that machine and your application installed on > > it.
> This is *exactly* what I had in mind when I began work on Capazon[1], > a cap extension library for EC2. I'll leave the best practices for the > stacks to those with more experience in the hosting vertical, but the > facility to easily deploy a pre-built stack to EC2 is already here.
> It's not a one step process in the public version of Capazon, but my > development version does this with a callback the provided Capistrano > 'deploy:setup' task in:
> task :before_setup do > ec2.setup_keypair > ec2.run_instance > ec2.authorize_ssh_and_web_access > ec2.setup_user > end
> This way, a user could require their choice of 'capazon' or > 'capaxen' (fictitious name) in their capistrano recipe then do the > normal 'cap setup; cap deploy:cold' and be up and running. I'd love to > hear what others think about this pattern.
On May 22, 2007, at 11:44 AM, jesse newland wrote:
> On May 21, 4:42 pm, DHH <david.heineme...@gmail.com> wrote: >> Even further out, I could see something like a XenAdapter or >> EC2Adapter where you could do something like cap xen:app:setup >> 123.123.123, be prompted for user name and password, and an app >> server >> would be configured on that machine and your application installed on >> it.
I was talking with Mike Bailey author of deprec at railsconf. He is going to be refactoring deprec to have pluggable backends. Meaning that you could have a gentoo stack with nginx,mongrel,mysql or a ubuntu with whatever webserver options and so on.
There is also a vmbuilder gem by Neil wilson(i think?) that automates setting up xen and friends on debian systems.
I think coming up with a common capistrano interface for building these stacks with pluggable backends is the way to go. So anyone with a nice stack can implement the backend nitty gritty tasks for setting it all up and people can use a common cap interface to deploy said stacks.
I can't see Jessie Newland's or Tobias Lutke's posts, so sorry if I repeat anything (google groups is being odd)
After talking to Jesse during the conference, I built an amazon EC2 Ami (ami-a38b6eca) as a proof of concept.
If it is launched with user data (-d in the java tools) specifying an svn server (and user/password, the three separated by pipe), and is able to connect to that server, it will export that repos and start 3 mongrels and nginx for that site.
if anyone wants to test it out, it's public, if anyone wants to just see it running, let me know I'll launch one.
On 5/22/07, Ezra Zygmuntowicz <ezmob...@gmail.com> wrote:
> On May 22, 2007, at 11:44 AM, jesse newland wrote:
> > On May 21, 4:42 pm, DHH <david.heineme...@gmail.com> wrote: > >> Even further out, I could see something like a XenAdapter or > >> EC2Adapter where you could do something like cap xen:app:setup > >> 123.123.123, be prompted for user name and password, and an app > >> server > >> would be configured on that machine and your application installed on > >> it.
> I was talking with Mike Bailey author of deprec at railsconf. He is > going to be refactoring deprec to have pluggable backends. Meaning > that you could have a gentoo stack with nginx,mongrel,mysql or a > ubuntu with whatever webserver options and so on.
> There is also a vmbuilder gem by Neil wilson(i think?) that > automates setting up xen and friends on debian systems.
> I think coming up with a common capistrano interface for building > these stacks with pluggable backends is the way to go. So anyone with > a nice stack can implement the backend nitty gritty tasks for setting > it all up and people can use a common cap interface to deploy said > stacks.
I have a bunch of images with rails running on EC2. I actually have one image running a rails application for the last 4 month. This included a multitude of deployments, migrations, fixture reloads, the works. I use capistrano with it.
I took the base Amazon FC4 image, upgrade it to FC6 (my newer images are now CentOS5), then install Postgres (yeah, I'm in THAT camp ;-) ), ruby, mongrel, gems, etc... Then I rebundled it with the changes, and now I have a ready image. When I need to deploy, all I do is launch it, create the DB, and using cap - deploy my application.
It has been working spendidly!
Bye,
Guy.
On May 21, 1:42 pm, DHH <david.heineme...@gmail.com> wrote:
> So Xen, EC2, Vmware, and other technologies are uprooting traditional > thoughts on system management. At RailsConf we had presentations by > Rails Machine, Engine Yard, and Amazon Web Services on the subject. > There's little doubt that this is the future.
> Thus, I think it's time to start a conversation about how we help > further this move by putting together a handful of official Rails > stacks. I'm envisioning something like a set of complete images for > web, app, and db servers respectively. Running perhaps nginx, mongrel, > and mysql for starters.
> Over time, the vision is that we can build even more infrastructure > around these stacks. And that they will become the convention for > Rails development, so Rails and capistrano can assume them in their > default setups.
> Even further out, I could see something like a XenAdapter or > EC2Adapter where you could do something like cap xen:app:setup > 123.123.123, be prompted for user name and password, and an app server > would be configured on that machine and your application installed on > it.
> But let's not get ahead of ourselves. First, lets just get this idea > going. I'm particularly interested in hearing from people with first- > hand experience running Rails under virtualization and from people > with experience building stacks.
I'm not so sure we should be thinking in terms of 'stacks' at all. The best approach is surely a Rails Appliance, where you:
- fire it up - navigate to the root with a web browser - Enter the path to the Rails app repository - Press Go.
And you have an instant web app.
The support function are then a Rails app that manages the backup/ restore of the databases, and provides some simple monitoring of the operating system - much like a simple Linux based router or firewall.
Bear in mind that EC2 alters the fundamental nature of server hosting, in that you pay for it at 10 cents an hour. Moreover the majority of apps people want to use have no "Web 2" collaborative nature whatsoever. Having them live 24/7 makes no sense at all - in fact it is just a big hassle.
What EC2 means is that every user can fire up their own Rails appliance just for the time that they need it and then shut it down again - just like the applications on your desktop, except that it can be shared immediately by several people. My current project is a Rails based UK payroll application that I fully expect to become an EC2 image that people fire up once a month on a Tuesday when they need to run the reports. Most of the time it would be suspended in Amazon S3 storage.
I see the lack of a decent volatile data backup mechanism as a huge failure in the current Rails stack designs. Having said that I reckon that anybody that can get Ubuntu's hibernate function to store its hibernate file on S3 is onto a winner.
I've done preliminary work with both my vmbuilder and multi-tenant projects (both on Rubyforge) which use Capistrano to automatically build Xen and EC2 debian Rails images from scratch using the standard Debian packages and Gems for the Ruby/Rails stuff. Multi-tenant adapts the standard Rails model so that Customers share a machine but have separate databases. However after trialling it, I've pretty much come to the conclusion that the one machine, one customer Appliance model is the way to go. It is far, far simpler to manage and with EC2 ridiculously simple to set up.
I know that there has been a lot of thought given to scaling sites to gazillions of users. My approach is the opposite because of the nature of the applications I'm interested in - just let the customer pay for their own personal replica so that the number of users per server stays small.
NeilW
On May 21, 9:42 pm, DHH <david.heineme...@gmail.com> wrote:
> So Xen, EC2, Vmware, and other technologies are uprooting traditional > thoughts on system management. At RailsConf we had presentations by > Rails Machine, Engine Yard, and Amazon Web Services on the subject. > There's little doubt that this is the future.
Thanks for starting this group. Rails is one of the first communities who understands the power of virtualization.
The best part about rails over all other frameworks is all the stuff we get for free pre-configured. If we take the same metaphor and replicate it in deployment world, I am sure we will be able to reach a common ground where we remove yet another roadblock for developers : deployment
I was thinking more on the lines of deploying a complete stack from loadbalancer to database in which Rails Developer need not worry about "what it will take to scale my app for 100K users".
Something similar to "database.yaml", we have "topology.yaml" - a self- describing network topology file that adapters can read to create a complete stack on-the-fly on single server or multiple servers. Rails distribution can provide some default ideal/standardized web-stacks (different topology.yaml files) that works OOTB and have been tested thoroughly. More simpler and easy we can make for rails developers, the better.
for example:
It would be ideal if we one could configure on the fly. but this is difficult to achieve now.
topology1.yaml 1 server server#1 : 127.0.0.1 Apache mod_proxy_balancer Mongrel cluster (3) 127.0.0.1:3000 127.0.0.1:3001 127.0.0.1:3002 Mysql master
This can be achieved via the parameterized launches feature of EC2 where you provide a limited amount of user specified data to instances at launch time. This makes it possible to build generic, reusable AMIs that receive additional instructions at launch time
Capazon does some of what you mentioned already with EC2 but one server/image at a time. I agree with jesse that we can create some more generic tools to make it more intuitive to develop and deploy.
I may be completely offtopic. Please Tell me - I am not on drugs :) just my 2 cents,
will be glad to help out, Jin
On May 21, 1:42 pm, DHH <david.heineme...@gmail.com> wrote:
> So Xen, EC2, Vmware, and other technologies are uprooting traditional > thoughts on system management. At RailsConf we had presentations by > Rails Machine, Engine Yard, and Amazon Web Services on the subject. > There's little doubt that this is the future.
> Thus, I think it's time to start a conversation about how we help > further this move by putting together a handful of official Rails > stacks. I'm envisioning something like a set of complete images for > web, app, and db servers respectively. Running perhaps nginx, mongrel, > and mysql for starters.
> Over time, the vision is that we can build even more infrastructure > around these stacks. And that they will become the convention for > Rails development, so Rails and capistrano can assume them in their > default setups.
> Even further out, I could see something like a XenAdapter or > EC2Adapter where you could do something like cap xen:app:setup > 123.123.123, be prompted for user name and password, and an app server > would be configured on that machine and your application installed on > it.
> But let's not get ahead of ourselves. First, lets just get this idea > going. I'm particularly interested in hearing from people with first- > hand experience running Rails under virtualization and from people > with experience building stacks.
