I’m proud to be talking about Leveraging the Cloud with Ruby at the Ruby Hoedown this coming August. Ruby Hoedown is a regional Ruby conference held this year in Huntsville, Alabama. There is a great lineup of speakers and it should be a great time.

I’ll be discussing cloud computing and showing rich code examples of how Ruby developers can use services such as Amazon Web Services to build highly scalable applications. Do you want to see something specific? If you’re attending Ruby Hoedown and would like to see something in particular leave a comment here. Register for the Ruby Hoedown today and I’ll see you in August!

Atlantic Dominion Solutions partner RightScale has raised $4.5M from Benchmark Capital. RightScale’s management platform simplifies the deployment and management of applications utilizing Amazon Web Services, and adds additional capabilities like auto-scaling. The RightScale tools are an integral part of ADS Mantis for EC2, which provides 24×7 monitoring and management of EC2-based applications.

Read more about the funding RightScale received on c|net and Giga OM.

The Amazon Web Services team announced yesterday a much anticipated feature for Amazon EC2 - persistent storage. This feature is not yet available, but coming soon. Per the Amazon Web Service Blog,

“…our forthcoming persistent storage feature will give you the ability to create reliable, persistent storage volumes for use with EC2. Once created, these volumes will be part of your account and will have a lifetime independent of any particular EC2 instance.”

The new persistent storage feature will allow for unformatted disk volumes ranging from 1 GB to 1 TB, with the ability to attach one or more to a single EC2 instance. If that wasn’t enough, you will also be able to make snapshot backups of the volumes and store them on S3. All volumes will be accessible via an API and the EC2 command-line tools: CreateVolume, DeleteVolume, AttachVolume, and CreateSnapshot.

What is very cool about this new feature is that the volumes are independent of EC2 instances. Did your EC2 instance just vaporize? No problem. Fire up a new one and attached the volume.

Atlantic Dominion Solutions partner RightScale is ready to roll with support for the new volumes. I talked with Thorsten who told me,

“We already have quite extensive support in our internal systems (for persistent storage). We have the Ruby gem (right_aws) already done, so we can release that as soon as Amazon goes public with the details. But the heavy lifting comes with the volumes and the snapshots, supporting multiple database use cases. To be effective, you need to automate the whole snapshot schedule stuff, else you drown and you have no clue where the best copy of your data is.”

Amazon continues to up the ante with the upcoming persistent storage feature. With Google AppEngine now in the mix as a competitor, we look for even more innovation to come from the Amazon Web Services team.

Trying to make sense of the Google App Engine and what possibilities it holds I have compiled a list of what seems to be good, bad, and what I’m not quite sure about.

First, what seems bad:

Some Bad Things

1. You can not access the file system.
2. You can not run background processes, not even CRON.
3. You can not spawn off another thread / process.
4. Requests that take more than a few seconds are terminated (why would you do this to your users anyway?).
5. You are only able to access other sites and services using HTTP(S), so no fancy sockets or anything like that (this doesn’t seem bad, but it is a limitation).
6. You can’t use the Django authentication, admin, or session middleware apps (more on this below).
7. You can not use Django models.
8. Some standard libraries have been disabled, so not all Python modules are available.
9. As of right now, you can not use Ruby, PHP, C#, Java, or anything else. Only Python (is that so bad actually?).

Some Good Things

1. You use Python.
2. You can use Django.
3. It uses Googles DataStore and they provide a nice API (it’s similar to Django models).
4. The Django admin and authentication is handled by Google.
5. It scales.
6. Deploys fast and easy.
7. Monitoring is part of the admin console provided by Google.
8. You can include any pure Python libraries when you upload your application.

Some Things I am Unsure About

1. Vendor lock-in? If I want to move my app someplace else for example.
2. How is domain management handled?
3. GQL? I’m not sure what to make of it, except it looks like SQL. Are there any Limitations compared to SQL?

How does this compare to Amazon’s web developer services? Well, it seems the 2 services are not really comparable.

The only thing that seems to compare is the hosting platforms, that is EC2 and the whole GAE package. An EC2 instance is like a VPS, you have much more control (what gets installed, libraries, languages, database, etc). Setting up an EC2 instance is not difficult, but definitely not as easy as using GAE. If you like Python and don’t have any issues with the limitations Google has with it’s sandbox it seems like a great service. Actually it’s a great service regardless of those limitations.

Regarding S3 and SimpleDB, I don’t see how Google is even competing or offering a similar service with GAE. Did I miss something? These 2 services can be used from anywhere on the internet. As far I can tell, GAE does not have any sort of S3 equivalent and DataTable seems limited to being used inside GAE only.

Regardless of any of it’s current limitations, Google App Engine seems like a very cool neat service and I can’t wait to take advantage of it someday.

My question now is what type of apps can take advantage of this service?

Oh, and if you aren’t a big fan of Django for some unknown various reason, you can always try Pylons.

The Amazon EC2 team announced that they added two new features yesterday:

• Elastic IP Addresses
• Availability Zones

Elastic IP Addresses are static IP addresses associated with your AWS account rather than your EC2 instances. These addresses can be programmatically mapped to any instances that you have. What this allows is easier recovery in case of failure as well presenting your users with a static IP address (i.e. less DNS headaches).

The second new feature, availability zones, are made for high availability:

“Each Amazon EC2 Availability Zone is a distinct location that is engineered to be insulated from failures in other Availability Zones. Previously, only very large companies had the scale to be able to distribute an application across multiple locations, but now it is as easy as changing a parameter in an API call. You can choose to run your application across multiple Availability Zones to be prepared for unexpected events such as power failures or network connectivity issues, or you can place instances in the same Availability Zone to take advantage of free data transfer and the lowest latency communication.”

Both of these features are very exciting as they address some of the current pitfalls of EC2. Read the official release here.

We received this email this morning from RightScale, an ADS partner.

—
Dear Customer,

At 7:35 EST Amazon S3 service started given 503 service unavailable messages. We have contacted Amazon and they have confirmed the problem and they are working on resolving it. The main effect on our customers is that 10-minute backups are not happening and operations that rely on S3 fail. Amazon has not giving an expected time to resolution yet. We will keep you informed. We apologize for the inconvenience.

The RightScale Team.
—

We received this soon after we noticed the outage ourselves. This type of information is extremely important to get out to clients when situations such as downtime occur, so that they are not surprised and you don’t hear the “why didn’t you tell me.” Transparency, honesty, and communication are the keys to great business relationships.

UPDATE

Soon after I put up the post S3 is back up. Happy days.

My last article for the Amazon Web Services Developer Connection was a how-to on creating a web spider using Ruby, Rails, Amazon S3, Amazon EC2, Amazon SQS, and then making it all searchable using Lucene. If you are wondering how to make that happen, check out the article: Using Amazon S3, EC2, SQS, Lucene, and Ruby for Web Spidering.

New EC2™ Management Solution from Atlantic Dominion Solutions Offers 24×7 Service

WINTER PARK, FL (December 19, 2007)…Web development innovator Atlantic Dominion Solutions (ADS) today introduced the only continuous monitoring and management solution for Rails apps deployed onto Amazon EC2 – ADS Mantis. Businesses can now use EC2 to scale on demand and still get the consistent, reliable, and personal service previously limited to traditional hosting solutions.

ADS Mantis combines dedicated 24×7 service with powerful management and monitoring tools from RightScale and FiveRuns, themselves leaders in Ruby on Rails and Amazon Web Services technologies. The only solution of its kind, ADS Mantis minimizes downtime by continuously monitoring EC2-deployed Rails apps, thus enabling the ADS team to prevent and, when necessary, quickly respond to issues.

“Clients had been asking us a lot about ‘EC2 hosting’ but there was really no precedent for it,” said ADS Project Director, Robert Dempsey. “We realized there was a big need for EC2 management and ADS is uniquely suited to provide such a service.”

EC2, part of the Amazon Web Services, provides virtual hosting using Amazon’s infrastructure – allowing businesses to scale their web applications up or down on demand and pay only for the capacity they use. But the flexibility and cost savings derived from Amazon EC2 do not come with the service and support normally included in traditional web hosting packages. To maintain the consistency and reliability required for the success of their web applications, businesses have often needed to hire additional IT staff to monitor and manage their deployments. ADS Mantis allows these businesses to realize maximum uptime without taking up valuable developer bandwidth or incurring the excessive costs of in-house staff.

“Rightscale is delighted to help ADS enable one-stop, personalized management and support for Rails on EC2 ,” said Michael Crandell, CEO of RightScale, an ADS partner. “With their ADS Mantis offering, ADS helps to deliver on the promise of cloud computing by adding a layer of managed services on top of Rightscale’s ability to launch and manage multi-server deployments.”

ADS partner FiveRuns, provider of the monitoring tool employed in ADS Mantis, was thrilled to get on board with the innovation in EC2 management. Dean Cruse, Vice President of Sales and Marketing for FiveRuns, talked about the opportunities afforded by the new service.

“We see tremendous momentum for EC2 within the Rails community, particularly due to its ability to scale on demand,” said Cruse. “However, businesses need to ensure their applications are performing properly in order to optimize their EC2 experience. FiveRuns welcomes the ADS Mantis solution for EC2 management and monitoring and we’re excited to be a part of the initiative to help EC2 customers manage, monitor and scale their applications.”

Dempsey echoed those benefits, and reiterated that the foundation of ADS Mantis is reliable personal service.

“EC2 is a flexible, powerful foundation for a great Rails app, and I highly recommend it to our clients” said Dempsey. “But not everyone has the manpower to manage it. With ADS Mantis, everyone can get the dedicated personal service necessary for their deployment and relax knowing their app is well taken care of.”

ADS Mantis offers three service levels for different deployment sizes. The Basic option monitors up to four concurrent virtual servers; the Plus level provides for up to eight servers and includes two hours of other premier services; and the Premium level works with up to 12 servers and includes four hours of premier services. For more information about ADS Mantis, visit www.techcfl.com/ads-mantis.

About Atlantic Dominion Solutions

ADS is a leading-edge web development firm that specializes in using Ruby on Rails to build advanced, scalable, database-backed applications for organizations of all sizes. Based on custom requirements, ADS creates forward-thinking and user-focused applications that can be deployed quickly, scaled easily, and maintained with minimal time or effort. ADS is a vanguard in enhancing Rails platforms with Amazon Web Services and, along with visionary web applications, ADS also provides data warehousing and data mining, graphic and interactive design, and business and technical consulting. For more information, visit www.techcfl.com.

Amazon Web Services released SimpleDB, a web service for “running queries on structured data in real time,” to limited BETA today. SimpleDB is a schema-less data storage service that provides automatic indexing of data, a RESTful API for storage and access, and a pay-as-you-go billing model.

Highlights

• Simple - RESTful interface makes using the service easy (we Rails folks love REST)
• Flexible - you aren’t locked into a schema, indexing is automatic
• Scalable - create new domains on demand
• Fast - built using Erlang; quick and efficient storage model
• Reliable - data is stored redundantly inside the Amazon high-availability data centers
• Ease of Integration - created to easily integrate with other Amazon Web Services
• Inexpensive - as with other offering, SimpleDB is pay-as-you-go

Terminology

• Domain - the “bucket” where your structured data is stored; you run queries against a domain
• Items - the structured data that you store in a domain; comprised of attribute-value (av) pairs; can have 256 attribute values; each attribute can be between 1 and 1,024 bytes

Functionality

• CREATE a new domain to store your items
• GET, PUT, and DELETE items in your domain with their av pairs; automatic indexing is performed as items are stored
• QUERY the data using =, !=, <, >, <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION, and UNION; query time is limited to 5 seconds (so it better be fast)

Pricing

Machine Utilization

• $0.14 per Amazon SimpleDB Machine Hour consumed (based on querying)

Data Transfer

• $0.10 per GB - all transfer in
• $0.18 per GB - first 10 TB / month data transfer out
• $0.16 per GB - next 40 TB / month data transfer out
• 0.13 per GB - data transfer out / month over 50 TB

Structured Data Storage

• $1.50 per GB-month

Final Thoughts

A relational database is a limiting factor for any web-based application. The database is limited to the partition on which it sits, and the ability to scale up depends on how quickly you can add to the storage pool on which the database server sits. If you are not in control of the hardware that houses your app, scalability is restricted.

Companies like Google use the file system (optimized of course) as the database, and solutions such as CouchDB and RDDB are very interesting to us. SimpleDB from Amazon is definitely a step in the right direction.

Here at ADS we work with Amazon Web Services quite extensively and are looking forward to using SimpleDB. We will stay on the lookout for a Ruby library and will post as soon as one is released.

The Amazon Web Services announced on Monday that the EC2 service is now open to all developers. Along with this excellent news, the AWS team released “Large” and “Extra Large” AMIs. Previously, all EC2 instances (AMIs) were created equal, giving you 1/2 an Opteron CPU, 1.75GB of memory, and 160GB disk. We now have some choices:

Small Instance (default - what we have now)

• 1.7 GB RAM
• 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
• 160 GB storage (150 GB plus 10 GB root partition)
• 32-bit platform
• $0.10 per instance hour

Large Instance (new)

• 7.5 GB RAM
• 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
• 850 GB storage (2 x 420 GB plus 10 GB root partition)
• 64-bit platform
• $0.40 per instance hour

Extra Large Instance (new)

• 15 GB RAM
• 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
• 1,690 GB instance storage (4 x 420 GB plus 10 GB root partition)
• 64-bit platform
• $0.80 per instance hour

Atlantic Dominion Solutions is leading the way with Ruby on Rails deployments on Amazon Web Services including EC2, S3, and SQS. ADS partner RightScale also announced on Wednesday that they are supporting the new instance types. For more information on scaling Ruby on Rails applications using Amazon Web Services, contact Robert Dempsey at ADS, and in the meantime, read our articles on the Amazon Web Services Developer Connection site:

Intro to AWS for Ruby Developers

Economical Use of Amazon S3 with Ruby on Rails