My company has made a fundamental shift towards Cloud computing and
specifically towards Amazon Web Services (AWS) in the last 12 months.
One needs a firm grip on any deployed architecture as under AWS pricing
it is the architecture that determines the cost per month. Get it wrong or miss
something and it reflects in an increase in the monthly cost. So, before we
started moving to the cloud we had to ask ourselves, is our architecture
flexible and easy enough to migrate over? More importantly, do we have the
system architecture documented; do we know what pieces we’re working with?
Of course, we couldn’t just then flip on our application and go live and
be done with it. The prototype worked but what about the ‘proper’ application
and data? The next step was to introduce the finalised architecture slowly. We
had decided on a ‘live’, ‘stage’ and ‘dev’ environment that was recommended by
our solution architect. Each environment would have its own elastic load
balancers and at least two standard instances behind each load balancer. That
is six instances now, the cost of the micro instance during the test was $0.035
per hour compared with the standard instance (confusing classed as ‘large’)
costing $0.48 per hour and having six of them, the hourly cost shot up by an
alarming 168%!
Despite the issues we had faced, everything was on still track and all
the instances created and ready to roll. It was at this time, with the
architecture on a semi-live footing that we ran through a very involved
performance testing process. We created a test harness both on a dedicated
cloud server (for cloud-to-cloud testing) and another server within our own LAN
environment. The latter test harness would probably be a more accurate
reflection of real life usage as the test would need to go navigate through network
traffic and the firewall before hitting the internet.
 | |||
| I think this is a picture of the servers for Facebook? |
Quoting the wiki (it’s always good to start with someone’s quote!):
Cloud computing is the delivery of
computing as a service rather than a product, whereby shared resources,
software, and information are provided to computers and other devices as a
utility (like the electricity grid) over a network, typically the Internet.
Why did we do it? At the technical level it was always something that we
wanted to do but like all projects it needs strategic buy-in. The potential
cost savings (as published in the media) as well as the flexible nature of the
cloud meant that it was foolish not to try the cloud if only to compare it with
the environment of ‘traditional’ hosting. At a strategic level, our parent
company had already migrated some of their main-line hosting applications to
the cloud via AWS so there was a path already set out (albeit with differences
in application and customers).
So the process of migrating to the cloud took a number of distinctive
steps.
1. Planning
2. Test Deployment
3. Parallel Running
4. Go Live
5. Update documentation and processes
PLANNING
 |
| Pencil and paper to start |
Unfortunately, what we had was not quite up to scratch so the first step
was an inventory of the systems ear-marked for 'cloud deployment'. It was a
useful exercise and took about four weeks to complete though the process should
now become an on-going one. At the end
of this exercise, we had a good idea of the proposed architecture. With this, I
had to sit down and justify to my project team and those above me, the cost of
this venture. Now, initially I went over to the product pricing pages (http://aws.amazon.com/ec2/#pricing
for EC2 pricing and http://aws.amazon.com/rds/pricing/
for RDS prices and threw together a tonne of spreadsheets and tried to work out
an annual cost. A bit of headache as AWS pricing also includes a lot of
variable related to data transfer, the use (or non-use) of Elastic IP addresses
and volume storage to name three. However, there is a handy monthly cost
calculator available which was a God-send (http://calculator.s3.amazonaws.com/calc5.html)
– if only I knew of this tool when I first started on our AWS adventure. Of
course, this tool’s accuracy is much improved if you can insert as much of the relevant
bits of information into the appropriate boxes. Hence the review and inventory
step was so useful furnishing me with this information to hand.
So I got my architecture and now a proposed monthly figure in Euros and added
15% contingency as you just know something will happen! I calculated the annual
cost and presented it for budgetary consideration. Since the decision to move to
the cloud was already made, surely, my budget would not detract them? Luckily,
my calculations were not that far off and well within the budget. So, green light
and all systems go! It was a near thing as most people didn’t think past the
initial headline costs (‘instances from $0.03 an hour!’) and were mildly
surprised at the proposed figure.
TEST DEPLOYMENT
 |
| Should be brighter colours.... |
With a plan now in mind and a road-map, it was time to prototype the
deployment. We had to get used to using
the AWS dashboard and the available tools as well as observing the performance
of the applications in this new and exciting environment. Now, back when we
started, a lot of the products were not available so the functionality was
limited.
A test environment of a SimpleDB with a micro instance (the smallest AWS
virtual machine available) was commissioned and then tested. SimpleDB is AWS’
simple and flexible non-relational datastore. It was quick and easy to set-up
and worked for the initial testing and deployment stage. We used it to stored
all our user and audit data. When we started, it was quite cheap but now,
SimpleDB is free the first 25 hours consumed and up to 1GB of storage
space per month! More details of Amazon SimpleDB can be found here: http://aws.amazon.com/simpledb/
The initial test did reveal a number of specific issues: we immediately
encountered problems with security groups and the incompatibility of SimpleDB to
our security model. However, the ease of deployment from some stock template
Amazon Machine Images (AMIs) meant that we could easily set-up a new
environment in a matter of minutes. The performance of the web site and application
was as fast as we had ever seen it and all for a few dollars; a bargain we
thought and we were impressed. Of course, there were still unanswered questions
over alerting and monitoring; management reporting and metrics but one step at
a time eh?
Other issues we had discovered when using SimpleDB were the relatively
slow interaction with the micro instance via remote desktop. Some additional gotchas: do not use the
Windows 2008 32-bit edition on a micro instance. With this version, everything
was just dreadfully slow. We were advised by AWS to switch to using the Windows
2008 64-bit edition. Once we switched over, things went a lot more smoothly. Of
course, the specifications of the micro instance just about meet the
recommended installation requirements for our ships no matter.
Out of interest we looked at the specifications of the micro instance and
this is what we found: 613 MB of memory,
up to 2 ECUs (for short periodic bursts), EBS storage only, 32-bit or 64-bit platform. Now, interestingly, from experience
the minimum RAM required for Windows 2008 is 512 MB of memory but in reality we
don’t commission anything with less than 4GB (32-bit) or 8GB (64-bit) and even
though theoretically we had about
100MB of RAM available, the micro instance once started up had about 5MB left
and would be frantically paging to disk. Basically, the micro instance cannot
be used for anything remotely useful commercially; which is a smart thing to do
by AWS.
We also realised how quickly Amazon Web Services were bringing out new
products almost every month!
RESERVED INSTANCES OR NO? (sorry digression warning!)
One year
|
|
On Demand Instances
|
|
Large Instance
|
|
Hourly Rate:
|
$0.48
|
Setup:
|
$0.00
|
Cost per year:
|
$4,204.80
|
Total
|
$4,204.80
|
It was at this time that we looked at reserved instances and the concept
was simple: pay AWS an upfront setup fee and get a much reduced hourly fee for
either a one-year or three-year period. If the planned utilisation of instances
was low (i.e. you would be running an instance only for a few hours per week)
than the reserved option was not economical. However if monthly utilisation is
high, such as 100% then reserved made financial sense and should be actively
considered. Here are some tables I had put together to convince my manager that
reserved instances is a good idea.
If we’re just paying the on-demand, pay-as-you-go fee then for one large
AWS instance the cost is $4,204.80 and this figure does not take into account
the use of other AWS products such as S3, RDS, Cloudwatch etc. This is the pure
cost of running one large
instance, 24/7.
Now, $4,204.80 sounds a lot but for this you get a very high performance
server, running windows 2008 64-bit Advanced. While they are virtual they are
equivalent to a quad core server, already licensed and ready to go.
However, if one pays a setup fee of $1,105; then the hourly cost drops
from $0.48 to $0.205 per hour; with 100% utilisation then the annual cost drops
to $1,795.00 per year and with the setup fee included this result in a net
saving of $1,304.00 per year per instance.
Oh, as of November 2011, AWS decided to add some more product
differentiation for the reserved instances. Previously there were two rates,
one for a one-year deal and second rate for a three-year deal. Well now we have
the same two rates but spread out into three ‘utilisation’ groups; light,
medium and heavy. The differences relate to how you are charged when you have a
reserved instance and it is NOT being used i.e. ‘lights on, no-one home‘ We
went with the heavy utilisation.
One year
|
Three year
|
|||||
Reserved Instances (Heavy Usage)
|
Reserved Instances (Heavy Usage)
|
|||||
Large Instance
|
Large Instance
|
|||||
Hourly Rate:
|
$0.205
|
Hourly Rate:
|
$0.205
|
|||
Setup:
|
$1,105.000
|
Setup:
|
$1,700.000
|
|||
Cost per year:
|
$1,795.800
|
Cost over 1 year:
|
$1,795.800
|
|||
Cost over 3 years:
|
$5,387.400
|
|||||
Total
|
$2,900.800
|
Total
|
$7,087.40
|
|||
Saves:
|
$1,304.00
|
Saving:
|
$5,527.00
|
|||
Pretty good numbers and as they say, a real no-brainer if you need to
save some money!
(edit: since this post was originally drafted, there's been further changes in the reserved pricing with high-, medium- and low-utilisation rates now available. Check them out here!)
(edit: since this post was originally drafted, there's been further changes in the reserved pricing with high-, medium- and low-utilisation rates now available. Check them out here!)
COMMISSIONING and then PARALLEL RUNNING
 |
| This has to be used in the next release phase! |
DATA, DATA, how it vexes me, DATA
Since I worked for a mapping company we had lots of data and one
challenge was to get the data into the cloud and onto the AWS instances. Two of
our instances were our mapping servers with all the data and server software
installed on each.
There were a number of options available to use and we considered them
all. The options included:
1.
Upload everything to S3.
2.
Use the AWS export / import service.
3.
Install an FTP client / server and use that to
move.
4.
Use the RDP copy function.
Using S3
Now, S3 (Simple Storage System) is Amazon’s fast, inexpensive cloud
storage offering that Amazon themselves use. You store things in a conceptual
folder, called a ‘bucket’. Like the other AWS products; one only needs to pay
for as much as one used and the cost of storage is available here (http://aws.amazon.com/s3/#pricing)
with additional cost for bundles of requests to and from the S3 bucket.
We had several hundred gigabytes to move into the cloud. We decided to
use the very useful Mozilla Firefox add-on, ‘S3Fox’ as the application to move
our data onto out S3 bucket. We then installed S3Fox onto each of the instances
so that we could then download the new data.
We took timings and not surprisingly, the speed to download the data
from an AWS S3 bucket to an AWS instance was significantly fast. The bottleneck
was our own internet connection’s speed (though not too shabby at 60mbps) and
any limitations to upload bandwidth.
There were some gotchas all to do with file sizes.
First the the S3 bucket was limited to objects no larger than 1TB in size[1]
and this posed a number of problems for us since our spatial data was already
in the terabyte range. So, we had to chunk up the data prior to transfer.
However, S3Fox itself appeared to be limited to files no bigger than 50GB per
file. Anything bigger and it would fail. We suspected that this was more to do
with the 32-bit operating systems we were using. This further compounded our
problem but it forced us to reduce the file of each chunk to no more than 50GB.
For a 1.2 TB file, it meant we had to create about 20 files which we would upload
sequentially to theS3 bucket in the cloud, then download it each chunk to our
selected instance before we stitched it back together again. The tools we used for this included 7zip[2]
and some linux tools.
The entire upload process to move our data from our own local servers to
the AWS instance via our S3 bucket took about a week to complete. Once the data
was successfully stitched together we copied the contents to a separate Elastic
Block Store [3](EBS)
and attached it to our mapping server instance as a new drive (volume ‘E’)
since the default instance only came with a 35GB C drive and a 100GB D drive,
both too small for our needs. We also made a copy of this volume. This new
volume would be attached to the second of our mapping server instances. This
would be a lot easier than downloading the data a second time.
AWS export/import service
Obviously, AWS had anticipated that some of their customers would need to
move large amounts of data into and out of AWS so they have offered a service
where one can use a portable storage device to load directly into AWS
high-speed network. We liked this idea as the cost of a 2TB drive is now
comfortably under $200.00 however, the service isn’t consistent: the AWS
import/export service only supports the import of files from the portable drive
into Amazon EBS for the US Regions. Everywhere else, the import/export puts
data into your Amazon S3 bucket. This obviously cuts out the slowest part of
the process (our upload time) but the fiddly stitching together of multiple
file still remained – we are keen to cut-out the need to chunk up the data and
reassembly the bits into a coherent file. All this takes time that we’re keen
to save.
Using the FTP client/server
This was tried and it worked fine; however it did require a change in the
AWS security group to allow FTP through as well as extra software installation
and configuration. Our aim was to keep the instances as lean and secure as
possible and installing FTP software would increase the vulnerability
footprint.
Using the RDP copy and paste function
This also worked but only for small (and I mean, small) files. It was unsuitable for the size of files we were
using.
At this point it was prudent for us to make a snapshot of this instance,
after all, losing this would set-up us back, we then discovered another
gotcha. With the new 1TB EBS volume
attached, taking a snapshot of the entire instance would 1) take a long time
and 2) most likely fail. Rather perturbed we tried again with the same result.
A quick chat to an AWS Solution Architect[4]
and it was recommended that we first dismount the new volume under Windows Disk
Management and then dismount the volume through the AWS dashboard. Then back up
the instance and EBS volume separately.
This worked but it added another step in an ever-growing work list.
Performance testing – does the application rock?
 |
| Faster, faster, faster! |
Will the
cloud server really be as fast as people claim to be?
The results
were very pleasing, on average the speed of response was less than one second
and the ability to handle a big jump in concurrent users, from one to 100 was
easily handled by the AWS infrastructure (ELB and EC2) – we didn’t even look at
the auto-scale functionality.
SECURITY
After the testing, we went through an involved round of security testing
from brute force hacking of the website (we warned AWS first in case they
responded!) to social engineering where we had some of our developers try to
hack into our email accounts in order to create sophisticated phishing
attempts. While I do not want to get into too much detail, the security is only
as strong as your personnel. AWS itself was perfectly fine.
BETA Programme
So, with testing complete it was time to for a limited beta-programme as
from our experience, there’s no testing quite like user testing. This was
opened to a select number of our clients and the feedback was invaluable – they
also discovered some bugs in the application itself that we had failed to
notice. The beta-programme should have for a couple of months but we extended
it by another three months with the notion that it will over-lap with the go
live date. We had some features in beta that would not make it to release due
to some legal issues hence the overlap.
GO LIVE
So the go live date arrived and it was actually a bit of an
anti-climax. With a marketing campaign kicking off and users signing up, the
site smoothly transitioned from beta to live. The usage patterns were very
pleasing with a steady upward curve in registrations and users.
THOUGHTS
We had replaced a bunch of physical servers with a new set of virtual
machines. We are running the live production servers 24/7; just like they are
physical servers so we're paying a constant fee each month. With a physical
machine, the initial cost is usually up-front (the capital cost) and the cost
(and value) of the machine drops over time, sometimes quite rapidly.
Conversely, the requirements of ever newer applications and software places
increased demands on hardware. The cloud
servers, (we've been informed) have a rolling programme where the underlying
hardware is constantly updated by the provider. Of course, one cannot control
this upgrade process and if you really need the latest and greatest hardware,
you cannot go out and get one[5]!
There is a misconception that I want to tackle, though I am not a myth
buster[6]!
The cloud is touted as being cheap and for some users, it is indeed, a cheap
and powerful resource. You treat computers and computing power like a utility,
turning it on and off whenever you need it. Pay for what you need! An excellent
idea and I cannot fault it. However for
a live production system (such as the one I am working on) - that doesn't actually
apply. Using the utility example further; our usage is more akin to using the
electricity to power our life-support system and there's no compelling reason
to switch it off to save money.
So it makes calculating the annual cost quite easy. The standard AMIs we
use are the large AMIs, currently at $0.48/hour which works out at $11.52 a day
and $4204.08 a year. The jump from a headline grabbing price of $0.48 per hour
(which doesn't sound much) to a more eye watering $4204.08 per year can be disconcerting
for those tracking the money closely. Now multiple this with the number of
instances one has in their production environment, their staging environment
and maybe a development environment and the cost starts to hit the level that
makes those in charge of budgets uncomfortable. Counting the cost of the
Elastic Block Store (EBS) for the storing all your extra data, throw in a
enterprise load balancer and lots of snapshots (you do want backups right?) and
a few 100 gigabytes worth of traffic to and from your AWS cloud and the cost
just keep going up.
CONCLUSION
The Cloud is here to stay, at least with my company and our parent
company. It is definitely more expensive than we had first expected, due to
many of our system architects being fooled into the cost savings of a
developing an ephemeral system.
My Bug Bears
1. CloudWatch has no SMS notification, only in the US East Region.
2. The import/export service where large files can be loaded into AWS?
Ideally they should be made available for the EBS but this is only available in
the US East Region. For those in Europe, we have to have the large file loaded
to S3 and then downloaded. Well, at least the
Now, we had some veterans in our company who looked at the cloud and
said, 'seen it before' and went back to cutting some obscure code.
However, the advantages of rapid up scaling either horizontally and/or
vertically meant that responses to architectural changes can be fast, safe and
efficient. If you have an excellent technical architecture, a clear understanding
of what you are delivering, support processes in the form of configuration
management, change management and release management. Then you're good to go.
[1]
This has now been raised to 5TB http://aws.typepad.com/aws/2010/12/amazon-s3-object-size-limit.html
[2] http://7-zip.org/ - a fantastic tool
[4] We
had the benefit of knowing someone at AWS rather than go through Customer
Support
[5]
Maybe the cloud provider (AWS in this case) can give an indication of this?
No comments:
Post a Comment