About this blog

'Going Spatial' is my personal blog, the views on this site are entirely my own and should in no way be attributed to anyone else or as the opinion of any organisation.

My tweets on GIS, Humanitarian, Tech, Games and Randomness

Tuesday 21 September 2010

Wildcard SSL certificates and EC2

I needed to secure our site with a SSL certificate and went shopping. Verisign was the first choice but the price was too high - so I was informed that GoDaddy would be a good choice. Indeed they were, with certificates in the dozens of pounds sterling rather than hundreds.

I also saw their 'wildcard' SSL certifcates where one can secure a domain rather than a FQDN. Example: I may have several sites that need HTTPS for credit card transactions but they are all off the same domain (*.mytestsite.com) - so a single wildcard certificate is sufficient to cover all sites running under the mytestsite.com domain!

Brilliant and for a competitive sum too!

The certificate arrived and the transaction went smoothly, we then imported the certificate to the server certificate store and bound the wildcard SSL to our website. All this was done in the IIS Management tool. Then, we ensured that HTTPS was enabled on the amazon firewall (or security group_ and voila it worked, https worked!

However, there's a catch. When the AMI is stopped or rebooted, it never comes back online. Somehow the wildcard SSL, once bound to the default website in IIS, prevents the AMI from ever coming back after a reboot or stop / start. I have some theories as to why this is but a bit of a disappointment for sure.

Am confident that a SSL certificate for the specific FQDN will work but this means purchasing a separate certificate per website. Tedious but doable.

Monday 20 September 2010

Amazon Web Services has new tags

Keeping track of AMIs is a hassle. Who knows what is running on 'instance i-7bf88e0c'? With no descriptive fields to help us, we've gone to using a spreadsheet to keep track of all the instances and what is running on them. Some organisations use different security groups ('live-site', 'staging' etc) as a proxy. 

Well, it was with some delight that I discovered that tags were now available to each instance!


Now one can add custom information and have then displayed alongside the rather cryptic ones more commonly seen. As you can see below, I have created a new tag called 'Notes' which I attached something useful and less cryptic to each of the installed instances.

You can view tags you need to enable them in you show/hide column however. Just click on the show/hide column icon on the dash board and a pop-up window will allow you to show/hide columns.




Very useful and a life saver. 


Oh, this tagging extends to volumes as well. 


Excellent.

Friday 17 September 2010

Amazon Elastic Load Balancers (ELB)


Setting up  Load Balancers inside the Amazon Cloud is deceptively easy but there are a number of 'gotchas'. First, they are called 'elastic load balancers or ELB'. They are also software load balancers of an unknown capacity.There's been some talk on how scalable they really are though I have lost track of the news item that stated this.

Like trying out anything new, I decided to have a delve around for the documentation, just to be sure that what I had in mind would work and oddly, I found this time round that the normally excellent Amazon Web Services documentation was a wee bit sparse. However, the GUI to set up a load balancer looks easy so I decided it probably wasn't too difficult. I also discovered that the GUI tool as provided in the AWS console was not available when this service first came out and all of the work had to be done by command line. I love command line so I read up on it too. So can you here in a nice blog post by Steve Evans.

As ever, I sketched out what I am trying to achieve: a simple website running on two (or more) instances behind a load balancer supporting HTTPS. I will also need to attach a much more memorable name (e.g. loadbalancer.esriuk.com)  to the DNS of the load balancer (currently something like: mytestloadbalancer123-678076149.eu-west-1.elb.amazonaws.com ) in order to make the site easier to use. This is the CNAME as explained in a previous post.

Creating the servers or instances
First step is to create the two instances and I opted to go for the small instances. They're cheaper and for my testing, perfectly adequate for the work in mind. I created one instance (server-01), installed IIS7 on it, patched and slapped on an anti-virus software on it and then took a snap-shot of it. This snap-shot creates an AMI file which I then used to create a second instance, indeed an almost identical copy of the server-01. Very useful. 

Once I had two instances running, I made sure that each one was functioning properly. Tests were done to see that IIS on both of the instances were working. I subtly changed the default landing page on one of the IIS instances to help me differentiate between the two. 
 
The 'secure 2' was part of the HTTPS test and also helped me to remember that we were on a new page.
Creating the Elastic Load Balancer
IMPORTANT!

Now before you even start the creation of the ELB, one needs to decide what protocols are to be used. Obviously, one would want HTTP (80) to be allowed through the load balancer and this is the default but if you want to load balance HTTPS (443) or some other protocol / port - these have to be defined during the creation of the ELB. One cannot add extra protocols / ports once the ELB has been created. It is a one-step process. 


The ELB I need must support HTTP and HTTPS so these had to be defined during the ELB creation stage. HTTPS is called ' Secure HTTP server'. When this is option is selected, the other options are automatically filled in though don't worry that the protocol drop-down still says 'HTTP'; the port changes to '443' - the well-known port for HTTPS so all is well. Am sure that the protocol drop-down error is a bug and a minor one at that. One can also see that you can also re-direct incoming traffic on a particular port (e.g. 80) to a specific one on the EC2 instances. This makes the 




The rest of the process is quite straight forward; though one should spend a bit of time understanding the instance health checks that the ELB will employ to ensure that it does not redirect traffic to a 'dead' or unresponsive instance. 
 

Instance Health Checks

The ELB is quite smart in that it needs a way to check that the instances it is redirecting traffic to are online and valid traffic should still be routed to them.  Otherwise if an instance died (or was accidentally switched off) the load balancer would still continue to route traffic to it, oblivious to the instance's 'dead' state and this would cause partial downtime. What's worse, it could be quite intermittent with one out of every two pages (for example) returning with a '404' error message. 


So how does the ELB work? It does it simply by checking a specific file, usually a web-page file on a set schedule to ensure that the instance is healthy.
There are six parameters that one needs to be configure: 

'Protocol' - you can use either HTTP or TCP as the protocol of choice. 
'Target' - this is the file that the ELB checks to ensure it is all OKAY, usually it will be a simple web-page such as the default.htm file.
'Interval' - this is how often the ELB will perform the above health check.
'Timeout' - this is how long to allow the instance to respond to the health check before a failure is recorded.
'Unhealthy threshold' - is the number of *consecutive* failures before the ELB marks the instance as 'failed' and 'OutOfService'.
'Health threshold' is the number of successful checks before the ELB marks the instance as 'working' and 'InService' 

Create CNAME Record for ELB Instance 

When an ELB instance is first created, AWS provides you a public DNS name. It is unique but quite long and difficult to remember. Much better to have something a bit friendly for it. To do this, you will need to create a CNAME record in DNS to redirect the friendly URL to this DNS name. I blogged about this in my previous post.

Thursday 16 September 2010

More Cloud
Back from leave and onto our cloud work. We've been looking at moving our ArcGIS Server Instances to version 10.0 and it was with some eagerness that on monday, we switched the older ArcGIS Server 9.3.1 instance to ArcGIS Server 10.0 - we had the latter instance running in a staging environment for a couple of weeks and have to admit, it looks very good. 

All things are green and working very well. I am happy.  

There are a few things that I need to look at - there will be no more changes to the base image as we're now on ArcGIS Server 10 but a lot around the effective management of these AMIs. First job is to remove all the older 9.3.1 images and ensure that we're running the minimum number of images. It all costs the business money. 

So what are we looking at now? Well - it is auto scaling; I need a method by which the ArcGIS Server images we have can automatically increase their resources in response to an increase in demand while removing resources when the demand drops. 

I know AWS has this ability so I went looking and it is called 'auto scaling'. However, my first idea that this would add CPU resources to a specific EC2 instance is wrong. It does not. The auto scaling adds instances (whole virtual machines) to your pre-configured pool as required. 

OK so how does this auto scaling work? 

CloudWatch
This is Amazon's monitoring service - it is quite simple but for $0.015/CPU, one can keep track of your instance health on a variety of dimensions such as CPU utilisation, Disk I/O, Availability of RAM, etc.
It would be nice to plug into this data stream directly and use ActiveXperts or something to send warning emails and SMS to those who need it but I think this is limited at the moment. Now CloudWatch is indirectly required for auto scaling as you create new rules based on the metrics you get back - these rules include adding or removing instances as required.

Auto scaling
Auto scaling is free! However, one needs to use CloudWatch (which isn't free) before you can get auto scaling to work. So it isn't as free as one would think.

With auto-scaling, you first have to define an auto scaling group, with virtual machines as members. You then create pre-defined CloudWatch parameters and Amazon will automatically launch and add virtual servers when certain conditions and thresholds are met.

All this is done through the 
This even works with the AWS load balancer. Good stuff and quite easy to implement.

Yes, Amazon has load balancers available.

Elastic Load Balancing
For $0.025/CPU hour plus $0.08/GB data transfer you get an easy to configure and almost instantly available. To configure the load balancer, one maps the external DNS of your choosing (in this case an 'a' record) using a CNAME mapping to the DNS name provided by AWS.

However, there's a great article on why auto scaling without any capacity planning or control is a bad idea. I also learnt a new term while reading this article: being 'SlashDotted'.