About this blog

'Going Spatial' is my personal blog, the views on this site are entirely my own and should in no way be attributed to anyone else or as the opinion of any organisation.

My tweets on GIS, Humanitarian, Tech, Games and Randomness

Thursday, 25 July 2013

Setting up your windows box to run the EC2 CLI and then using it for ArcGIS Server!

Installing the Amazon EC2 Command Line Tools on Windows

Not quite my command line but kinda cool....
In my own style, I will describe how to install the Amazon EC2 command line tools onto a Windows 7 Box. If you're using another version of Windows or Linux, the steps will be slightly different. I intend to install and configure the same tools on my Mint Linux box in the near future. Once completed, I am going to use a few easy to understand scripts to perform some automated steps to my ArcGIS Server 10.1 AWS instance. Soon you will be a guru. All from the comfort of my command line.

 

Why the command line?

Yes indeed a very good question, why use the command line - surely everything is point and click (or look and blink)? Well, there are probably dozens of reasons but for me it is: quicker, easy to script (and hence batch up and repeat), one has more control over options and it is just more efficient. You can also perform a lot of your work on a number of different computers and location; for example via SSH to a Linux box. Finally, being able to master (or at least) tame the command line will give you, the user, more confidence over the box(es) in front of you. 

There, pep talk over. Onto the installation steps!

Process for Installing the Command Line API Tools

Step 1: Download the Command Line Tools (API Tools) zip file
The command line tools are available as a ZIP file on this site: Amazon EC2 CLI Tools. The tools are written in Java and include shell scripts for both Windows and Linux. The ZIP file is self-contained and you simply download the file and then unzip it. I popped the files to D:\TOOLS\AWS but anywhere sensible will do.

Some additional setup is required before you can use the tools; you didn't think it would be that easy do you? These extra steps are discussed next.

Step 2: Set the JAVA_HOME Environment Variable
The Amazon EC2 command line tools read an environment variable (JAVA_HOME) computer to locate the Java runtime. Environmental variables makes things so much easier; you avoid the need to tap out the entire path for a program to execute which saves time and your keyboard! So get to it.

Step 3: To set the JAVA_HOME environment variable
If you don't have Java 1.7 or later installed then download and install Java. To view and download JREs for a range of platforms, go to http://java.oracle.com
  1. Set JAVA_HOME to the full path of the Java home directory. This is the directory that contains the 'bin' directory that contains the Java executable you installed (java.exe). For example, if your Java executable is in D:\JAVA\BIN, set JAVA_HOME to D:\JAVA 
  2. Don't include the bin directory in JAVA_HOME! This is a common mistake, but the command line tools won't work if you do this! I know, I have tried this and learnt my lesson.
  3. Open a new Command Prompt window and verify your JAVA_HOME setting using this command.
  4. C:\%JAVA_HOME%\bin\java -version
    If you've set the environment variable correctly, the output looks something like this.
    Love the Matrix-like colours
     Otherwise, check the setting of JAVA_HOME, fix any errors, open a new Command Prompt window, and try the command again and also, try some of the following:
    • Add the BIN directory, that contains the Java executables to your path.
    • In System variables, select Path, and then click Edit. 
    • In Variable values, before any other versions of Java add ;%JAVA_HOME%\bin.  
    • Reboot?

    Step 4: Set the AWS_ACCESS_KEY and AWS_SECRET_KEY Environment Variables
    When you sign up for an AWS account, AWS creates access credentials for you so that you can make secure requests to AWS without the need to type in your username and password. You must provide these credentials to the Amazon EC2 command line tools so that they know that the commands that you issue come from your account.

    The following steps describes how you can view your access credentials before setting them up to be used by your scripts etc. Keep these credentials as safe; if someone can nick them and use them for nefarious purposes - it is on your head. To improve it, create a new AWS EC2 user account and allocate just resource access rights and no more. There's another post coming on how I use AWS Integrated Access Management (IAM) resources.

    Step 5: To view your AWS access credentials
    1. Go to the Amazon Web Services website at http://aws.amazon.com.
    2. Click My Account/Console, and then click Security Credentials.
    3. Under Your Account, click Security Credentials.
    4. In the spaces provided, type your user name and password, and then click Sign in using our secure server.
    5. Under Access Credentials, on the Access Keys tab, your access key ID is displayed. To view your secret key, under Secret Access Key, click Show.
    You can specify these credentials with the --aws-access-key and --aws-secret-key (or -O and -W) options every time you issue a command. However, it's easier to specify your access credentials using the following environment variables:
    • AWS_ACCESS_KEY—Your access key ID
    • AWS_SECRET_KEY—Your secret access key
    If these environment variables are set properly, their values serve as the default values for these required options, so you can omit them from the command line. This saves typing them in each time AND keeps the keys secure as they are now referenced only as environmental variables. However, definitely keep a copy of both keys in a secure fashion and not copied to a postit note!

    Right, let's create these environmental variables that specify your access credentials:

    Step 6: To set up your environment variables (it's dead easy...)
    1. On the computer you are using to connect to Amazon Web Services, click Start, right-click Computer, and then click Properties
    2. Click Advanced system settings.
    3. Click Environment Variables.
    4. Under System variables, click New.
    5. In Variable name, type 'AWS_ACCESS_KEY'.
    6. In Variable value, specify your access key ID.
    7. Under System variables, click New.
    8. In Variable name, type 'AWS_SECRET_KEY'. 
    9. In Variable value, specify your secret access key.
    Step 7: Set the EC2_HOME Environment Variable
    The Amazon EC2 command line tools read an environment variable (EC2_HOME) to locate supporting libraries. You'll need to set this environment variable before you can use the tools with confidence from the command line. You can do it. 

    Step 8: To set the EC2_HOME environment variable
    1. Click Advanced system settings. 
    2. Click Environment Variables. 
    3. Under System variables, click New. 
    4. In Variable name, type EC2_HOME. 
    5. In Variable value, type the path to the directory where you installed the command line tools. For example, C:\AWS\EC2\ec2-api-tools-1.5.4.0
    6. Open a new Command Prompt window and verify your EC2_HOME setting using this command:
    1. C:\dir %EC2_HOME% 

    If you get a File Not Found error, check the setting of EC2_HOME, fix any errors, open a new Command Prompt window, and try the command again. Typos mate, I reckon - it's typos....
    1. Add the bin directory for the tools to your system Path environment variable. The rest of this guide assumes that you've done this.
      You can update your Path as follows:
      1. In System variables, select Path, and then click Edit.
      1. In Variable values, add ;%EC2_HOME%\bin.
      Open a new Command Prompt window and verify your update to the Path environment variable using this command.

      C:\> ec2-describe-regions 
      If all your environment variables are set correctly, you'll see output that looks something like this.
      If you get an error that this command is not recognized as an internal or external command, check the setting of Path, fix any errors, open a new Command Prompt window, and try the command again. Trust me, it's the path variable; there's a typo!

      If you get an error that required option -O is missing, check the setting of AWS_ACCESS_KEY, fix any errors, open a new Command Prompt window, and try the command again.

      If you get an error that required option -W is missing, check the setting of AWS_SECRET_KEY, fix any errors, open a new Command Prompt window, and try the command again.
        Step 9: Set the Region
        By default, the Amazon EC2 tools use the US East (Northern Virginia) Region (us-east-1) with the ec2.us-east-1.amazonaws.com service endpoint URL. If your instances are in a different region (like ours), you must specify the region where your instances reside. For example, if your instances are in Europe, you must specify the eu-west-1 Region by using the --region eu-west-1 option or by setting the EC2_URL environment variable.

        This section describes how to permanently specify a different region by changing the service endpoint URL.

        Step 10: To specify a different region

        1. To view available Regions, see Regions and Endpoints in the Amazon Web Services General Reference.
        2. To change the service endpoint, set the EC2_URL environment variable.
        3. The following example sets EC2_URL to the EU (Ireland) Region.
          • On the computer you'll use to connect to Amazon Web Services, click Start, right-click Computer, and then click Properties.
          • Click Advanced system settings.
          • Click Environment Variables.
          • Under System variables, click New.
          • In Variable name, type EC2_URL.
          • In Variable value, type https://ec2.eu-west-1.amazonaws.com.
        Boom, you're in the EU region!

        Basic Amazon EC2 scripting examples

        Right, yes - you're now on the way to being a command line guru. However, let's get something useful and ops related. The command line is there to help you execute your regular ops more efficiently. So, below are a few simple scripts for some of our common workflows:

        1) stop an instance (our regular 18:00 evening script),
        2) start an instance and associate an elastic IP (our regular 'Good morning' script),
        3) stop multiple instances and then terminate them.

        Stop an instance

        The simplest script is one that stops an already running instance. You can use the AWS Dashboard but what if you need to stop it in the evening? Of course, you can login each night but you must have better things to do. I know I would prefer playing computer games in the evenings, safe in the knowledge that my scripts are executing my commands automatically. The simple example calls the EC2-STOP-INSTANCES command and passes the instance name that I want to stop. Also, make sure that the AWS credentials you are using have the necessary permissions. You need to get into AWS IAM to manage this and I will get together a new post on this. For now, am assuming you're an IAM expert, or knows someone who is. Back to the script. I’m using sample values for instance names and other parameters, you would simply substitute your information for the placeholders in the examples below.

        ec2-stop-instances i-abcd1234

        Easy, right? In this case we just specify the stop command and an instance name, and the instance is stopped. The instance ID is specific to your instance and is NOT the host name of the instance nor is it the public DNS name. You need to check the instance-id. Check out the screenshot below: 
        'scuse the red ink but I love using paint...
        Ok, now let’s try something a little more interesting.

         

        Start an instance and associate an elastic IP

        This next script starts an instance, then associates an elastic IP address with that instance. This is a very common workflow because when you stop and start an EC2 instance, the IP address changes. Don't ask me why, but it does - the public IP address and the FQDN is NOT permanent when it comes to EC2 instances. However, you may want to permanently 'anchor' an IP address. An Amazon Elastic IP gives you a way to maintain an unchanging address for your instances. The only requirement is that whenever you stop and start the instance, you have to re-associate the address. Here’s the script to do it:

        call ec2-start-instances i-a5cw1214 timeout 300 ec2-associate-address 0.0.0.0 -i i-qwesdff4 ec2-reboot-instance i-aee1884

        The first line calls the EC2-START-INSTANCES command, which has a similar syntax to the EC2-STOP_INSTANCES command showed above. In this case, since we want to run several commands in sequence, we run the EC2 command in its own shell using the CALL command. The EC2 batch commands are actually shell scripts that run Java code, and CALL is needed to return control to the original batch file. Yes, my Padawan, the scripting starts to get more useful. If the CALL command is not used, the batch process terminates after the first EC2 command (ec2-start-instances), and the rest of the commands are never executed and that is a bad thing. There are other options but this is the most elegant.

        What else? The TIMEOUT command  waits five minutes (300 seconds) to give the instance time to start. It is quite generous but some instances that we have take even longer, so do your research. When an instance is first started, the status is classed as “pending” until the EC2 startup process is complete. When it reaches this stage the startup process hands off to the next process and the status changes to “running.” An elastic IP can only be associated with a running instance.

        Next, EC2-ASSOCIATE_ADDRESS is used to associate the elastic IP with the instance, and, finally, the last command in this script reboots the instance. Rebooting is required for ArcGIS Server to recognize the elastic IP. Rebooting is not needed for an enterprise geodatabase instance.

         

        Start/stop several instances

        Of course these commands can be chained together to stop or start several instances. One way is to add a separate command for each instance; however, the EC2 commands enable you to reference several instances in the same command. This is very handy. Saves typing. See the example below, which would stop three instances:

        ec2-stop-instances i-aopl4434 i-yygh1234 i-tykl2334

         

        Scheduling scripts

        Once you have some scripts to work with your EC2 instances, you can use operating system tools to schedule them to run at regular times (such as Friday nights or Monday mornings). For example, Windows Task Scheduler is available on the various desktop and server editions of Windows and gives you an easy-to-use GUI environment for scheduling a script.

        You can set a BAT file to run in the Task Scheduler by creating a Basic Task (Action> Create Basic Task). Once you name your task and set the schedule, you’ll be prompted to select an action to perform. Choose Start a Program and point to your BAT file. Your script is now set to run.

        Keep in mind that you can always manually launch a script by just double-clicking it, or running it from the task scheduler. An advantage of running the script from the Task Scheduler is that a history will be recorded of when the task was run. This information is accessible in the properties of any task under the History tab.

         

        Summary

        In summary, Amazon EC2 includes scripting tools that can help you automate your work. Scripts make it easier to administer your servers remotely, and, in many cases, allow you to cut costs. Once you have a useful script, you can use operating system tools to run it automatically.

        This post is meant as an introduction to scripting, but going further you can use scripting to launch or destroy new instances, create security groups, attach volumes, and so on. Stay tuned to this blog for more scripting tips.