I’ve been playing with Amazon’s Simple Storage Service for several months now. In case you’re not familiar with S3, it’s a storage service for files or any kind of data that you want to put out there. The data can be kept out on Amazon’s servers for your personal use, or you can make them publicly available. For example, for some of my installed programs or content on my website, I store the static data, downloads, and other support files out on the Amazon S3 service. My web server on my shared hosting account is only used for application logic and processing requests. The static data is getting downloaded from (or will be downloaded from soon) the Amazon S3 servers. The benefit to this is that I can take advantage of the lower cost bandwidth fees and pay-for-what-you-use space on Amazon’s servers, and I can keep the main web application server for doing the more complex operations. Plus, in the future, I can expand and take advantage of Amazon’s CloudFront content delivery network to serve my content faster depending on where in the world people are located and viewing my web site.

Amazon’s S3 service has two APIs for interacting with the service and uploading or downloading data from the storage service. The first API is SOAP-based web services, and in the past this would have probably been the preferred way of interacting with S3. However, with .NET 3.5, the REST-based APIs are simple to implement and very easy to use. Over the next several posts, I’ll be showing you how to interact with different Amazon S3 services in order to store and retrieve data. I’ll also be explaining more about how I’m planning on utilizing S3 storage for the new web site that I’m building using Umbraco.

In this first post, I’ll show you how to connect to the Amazon S3 service to view the list of buckets associated with an account. For those new to the concepts of S3, S3 works by storing objects in buckets. A bucket might be similar to a disk drive, for example. It’s a place where you’re going to build a file system and store files. An object is a file or a BLOB. It’s basically a large group of bytes that can be anything that you want. With your Amazon S3 account, you can create any number of buckets that you want for different purposes, and you can store as many objects as you want in your bucket. Once the objects are in a bucket, they can be accessed over HTTP (or HTTPS) because Amazon S3 supports basic web server-like features.

In this first post, we’re going to assume that we have an existing Amazon S3 account with a number of buckets. Using the REST API, we’re going to query the Amazon S3 web service to get back the list of buckets associated with our account. The response from the Amazon S3 web service is going to look like this:

    
        ...
        ...
    
    
        
            clickonce.imaginaryrealities.com
            date
        
        
            downloads.imaginaryrealities.com
            date
        
        
            media.imaginaryrealities.com
            date
        
        
            symbols.imaginaryrealities.com
            date
        
        
            sourcecode.imaginaryrealities.com
            date
        
    

In the above example output, you can see that I have five buckets associated with my account. Bucket names are globally unique across all users of the Amazon S3 service, so it’s recommended to suffix the bucket names with a domain name. One added advantage of this is that I could create CNAME records in my DNS to redirect requests for media.imaginaryrealities.com, for example, to media.imaginaryrealities.com.s3.amazonaws.com. This allows my to specify URLs using media.imaginaryrealities.com without ever having to let anyone in on the fact that I’m using Amazon S3.

The next trick is that we have to define the WCF service contract interface to query the Amazon S3 service:

/// 

/// Defines the operations for the Amazon Storage Services' REST API.
/// 
[ServiceContract(Name = "Amazon Storage Service")]
public interface IAmazonStorageService
{
    /// 

    /// Queries the Amazon Storage Service for the list of buckets for
    /// the specified account.
    /// 
    /// 
    /// Returns a  object containing
    /// the list of buckets.
    /// 
    [OperationContract]
    [WebGet(UriTemplate = "/")]
    ListAllMyBucketsResult GetBuckets();
}

And the data contracts are shown below:

using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Runtime.Serialization;

/// 

/// Response from the 
/// web service operation.
/// 
[DataContract(
    Namespace = "http://s3.amazonaws.com/doc/2006-03-01/",
    Name = "ListAllMyBucketsResult")]
[SuppressMessage(
    "Microsoft.Performance",
    "CA1812:AvoidUninstantiatedInternalClasses",
    Justification = "This class is instantiated by WCF.")]
public class ListAllMyBucketsResult
{
    /// 

    /// Gets or sets the owner of the account.
    /// 
    /// 
    /// The value of this property is an  object
    /// describing the account owner.
    /// 
    [DataMember(Order = 1)]
    public Owner Owner
    {
        get;
        set;
    }

    /// 

    /// Gets or sets the list of buckets created on the account.
    /// 
    /// 
    /// The value of this property is a  object
    /// containing the list of buckets.
    /// 
    [DataMember(Order = 2, Name = "Buckets")]
    public BucketList Buckets
    {
        get;
        set;
    }
}


/// 

/// Stores information about the Amazon Web Services account owner.
/// 
[SuppressMessage(
    "Microsoft.Performance",
    "CA1812:AvoidUninstantiatedInternalClasses",
    Justification = "This class is instantiated by WCF.")]
[DataContract(Namespace = "http://s3.amazonaws.com/doc/2006-03-01/")]
public class Owner
{
    /// 

    /// Gets or sets the display name of the owner.
    /// 
    [DataMember(Order = 2)]
    public string DisplayName
    {
        get;
        set;
    }

    /// 

    /// Gets or sets the unique identifier of the owner.
    /// 
    [DataMember(Order = 1, Name = "ID")]
    public string Id
    {
        get;
        set;
    }
}

/// 

/// Collection of  objects.
/// 
[CollectionDataContract(
    Namespace = "http://s3.amazonaws.com/doc/2006-03-01/")]
[SuppressMessage(
    "Microsoft.Performance",
    "CA1812:AvoidUninstantiatedInternalClasses",
    Justification = "This class is instantiated by WCF.")]
public class BucketList : List
{
}

/// 

/// Stores information about a bucket on the Amazon Storage Service.
/// 
[SuppressMessage(
    "Microsoft.Performance",
    "CA1812:AvoidUninstantiatedInternalClasses",
    Justification = "This class is instantiated by WCF.")]
[DataContract(
    Namespace = "http://s3.amazonaws.com/doc/2006-03-01/")]
public class Bucket
{
    /// 

    /// Gets or sets the date and time that the bucket was created.
    /// 
    /// 
    /// The value of this property is a  value
    /// specifying when the bucket was created.
    /// 
    [DataMember(Order = 2)]
    public DateTime CreationDate
    {
        get;
        set;
    }

    /// 

    /// Gets or sets the name of the bucket.
    /// 
    /// 
    /// The value of this property is the name of the bucket.
    /// 
    [DataMember(Order = 1)]
    public string Name
    {
        get;
        set;
    }
}

If you’re looking at the technical documentation for the Amazon S3 web service, please be sure to note that the XML namespace for the ListAllMyBucketsResult data structure is incorrect. The correct namespace is http://s3.amazonaws.com/doc/2006-03-01/.

With the service contract defined and the data contracts implemented, we should be all set now to call the web service and get back the list of buckets, correct? Well, almost. First, there’s a little hurdle that we need to get past called authentication. We don’t want anyone to be manipulating our Amazon S3 account, so we want to make sure that Amazon Web Services know who we are. To do that, we need to authenticate our request.

In the Amazon S3 REST API, the authentication token is an HMAC-SHA1 signature of a parameter string that is used to describe the request. Before submitting the request, we’re going to create the signature. Then we’re going to attach the signature to the Authorization HTTP header, and then submit the request. The signature is generated using a secret key that is shared between us and the Amazon Web Services service. When you sign up for an Amazon S3 account, you’ll receive your secret key and will also receive a second value, the access key identifier, that is used to identify yourself to the Amazon Web Services service.

Were going to start off by creating the client proxy that we’ll use to communicate with the Amazon S3 web service:

var serviceUri = new Uri("http://s3.amazonaws.com");
var channelFactory =
    new WebChannelFactory(serviceUri);
var service = channelFactory.CreateChannel();

Next, we need to generate the authentication signature. The authentication signature is going to look like this:

GET

application/xml; charset=utf-8

x-amz-date:Sat, 07 Feb 2009 20:00:00 GMT
/

Notice the x-amz-date code in the signature text. x-amz-date is a special Amazon Web Services HTTP header that is used to communicate the time stamp for the request over the wire. The time stamp is used during authentication to prevent replay attacks on your account. Amazon requires the time stamp to be within fifteen minutes of the system clock on the Amazon Web Services servers.

The code to build our signature is below:

var key = Encoding.UTF8.GetBytes(secretAccessKey);
var hmacsha1 = new HMACSHA1(key);

var stringToSign = new StringBuilder();
var timestamp = DateTime.UtcNow.ToString(
    "R",
    CultureInfo.InvariantCulture);
stringToSign.Append(
    "GET\n\napplication/xml; charset=utf-8\n\nx-amz-date:")
    .Append(timestamp)
    .Append("\n/");
var bytesToSign =
    Encoding.UTF8.GetBytes(stringToSign.ToString());
var signatureBytes = hmacsha1.ComputeHash(bytesToSign);
var signature = Convert.ToBase64String(signatureBytes);
var authorization = String.Format(
    CultureInfo.InvariantCulture,
    "AWS {0}:{1}",
    accessKeyId,
    signature);

First, I took the secret access key that Amazon gave to me and I converted that into a byte array using Encoding.UTF8.GetBytes(). I then initialized the HMACSHA1 object using the bytes for the secret key. Next, I created the time stamp by taking the local time, converting it to UTC time, and then converting the time stamp into the RFC 1123 format. I build the signature string on lines 8 through 11. I converted the signature string into a UTF-8-encoded byte array. I then signed the byte array using the HMAC-SHA1 algorithm. Finally, I converted the signature to a BASE-64 string to send to the Amazon S3 web service.

With the signature calculated, we can authenticate our request. Now for sending the message to the Amazon S3 web service:

using (new OperationContextScope((IClientChannel) service))
{
    var context = WebOperationContext.Current;
    context.OutgoingRequest.Headers.Add(
        "Authorization",
        authorization);
    context.OutgoingRequest.Headers.Add(
        "x-amz-date",
        timestamp);

    result = service.GetBuckets();
}

To authenticate, I need to add the signature that I calculated to the request as an HTTP header. I also needed to send the time stamp that I used to sign the request, because the Amazon S3 web service will need that to validate my signature. To send the HTTP header, I created a new OperationContextScope object. I then added the Authorization and x-amz-date HTTP headers to the outgoing request’s HTTP headers. Finally, with the headers in place, I was able to make the web service call and get back the list of buckets.

In my next post, I’ll show you how to upload an object to a bucket using the WCF REST APIs.