Category Archives: Amazon ElastiCache

Membase Cluster instead of ElastiCache in 5 minutes

Want to have an ElastiCache like service in all regions, not just US-EAST?Memby
Want to utilize the new reserved instances utilization model to lower your costs?
Want to have your cache persistent and easily backup and restore it?
Want to add (or remove) servers from your cache cluster with no down time?

We have just the solution for you. A simple CloudFormation template to create a Membase cluster which gives you all of the ElasticCache benefits and a lot more including:

  • Support for ALL regions (not just US-EAST Apparently Amazon beat me to the punch with support at US West (N. California), EU West (Dublin), Asia Pacific (Singapore), and Asia Pacific (Tokyo))
  • Support for reserved instances including the new utilization based model
  • Supports adding and removing servers to the cluster with no downtime and automatic rebalancing of keys amont the cluster’s servers
  • Support persistency (if you wish)
  • Supports multi-tenancy and SASL authentication
  • Supports micro instances (not recommended, but good for low volume environments and testing environments)
  • Easily backup and restore
  • Install the cluster in multiple availability zones and regions for maximum survivability and have all of them talk to each other
  • Using a vBucket aware memcache client or running through a Moxi proxy changes in topology will get communicated automatically with no need for code or configuration changes!
  • No need for a single address (the CNAME you get from ElastiCache) because if you are using a vBucket aware client (or going through Moxi Proxy) topology changes are communicated to your client.
Based on the CloudFormation scripts created by Couchbase Labs, this script is more generic and utilize the complete available RAM (80% of available ram) for each type of instance.

Notes:

  • All instances run the latest Amazon Linux (2011-09)
  • 64bit instances use instance-store for storage
  • m1.micro uses 32bit to utilize maximum amount of RAM (we don’t want those 64bit pointers eating our RAM)
  • m1.small is 32bit and while it does have instance-store support, we wanted to have a one formation script to rule them all so it uses an EBS supported AMI.
  • The CloudFormation script changes the names of the instance to their public DNS names so they are available from anywhere in the world and any Availability Zone and Region in AWS so you can have repl

Security (VERY IMPORTANT!):

  • The default script will create a security group for you which allows access to ALL of the servers.
  • If you created a default bucket via the script – that bucket, which uses port 11211 is open to everyone. Make sure to protect it or delete it and create a bucket with SASL protection.
  • In general, if you are creating a non SASL protected bucket, make sure to protect the port by updating the security group!

Download:

Instructions:

  1. Grab one of the existing templates from the GitHub repository or run gen-pack.py to generate a template with any number of instances in it.
  2. Go to the CloudFromation Console
  3. Click on “Create New Stack”
  4. Name your stack
  5. Select “Upload a Temaplate File”
  6. Click “Browse” and select the template file (use one of the defaults named “membase-pack-X”)
  7. Click “Continue”
  8. In “Specify Parameters” step the minimum parameters you are required to fill are:
    • RESTPassword – the password for the REST API and management console
    • KeyName – the name of the KeyPair you are using when creating instances (usually the name of the .pem file without the “.pem” suffix)
    • Instance Type – choose the size of the instance (t1.micro by default)
  9. Click “Continue”
  10. Sit back and relax. CloudFormation will do the rest of the work for you
  11. Select the “Outputs” tab (click refresh if it doesn’t show anything). It should contain the URL of the Membase management console
  12. Clap your hands with joy!

The cloud is more than just the hardware – its also about services!

When talking about the cloud, most people talk about running the servers on the cloud. They talk about the fact that they can start or stop virtual servers with a simple API call.

Cloud Services Toolbox
taken by keepthebyte

A lot of Cloud Providers do provide you with virtual servers, virtual load balancer and infinite (and/or scalable) storage, but these are all building block of servers. You still need to do the heavy lifting of taking a bunch of servers and making it do the work for you. You need to install the software (not just the one you wrote), mange, handling disaster recovery, backups, logging, etc.

What makes Amazon’s cloud unique on top of the servers and storage they provide is the fact that it provides a set of cloud services that can be used as black boxes for your application/service reducing the code you need to write/maintain as well as the number of servers you need to administer. These services are also available for use outside of Amazon’s cloud but there are benefits of using them within your EC2 instances.

The Amazon non-hardware cloud services offering is split into two categories:

  • Off the shelf services – These are preconfigured services that Amazon takes most of the hassle off of managing it. Such services are
    • Amazon Relational Database Services (RDS) – Hosted MySQL which Amazon manage, patches and helps you backup and restore as well as deploy in a smart way across multiple availability zones.
    • Amazon ElastiCache – Hosted Memcached which Amazon manages, allows to resize (add or remove servers) as well as patch with the latest software.
  • Development Building Blocks (Black Boxes) – Web services that provides functionality which you can mix and match to create you service, removing the need for you to handle the load, machines and configuration of these services. Such services are:
    • Amazon SimpleDB – an Amazon written key/value datastore that is hosted and operated by Amazon. Scalable and simple. It just works
    • Amazon Simple Queue Service (SQS) – A reliable, scalable, persistent queue service. Great for talking with distributed async components.
    • Amazon Simple Email Service (SES) – A scalable Email service which allows you to send Email (for humans and machines)
    • Amazon Simple Notification Service (SNS) –  A web service to send notifications to people/machines/service/etc. It allows to send the notification out as an Email or as an HTTP request as well as post it to an SQS queue

What I love the most about Amazon Web Services is the fact that when they do provide a certain Development Building Block such as Simple Email Service (SES), they do so without killing or harming the competition. There is still enough value and features in other Email services such as mailgun, SendGrid and MailChimp for them to co-exist with SES.

Not stepping (too much) on web services developers toes is not something to dismiss and I would love to see the innovation that comes out of Cloud based web services in the future.

Go cloud services! 🙂