Category Archives: Code

Using Google Cloud SQL from Go with GORM in Google Container Engine and Google Compute Engine

One of the things I find a bit difficult with CloudSQL is surprisingly the first thing you need to do – connect.

Being able to connect securely to your CloudSQL instance is sometimes a bit challenging not just in Go, but in Python as well.

In Go, I’ve found out conflicting suggestions and examples and after much trial and error I found a reasonable solution that works well.

After much searching I found CloudSQL Proxy which takes all of the IP white listing and SSL mess away by using direct OAuth APIs to obtain access to the instance.

While my example works specifically with GORM, which requires a nice database connection string, there are examples of using plain database/sql syntax with CloudSQL Proxy.

I know it looks simple enough, but without importing cloudsql-proxy it wouldn’t have been possible to use the “[email protected](project-id:zone:instance-name)/db” syntax style that is scattered around the net.

MongoDB ReplicaSet Backup in the Cloud

MongoDB replicaset is a great way to handle scalability and redundancy. In the age of the cloud nodes are added and removed from a replicaset easily and quickly and in most cases all are created from the same image.

So how can we make sure that we are always running backup from a non MASTER replica set node?

Below is a small script that will only run backup on non master replica set node.

It will also archive and compress the backup and upload it to a Google Cloud Storage bucket. You can easily modify the last part to upload the file to an AWS S3 bucket using s3cp or s3cmd.

This is a template that works best for a typical small replica set – 2 nodes and an arbiter. You will install it on both nodes, schedule it using cron and it will only run on the non master one. Even if you flip the master role between servers the script will still work well without changing a thing.

A simple and elegant solution if I may say so myself 🙂

Quick & Dirty API for Accessing Amazon Web Services (AWS) EC2 Pricing Data

Continuing my post about the JSON files used in the Amazon EC2 Page, I’ve created a small Python library that also acts as a command line interface to get the data.
The data in the JSON files does not contain the same values as the EC2 API for things like region name and instance types so the library/cli translates these values to their corresponding values in the EC2 API.

You can filter the output by region, instance type and OS type.

The command line output support a human readable table format, JSON and CSV.

To use the command line you’ll need to install the following Python libraries:

  • argparse – only if you are using Python < 2.7 (argparse is included in Python 2.7 and 3.x)
  • prettytable – if you want to print the human readable and pretty good looking ASCII table output

Both libraries can be installed using pip.
Grab the code from Github

Don’t forget to send feedback! Enjoy!

 

Membase Cluster instead of ElastiCache in 5 minutes

Want to have an ElastiCache like service in all regions, not just US-EAST?Memby
Want to utilize the new reserved instances utilization model to lower your costs?
Want to have your cache persistent and easily backup and restore it?
Want to add (or remove) servers from your cache cluster with no down time?

We have just the solution for you. A simple CloudFormation template to create a Membase cluster which gives you all of the ElasticCache benefits and a lot more including:

  • Support for ALL regions (not just US-EAST Apparently Amazon beat me to the punch with support at US West (N. California), EU West (Dublin), Asia Pacific (Singapore), and Asia Pacific (Tokyo))
  • Support for reserved instances including the new utilization based model
  • Supports adding and removing servers to the cluster with no downtime and automatic rebalancing of keys amont the cluster’s servers
  • Support persistency (if you wish)
  • Supports multi-tenancy and SASL authentication
  • Supports micro instances (not recommended, but good for low volume environments and testing environments)
  • Easily backup and restore
  • Install the cluster in multiple availability zones and regions for maximum survivability and have all of them talk to each other
  • Using a vBucket aware memcache client or running through a Moxi proxy changes in topology will get communicated automatically with no need for code or configuration changes!
  • No need for a single address (the CNAME you get from ElastiCache) because if you are using a vBucket aware client (or going through Moxi Proxy) topology changes are communicated to your client.
Based on the CloudFormation scripts created by Couchbase Labs, this script is more generic and utilize the complete available RAM (80% of available ram) for each type of instance.

Notes:

  • All instances run the latest Amazon Linux (2011-09)
  • 64bit instances use instance-store for storage
  • m1.micro uses 32bit to utilize maximum amount of RAM (we don’t want those 64bit pointers eating our RAM)
  • m1.small is 32bit and while it does have instance-store support, we wanted to have a one formation script to rule them all so it uses an EBS supported AMI.
  • The CloudFormation script changes the names of the instance to their public DNS names so they are available from anywhere in the world and any Availability Zone and Region in AWS so you can have repl

Security (VERY IMPORTANT!):

  • The default script will create a security group for you which allows access to ALL of the servers.
  • If you created a default bucket via the script – that bucket, which uses port 11211 is open to everyone. Make sure to protect it or delete it and create a bucket with SASL protection.
  • In general, if you are creating a non SASL protected bucket, make sure to protect the port by updating the security group!

Download:

Instructions:

  1. Grab one of the existing templates from the GitHub repository or run gen-pack.py to generate a template with any number of instances in it.
  2. Go to the CloudFromation Console
  3. Click on “Create New Stack”
  4. Name your stack
  5. Select “Upload a Temaplate File”
  6. Click “Browse” and select the template file (use one of the defaults named “membase-pack-X”)
  7. Click “Continue”
  8. In “Specify Parameters” step the minimum parameters you are required to fill are:
    • RESTPassword – the password for the REST API and management console
    • KeyName – the name of the KeyPair you are using when creating instances (usually the name of the .pem file without the “.pem” suffix)
    • Instance Type – choose the size of the instance (t1.micro by default)
  9. Click “Continue”
  10. Sit back and relax. CloudFormation will do the rest of the work for you
  11. Select the “Outputs” tab (click refresh if it doesn’t show anything). It should contain the URL of the Membase management console
  12. Clap your hands with joy!

Script to update Beanstalkd work queue statistics in CloudWatch

I’ve written a small Python script which uses the excellent boto libraryto monitor Beanstalkd server statistics as well as a specific beanstalkd tube (queue) statistics.

Beanstalkd
By Evil Cheese Scientist (Back Intact!)

You can get the code herehttps://github.com/erans/beanstalkdcloudwatch

The easiest way to run it is via a cron job. I run it every 1 minute to monitor the “reserved” and “buried” state of a few of the tubes I use (if you want to read more about how beanstalkd works I suggest reading the protocol document).

I highly recommending checking beanstalkd out if you need a queue for offloading tasks to other processes/machines. It’s really geared towards that task and have a couple of dedicated features such as the buried state, time-to-live and time-to-run which makes managing this really painless.

ec2ssh – Quicker SSH-ing Into Your EC2 Instances

Ever since Amazon introduced tags in EC2 I felt such a relief that I can name my instances and actually remember which one is which.

It took a while for Name tags to find its way to various aspects of the AWS Console, however, connecting to machines still requires a way to find the IP address from the console or via a command line using the EC2 API Tools.

I thought that it would be much easier for me and others to utilize the Name tags to connect more easily to the machines.

Initially, the script was a simple bash script which utilizes a the ec2-describe-instances command with a flag that matched the Name attribute, however, managed various other command line parameters such as the user name to connect with (ubuntu images, for example, users the ‘ubuntu’ user instead of ‘root’. Amazon Linux AMI uses ‘ec2user’, etc) so I’ve decided to rewrite it in Python and use the magnificent Boto Python library.

This allowed better argument handling as well as remove the need to install Java and the EC2 API Tools to access the EC2 API.

Grab the code here

Don’t forget to send some feedback!

 

Monitor Your Amazon Web Services Simple Queue Service (SQS) Queue Length in Amazon CloudWatch

UPDATE (2011-07-16): I just got a newsletter Email from Amazon stating that they have added SQS and SNS to CloudWatch which allows monitor SQS queues not just for the length of the queue, but for others metrics as well, so there is no real need in my script. Unless you really really want to use it 🙂

All you have to do is select SQS in the metrics type drop down and you will see a set of metrics to select from for all of your existing queues.

 

 

Amazon’s CloudWatch is a great tool for monitor various aspects of your service. Last May Amazon introduced custom metrics to CloudWatch which allows sending any metrics data you wish to CloudWatch. You can then store it, plot it and also create CloudWatch Alerts based on it.

One of the things missing from CloudWatch is Simple Queue Service (SQS) monitoring, so I’ve written a small script to update a queue’s count in a CloudWatch custom metric.
Having the queue’s count in CloudWatch allows adding alerts and actions based on the queue’s length.

For example, if the queue’s length is above a certain amount of a certain period of time, one of 2 things happened:

  1. There is a bug in the code causing the worker processes that process the queue’s message to fail
  2. There is a higher than usual load on the system causing the queue fill up and get more and more messages while there aren’t enough worker processes to process these messages in reasonable time

If the load is higher than usual you can easily tell via a CloudWatch alert to add an additional machine instance running more worker processes or simply send an Email alert saying there is something wrong.

The script is very easy to use and can be run from a cron job. I’m running it as a cron job in 1 minute intervals and have set up various CloudWatch alerts to better monitor my queue.

Grab the script on Github at:   SQS Cloud Watch Queue Count.