MongoDB replicaset is a great way to handle scalability and redundancy. In the age of the cloud nodes are added and removed from a replicaset easily and quickly and in most cases all are created from the same image.
So how can we make sure that we are always running backup from a non MASTER replica set node?
Below is a small script that will only run backup on non master replica set node.
It will also archive and compress the backup and upload it to a Google Cloud Storage bucket. You can easily modify the last part to upload the file to an AWS S3 bucket using s3cp or s3cmd.
This is a template that works best for a typical small replica set – 2 nodes and an arbiter. You will install it on both nodes, schedule it using cron and it will only run on the non master one. Even if you flip the master role between servers the script will still work well without changing a thing.
A simple and elegant solution if I may say so myself 🙂
Ever since Amazon introduced tags in EC2 I felt such a relief that I can name my instances and actually remember which one is which.
It took a while for Name tags to find its way to various aspects of the AWS Console, however, connecting to machines still requires a way to find the IP address from the console or via a command line using the EC2 API Tools.
I thought that it would be much easier for me and others to utilize the Name tags to connect more easily to the machines.
Initially, the script was a simple bash script which utilizes a the ec2-describe-instances command with a flag that matched the Name attribute, however, managed various other command line parameters such as the user name to connect with (ubuntu images, for example, users the ‘ubuntu’ user instead of ‘root’. Amazon Linux AMI uses ‘ec2user’, etc) so I’ve decided to rewrite it in Python and use the magnificent Boto Python library.
This allowed better argument handling as well as remove the need to install Java and the EC2 API Tools to access the EC2 API.
Grab the code here
Don’t forget to send some feedback!
UPDATE (2011-07-16): I just got a newsletter Email from Amazon stating that they have added SQS and SNS to CloudWatch which allows monitor SQS queues not just for the length of the queue, but for others metrics as well, so there is no real need in my script. Unless you really really want to use it 🙂
All you have to do is select SQS in the metrics type drop down and you will see a set of metrics to select from for all of your existing queues.
Amazon’s CloudWatch is a great tool for monitor various aspects of your service. Last May Amazon introduced custom metrics to CloudWatch which allows sending any metrics data you wish to CloudWatch. You can then store it, plot it and also create CloudWatch Alerts based on it.
One of the things missing from CloudWatch is Simple Queue Service (SQS) monitoring, so I’ve written a small script to update a queue’s count in a CloudWatch custom metric.
Having the queue’s count in CloudWatch allows adding alerts and actions based on the queue’s length.
For example, if the queue’s length is above a certain amount of a certain period of time, one of 2 things happened:
- There is a bug in the code causing the worker processes that process the queue’s message to fail
- There is a higher than usual load on the system causing the queue fill up and get more and more messages while there aren’t enough worker processes to process these messages in reasonable time
If the load is higher than usual you can easily tell via a CloudWatch alert to add an additional machine instance running more worker processes or simply send an Email alert saying there is something wrong.
The script is very easy to use and can be run from a cron job. I’m running it as a cron job in 1 minute intervals and have set up various CloudWatch alerts to better monitor my queue.
Grab the script on Github at: SQS Cloud Watch Queue Count.