Category Archives: DevOps

Google Cloud Network Load Balancer Thinks Server Is Unhealthy

Recently I bumped twice into an issue with Google Cloud Network Load Balancer in which it decided a server was unhealthy regardless of if there was a health check or not.
The first time I cloned the machine and started a new one – which obviously fixed the issue.

The second time I’ve decided to try a bit harder.

I’ve bumped into this old bug detailing a race condition in google-address-manager.

google-address-manager is part of the Google Daemon, a set of processes that runs inside an image running in Google Compute Engine.

If google-address-manager is down, Network Load Balancing may not work at all, marking your instance as unhealthy even though it is working well.

If that happens, you might want to restart the service but issuing:

sudo service google-address-manager restart

or

sudo /etc/init.d/google-address-manager restart

This should make sure Network Load Balancing will work as it should. Also note that without this service, SSH keys will not get provisioned.

While I do not know what the process crashed (I couldn’t find any reasonable log) I suspect it had something to do with low disk space on that drive.

MongoDB ReplicaSet Backup in the Cloud

MongoDB replicaset is a great way to handle scalability and redundancy. In the age of the cloud nodes are added and removed from a replicaset easily and quickly and in most cases all are created from the same image.

So how can we make sure that we are always running backup from a non MASTER replica set node?

Below is a small script that will only run backup on non master replica set node.

It will also archive and compress the backup and upload it to a Google Cloud Storage bucket. You can easily modify the last part to upload the file to an AWS S3 bucket using s3cp or s3cmd.

This is a template that works best for a typical small replica set – 2 nodes and an arbiter. You will install it on both nodes, schedule it using cron and it will only run on the non master one. Even if you flip the master role between servers the script will still work well without changing a thing.

A simple and elegant solution if I may say so myself 🙂