I’ve recently encountered a use case where I need to reliably send a message to a worker process that should handle the following requirements:
- Message should persist until the action it specified is done.
- Message should be processed (or wait to be processed) when the worker processes have a bug or are down
- Message processing should be as fast as possible – process the message ASAP.
Using Amazon’s Simple Queue Service (SQS)
SQS can provide the persistency needed. The message will be available until it is deleted (at least up to ~3 days) even if the worker processes are down or have a bug.
Using Amazon’s Simple Notification Service (SNS)
- Create a topic
- Subscribe an SQS queue to that topic
- Subscribe a worker process that work via HTTP/S to that topic (for increased reliability this can be an Elastic Load Balancer (ELB) that hides a set of machines)
- Submit a message to the topic
- SNS will publish the message to the SQS queue
- SNS will then notify via HTTP/S POST to the handler to start processing the message
This solution covers the original 3 requirements of message reliability, handling cases where workers are down or have bugs and handling messages as soon as they are sent.