AWS (SQS) in a BASH script
» » » AWS (SQS) in a BASH script

AWS (SQS) in a BASH script

posted in: BASH Scripting, Coding | 0

Today I spent most of the day working on a BASH script to action messages from an AWS SQS queue.  Due to NDAs at wok I can’t go into the detail of what it does, but I’d like to cover some of the tools I’m using and the challenges I’ve encountered.

The first tool I used is created by Timothy Kay.  It is a command line tool for interfacing with a number of AWS services, but I am primarily using it to interface to SQS.  The tool is simply called AWS.  It is written in Perl, and based on my limited knowledge of Perl in the last 10 years, seems to be well written.  Some aspects of this script seem well documented, but the SQS aspect has limited documentation available.

To implement the queue I set-up two shell scripts.  The first shell script is fairly simple and straight forward. I intend to remove the sleep 10 from this script when I get a response to the issue I submitted at GitHub about not knowing how to implement long polling. The other problem I raised in this issue ticket is the auto deleting of messages when --exec is used the region is not passed to the the second phase of the script and the message is never deleted.

The second script I have written does all the grunt work. My producer (the script that populates the queue) provides JSON messages, so my second script utilises jq to interpret the message body and set local variables. I then perform all my required actions and upon completion I use a command like aws --region="us-west-2" send "/1234/queue-name" --message="${JSON}" to pass the required data to the next process. The second script also uses the same code to post messages to an different queue for logging and debugging.

The final aspect of the configuration is the implementation of a dead-letter queue. I have configured SQS to allow messages to be processed a maximum of 3 times before they are passed to the dead-letter queue. While I am testing my implementation I am not actioning the dead-letter queue, but prior to production I intend to connection it to an SNS topic I utilise for monitoring.

Update: 10 hours after publishing

Timothy Kay advised me that I was incorrectly using the --wait parameter. He also supplied a fix for the region not being passed to the delete phase of the script. My new code for the consumer follows.