In HowTo: Rotate Logs to S3 we saw how to rotate logs to S3, and amongst those logs were the logs for apache web servers. In that article, the S3 path had the website hard coded in the logrotate configuration file; here we will see how it can be set based off of an EC2 tag.

This article will talk about how to use logrotate to rotate your logs to S3. Here we specifically are using Gentoo Linux, and we can find EC2 AMIs on the Pygoscelis Papua page. We will be using s3cmd to actually move the files to S3, if you do not have it installed already, see HowTo: Install AWS CLI - Amazon Simple Storage Service (S3) - S3cmd.

While dnscurl.pl gives you access to Amazon Route 53, it does require reading and writing XML to actually interface with the service. cli53 from Barnaby Gray gives a simpler way to manage Route 53 on the command line as well as being able to work with BIND format zone files.

While s3curl.pl allows command line access to Amazon S3, it will only work one file at a time, we have to read XML for “directory” listings, and requires we know details about S3 (like how to list more than 1000 objects in a single request, or how to list a “directory”/start with a prefix). s3cmd, from Michal Ludvig, gives a much simpler way of interfacing with Amazon S3 on the command line.

Through this series we see how to extract information from the Elastic MapReduce ruby client, and use it to create the same command using the AWS CLI tool. In this article, we will look specifically at running an HBase database.