Amazon RDS Back Ups over VPC for Internal Network Storage

We have our hosting and database storage on the Amazon cloud.  This is a very strong, functional platform.  However like all remote systems, it can sometime be painful to get local copies of the data.  When your database size in calculated in the GB realm, you have very few options other than wait when making backups.

Amazons automated backups is a nice touch ensuring you always have a nightly version of your data in a mission critical case, but it also allows you the ability to be able to build a temporary RDS to export your data from, thus removing any impact on your production environments. This solution is great, but it is time consuming.  You need to build your RDS from the snap shot, change a bunch of settings to make it available to your client, perform a long winded mySQLdump to get your data out, then copy it back down across the wire.  That is time you can spend more wisely on other higher priority tasks.

So this is what we were tasked with.  Automate a backup of the data nightly from Amazons RDS through the leveraging of their CLI tools, and get it back into the office network to allow much faster access for local development setups.

Not long ago, we went through an entire Amazon Infrastructure rebuild implementing both a public and private VPC for our network in the office, this was clutch to make the following solution work.  Without this bridge in place you would still have to go through the drawn out process of copying your data back locally, or run the rusk of exposing file shares to a secured internal network to the world (which I highly suggest you don’t).


This utility will perform the following for you with no interaction required from yourself once it is configured and running.

  1. Find the most recent automatic backup for your RDS DB Instance Group by queuing the list of automatic snapshots, sorting them created date, and taking the most recent one.
  2. Mount this backup to a new RDS Instance
  3. Set the Security Group for the RDS Instance making it available to your SQL EXPORT server
  4. Set the DB Parameter Group for this instance to allow you to build custom functions through your sanitization script
  5. Export your database using mysqldump, and compressing it with gzip
  6. Copy this backup gzip to a mapped folder for an internal server within your corporate network
  7. Sanitize the database with the supplied SQL script
  8. Export and compress the sanitized version
  9. Copy the sanitized version to a different share point across your corporate network
  10. Destroy the RDS that was created.


What I will walk you through here below is the contents and scripts to make your lives MUCH simpler with you backups.  I am going to be writing this as if you have a somewhat sound understanding of Amazons Infrastructure and VPC configurations.  I will walk you through the steps needed to configure a working nightly backup that will also allow you to clean and sanitize any data that will be available to your developers.  Source for this can be found here ( for your reference / usage to get this setup running for your day to day usage.

A couple things I am going to assume before we start this:

  1. You have an Amazon account setup and know how to get into it.
  2. You have set up a public and a private VPC in the infrastructure of Amazon.  If you haven’t done this yet, Amazon has great documentation on how to go about setting this up and configuring it from start to finish.  You can find their guide here (
  3. You have your internal network setup within you private VPC.
  4. You have a database you want to export that has auto backups configured.
  5. You have created a “backup” user on your database that you wish to export with the following abilities:
    • show grants for ‘backup’@‘[host ip]’;


First off, lets get you setup and configured on an S3 instance for your exporting needs.  This doesn’t have to be a super large instance, although something with some power isn’t a bad move, I believe we went with a “t2.medium” and it seems to be able to handle the work load without too much of a problem.  Install the instance with an the standard Ubuntu 14.04 image available on Amazon.  This instance should exist in your PRIVATE VPC, without an PUBLIC IP address.  This box needs to be secure as it will be connecting to your internal network.

Next you will need to install a NAT TRANSLATION BOX into your public VPC and use this to allow the connection for updates from your PRIVATE SQL EXPORT box.  Amazon has a great walk through on this that can be located here (  Once you have this box set up and installed with the recommended image, you can now use this to set the access for your PRVATE SQL EXPORT box.

Moving back to your SQL EXPORT box, you will need to install the following packages by performing the following commands:

  1. sudo apt-get update
  2. sudo apt-get install mysql-client-5.5
  3. sudo apt-get install cifs-utils
  4. sudo apt-get install awscli

You should also go ahead and create a .my.cnf file located in your home directory for the ubuntu account on the system.  In this file you will need to include your backup users credentials to allow the mysql commands to fire from the command line without having to enter your login credentials, as well as preventing you from having to have them in the command and committed to Github, because we all know username and password in Github is a bad thing.  The file should contain the following:


Once you have these packages installed, you will need to configure your Amazon access with your Key ID and Secret Access Key ID.  Again, there is a great walk through on this located here: (

When you have your keys set up you will need to enter the following to configure your Amazon CLI access.
aws configure

This will prompt you for the following information.  Enter the information from the credentials on your previous step.
AWS Access Key ID [None]: XXXX123456789XXXX
Default region name [None]: [Your hosting region for your VPC]
Default output format [None]: table

Now you will have access through the Amazon CLI to you instances.  You will be able to run the scripts now against Amazon and have the control you need.

You should not proceed to mapping your drive into your internal server.  You again should create a local credentials file in your home directory to avoid having to store these in Github.

Create a file named “.credentials” in your home directory and enter:

username=[Your network user with access to your shared drive for read and write]
domain=[Your domain]

This will allow us to target this file when attempting to map the network drive.  So in your home directory, create a folder names “RDSBackups”.  Once you have this created run the following command to map the directory to the share on the server you are wanting to copy your backups to.

sudo mount //[Server IP]/RDSBackups$ /home/ubuntu/RDSBackups -t cifs -o credentials=/home/ubuntu/.credentials,uid=1000,gid=1000

This will map the folder share for this session.  Ensure you can actually copy, create and delete files from this share.  Once that is confirmed you can move on to adding this record to your stab to ensure that this drive path is mapped after each reboot.

sudo pico /etc/fstab//[Server IP]/RDSBackups$ /home/ubuntu/RDSBackups cifs credentials=/home/ubuntu/.credentials,uid=1000,gid=1000

Reboot your server now and confirm that this mapping is actually working and remapping on a reboot.  This will be necessary if you ever move this process to OpsWorks to be a time based server to start and stop automatically.

TIP: If you are planning on doing a sanitized version of your database, then you will need to create another drive mapping to a different folder that you can make accessible to the developers, restricting access to the primary folder that will have the confidential data within it.  I would suggest a mapping for RDABackupDevs$ as a share point, then the corresponding folder on your SQL EXPORT machine mapped through the same user in the ‘stab’ file.

We can now go ahead and clone the repo from Github and start our configuration of the scripts to allow for the backups to auto backup.

git clone rds-exporter

In this folder you will see several different partial included files as well as the primary file, which for this case we will copy and use ‘’ as our working template to set up your first export.

Copy over the ‘’ file and name it to match the db instance name for clarity that it will be actioning on. Example, if your RDS DB Instance is named ‘website’, then use this name. Once you have the file copied, be sure to open it and change the instance name in the top of the file accordingly:

export SNAPSHOT_GROUP_NAME=’website_production’

Be sure to update the mappings to the save folders for where you want to copy the exported data to on your local machine (which is the mapped folder to your server on your VPC).

mkdir -p ~/RDSBackups/${CURRENT_DATE}
mkdir -p ~/RDSBackupDevs/${CURRENT_DATE}

If you wish to create a sanitized version of the data for your developers, you can also copy over the ‘database_name.sql’ file in the sanitizer folder to and name it accordingly to match your DB Instance name, ‘website.sql’.  Within this file you can write your custom SQL to clean and sanitize the data for your developers, allowing you to ensure that secure information isn’t left out in the wild.

You will need to go and update the information in the ‘’ file before going forward.  This information is what will be used for setting up and assigning the proper settings to your RDS instance to allow the export to work properly.  Most of these should be self explanatory, by default all of these ARE REQUIRED unless you edit the source scripts to remove the option that are not applicable to you.

export CURRENT_DATE=$(date +”%Y_%m_%d”)
export LOG_FILE=”$(dirname $0)/logs/${CURRENT_DATE}-${SNAPSHOT_GROUP_NAME}.log”

IMPORTANT: One of the issues I came across when I was building this was the fact that Amazon RDS doesn’t really allow user accounts without SUPER rights to be able to create FUNCTIONS within their databases.  This is a security thing, and it makes sense.  That being said with the process of sanitizing the data from the database before exporting it we need to ensure that the sensitive information is removed.  You will need to go into your Amazon Account -> RDS -> Parameter Group, and clone the existing parameter group you are using in your production environment.  Once you have this clones, edit the ‘log_bin_trust_function_creators’ and set this value to 1.  This will allow your users of the RDS instance on this parameter group the ability to be able to create the function we will need to sanitize the data.

You can now test run your setup by running the following command:


What you will see is the output in the console, as well as the contents will be logged into the /logs folder instance and time stamping your log files.

Once you have this running with success, go ahead and check the time on your instance for when it is set to do your auto backups. Depending on the size of your database, you can then schedule this script to run at an appropriate time through the crontab.

This is still a very much work in progress, any changes that I make I will do my best to ensure the code base and this article are updated to reflect the most current version.

Please feel free to submit any suggestions / comments below or on the github repository.

Top Ten Tips for a Successful Hackathon

Here’s Info-Tech’s Top 10 Tips for a successful hackathon.

  1. Get building faster, by using vendors and mentors. Vendors and their mentors offer a great opportunity for learning new skills, getting new perspectives, and sometimes even offering new tools (such as APIs) that can help you get your project up and running quickly. Info-Tech’s team of mentors includes Business Analysts, Design and UX experts, and experienced Developers that live and breathe product development day in and day out.
  2. Make something valuable by creating something YOU would want. Every reputable source on Innovation says that fully understanding your target user, their problems, and their use cases is essential to building a successful problem. One easy way to make this happen is to build a product for someone like you. Otherwise, recruit a few relevant target users to have available to bounce ideas off of to ensure the problem you are solving is compelling and valuable.
  3. Start from “The Compelling Pitch” to focus efforts on impactful work. A common pitfall of any hackathon is to assume the product will “speak for itself”. It won’t. When drawing up plans think through how the creativity, technical difficulty, polish, and usefulness of the project will be communicated, as this will determine the judges’ final score. Building out a fully featured product with elegant code is important in the real world, but in this short timeframe, it’s more important to be able to elegantly communicate the user experience, the primary design components, and most importantly the business value of the product. If you decide to continue building post-event, you can add all the bells and whistles later.
  4. Create a simple, valuable, working product by reigning in scope. Complexity is your enemy. When determining all the desired features of the product, organize them into product releases. What is the minimal viable product that you can build into a working product? Start with that. There’s nothing worse than having a fully functioning design that does nothing. In a typical organization, there are 3 things you can control on a project: Time, Resources and Scope. Unfortunately, you don’t have that luxury, so be aggressive in keeping the features to a maintainable number.
  5. Eliminate last minute failure by getting to production fast and iterating. Get your production environment up and running early. Code always runs differently on a local machine than it does in production. In addition, knowing that your have a working demo will keep stress levels down as deadline approaches. In the perfect world, the last few hours would be spent polishing design and ironing out bugs. But let’s be real, there’s always the temptation to do a little more in the way of features at the end, and this just isn’t possible if you haven’t yet promoted your code. Use a versioning tool, like GitHub, and deploy and smoke test regularly.
  6. Convincingly communicate your coolness, by practicing the perfect pitch and eliminating dependencies. Arguably the most important element of the project, this often gets left to the last minute. By thinking of this up front, you can start thinking about what should go in your pitch. When your main product components are completed start putting together your messaging, and practice your demo. Be careful not to rely on wifi, as there may be many people demoing at once.
  7. Here is a list of things to consider in your pitch:
    • The hook: Share a story or scenario in which the user of your product experienced some kind of pain or desired something better.
    • The basics: Who are the target users? What is the goal of the product? What is the broader pain point you are solving? Don’t assume everyone understands the problem you are trying to solve, as there are many teams and judges may not be experts in your area.
    • The walkthrough: How does it work? What are main features? What are the standout features? Why is it better than comparable products? Are there any cool, sophisticated or elegant components to your design or architecture?
    • The tagline: What do you want to leave them with? Why would they want to share or talk about your product to a colleague or the broader public? Don’t feel the need to go beyond your key message, as this will water it down.
  8. Get the most value out of the event by getting your priorities straight. Six months from how, what will you have gained from this event. Hackathons are about having fun, meeting new people, learning new things, and getting new ideas.
  9. Don’t over commit.  We want you to be able to accomplish what is reasonable in your dedicated time.

What will be more valuable when the event is over? New skills learned, new connections, or a fun new toy. The new toy will be fun for a while, but that skill or that connection may change your life.

Happy Hacking!