Could you create an S3 FTP file backup/transfer solution without the normal administration headache on top of Amazon’s Simple Storage Service?

S3 FTP serverFTP (File Transfer Protocol) is a fast and convenient way to transfer large files over the Internet. You might, at some point, have configured an FTP server and used block storage, NAS, or an SAN as your backend. But using this kind of storage requires infrastructure support and can cost you a fair amount of both time and money.

Could an S3 FTP solution work better? Since their reliable and competitively priced infrastructure is just sitting there waiting to be used, I was curious to see whether AWS can give us what we need without the administration headache.

Why S3 FTP?

Amazon S3 is reliable and accessible, that’s why.

NOTE: FTP is not a secure protocol and should not be used to transfer sensitive data. You might consider using the SSH File Transfer Protocol (sometimes called SFTP) for that.

Using S3 FTP: object storage as filesystem

SAN, iSCSI, and local disks are block storage devices. That means block storage volumes that are attached directly to an machine running an operating system that drives your filesystem operations. But S3 is built for object storage. This mean interactions occur at the application level via an API or command lines. You can’t mount S3 directly on your operating systems.

So to mount S3 on your server and use it as an FTP server storage backend, we’ll need some help.  S3fuse will let us mount a bucket as a local filesystem with read/write access. On s3fs-mounted files systems, we can simply use cp, mv, and ls – and all the basic Unix file management commands – to manage resources on locally attached disks. S3fuse is FUSE-based file system that enables fully functional filesystems in a userspace program.

So it seems that we’ve got all the pieces for an S3 FTP solution. How will it actually work?

Installing s3fs

1. Install the packages we’ll need:

2. Download and complile s3fs:

3. Configure AWS credentials:

4. Mount your S3 Bucket:

Now copy some files to the /home/ec2/s2bucket directory so you’ll have something to upload to your S3 bucket.

5. Install Vsftpd  and configure storage as an S3 mounted bucket:

Adjust the configuration to read like this (substitute the public IP address of your FTP server for xxx…):

6. Add the ftpuser and set its home directory as /home/ec2/s3bucket/

Finally, you can connect to the server via FTP using the username and passwords created earlier. Now whatever files you have copied to your s3bucket directory will be automatically uploaded to Amazon S3. Voila! An S3 FTP server!

If you want to deepen your understanding of how S3 works, this is your go-to course: AWS Storage Fundamentals – Simple Storage Service (S3)