Lab Overview

FTP (File Transfer Protocol) is a fast and convenient way to transfer large files over the Internet. You might, at some point, have configured an FTP server and used block storage, NAS, or a SAN as your backend. But using this kind of storage requires infrastructure support and can cost you a fair amount of both time and money.

Since November 2018 AWS has offered a managed SFTP service called AWS Transfer for SFTP. It offers the convenience of being highly available and automatically scaled. It also has built-in integration with other AWS services such as IAM and CloudWatch. If cost is not an issue, the managed service may be a good choice for you. This Lab explores a fully customizable, open source solution for using FTP to transfer files to an S3 bucket. The solution in this Lab also covers scenarios when you want to mount an S3 bucket on your local file system.

S3FS 

In this lab, you will learn to download, compile, and install the S3FS Fuse utility onto an Amazon Linux 2 based EC2 instance. s3fs allows Linux based operating systems to mount an S3 bucket via FUSE. S3FS is an open source project maintained and located as a public git repository on GitHub. In this lab, you will download the s3fs source code, then compile and install the resulting s3fs executable, which will then be used to mount a Linux file system directory to an S3 bucket.

This lab is for anyone that is interested in configuring an FTP service to use Amazon's S3 service as its backend storage layer.

Lab Objectives

Upon completion of this lab, you will be able to:

• Install and configure an FTP service
• Download, compile and install the S3FS Fuse based executable
• Configure and mount a Linux filesystem directory to an S3 bucket
• Use both command line and desktop based FTP clients to upload and download files - which will, in turn, be transferred to and from a mounted S3 bucket

Lab Prerequisites

You should be familiar with:

• Be comfortable with SSH to remotely administer a Linux based server
• Basic Linux administration
• Basic FTP commands

Lab Environment

This lab will start with the following AWS resources being provisioned automatically for you:

• IAM Role (ec2-instance-role)
• IAM Policy (s3-bucket-policy)
• EC2 Amazon Linux 2 based instance - with ec2-instance-role attached
• S3 Bucket

To achieve the lab end state, you will be walked through the process of:

• Downloading, compiling, and installing the s3fs executable
• Install and configure an FTP service
• Mount a Linux directory using the s3fs command to the provided S3 bucket
• Perform FTP testing from your local workstation - resulting in locally hosted files being FTP'd up to the remote EC2 instance. The s3fs process running on the EC2 instance will in turn propagate the files across to the configured S3 bucket.

Updates

December 30th, 2022 - Updated lab to use EC2 Instance Connect

January 10th, 2019 - Added a validation Lab Step to check the work you perform in the Lab

November 30th, 2018 - Added clarification around the differences between AWS Transfer for SFTP and the approach taken in this Lab.