Configure FTP service with S3 backend using S3FS

Lab Steps

keyboard_tab
lock
Logging in to the Amazon Web Services Console
lock
Connecting to the Virtual Machine using SSH
lock
Building and Installing the S3FS Executable from Source
lock
Configuring an FTP User Account and Home Directory
lock
Install and Configure FTP Service
lock
Testing FTP using a Command Line Client
lock
Starting Up S3FS and Mounting a Directory
lock
Performing an End-to-End Test of S3 FTP
lock
Validate AWS Lab

Ready for the real environment experience?

DifficultyIntermediate
Max Duration1h
Students94
Ratings
4.3/5
star star star star star-half

Description

Lab Overview

FTP (File Transfer Protocol) is a fast and convenient way to transfer large files over the Internet. You might, at some point, have configured an FTP server and used block storage, NAS, or a SAN as your backend. But using this kind of storage requires infrastructure support and can cost you a fair amount of both time and money.

Since November 2018 AWS has offered a managed SFTP service called AWS Transfer for SFTP. It offers the convenience of being highly available and automatically scaled. It also has built-in integration with other AWS services such as IAM and CloudWatch. If cost is not an issue, the managed service may be a good choice for you. This Lab explores a fully customizable, open source solution for using FTP to transfer files to an S3 bucket. The solution in this Lab also covers scenarios when you want to mount an S3 bucket on your local file system.

S3FS 

In this Lab, you will learn to download, compile, and install the S3FS Fuse utility onto an Amazon Linux 2 based EC2 instance. s3fs allows Linux based operating systems to mount an S3 bucket via FUSE. S3FS is an open source project maintained and located as a public git repository on GitHub. In this lab you will download the s3fs source code, then compile and install the resulting s3fs executable, which will then be used to mount a Linux file system directory to an S3 bucket.

This Lab is for anyone that is interested in configuring an FTP service to use Amazon's S3 service as its backend storage layer.

Lab Objectives

Upon completion of this Lab, you will be able to:

  • Install and configure an FTP service
  • Download, compile and install the S3FS Fuse based executable
  • Configure and mount a Linux filesystem directory to an S3 bucket
  • Use both command line and desktop based FTP clients to upload and download files - which will, in turn, be transferred to and from a mounted S3 bucket

Lab Prerequisites

You should be familiar with:

  • Be comfortable with SSH to remotely administer a Linux based server
  • Basic Linux administration
  • Basic FTP commands

Lab Environment

This Lab will start with the following AWS resources being provisioned automatically for you:

  • IAM Role (ec2-instance-role)
  • IAM Policy (s3-bucket-policy)
  • EC2 Amazon Linux 2 based instance - with ec2-instance-role attached
  • S3 Bucket

To achieve the Lab end state, you will be walked through the process of:

  • Downloading, compiling, and installing the s3fs executable
  • Install and configure an FTP service
  • Mount a Linux directory using the s3fs command to the provided S3 bucket
  • Perform FTP testing from your local workstation - resulting in locally hosted files being FTP'd up to the remote EC2 instance. The s3fs process running on the EC2 instance will in turn propagate the files across to the configured S3 bucket.

 

Updates

January 10th, 2019 - Added a validation Lab Step to check the work you perform in the Lab

November 30th, 2018 - Added clarification around the differences between AWS Transfer for SFTP and the approach taken in this Lab.

About the Author

Students11150
Labs28
Courses65
Learning paths15

Jeremy is the DevOps Content Lead at Cloud Academy where he specializes in developing technical training documentation for DevOps.

He has a strong background in software engineering, and has been coding with various languages, frameworks, and systems for the past 20+ years. In recent times, Jeremy has been focused on DevOps, Cloud, Security, and Machine Learning.

Jeremy holds professional certifications for both the AWS and GCP cloud platforms.