CloudAcademy

Configure FTP service with S3 backend using S3FS

Lab Steps

keyboard_tab
lock
Logging in to the Amazon Web Services Console
lock
Connecting to the Virtual Machine using SSH
lock
Building and Installing the S3FS Executable from Source
lock
Configuring an FTP User Account and Home Directory
lock
Install and Configure FTP Service
lock
Testing FTP using a Command Line Client
lock
Starting Up S3FS and Mounting a Directory
lock
Performing an End-to-End Test of S3 FTP

Ready for the real environment experience?

DifficultyIntermediate
Duration1h
Students8

Description

Lab Overview

FTP (File Transfer Protocol) is a fast and convenient way to transfer large files over the Internet. You might, at some point, have configured an FTP server and used block storage, NAS, or a SAN as your backend. But using this kind of storage requires infrastructure support and can cost you a fair amount of both time and money.

Since November 2018 AWS has offered a managed SFTP service called AWS Transfer for SFTP. It offers the convenience of being highly available and automatically scaled. It also has built-in integration with other AWS services such as IAM and CloudWatch. If cost is not an issue, the managed service may be a good choice for you. This Lab explores a fully customizable, open source solution for using FTP to transfer files to an S3 bucket. The solution in this Lab also covers scenarios when you want to mount an S3 bucket on your local file system.

S3FS 

In this Lab, you will learn to download, compile, and install the S3FS Fuse utility onto an Amazon Linux 2 based EC2 instance. s3fs allows Linux based operating systems to mount an S3 bucket via FUSE. S3FS is an open source project maintained and located as a public git repository on GitHub. In this lab you will download the s3fs source code, then compile and install the resulting s3fs executable, which will then be used to mount a Linux file system directory to an S3 bucket.

This Lab is for anyone that is interested in configuring an FTP service to use Amazon's S3 service as its backend storage layer.

Lab Objectives

Upon completion of this Lab, you will be able to:

  • Install and configure an FTP service
  • Download, compile and install the S3FS Fuse based executable
  • Configure and mount a Linux filesystem directory to an S3 bucket
  • Use both command line and desktop based FTP clients to upload and download files - which will, in turn, be transferred to and from a mounted S3 bucket

Lab Prerequisites

You should be familiar with:

  • Be comfortable with SSH to remotely administer a Linux based server
  • Basic Linux administration
  • Basic FTP commands

Lab Environment

This Lab will start with the following AWS resources being provisioned automatically for you:

  • IAM Role (ec2-instance-role)
  • IAM Policy (s3-bucket-policy)
  • EC2 Amazon Linux 2 based instance - with ec2-instance-role attached
  • S3 Bucket

To achieve the Lab end state, you will be walked through the process of:

  • Downloading, compiling, and installing the s3fs executable
  • Install and configure an FTP service
  • Mount a Linux directory using the s3fs command to the provided S3 bucket
  • Perform FTP testing from your local workstation - resulting in locally hosted files being FTP'd up to the remote EC2 instance. The s3fs process running on the EC2 instance will in turn propagate the files across to the configured S3 bucket.

 

Updates

November 30th, 2018 - Added clarification around the differences between AWS Transfer for SFTP and the approach taken in this Lab.

About the Author

Students4346
Labs2
Courses35
Learning paths7

Jeremy is a Cloud Researcher and Trainer at Cloud Academy where he specializes in developing technical training documentation for security, AI, and machine learning for both AWS and GCP cloud platforms.

He has a strong background in development and coding, and has been hacking with various languages, frameworks, and systems for the past 20+ years.

In recent times, Jeremy has been focused on Cloud, Security, AI, Machine Learning, DevOps, Infrastructure as Code, and CICD.

Jeremy holds professional certifications for both AWS and GCP platforms.