hands-on lab

Configure FTP service with S3 backend using S3FS

Intermediate
1h
313
4.6/5
Get guided in a real environmentPractice with a step-by-step scenario in a real, provisioned environment.
Learn and validateUse validations to check your solutions every step of the way.
See resultsTrack your knowledge and monitor your progress.
Lab description

Lab Overview

FTP (File Transfer Protocol) is a fast and convenient way to transfer large files over the Internet. You might, at some point, have configured an FTP server and used block storage, NAS, or a SAN as your backend. But using this kind of storage requires infrastructure support and can cost you a fair amount of both time and money.

Since November 2018 AWS has offered a managed SFTP service called AWS Transfer for SFTP. It offers the convenience of being highly available and automatically scaled. It also has built-in integration with other AWS services such as IAM and CloudWatch. If cost is not an issue, the managed service may be a good choice for you. This Lab explores a fully customizable, open source solution for using FTP to transfer files to an S3 bucket. The solution in this Lab also covers scenarios when you want to mount an S3 bucket on your local file system.

S3FS 

In this lab, you will learn to download, compile, and install the S3FS Fuse utility onto an Amazon Linux 2 based EC2 instance. s3fs allows Linux based operating systems to mount an S3 bucket via FUSE. S3FS is an open source project maintained and located as a public git repository on GitHub. In this lab, you will download the s3fs source code, then compile and install the resulting s3fs executable, which will then be used to mount a Linux file system directory to an S3 bucket.

This lab is for anyone that is interested in configuring an FTP service to use Amazon's S3 service as its backend storage layer.

Lab Objectives

Upon completion of this lab, you will be able to:

  • Install and configure an FTP service
  • Download, compile and install the S3FS Fuse based executable
  • Configure and mount a Linux filesystem directory to an S3 bucket
  • Use both command line and desktop based FTP clients to upload and download files - which will, in turn, be transferred to and from a mounted S3 bucket

Lab Prerequisites

You should be familiar with:

  • Be comfortable with SSH to remotely administer a Linux based server
  • Basic Linux administration
  • Basic FTP commands

Lab Environment

This lab will start with the following AWS resources being provisioned automatically for you:

  • IAM Role (ec2-instance-role)
  • IAM Policy (s3-bucket-policy)
  • EC2 Amazon Linux 2 based instance - with ec2-instance-role attached
  • S3 Bucket

To achieve the lab end state, you will be walked through the process of:

  • Downloading, compiling, and installing the s3fs executable
  • Install and configure an FTP service
  • Mount a Linux directory using the s3fs command to the provided S3 bucket
  • Perform FTP testing from your local workstation - resulting in locally hosted files being FTP'd up to the remote EC2 instance. The s3fs process running on the EC2 instance will in turn propagate the files across to the configured S3 bucket.

Updates

December 30th, 2022 - Updated lab to use EC2 Instance Connect

January 10th, 2019 - Added a validation Lab Step to check the work you perform in the Lab

November 30th, 2018 - Added clarification around the differences between AWS Transfer for SFTP and the approach taken in this Lab.

Environment before
Environment after
About the author
Avatar
Jeremy Cook
Content Lead Architect
Students
159,312
Labs
80
Courses
108
Learning paths
212

Jeremy is a Content Lead Architect and DevOps SME here at Cloud Academy where he specializes in developing DevOps technical training documentation.

He has a strong background in software engineering, and has been coding with various languages, frameworks, and systems for the past 25+ years. In recent times, Jeremy has been focused on DevOps, Cloud (AWS, Azure, GCP), Security, Kubernetes, and Machine Learning.

Jeremy holds professional certifications for AWS, Azure, GCP, Terraform, Kubernetes (CKA, CKAD, CKS).

Covered topics
Lab steps
Logging In to the Amazon Web Services Console
Connecting to an EC2 Instance Using Amazon EC2 Instance Connect
Building and Installing the S3FS Executable from Source
Configuring an FTP User Account and Home Directory
Install and Configure FTP Service
Testing FTP using a Command Line Client
Starting Up S3FS and Mounting a Directory
Performing an End-to-End Test of S3 FTP