S3 FTP: build a reliable and inexpensive FTP server using Amazon’s S3


Could you create an S3 FTP file backup/transfer solution without the normal administration headache on top of Amazon’s Simple Storage Service?

S3 FTP serverFTP (File Transfer Protocol) is a fast and convenient way to transfer large files over the Internet. You might, at some point, have configured an FTP server and used block storage, NAS, or an SAN as your backend. But using this kind of storage requires infrastructure support and can cost you a fair amount of both time and money.

Could an S3 FTP solution work better? Since their reliable and competitively priced infrastructure is just sitting there waiting to be used, I was curious to see whether AWS can give us what we need without the administration headache.

Why S3 FTP?

Amazon S3 is reliable and accessible, that’s why.

NOTE: FTP is not a secure protocol and should not be used to transfer sensitive data. You might consider using the SSH File Transfer Protocol (sometimes called SFTP) for that.

Using S3 FTP: object storage as filesystem

SAN, iSCSI, and local disks are block storage devices. That means block storage volumes that are attached directly to an machine running an operating system that drives your filesystem operations. But S3 is built for object storage. This mean interactions occur at the application level via an API or command lines. You can’t mount S3 directly on your operating systems.

So to mount S3 on your server and use it as an FTP server storage backend, we’ll need some help.  S3fuse will let us mount a bucket as a local filesystem with read/write access. On s3fs-mounted files systems, we can simply use cp, mv, and ls – and all the basic Unix file management commands – to manage resources on locally attached disks. S3fuse is FUSE-based file system that enables fully functional filesystems in a userspace program.

So it seems that we’ve got all the pieces for an S3 FTP solution. How will it actually work?

Installing s3fs

1. Install the packages we’ll need:

2. Download and complile s3fs:

3. Configure AWS credentials:

4. Mount your S3 Bucket:

Now copy some files to the /home/ec2/s2bucket directory so you’ll have something to upload to your S3 bucket.

5. Install Vsftpd  and configure storage as an S3 mounted bucket:

Adjust the configuration to read like this (substitute the public IP address of your FTP server for xxx…):

6. Add the ftpuser and set its home directory as /home/ec2/s3bucket/

Finally, you can connect to the server via FTP using the username and passwords created earlier. Now whatever files you have copied to your s3bucket directory will be automatically uploaded to Amazon S3. Voila! An S3 FTP server!

If you want to deepen your understanding of how S3 works, this is your go-to course: AWS Storage Fundamentals – Simple Storage Service (S3)

Nitheesh Poojary

Nitheesh Poojary

My professional IT career began nine years back when I was just out of my college. I worked with a great team as an infrastructure management engineer, managing hundreds of enterprise application servers. I found my passion when I got the opportunity to work with Cloud technologies: I'm addicted to AWS Cloud Services, DevOps engineering, and all the cloud tools and technologies that make engineers' lives easier. Currently I am working as a Technical Specialist in a respected firm in Bangalore. I'm a Certified AWS and SysOps Engineer, happily helping fellow engineers across the globe through my blogs and answering questions in various forums. When not solving problems for my projects, I play sports, hang out with friends or travel to places on my wish list.

More Posts - Website

Follow Me:
TwitterLinkedIn

  • J Mulreany

    Nitheesh
    One suggestion around API keys. never use the master AWS account keys. Make an IAM user with keys and no login privileges with just enough access to access the S3 bucket you are using.

    • David Clinton

      That’s an excellent point.

    • Nitheesh Poojary

      Agree Mulreany . Thanks for Suggestion•

    • Darryn Campbell

      Better still, use an IAM role and the iam_role option to s3fs. I chose to add my mount via fstab, so that it automatically mounts and starts the service at boot/reboot, so something like

      echo bucket-name /path/to/mount/point fuse.s3fs rw,nosuid,nodev,iam_role=some-iam-role,uid=ftpuser,gid=ftpuser,allow_other 0 0 >> /etc/fstab

      worked for me with no AWS keys required.

  • Giuseppe Borgese

    I did something similar on the past not with ftp but with a server to host images. I had two problems:

    1 – the access to the files with the mount is very very slow

    2 – how can I assure that the file is always mount? From my experience this kind of mount is not so much reliable . If you look this article https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/
    “Avoid filesystem mounts (FUSE, etc).
    I’ve found them to be about as reliable as a large government department when used in critical applications. Use the SDK instead.”

    In my opinion this can be a good way when the NFS as a service “Elastic file system” of aws will be available (now it is still in preview only in Oregon) .

    What do you think in particular for the reliability of the s3fs? I’m very interested to know more about this kind of solution.

    Regards from a big fan of your platform

    • David Clinton

      Hi Guiseppe,
      I’ve also seen complaints about the reliability of s3fs – especially under heavy use scenarios. Perhaps you might try out AWS’s s3cmd – it’s not a mount, but I’ve found it to be really efficient. You could create an alias or some kind of front end for it and you’d never know the files weren’t local. :)
      I wrote about using s3cmd in a cron job for a backup solution here:
      http://cloudacademy.com/blog/data-backup-s3cmd/

      Regards,
      David

    • Nitheesh Poojary

      Yes in your case “Elastic file system” would be ideal solution . But EFS is very expensive. You can have a look @GlusterFS http://www.gluster.org/community/documentation/index.php/Getting_started_setup_aws.Connect with me if you need more help

  • sunil951

    Connection established, waiting for welcome message…

    Response: 220 (vsFTPd 2.2.2)

    Command: USER backup

    Response: 530 Permission denied.

    Error: Could not connect to server

    Getting error after connecting to ftp. User default directory set as s3 bucket mounted folder.