Moving Data to S3 with Apache NiFi

Moving data to the cloud is one of the cornerstones of any cloud migration. Apache NiFi is an open source tool that enables you to easily move and process data using a graphical user interface (GUI).  In this blog post, we will examine a simple way to move data to the cloud using NiFi complete with practical steps. Calculated Systems offers a cloud-first version of NiFi that you can use to follow along.

Cloud Object Storage

There are many ways to store data on the cloud, but the easiest are the object stores. All three major cloud providers have them:
These is an ideal starting point for files as you can typically land the files without too much forethought or capacity planning. Additionally, these object stores are extremely robust, featuring multiple levels of durability and availability.
NiFi Cloud Migration
For the purposes of this tutorial, we will start with the most common object store: Amazon Simple Storage Service (Amazon S3).

Amazon S3 Terminology

Before we get started moving data, let’s establish some basic terminology:
  • Identity and Access Management (IAM) – Controls for making and controlling who and what can interact with your AWS resources.   
  • Access Keys – These are your access credentials to use AWS. These are not your typical username/password — they are generated using access identity management. 
  • Bucket – A grouping of similar files that must have a unique name. These can be made publicly accessible and are often used to host static objects.
  • Folder – Much like an operating system folder, these exist within a bucket to enable organization.

Creating an Access Key

For NiFi to have permission to write to S3, we must set it up with an access key pair. There are many ways to do this, but the best practice is to create a new IAM user. To get to the IAM user screen, navigate to the IAM homepage
  1. Select “Add user” and check “Programmatic access”
  2. Enter a new name such as “NiFi_demo”
  3. Click “Next: Permissions”
  4. Click “Create Group” and you will be presented with a list of permissions you can add to this new user
  5. Enter a group name such as “Nifi_Demo_Group”
  6. Next to filter policies search for S3 and check “AmazonS3FullAccess” > Click “Create Group”
  7. At the bottom right, select “Next:Tags” > Click through to “Next:Review”
  8. Click “Create user” to finish making an IAM User
The access key ID and secret access key are very important to setting up your data transfer. You can download them as a .CSV file or save them somewhere safe.
IMPORTANT: Be sure to record your secret access key as this is the only time it can be viewed. 

Creating an S3 Bucket

Although we will cover the basics of creating your S3 bucket in this post, you can check out Cloud Academy’s Storage Fundamentals of AWS for an in-depth overview. Now that we have credentials for AWS, we need a place to land them. To put it simply, we need to create a new S3 bucket if you do not already have one. Go to the AWS S3 Console
  1. Click “+ Create Bucket”
  2. Enter a unique bucket name and the region you are creating it in
  3. Click through until the bucket is created (default options are fine to use)
  4. Click on your new bucket and you should be able to see its contents — which will be empty

Setting up your NiFi & AWS Credential Service or Processor Controls

NiFi can be setup several ways including download from the Apache website or using a pre-made solution like Calculated System’s AWS Marketplace Offering.
NiFi has many ways to provide access to AWS either through an overarching credential service or parameters set to a specific processor. The credential service is ideal when you have multiple processors all relying on the same keys. For the scope of this tutorial, we will not be using the service, but it is ideal when moving into a production setting.
  1. To get started, click and drag in a new processor “PutS3Object” > right-click “Configure the processor”
  2. Under the Settings tab, you will see Automatically Terminate Relations > check the boxes next to “failure” and “success” since this is the last processor in the flow.
  3. Under the Properties tab, configure the following properties:
    • Access Key ID – From the User you created earlier and noted down
    • Secret Access Key ID – From the User you created earlier and noted down
    • Bucket – Put the name of the bucket you created
    • Region – The region your bucket is located; often U.S. East (N. Virginia)

    Processor Configuration

  4. Click “Apply” to finish up configuring the processor. 

Setting Up Your Flow

For the purposes of this sample flow, let’s replicate NiFi’s own configuration directory to S3. To accomplish this, we need two additional processors: ListFiles and FetchFiles. Connect and configure them as shown below.
List File Flow
ListFile
  • Properties tab – Set “Input Directory” to /nifi/docs/html
  • Drag a connection from ListFile to FetchFile for relationship success
FetchFile
  • Settings tab – Check the boxes next to “Failure,” “not.found,” * “permission.denied”
  • Drag a connection from FetchFile to PutS3Object for relationship success
Running Your Flow
  • Right-click each of the processors > click “Start”
  • Let this run for a few seconds. If you want to track the progress, right-click into any blank space of your NiFi canvas and press “refresh.” You should see each processor reporting flowfiles “in” and “out”
  • For the purpose of this demo, right-click “Stop list files.” In production, you can leave this task running, but it is always best to stop demos when done. This stops the demo from producing sample files after you stopped using the program.

Viewing the Objects in S3

If you return to your bucket, you should see your files listed. Note: You may have to refresh button the page depending on your browser/settings.
S3 Bucket

[Optional] Security Cleanup

As an optional step, you may wish to revoke the access keys you gave to this Nifi Demo. It is general best practice to remove unused keys when done. To revoke the keys, go the AWS Console.
  • Click on the user you created earlier in the tutorial
  • Go to the Security Credentials tab and search for the Access Keys subsection. Here you can inactivate, delete, or even make new keys.
  • As a best practice, make the key inactive or delete the key.
Chris Gambino

Written by

Chris Gambino

Chris has been focused on the big data ecosystem for years. Starting with simple databases he focused on Hadoop for several years before switching to a cloud-first approach. An Author of "Nifi for Dummies", his approach involves a holistic evaluation of the problem before assigning technology. https://www.calculatedsystems.com/


Related Posts

Amanda Cross
Amanda Cross
— January 7, 2021

New Content: AWS Terraform, Java Programming Lab Challenges, Azure DP-900 & DP-300 Certification Exam Prep, Plus Plenty More Amazon, Google, Microsoft, and Big Data Courses

This month our Content Team continues building the catalog of courses for everyone learning about AWS, GCP, and Microsoft Azure. In addition, this month’s updates include several Java programming lab challenges and a couple of courses on big data. In total, we released five new learning...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Avatar
Stuart Scott
— December 17, 2020

Where Should You Be Focusing Your AWS Security Efforts?

Another day, another re:Invent session! This time I listened to Stephen Schmidt’s session, “AWS Security: Where we've been, where we're going.” Amongst covering the highlights of AWS security during 2020, a number of newly added AWS features/services were discussed, including: AWS Audit...

Read more
  • AWS
  • AWS re:Invent
  • cloud security
Joe Nemer
Joe Nemer
— December 4, 2020

AWS re:Invent: 2020 Keynote Top Highlights and More

We’ve gotten through the first five days of the special all-virtual 2020 edition of AWS re:Invent. It’s always a really exciting time for practitioners in the field to see what features and services AWS has cooked up for the year ahead.  This year’s conference is a marathon and not a...

Read more
  • AWS
  • AWS Glue Elastic Views
  • AWS re:Invent
Bryony Harrower
Bryony Harrower
— November 6, 2020

WARNING: Great Cloud Content Ahead

At Cloud Academy, content is at the heart of what we do. We work with the world’s leading cloud and operations teams to develop video courses and learning paths that accelerate teams and drive digital transformation. First and foremost, we listen to our customers’ needs and we stay ahea...

Read more
  • AWS
  • Azure
  • content roadmap
  • GCP
Joe Nemer
Joe Nemer
— October 25, 2020

Excelling in AWS, Azure, and Beyond – How Danut Prisacaru Prepares for the Future

Meet Danut Prisacaru. Danut has been a Software Architect for the past 10 years and has been involved in Software Engineering for 30 years. He’s passionate about software and learning, and jokes that coding is basically the only thing he can do well (!). We think his enthusiasm shines t...

Read more
  • AWS
  • careers
  • champions
  • upskilling
Joe Nemer
Joe Nemer
— October 14, 2020

New Content: AWS Data Analytics – Specialty Certification, Azure AI-900 Certification, Plus New Learning Paths, Courses, Labs, and More

This month our Content Team released two big certification Learning Paths: the AWS Certified Data Analytics - Speciality, and the Azure AI Fundamentals AI-900. In total, we released four new Learning Paths, 16 courses, 24 assessments, and 11 labs.  New content on Cloud Academy At any ...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Joe Nemer
Joe Nemer
— September 15, 2020

New Content: Azure DP-100 Certification, Alibaba Cloud Certified Associate Prep, 13 Security Labs, and Much More

This past month our Content Team served up a heaping spoonful of new and updated content. Not only did our experts release the brand new Azure DP-100 Certification Learning Path, but they also created 18 new hands-on labs — and so much more! New content on Cloud Academy At any time, y...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Joe Nemer
Joe Nemer
— August 28, 2020

AWS Certification Practice Exam: What to Expect from Test Questions

If you’re building applications on the AWS cloud or looking to get started in cloud computing, certification is a way to build deep knowledge in key services unique to the AWS platform. AWS currently offers 12 certifications that cover major cloud roles including Solutions Architect, De...

Read more
  • AWS
  • AWS Certifications
Patrick Navarro
Patrick Navarro
— August 25, 2020

Overcoming Unprecedented Business Challenges with AWS

From auto-scaling applications with high availability to video conferencing that’s used by everyone, every day —  cloud technology has never been more popular or in-demand. But what does this mean for experienced cloud professionals and the challenges they face as they carve out a new p...

Read more
  • AWS
  • Cloud Adoption
  • digital transformation
Avatar
Andrew Larkin
— August 18, 2020

Constant Content: Cloud Academy’s Q3 2020 Roadmap

Hello —  Andy Larkin here, VP of Content at Cloud Academy. I am pleased to release our roadmap for the next three months of 2020 — August through October. Let me walk you through the content we have planned for you and how this content can help you gain skills, get certified, and...

Read more
  • alibaba
  • AWS
  • Azure
  • content roadmap
  • Content updates
  • DevOps
  • GCP
  • Google Cloud
  • New content
Alisha Reyes
Alisha Reyes
— August 5, 2020

New Content: Alibaba, Azure AZ-303 and AZ-304, Site Reliability Engineering (SRE) Foundation, Python 3 Programming, 16 Hands-on Labs, and Much More

This month our Content Team did an amazing job at publishing and updating a ton of new content. Not only did our experts release the brand new AZ-303 and AZ-304 Certification Learning Paths, but they also created 16 new hands-on labs — and so much more! New content on Cloud Academy At...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Alisha Reyes
Alisha Reyes
— July 16, 2020

Blog Digest: Which Certifications Should I Get?, The 12 Microsoft Azure Certifications, 6 Ways to Prevent a Data Breach, and More

This month, we were excited to announce that Cloud Academy was recognized in the G2 Summer 2020 reports! These reports highlight the top-rated solutions in the industry, as chosen by the source that matters most: customers. We're grateful to have been nominated as a High Performer in se...

Read more
  • AWS
  • Azure
  • blog digest
  • Certifications
  • Cloud Academy
  • OWASP
  • OWASP Top 10
  • Security
  • VPCs