HDInsight – Azure’s Hadoop Big Data Service

How can Azure HDInsight solve your big data challenges?

Big data refers to large volumes of fast-moving data in any format that haven’t yet been handled by your traditional data processing system. In other words, it refers to data which have Volume, Variety and Velocity (commonly termed as V3 in Big Data circles). Data can come from just about anywhere: application logs, sensor data, archived images, videos, streaming data like twitter trends, weather forecast data, astronomical data, biological genomes, and almost anything generated by human or machine. Handling data on this scale is a relatively new problem. Azure’s HDInsight is, appropriately, a new tool that aims to address this problem.

New Challenges and New Solutions: Coping with Variety and Velocity

Whether human-readable or not, managing fast moving data generated at a massive scale – while maintaining data integrity – will require a different kind of processing mechanism than we would have been used for a traditional database or data-mining. Older solutions handled Volume well enough, but Variety and Velocity are relatively new problems. Since not everyone can afford super computers, brute force approaches probably aren’t going to be practical. This challenge inspired the development of Hadoop, which made it possible to process big data on industry-standard servers, while guaranteeing reliable and scalable parallel and distributed computing on smaller budgets.
Without diving too deeply into the technologies of big data management, we can’t go on without at least mentioning the biggest and most important players:

  • Hadoop (batch processing),
  • NoSQL (HBase, MongoDB, and Cassandra for distributed non-ACID databases),
  • Storm and Kafka (real time streaming data),
  • Spark (in-memory distributed processing), and
  • Pig scripts and Hive queries.

Besides the standard Apache Hadoop framework, there are many vendors who provide customized Hadoop distributions, like Cloudera, Hortonworks, MapR, and Greenplum. Many established cloud vendors provide some variety of Hadoop Platform as a Service (PaaS), like AWS’s Amazon Elastic Map Reduce (EMR), and Google Big Query.
The purpose of this post is to introduce you to Azure HDInsight, which is based on the Hortonworks Data Platform (HDP). The flexibility of Azure cloud and the innovation of Hortonworks can make Azure HDInsight a very interesting and productive space to process your big data.

Azure’s Big Data Solutions

Azure provides various big data processing services. The most popular of them is HDInsight, which is an on-demand Hadoop platform powered by Hortonworks Data Platform (HDP). Besides HDInsight (on which we’re going to focus our attention in this post) Azure also offers:

  • Data Lake Analytics
  • Data Factory
  • SQL Data Warehouse
  • Data Catalog

Azure HDInsight: A Comprehensive Managed Apache Hadoop, Spark, R, HBase, and Storm Cloud Service

As we mentioned, Azure provides a Hortonworks distribution of Hadoop in the cloud. “Hadoop distribution” is a broad term used to describe solutions that include some MapReduce and HDFS platform, in addition to a full stack featuring Spark, NoSQL, Pig, Sqoop, Ambari, and Zookeeper. Azure HDInsight provides all those technologies as part of its Big Data service, and also integrates with business intelligence (BI) tools like Excel, SQL Server Analysis Services, and SQL Server Reporting Services.
As a distribution, HDInsight comes in two flavors: Ubuntu and Windows Server 2012. Users can manage Linux-based HDInsight clusters using Apache Ambari and, for Windows users, Azure provides a cluster dashboard.
Cluster configuration for HDInsight is categorized into four different offerings:
1. Hadoop Cluster for query and batch processing with MapReduce and HDFS.
HDInsight - hadoop cluster

Hadoop clusters are made up of two types of nodes: Head Nodes that run Name Nodes and Job Trackers in a cluster (two nodes/cluster minimum); and Worker Nodes – also called Data Nodes – representing a Hadoop cluster’s true work horses. The minimum number of worker nodes is one, and you can scale worker nodes up or down according to need.

2. HBase for NoSQL-based workloads. NoSQL is a term defining a certain kind of data store and data processing engine that do not rely on traditional ACID properties but, instead, on CAP theorem.
HDInsight - hbase cluster
3. Storm for streaming data processing (remember the twitter data or sensor data use-case).
HDInsight - storm cluster
4. HDInsight Spark for in-memory parallel processing for big data analytics. Use cases for HDInsight Spark are Interactive data analysis and BI, Iterative Machine Learning, Streaming and real-time data analysis etc. In recent days, Apache Spark has taken over Hadoop based data analytics because of its capability to handle complex algorithms, faster in-memory processing and graph computing.
Azure Spark services diagram
You can also customize the cluster with scripts employing various technologies and languages. Spark, Solr, R, Giraph, and Hue can be used to install additional components.  HDInsight clusters also include the following (Apache) components:

  • Amabri for cluster provisioning, management, and monitoring.
  • Hive – and SQL query-like language that runs against data which, in turn, converts to MapReduce algorithms to process the data on HDFS.
  • Pig for data processing using user-created scripts.
  • Zookeeper for cluster co-ordination and management in a distributed environment.
  • Oozie for workflow management.
  • Mahout for machine learning.
  • Sqoop (SQL on Hadoop) for data import and export from SQL storage.
  • Tez – a successor of MapReduce that runs on YARN (Yet-Another-Resource-Negotiator) for complex and acyclic graph processing.
  • Phoenix – a layer over HBase to query and analyze data kept in SQL stores. Unlike Hive, Phoenix transforms queries into NoSQL API calls for processing and then converts them for MapReduce programming.
  • HCatalog provides a relational view of data in HDFS. It often used with Hive.
  • Avro – a data serialization format for the .NET environment.

The current version of HDInsight is 3.4, other software versions in the stack are up to these version numbers:

HDInsight software stack versions
  (HDInsight software stack)

Conclusion

Azure HDInsight offers all the best big data management features for the enterprise cloud, and has become one of the most talked about Hadoop Distributions in use. While users can quickly scale clusters up or down according to their needs, they will pay only for resources they actually use, and avoid the capital costs required to provision complex hardware configurations and the professionals needed to maintain them. HDInsight lets you crunch all kinds of data – whether structured or not – overcoming the burden of Hadoop cluster configuration.

If you’d like to learn more about Big Data and Hadoop, why not take our Analytics Fundamentals for AWS course? We’re currently building some great Azure content, but in the meantime this course provides a solid foundation in Big Data analytics for AWS. Check it out today!

Learn Analytics Fundamentals for AWS: EMR

 

Avatar

Written by

Chandan Patra

Cloud Computing and Big Data professional with 10 years of experience in pre-sales, architecture, design, build and troubleshooting with best engineering practices. Specialities: Cloud Computing - AWS, DevOps(Chef), Hadoop Ecosystem, Storm & Kafka, ELK Stack, NoSQL, Java, Spring, Hibernate, Web Service


Related Posts

Joe Nemer
Joe Nemer
— October 14, 2020

New Content: AWS Data Analytics – Specialty Certification, Azure AI-900 Certification, Plus New Learning Paths, Courses, Labs, and More

This month our Content Team released two big certification Learning Paths: the AWS Certified Data Analytics - Speciality, and the Azure AI Fundamentals AI-900. In total, we released four new Learning Paths, 16 courses, 24 assessments, and 11 labs.  New content on Cloud Academy At any ...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Joe Nemer
Joe Nemer
— September 15, 2020

New Content: Azure DP-100 Certification, Alibaba Cloud Certified Associate Prep, 13 Security Labs, and Much More

This past month our Content Team served up a heaping spoonful of new and updated content. Not only did our experts release the brand new Azure DP-100 Certification Learning Path, but they also created 18 new hands-on labs — and so much more! New content on Cloud Academy At any time, y...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Joe Nemer
Joe Nemer
— August 28, 2020

AWS Certification Practice Exam: What to Expect from Test Questions

If you’re building applications on the AWS cloud or looking to get started in cloud computing, certification is a way to build deep knowledge in key services unique to the AWS platform. AWS currently offers 12 certifications that cover major cloud roles including Solutions Architect, De...

Read more
  • AWS
  • AWS Certifications
Patrick Navarro
Patrick Navarro
— August 25, 2020

Overcoming Unprecedented Business Challenges with AWS

From auto-scaling applications with high availability to video conferencing that’s used by everyone, every day —  cloud technology has never been more popular or in-demand. But what does this mean for experienced cloud professionals and the challenges they face as they carve out a new p...

Read more
  • AWS
  • Cloud Adoption
  • digital transformation
Avatar
Andrew Larkin
— August 18, 2020

Constant Content: Cloud Academy’s Q3 2020 Roadmap

Hello —  Andy Larkin here, VP of Content at Cloud Academy. I am pleased to release our roadmap for the next three months of 2020 — August through October. Let me walk you through the content we have planned for you and how this content can help you gain skills, get certified, and...

Read more
  • alibaba
  • AWS
  • Azure
  • content roadmap
  • Content updates
  • DevOps
  • GCP
  • Google Cloud
  • New content
Alisha Reyes
Alisha Reyes
— August 5, 2020

New Content: Alibaba, Azure AZ-303 and AZ-304, Site Reliability Engineering (SRE) Foundation, Python 3 Programming, 16 Hands-on Labs, and Much More

This month our Content Team did an amazing job at publishing and updating a ton of new content. Not only did our experts release the brand new AZ-303 and AZ-304 Certification Learning Paths, but they also created 16 new hands-on labs — and so much more! New content on Cloud Academy At...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Alisha Reyes
Alisha Reyes
— July 16, 2020

Blog Digest: Which Certifications Should I Get?, The 12 Microsoft Azure Certifications, 6 Ways to Prevent a Data Breach, and More

This month, we were excited to announce that Cloud Academy was recognized in the G2 Summer 2020 reports! These reports highlight the top-rated solutions in the industry, as chosen by the source that matters most: customers. We're grateful to have been nominated as a High Performer in se...

Read more
  • AWS
  • Azure
  • blog digest
  • Certifications
  • Cloud Academy
  • OWASP
  • OWASP Top 10
  • Security
  • VPCs
Avatar
Cloud Academy Team
— July 9, 2020

Which Certifications Should I Get?

The old AWS slogan, “Cloud is the new normal” is indeed a reality today. Really, cloud has been the new normal for a while now and getting credentials has become an increasingly effective way to quickly showcase your abilities to recruiters and companies. With all that in mind, the s...

Read more
  • AWS
  • Azure
  • Certifications
  • Cloud Computing
  • Google Cloud Platform
Alisha Reyes
Alisha Reyes
— July 2, 2020

New Content: AWS, Azure, Typescript, Java, Docker, 13 New Labs, and Much More

This month, our Content Team released a whopping 13 new labs in real cloud environments! If you haven't tried out our labs, you might not understand why we think that number is so impressive. Our labs are not “simulated” experiences — they are real cloud environments using accounts on A...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Joe Nemer
Joe Nemer
— June 19, 2020

Kickstart Your Tech Training With a Free Week on Cloud Academy

Are you looking to make a jump in your technical career? Want to get trained or certified on AWS, Azure, Google Cloud Platform, DevOps, Kubernetes, Python, or another in-demand skill? Then you'll want to mark your calendar. Starting Monday, June 22 at 12:00 a.m. PDT (3:00 a.m. EDT), ...

Read more
  • AWS
  • Azure
  • cloud academy content
  • complimentary access
  • GCP
  • on the house
Alisha Reyes
Alisha Reyes
— June 11, 2020

New Content: AZ-500 and AZ-400 Updates, 3 Google Professional Exam Preps, Practical ML Learning Path, C# Programming, and More

This month, our Content Team released tons of new content and labs in real cloud environments. Not only that, but we introduced our very first highly interactive "Office Hours" webinar. This webinar, Acing the AWS Solutions Architect Associate Certification, started with a quick overvie...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Rebecca Willis
Rebecca Willis
— June 3, 2020

Azure vs. AWS: Which Certification Provides the Brighter Future?

More and more companies are using cloud services, prompting more and more people to switch their current IT position to something cloud-related. The problem is most people only have that much time after work to learn new technologies, and there are plenty of cloud services that you can ...

Read more
  • AWS
  • Azure
  • certification