How can Azure HDInsight solve your big data challenges?
Big data refers to large volumes of fast-moving data in any format that haven’t yet been handled by your traditional data processing system. In other words, it refers to data which have Volume, Variety and Velocity (commonly termed as V3 in Big Data circles). Data can come from just about anywhere: application logs, sensor data, archived images, videos, streaming data like twitter trends, weather forecast data, astronomical data, biological genomes, and almost anything generated by human or machine. Handling data on this scale is a relatively new problem. Azure’s HDInsight is, appropriately, a new tool that aims to address this problem.
New Challenges and New Solutions: Coping with Variety and Velocity
Whether human-readable or not, managing fast moving data generated at a massive scale – while maintaining data integrity – will require a different kind of processing mechanism than we would have been used for a traditional database or data-mining. Older solutions handled Volume well enough, but Variety and Velocity are relatively new problems. Since not everyone can afford super computers, brute force approaches probably aren’t going to be practical. This challenge inspired the development of Hadoop, which made it possible to process big data on industry-standard servers, while guaranteeing reliable and scalable parallel and distributed computing on smaller budgets.
Without diving too deeply into the technologies of big data management, we can’t go on without at least mentioning the biggest and most important players:
- Hadoop (batch processing),
- NoSQL (HBase, MongoDB, and Cassandra for distributed non-ACID databases),
- Storm and Kafka (real time streaming data),
- Spark (in-memory distributed processing), and
- Pig scripts and Hive queries.
Besides the standard Apache Hadoop framework, there are many vendors who provide customized Hadoop distributions, like Cloudera, Hortonworks, MapR, and Greenplum. Many established cloud vendors provide some variety of Hadoop Platform as a Service (PaaS), like AWS’s Amazon Elastic Map Reduce (EMR), and Google Big Query.
The purpose of this post is to introduce you to Azure HDInsight, which is based on the Hortonworks Data Platform (HDP). The flexibility of Azure cloud and the innovation of Hortonworks can make Azure HDInsight a very interesting and productive space to process your big data.
Azure’s Big Data Solutions
Azure provides various big data processing services. The most popular of them is HDInsight, which is an on-demand Hadoop platform powered by Hortonworks Data Platform (HDP). Besides HDInsight (on which we’re going to focus our attention in this post) Azure also offers:
- Data Lake Analytics
- Data Factory
- SQL Data Warehouse
- Data Catalog
Azure HDInsight: A Comprehensive Managed Apache Hadoop, Spark, R, HBase, and Storm Cloud Service
As we mentioned, Azure provides a Hortonworks distribution of Hadoop in the cloud. “Hadoop distribution” is a broad term used to describe solutions that include some MapReduce and HDFS platform, in addition to a full stack featuring Spark, NoSQL, Pig, Sqoop, Ambari, and Zookeeper. Azure HDInsight provides all those technologies as part of its Big Data service, and also integrates with business intelligence (BI) tools like Excel, SQL Server Analysis Services, and SQL Server Reporting Services.
As a distribution, HDInsight comes in two flavors: Ubuntu and Windows Server 2012. Users can manage Linux-based HDInsight clusters using Apache Ambari and, for Windows users, Azure provides a cluster dashboard.
Cluster configuration for HDInsight is categorized into four different offerings:
1. Hadoop Cluster for query and batch processing with MapReduce and HDFS.
Hadoop clusters are made up of two types of nodes: Head Nodes that run Name Nodes and Job Trackers in a cluster (two nodes/cluster minimum); and Worker Nodes – also called Data Nodes – representing a Hadoop cluster’s true work horses. The minimum number of worker nodes is one, and you can scale worker nodes up or down according to need.
2. HBase for NoSQL-based workloads. NoSQL is a term defining a certain kind of data store and data processing engine that do not rely on traditional ACID properties but, instead, on CAP theorem.
3. Storm for streaming data processing (remember the twitter data or sensor data use-case).
4. HDInsight Spark for in-memory parallel processing for big data analytics. Use cases for HDInsight Spark are Interactive data analysis and BI, Iterative Machine Learning, Streaming and real-time data analysis etc. In recent days, Apache Spark has taken over Hadoop based data analytics because of its capability to handle complex algorithms, faster in-memory processing and graph computing.
You can also customize the cluster with scripts employing various technologies and languages. Spark, Solr, R, Giraph, and Hue can be used to install additional components. HDInsight clusters also include the following (Apache) components:
- Amabri for cluster provisioning, management, and monitoring.
- Hive – and SQL query-like language that runs against data which, in turn, converts to MapReduce algorithms to process the data on HDFS.
- Pig for data processing using user-created scripts.
- Zookeeper for cluster co-ordination and management in a distributed environment.
- Oozie for workflow management.
- Mahout for machine learning.
- Sqoop (SQL on Hadoop) for data import and export from SQL storage.
- Tez – a successor of MapReduce that runs on YARN (Yet-Another-Resource-Negotiator) for complex and acyclic graph processing.
- Phoenix – a layer over HBase to query and analyze data kept in SQL stores. Unlike Hive, Phoenix transforms queries into NoSQL API calls for processing and then converts them for MapReduce programming.
- HCatalog provides a relational view of data in HDFS. It often used with Hive.
- Avro – a data serialization format for the .NET environment.
The current version of HDInsight is 3.4, other software versions in the stack are up to these version numbers:
Azure HDInsight offers all the best big data management features for the enterprise cloud, and has become one of the most talked about Hadoop Distributions in use. While users can quickly scale clusters up or down according to their needs, they will pay only for resources they actually use, and avoid the capital costs required to provision complex hardware configurations and the professionals needed to maintain them. HDInsight lets you crunch all kinds of data – whether structured or not – overcoming the burden of Hadoop cluster configuration.
If you’d like to learn more about Big Data and Hadoop, why not take our Analytics Fundamentals for AWS course? We’re currently building some great Azure content, but in the meantime this course provides a solid foundation in Big Data analytics for AWS. Check it out today!
New Content: AWS Terraform, Java Programming Lab Challenges, Azure DP-900 & DP-300 Certification Exam Prep, Plus Plenty More Amazon, Google, Microsoft, and Big Data Courses
This month our Content Team continues building the catalog of courses for everyone learning about AWS, GCP, and Microsoft Azure. In addition, this month’s updates include several Java programming lab challenges and a couple of courses on big data. In total, we released five new learning...
Where Should You Be Focusing Your AWS Security Efforts?
Another day, another re:Invent session! This time I listened to Stephen Schmidt’s session, “AWS Security: Where we've been, where we're going.” Amongst covering the highlights of AWS security during 2020, a number of newly added AWS features/services were discussed, including: AWS Audit...
AWS re:Invent: 2020 Keynote Top Highlights and More
We’ve gotten through the first five days of the special all-virtual 2020 edition of AWS re:Invent. It’s always a really exciting time for practitioners in the field to see what features and services AWS has cooked up for the year ahead. This year’s conference is a marathon and not a...
WARNING: Great Cloud Content Ahead
At Cloud Academy, content is at the heart of what we do. We work with the world’s leading cloud and operations teams to develop video courses and learning paths that accelerate teams and drive digital transformation. First and foremost, we listen to our customers’ needs and we stay ahea...
Excelling in AWS, Azure, and Beyond – How Danut Prisacaru Prepares for the Future
Meet Danut Prisacaru. Danut has been a Software Architect for the past 10 years and has been involved in Software Engineering for 30 years. He’s passionate about software and learning, and jokes that coding is basically the only thing he can do well (!). We think his enthusiasm shines t...
New Content: AWS Data Analytics – Specialty Certification, Azure AI-900 Certification, Plus New Learning Paths, Courses, Labs, and More
This month our Content Team released two big certification Learning Paths: the AWS Certified Data Analytics - Speciality, and the Azure AI Fundamentals AI-900. In total, we released four new Learning Paths, 16 courses, 24 assessments, and 11 labs. New content on Cloud Academy At any ...
New Content: Azure DP-100 Certification, Alibaba Cloud Certified Associate Prep, 13 Security Labs, and Much More
This past month our Content Team served up a heaping spoonful of new and updated content. Not only did our experts release the brand new Azure DP-100 Certification Learning Path, but they also created 18 new hands-on labs — and so much more! New content on Cloud Academy At any time, y...
AWS Certification Practice Exam: What to Expect from Test Questions
If you’re building applications on the AWS cloud or looking to get started in cloud computing, certification is a way to build deep knowledge in key services unique to the AWS platform. AWS currently offers 12 certifications that cover major cloud roles including Solutions Architect, De...
Overcoming Unprecedented Business Challenges with AWS
From auto-scaling applications with high availability to video conferencing that’s used by everyone, every day — cloud technology has never been more popular or in-demand. But what does this mean for experienced cloud professionals and the challenges they face as they carve out a new p...
Constant Content: Cloud Academy’s Q3 2020 Roadmap
Hello — Andy Larkin here, VP of Content at Cloud Academy. I am pleased to release our roadmap for the next three months of 2020 — August through October. Let me walk you through the content we have planned for you and how this content can help you gain skills, get certified, and...
New Content: Alibaba, Azure AZ-303 and AZ-304, Site Reliability Engineering (SRE) Foundation, Python 3 Programming, 16 Hands-on Labs, and Much More
This month our Content Team did an amazing job at publishing and updating a ton of new content. Not only did our experts release the brand new AZ-303 and AZ-304 Certification Learning Paths, but they also created 16 new hands-on labs — and so much more! New content on Cloud Academy At...
Blog Digest: Which Certifications Should I Get?, The 12 Microsoft Azure Certifications, 6 Ways to Prevent a Data Breach, and More
This month, we were excited to announce that Cloud Academy was recognized in the G2 Summer 2020 reports! These reports highlight the top-rated solutions in the industry, as chosen by the source that matters most: customers. We're grateful to have been nominated as a High Performer in se...