The Azure Data Lake Store service provides a platform for organizations to park – and process and analyse – vast volumes of data in any format. Find out how.
With increasing volumes of data to manage, enterprises are looking for appropriate infrastructure models to help them apply analytics to their big data, or simply to store them for undetermined future use. In this post, we’re going to discuss Microsoft’s entry into the data lake market, Azure Data Lake, and in particular, Azure Data Lake store.
What is a data lake?
In simple terms, a data lake is a repository for large quantities and varieties of both structured and unstructured data in their native formats. The term data lake was coined by James Dixon, CTO of Pentaho, to contrast what he called “data marts”, which handled the data reporting and analysis by identifying “the most interesting attributes, and to aggregate” them. The problems with this approach are that “only a subset of the attributes is examined, so only pre-determined questions can be answered,” and that “data is aggregated, so visibility into the lowest levels is lost.”
A data lake, on the other hand, maintains data in their native formats and handles the three Vs of big data (Volume, Velocity and Variety) while providing tools for analysis, querying, and processing. Data lake eliminates all the restrictions of a typical data warehouse system by providing unlimited space, unrestricted file size, schema on read, and various ways to access data (including programming, SQL-like queries, and REST calls).
With the emergence of Hadoop (including HDFS and YARN), the benefits of data lake – previously available only to the most resource-rich companies like Google, Yahoo, and Facebook – became a practical reality for just about anyone. Now, organizations who had been generating and gathering data on a large scale but had struggled to store and process them in a meaningful way, have more options.
Azure Data Lake
Azure Data Lake is the new kid on the data lake block from Microsoft Azure. Here is some of what it offers:
- The ability to store and analyse data of any kind and size.
- Multiple access methods including U-SQL, Spark, Hive, HBase, and Storm.
- Built on YARN and HDFS.
- Dynamic scaling to match your business priorities.
- Enterprise-grade security with Azure Active Directory.
- Managed and supported with an enterprise-grade SLA.
Azure Data Lake can, broadly, be divided into three parts:
- Azure Data Lake store – The Data Lake store provides a single repository where organizations upload data of just about infinite volume. The store is designed for high-performance processing and analytics from HDFS applications and tools, including support for low latency workloads. In the store, data can be shared for collaboration with enterprise-grade security.
- Azure Data Lake analytics – Data Lake analytics is a distributed analytics service built on Apache YARN that compliments the Data Lake store. The analytics service can handle jobs of any scale instantly with on-demand processing power and a pay-as-you-go model that’s very cost effective for short term or on-demand jobs. It includes a scalable distributed runtime called U-SQL, a language that unifies the benefits of SQL with the expressive power of user code.
- Azure HDInsight – Azure HDInsight is a full stack Hadoop Platform as a Service from Azure. Built on top of Hortonworks Data Platform (HDP), it provides Apache Hadoop, Spark, HBase, and Storm clusters.
We’ve already been introduced to HDInsight in this series. Now we will discuss Azure Data Lake Store…which is still in Preview Mode.
Azure Data Lake Store
According to Microsoft, Azure Data Lake store is a hyper-scale repository for big data analytics workloads and a Hadoop Distributed File System (HDFS) for the cloud. It…
- Imposes no fixed limits on file size.
- Imposes no fixed limits on account size.
- Allows unstructured and structured data in their native formats.
- Allows massive throughput to increase analytic performance.
- Offers high durability, availability, and reliability.
- Is integrated with Azure Active Directory access control.
Some have compared Azure Data Lake store with Amazon S3 but, beyond the fact that both provide unlimited storage space, the two really don’t share all that much in common. If you want to compare S3 to an Azure service, you’ll get better mileage with the Azure Storage Service. Azure Data Lake store, on the other hand, provides an integrated analytics service and places no limits on file size. Here’s a nice illustration:
(Image Courtesy: Microsoft)
Azure Data Lake store can handle any data in their native format, as is, without requiring prior transformations. Data Lake store does not require a schema to be defined before the data is uploaded, leaving it up to the individual analytic framework to interpret the data and define a schema at the time of the analysis. Being able to store files of arbitrary size and formats makes it possible for Data Lake store to handle structured, semi-structured, and even unstructured data.
Azure Data Lake store file system (adl://)
Azure Data Lake Store can be accessed from Hadoop (available with an HDInsight cluster) using the WebHDFS-compatible REST APIs. However, Azure Data Lake store introduced a new file system called AzureDataLakeFilesystem (adl://). adl:// is optimized for performance and available in HDInsight. Data is accessed in the Data Lake store using:
Azure Data Lake store security:
Azure Data Lake store uses Azure Active Directory (AAD) for authentication and Access Control Lists (ACLs) to manage access to your data. Azure Data Lake benefits from all AAD features including Multi-Factor Authentication, conditional access, role-based access control, application usage monitoring, security monitoring and alerting. Azure Data Lake store supports the OAuth 2.0 protocol for authentication within the REST interface. Similarly, Data Lake store provides access control by supporting POSIX-style permissions exposed by the WebHDFS protocol.
Azure Data Lake store pricing
Data Lake Store is currently available in US-2 region and offers preview pricing rates (excluding Outbound Data transfer):
Azure Data Lake is an important new part of Microsoft’s ambitious cloud offering. With Data Lake, Microsoft provides service to store and analyze data of any size at an affordable cost. In related posts, we will learn more about Data Lake Store, Data Lake Analytics, and HDInsight.
Application Load Balancer vs. Classic Load Balancer
What is an Elastic Load Balancer? This post covers basics of what an Elastic Load Balancer is, and two of its examples: Application Load Balancers and Classic Load Balancers. For additional information — including a comparison that explains Network Load Balancers — check out our post o...
Advantages and Disadvantages of Microservices Architecture
What are microservices? Let's start our discussion by setting a foundation of what microservices are. Microservices are a way of breaking large software projects into loosely coupled modules, which communicate with each other through simple Application Programming Interfaces (APIs). ...
Kubernetes Services: AWS vs. Azure vs. Google Cloud
Kubernetes is a popular open-source container orchestration platform that allows us to deploy and manage multi-container applications at scale. Businesses are rapidly adopting this revolutionary technology to modernize their applications. Cloud service providers — such as Amazon Web Ser...
AWS Internet of Things (IoT): The 3 Services You Need to Know
The Internet of Things (IoT) embeds technology into any physical thing to enable never-before-seen levels of connectivity. IoT is revolutionizing industries and creating many new market opportunities. Cloud services play an important role in enabling deployment of IoT solutions that min...
Which Certifications Should I Get?
As we mentioned in an earlier post, the old AWS slogan, “Cloud is the new normal” is indeed a reality today. Really, cloud has been the new normal for a while now and getting credentials has become an increasingly effective way to quickly showcase your abilities to recruiters and compan...
How to Go Serverless Like a Pro
So, no servers? Yeah, I checked and there are definitely no servers. Well...the cloud service providers do need servers to host and run the code, but we don’t have to worry about it. Which operating system to use, how and when to run the instances, the scalability, and all the arch...
AWS Security: Bastion Hosts, NAT instances and VPC Peering
Effective security requires close control over your data and resources. Bastion hosts, NAT instances, and VPC peering can help you secure your AWS infrastructure. Welcome to part four of my AWS Security overview. In part three, we looked at network security at the subnet level. This ti...
Top 13 Amazon Virtual Private Cloud (VPC) Best Practices
Amazon Virtual Private Cloud (VPC) brings a host of advantages to the table, including static private IP addresses, Elastic Network Interfaces, secure bastion host setup, DHCP options, Advanced Network Access Control, predictable internal IP ranges, VPN connectivity, movement of interna...
Big Changes to the AWS Certification Exams
With AWS re:Invent 2019 just around the corner, we can expect some early announcements to trickle through with upcoming features and services. However, AWS has just announced some big changes to their certification exams. So what’s changing and what’s new? There is a brand NEW ...
New on Cloud Academy: ITIL® 4, Microsoft 365 Tenant, Jenkins, TOGAF® 9.1, and more
At Cloud Academy, we're always striving to make improvements to our training platform. Based on your feedback, we released some new features to help make it easier for you to continue studying. These new features allow you to: Remove content from “Continue Studying” section Disc...
AWS Security Groups: Instance Level Security
Instance security requires that you fully understand AWS security groups, along with patching responsibility, key pairs, and various tenancy options. As a precursor to this post, you should have a thorough understanding of the AWS Shared Responsibility Model before moving onto discussi...
Cloud Migration Risks & Benefits
If you’re like most businesses, you already have at least one workload running in the cloud. However, that doesn’t mean that cloud migration is right for everyone. While cloud environments are generally scalable, reliable, and highly available, those won’t be the only considerations dri...