Perhaps surprisingly, Amazon Elasticsearch is hardly overwhelming, coming with a very basic tool kit and an outdated release version. And it’s expensive.
Just a month ago, AWS launched their Amazon Elasticsearch Service. Elasticsearch itself is an open source scalable, distributed, real-time search and analytics engine from Elastic, the creators of Logstash, Beats, and Kibana. Elasticsearch makes an excellent alternative to Splunk.
According to AWS Elasticsearch documentation:
“Amazon Elasticsearch Service (Amazon ES) is a managed service that makes it easy to deploy, operate, and scale Elasticsearch in the AWS cloud…You can set up and configure your Amazon Elasticsearch cluster in minutes from the AWS Management Console. Amazon ES provisions all the resources for your cluster and launches it…Amazon ES allows you to easily scale your cluster via a single API call or a few clicks in the AWS Management Console.”
Amazon Elasticsearch features
According to their documentation, the Amazon Elasticsearch Service provides the following features:
- A full range of instance types from which to build your clusters.
- Magnetic, General Purpose, and Provisioned IOPS EBS volumes.
- Clusters spanning multiple regions and Availability Zones.
- Security through IAM-based access control.
- Dedicated master nodes to improve cluster stability.
- Domain snapshots to back up and restore Elasticsearch domains and replicate domains across Availability Zones.
- Kibana for data visualization.
- Integration with Amazon CloudWatch for monitoring Elasticsearch domain metrics.
- Integration with AWS CloudTrail for auditing configuration API calls to Elasticsearch domains.
Amazon Elasticsearch currently uses following package versions:
- Elasticsearch 1.5.2
- Kibana 4 (also Kibana 3 as a plugin).
- Plugins: jetty, cloud-aws, kuromoji, and icu.
- The following APIs:
/_alias, /_aliases, /_all, /_analyze, /_bulk, /_cat, /_cluster/health, /_cluster/settings, /_cluster/stats, /_count, /_flush, /_mapping, /_mget, /_msearch, /_nodes, /_plugin/kibana, /_plugin/kibana3, /_percolate, /_refresh, /_search, /_snapshot, /_stats, /status, /_template
Amazon Elasticsearch: limits
Amazon Elasticsearch has a few built-in limitations which you need to be aware of before you start:
Older version of Elasticsearch
Elasticsearch 1.5.2 – the version used by Amazon – is actually quite old when compared with the current stable version is (1.7.2). And Elasticsearch 2.0.0 beta, which is just around the corner, will address many more bugs. Since Amazon Elasticsearch is a managed service, there is no way for you to upgrade your clusters on your own. If you were to host Elasticsearch yourself, upgrades would be as simple as updating the jar files in ES_HOME/lib folder.
Elasticsearch 1.5.x and other versions have critical bugs
In the release notes to Elasticsearch 1.7.1, more than a dozen bugs are identified as fixed. Users are advised to upgrade their clusters as soon as possible. Here are just a couple of examples:
IP range aggregation issue:
ip_range aggregation with mask of 0.0.0.0/0 gets treated as 0.0.0.0/32. This was resolved with the 1.7.x release.
Elasticsearch 1.7.x has addressed many problems from 1.5.2, including one which could result in the loss of an entire index if you suffer a multiple node failure while having idle shards. This might be a particularly serious concern with a cloud setup, where node failures due to Availability Zone outages are not uncommon. Although these are rare cases, Elasticsearch Support did send this email alert to their customers:
EBS volume size
You can attach a maximum of 512 GB of storage to a single I or R series node (i2.2xlarge, r3.8xlarge etc). For M series nodes, however, you are limited to a maximum of 100 GBs. Besides the fact that I and R series nodes are expensive, they only come as large, instance-store volumes. This is an obvious problem if you intend to shut down, and then reuse your Elasticsearch cluster at some future time.
There are two major limitations with the instance types available for Amazon Elasticsearch. The first is that you can only run a maximum of ten instances per cluster. If you want more, you’ll have to submit a service request for an increase. The second problem concerns node memory. Here’s what Elasticsearch’s documentation says:
“A machine with 64 GB of RAM is the ideal sweet spot, but 32 GB and 16 GB machines are also common. Less than 8 GB tends to be counterproductive (you end up needing many, many small machines)”.
Seeing how AWS offers us r3.2xlarge instances (and higher) and i2.2xlarge, fits nicely with Elastic’s ideal for cluster nodes, but they will be very expensive. An EC2 r3.8xlarge on-demand RHEL instance costs $2.903 per hour, and the r3.8xlarge.elasticsearch will cost you $4.704 per hour!
No Shield, Watcher, and Marvel support
Elasticsearch has released many commercial products: Shield for security, Watcher for alerts and notifications, and Marvel for cluster monitoring. They are really useful and come out-of-box with Elasticsearch. There are many such plugins, like Sense, kopf, and river, that were developed for Elasticsearch administrators and developers. You can certainly use AWS’s IAM and Cloudwatch in place of Shield and Marvel, but choosing those will sometimes add extra costs and often new skills. If you already have Shield, Watcher, and Marvel licenses, and you’re just moving your existing Elasticsearch cluster to Amazon, then those licenses will be of no use.
No River Plugin support:
River plugins are helpful for supporting data migration from a source to an Elasticsearch cluster (like MongoDB River and jdbc River). Again, not all of those are available for Amazon Elasticsearch installations.
Perhaps surprisingly, Amazon Elasticsearch is hardly overwhelming. It certainly looks nice, but it comes with a very basic tool kit and, as we’ve seen, lacks access to some fairly critical features. In my opinion, Amazon Elasticsearch does deliver an agile offering with faster cluster set up and automated snapshot and restore process, but it is not yet cost-effective.
Setting up Elasticsearch on your own VM (including EC2 instances) is not at all difficult. You can decompress the zip or tar files and, with a minimum of administration knowledge, make the light modifications to the elasticsearch.yml file. You’ll have your cluster up and running in minutes. With your own setup, you have more control over your cluster. You can change the parameters and reconcile your cluster with releases from Elasticsearch.
However, this is Amazon, and this is just a 1.0 release. We can certainly expect to see something significantly more robust in the coming months.
New Content: AWS Terraform, Java Programming Lab Challenges, Azure DP-900 & DP-300 Certification Exam Prep, Plus Plenty More Amazon, Google, Microsoft, and Big Data Courses
This month our Content Team continues building the catalog of courses for everyone learning about AWS, GCP, and Microsoft Azure. In addition, this month’s updates include several Java programming lab challenges and a couple of courses on big data. In total, we released five new learning...
Where Should You Be Focusing Your AWS Security Efforts?
Another day, another re:Invent session! This time I listened to Stephen Schmidt’s session, “AWS Security: Where we've been, where we're going.” Amongst covering the highlights of AWS security during 2020, a number of newly added AWS features/services were discussed, including: AWS Audit...
AWS re:Invent: 2020 Keynote Top Highlights and More
We’ve gotten through the first five days of the special all-virtual 2020 edition of AWS re:Invent. It’s always a really exciting time for practitioners in the field to see what features and services AWS has cooked up for the year ahead. This year’s conference is a marathon and not a...
WARNING: Great Cloud Content Ahead
At Cloud Academy, content is at the heart of what we do. We work with the world’s leading cloud and operations teams to develop video courses and learning paths that accelerate teams and drive digital transformation. First and foremost, we listen to our customers’ needs and we stay ahea...
Excelling in AWS, Azure, and Beyond – How Danut Prisacaru Prepares for the Future
Meet Danut Prisacaru. Danut has been a Software Architect for the past 10 years and has been involved in Software Engineering for 30 years. He’s passionate about software and learning, and jokes that coding is basically the only thing he can do well (!). We think his enthusiasm shines t...
New Content: AWS Data Analytics – Specialty Certification, Azure AI-900 Certification, Plus New Learning Paths, Courses, Labs, and More
This month our Content Team released two big certification Learning Paths: the AWS Certified Data Analytics - Speciality, and the Azure AI Fundamentals AI-900. In total, we released four new Learning Paths, 16 courses, 24 assessments, and 11 labs. New content on Cloud Academy At any ...
New Content: Azure DP-100 Certification, Alibaba Cloud Certified Associate Prep, 13 Security Labs, and Much More
This past month our Content Team served up a heaping spoonful of new and updated content. Not only did our experts release the brand new Azure DP-100 Certification Learning Path, but they also created 18 new hands-on labs — and so much more! New content on Cloud Academy At any time, y...
AWS Certification Practice Exam: What to Expect from Test Questions
If you’re building applications on the AWS cloud or looking to get started in cloud computing, certification is a way to build deep knowledge in key services unique to the AWS platform. AWS currently offers 12 certifications that cover major cloud roles including Solutions Architect, De...
Overcoming Unprecedented Business Challenges with AWS
From auto-scaling applications with high availability to video conferencing that’s used by everyone, every day — cloud technology has never been more popular or in-demand. But what does this mean for experienced cloud professionals and the challenges they face as they carve out a new p...
Constant Content: Cloud Academy’s Q3 2020 Roadmap
Hello — Andy Larkin here, VP of Content at Cloud Academy. I am pleased to release our roadmap for the next three months of 2020 — August through October. Let me walk you through the content we have planned for you and how this content can help you gain skills, get certified, and...
New Content: Alibaba, Azure AZ-303 and AZ-304, Site Reliability Engineering (SRE) Foundation, Python 3 Programming, 16 Hands-on Labs, and Much More
This month our Content Team did an amazing job at publishing and updating a ton of new content. Not only did our experts release the brand new AZ-303 and AZ-304 Certification Learning Paths, but they also created 16 new hands-on labs — and so much more! New content on Cloud Academy At...
Blog Digest: Which Certifications Should I Get?, The 12 Microsoft Azure Certifications, 6 Ways to Prevent a Data Breach, and More
This month, we were excited to announce that Cloud Academy was recognized in the G2 Summer 2020 reports! These reports highlight the top-rated solutions in the industry, as chosen by the source that matters most: customers. We're grateful to have been nominated as a High Performer in se...