Elasticsearch vs. CloudSearch: What’s the main difference?
Let’s compare AWS-based cloud tools: Elasticsearch vs. CloudSearch. While both services use proven technologies, Elasticsearch is more popular, open source, and has a flexible API to use for customization; in comparison, CloudSearch is fully managed and benefits from managed service features such as (near) plug-and-play startup and auto-patching and updating.
Want to jump-start your learning on search engines and analytics in general? Get some direct Elastisearch knowledge in our Analytics Fundamentals for AWS course:
In part one of this series, we described what search engines are, how they solve the problem of accessing content stretched across large websites, and how Amazon CloudSearch provides a solution for a cloud environment. AWS CloudSearch is certainly a powerful and appealing service from Amazon. However, there are also other popular players in the search engine market, and Elasticsearch ranks right behind Solr as the most popular search and analytics engine. We’ll explore the battle of the Amazon search providers: Elasticsearch vs. CloudSearch.
Elasticsearch vs. CloudSearch: Provisioning
Both Elasticsearch and CloudSearch are provided by Amazon as AWS services. However, Elasticsearch is an independent product developed by elastic.co, which means you can set up Elasticsearch independently by downloading and extracting the tarball, or through a yum/apt-get install.
Amazon CloudSearch, on the other hand, is fully managed by AWS, which, once you choose your instance type, handles the complete provisioning. Users are able to select High-Availability (AZ level), replication, and partitioning options through the AWS Management Console or AWS CLI.
Elasticsearch vs. CloudSearch: Upgrading
Elasticsearch is easy to upgrade. The process can be as easy as replacing the lib folder of an older version with a new version.
Updates of Amazon CloudSearch are pushed by AWS, relieving users of the responsibility. However, this can result in delayed upgrades of new releases, though with delay comes stability.
Elasticsearch vs. CloudSearch: Data Import/Export
When existing data need to be searchable, they should be imported to the search engines. In Elasticsearch, there are plugins called “rivers” to push data into a cluster. There are many popular river plugins available such as elasticsearch–river- mongodb, elasticsearch-river-couchdb, Elasticsearch-jdbc. However, for various reasons, river plugins are being deprecated.
Logstash Forwarders, are normally used to push logs from application or database servers to Elasticsearch. This makes them available for searching logs or to plot graphs in Kibana. Recently, Logstash and input_changes plugins have taken center stage to replace rivers as tools to push data to Elasticsearch, too. Some of the recently developed input_changes are couchdb_changes, Twitter, and rabbitmq.
In Amazon CloudSearch, data and documents (in either XML or JSON format) are pushed in batches. Data can also be pushed to S3, with the data path given to index the documents.
Elasticsearch vs. CloudSearch: Data and Index Backup
In Elasticsearch, data is backed up (and restored) using the Snapshot and Restore module. Usually, users are required to define a shared mount path. In the cloud, they can instead opt for Amazon S3, HDFS, or Azure storage. Curator is a tool that acts as a cron job manager that users can set to automate the backup process.
In Amazon CloudSearch, the service itself takes care of the whole backup process, once again sparing users the bother. Unlike Elasticsearch, where users must manually run the restore activity from backed up indexes, CloudSearch does it automatically.
Elasticsearch vs. CloudSearch: Security and User Management
Elasticsearch provides a plugin called shield to handle authentication and authorization. Shield also provides features like encryption, role-based access control, IP filtering, and auditing. However, shield is a licensed product that must be purchased.
You can also integrate your AD server to control access locally.
Amazon CloudSearch provides IAM-based access control (and all the granularity this affords, such as restricting access to specific domains). CloudSearch also supports HTTPS for all requests.
Elasticsearch vs. CloudSearch: Cluster Management
In Elasticsearch, adding or deleting nodes within a cluster must be done manually. If the cluster instances are upgraded – i.e. vertical scaling – then you’ll need to run through the setup process from scratch. Old data must be backed up and restored to the new cluster. In the case of horizontal scaling, where servers are added or removed from the cluster, cluster rebalancing and resharding are mandatory. These, too, are manual processes. Users need to be very careful during the process.
Amazon CloudSearch, on the other hand, has built-in scaling and upgrade tools. When a server in a CloudSearch service reaches its threshold, it automatically upgrades to the next larger instance type. And when the capacity goes beyond the largest available instance types, the index is partitioned into multiple instances.
Elasticsearch vs. CloudSearch: Monitoring
In Elasticsearch, there are cluster monitoring tools like Marvel which allow a user to send RESTful queries to check cluster health. Another product called Watcher provides an alerting mechanism. These tools are all provided by Elasticsearch itself. Users can, of course, also bring their own monitoring tools, like SPM or the New Relic plugin for Elasticsearch to keep an eye on their clusters.
Amazon CloudSearch is fully integrated with Amazon CloudWatch, which can monitor metrics like SuccessfulRequests, Searchable Documents, Index Utilization, and Partition Count. Like Watcher in Elasticsearch, AWS Simple Notification Service (SNS) can be integrated with CloudSearch for alerting.
Elasticsearch vs. CloudSearch: High Availability
As they’re both built for running search engines in the cloud, Elasticsearch and CloudSearch are designed for high availability (HA).
Elasticsearch is built for distributed computing where the cluster grows horizontally. The indexes are split into shards and replication factors provide shard redundancy. Whenever a node fails, the replicated shards are used to replace lost data.
Elasticsearch employs a technique called zen discovery, where all the nodes communicate with each other through an “elected” master. In case the master node fails, another node takes over as master.
A similar architecture is followed in CloudSearch to handle failure and provide HA. CloudSearch also has an optional feature for multi-AZ replication within a single region to provide HA and Availability Zone failover.
Elasticsearch vs. CloudSearch: Search and Indexing
In Elasticsearch, searching happens on both index and types using a search API. The search API also includes Faceting and Filtering for searching data.
In CloudSearch, users create a search domain that includes sub-services to upload documents. A search service provides the means to search indexed data.
In Elasticsearch, many built-in libraries are provided for analyzers, tokenizers, and filters for indexing. Amazon CloudSearch, on the other hand, provides a much simpler configuration service for all indexing operations and relevance ranking.
Elasticsearch vs. CloudSearch: Client Libraries
Amazon CloudSearch supports many SDKs along with RESTful API calls. The most popular SDKs are in Java, Ruby, Python, .Net, PHP, and Node.js.
Elasticsearch vs. CloudSearch: Cost
As Elasticsearch requires manual set up, the true cost of deployment must include infrastructure costs, licensing for all non-open source software tools and the OS, and the Elasticsearch binary. This may require a large operational expenditure to cover skilled Elasticsearch admins and a monitoring team.
Amazon CloudSearch is priced according to the search instance size. Here’s an example:
With Multi-AZ enabled, the cost of redundant search instances will also be added. If an index is partitioned, the cost of each new search instance in each AZ is also added to the cost.
CloudSearch Document Batch Uploads
Document batch upload costs are $0.10 per 1,000 Batch Upload Requests (the maximum size for each batch is 5 MB).
CloudSearch IndexDocuments Requests
Re-indexing is required for indexes when a new field is added to the index. The charge for a re-indexing request is $0.98 per GB of data stored in your search domain.
CloudSearch Data Transfer
Inbound data transfers are free between Amazon CloudSearch and other AWS Services. There are charges for outbound data transfers:
- Data transferred between Amazon CloudSearch and AWS services in different regions will be charged as Internet Data Transfers on both sides of the transfer.
- Traffic sent between Amazon CloudSearch and Amazon EC2 instances in the same region is only billed for the Data Transfer in and out of the Amazon EC2 instances. Standard Amazon EC2 Regional Data Transfer charges apply.
Elasticsearch vs. CloudSearch: How to get the most benefit from your choice
Both Elasticsearch and Amazon CloudSearch are built on proven technologies and are the choice of many demanding organizations. Because of its flexibility and active developer community, Elasticsearch is more popular. But Amazon CloudSearch scores when it comes to operational efficiency.
As a side note, because of its popularity, AWS provides Elasticsearch as a Service (Amazon Elasticsearch Service) which, in many ways, provides the best of both worlds. Elastic.co also provides Elasticsearch as a cloud service Found.
Your search solution is an important part of the overall architecture plan for your application or product. Keep your momentum going with AWS and all its changing parts by checking out a Solutions Architect Certification learning path. They’re constantly updated, give you hands-on practice, and provide guidance to as you try to organize the big projects in your career.
New Content: Platforms, Programming, and DevOps – Something for Everyone
This month our team of expert certification specialists released three new or updated learning paths, 16 courses, 13 hands-on labs, and four lab challenges! New content on Cloud Academy You can always visit our Content Roadmap to see what’s just released as well as what’s coming soon....
Mastering AWS Organizations Service Control Policies
Service Control Policies (SCPs) are IAM-like policies to manage permissions in AWS Organizations. SCPs restrict the actions allowed for accounts within the organization making each one of them compliant with your guidelines. SCPs are not meant to grant permissions; you should consider ...
New Content: Focus on DevOps and Programming Content this Month
This month our team of expert certification specialists released 12 new or updated learning paths, 15 courses, 25 hands-on labs, and four lab challenges! New content on Cloud Academy You can always visit our Content Roadmap to see what’s just released as well as what’s coming soon. Ja...
New Content: Get Ready for the CISM Cert Exam & Learn About Alibaba, Plus All the AWS, GCP, and Azure Courses You Know You Can Count On
This month our team of intrepid certification specialists released five learning paths, seven courses, 19 hands-on labs, and three lab challenges! One particularly interesting new learning path is Certified Information Security Manager (CISM) Foundations. After completing this learn...
Which Certifications Should I Get?
The old AWS slogan, “Cloud is the new normal” is indeed a reality today. Really, cloud has been the new normal for a while now and getting credentials has become an increasingly effective way to quickly showcase your abilities to recruiters and companies. With all that in mind, the s...
The 12 AWS Certifications: Which is Right for You and Your Team?
As companies increasingly shift workloads to the public cloud, cloud computing has moved from a nice-to-have to a core competency in the enterprise. This shift requires a new set of skills to design, deploy, and manage applications in cloud computing. As the market leader and most ma...
AWS Certified Solutions Architect Associate: A Study Guide
Want to take a really impactful step in your technical career? Explore the AWS Solutions Architect Associate certificate. Its new version (SAA-C02) was released on March 23, 2020. The AWS Solutions Architect - Associate Certification (or Sol Arch Associate for short) offers some ...
New Content: AWS Terraform, Java Programming Lab Challenges, Azure DP-900 & DP-300 Certification Exam Prep, Plus Plenty More Amazon, Google, Microsoft, and Big Data Courses
This month our Content Team continues building the catalog of courses for everyone learning about AWS, GCP, and Microsoft Azure. In addition, this month’s updates include several Java programming lab challenges and a couple of courses on big data. In total, we released five new learning...
Where Should You Be Focusing Your AWS Security Efforts?
Another day, another re:Invent session! This time I listened to Stephen Schmidt’s session, “AWS Security: Where we've been, where we're going.” Amongst covering the highlights of AWS security during 2020, a number of newly added AWS features/services were discussed, including: AWS Audit...
AWS re:Invent: 2020 Keynote Top Highlights and More
We’ve gotten through the first five days of the special all-virtual 2020 edition of AWS re:Invent. It’s always a really exciting time for practitioners in the field to see what features and services AWS has cooked up for the year ahead. This year’s conference is a marathon and not a...
WARNING: Great Cloud Content Ahead
At Cloud Academy, content is at the heart of what we do. We work with the world’s leading cloud and operations teams to develop video courses and learning paths that accelerate teams and drive digital transformation. First and foremost, we listen to our customers’ needs and we stay ahea...
Excelling in AWS, Azure, and Beyond – How Danut Prisacaru Prepares for the Future
Meet Danut Prisacaru. Danut has been a Software Architect for the past 10 years and has been involved in Software Engineering for 30 years. He’s passionate about software and learning, and jokes that coding is basically the only thing he can do well (!). We think his enthusiasm shines t...