As architectural patterns change and become more complex, from the monolith to microservices, observability has become more and more challenging. With the distributed nature of applications, you now have to manage 10 times more infrastructure, all potentially with different programming languages supported by different teams. 

With observability, this means teams might be using different SDKs or agents to instrument their applications.  Because of this, the data they collect from the agents or SDKs may not be in a standardized data format.

This leads to two big issues: 

1. Running multiple agents can create extra costs, and can be frustrating for engineering teams

2. And having to correlate data from all the sources can be very slow and time-consuming  

This is why the OpenTelemetry standard was created. OpenTelemetry states that instead of installing and running multiple SDKs and agents, you can instrument your applications just once. Then, you collect metrics and traces all in the same standardized format, and can then send that data to the monitoring platforms of your choice. This set of standards, which is part of the Cloud Native Computing Foundation, is becoming very popular with organizations that are implementing observability.  So much so that AWS decided to develop a distribution of the OpenTelemetry standard called AWS Distro for OpenTelemetry. 

The main benefit of using AWS’ distribution instead of running your own is that AWS ensures that it is predictable and secure through their own continuous testing. The other benefit is that if you run into any issues, you then have the ability to reach out to AWS Support for service-related help. You can use AWS Distro for OpenTelemetry on basically any type of compute, including ECS, EKS, Fargate, EC2, Lambda, and on-premises services. From there, you can instrument your applications with the AWS Distro for OpenTelemetry SDKS, agents, collectors, and exporters. 

But once you instrument your applications with AWS Distro for OpenTelemetry, where do you send that data? You have four main options: you can send the data to Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch Service, and Amazon Managed Service for Prometheus.

Let’s talk about the last option, Amazon Managed Service for Prometheus. Prometheus is an open-source monitoring system that has become very popular for gathering metrics and creating alerts. It has become the go-to monitoring platform for a lot of containerized workloads - especially Kubernetes workloads. 

Because of this, Amazon has created its own compatible version of Prometheus called Amazon managed Service for Prometheus, or AMP. AMP is a metric-only service and does not support logging or tracing. Once you collect data using AWS Distro for OpenTelemetry, you can send the metrics from your application to AMP.  AMP is generally recommended in two cases:

1. If you’re already using Prometheus and running it in AWS, you can switch to the managed service to reduce burden

2. Or if you’re looking for a service that is better optimized to monitor your container workloads. 

Keep in mind that Prometheus monitoring data can be integrated with CloudWatch, so that you can query and alarm based on that data. 

Since Prometheus is used mainly as a reporting service, often more visualization of metrics is needed. This is where customers can choose to integrate another service. A popular open-source service for analysis and visualization is called Grafana.

You can host a Grafana server yourself the DIY way using Amazon EC2 or you can use the Amazon Managed Grafana service….which I’m sure you can guess through the name is AWS’ version of a managed Grafana server. With Amazon Managed Grafana, you can create dashboards and other visualizations to gain insight into your metrics, logs, and traces. With Amazon Managed Grafana, it lessens the burden of management, and you still get integration with the AWS landscape with the ability to use IAM, multi-accounts, SSO, CloudTrail, and more. 

So, if you have an open source requirement for observability and still want to use AWS, you have another option for an observability stack in AWS by using AWS Distro for OpenTelemetry for data collection, Prometheus for data reporting, and Grafana for data visualization.