Cognitive Services Overview
Cognitive Services Overview

Artificial Intelligence is not a future or distant concept; it is here and now, and being used by many companies of various sizes and industries. The foundational theory for AI was actually developed several decades ago, but recent advancements in big data, computing power, cloud, and algorithms have made it affordable and widespread today. With AI and Machine Learning, computers are now able to start reasoning, understanding, and interacting in ways that were never possible before.

Microsoft has created a predefined set of AI models available for companies of all sizes to start with called Cognitive Services, and best of all, they require little to no knowledge of data science. In this course, you will learn how to infuse your apps—on an architectural level—with the intelligence that Cognitive Services provide. We will cover what Cognitive Services are and how to use the various solutions they provide, including Vision, Speech, Language, Decision, and Web Search.

Learning Objectives

  • Understand the functionality provided by Azure Cognitive Services
  • Learn how to incorporate these services into your apps

Intended Audience

  • People who want to learn more about Azure Cognitive Services


  • Knowledge of Azure
  • Knowledge of at least one programming language
  • Experience using REST APIs

Before we talk about Cognitive Services, let me give you a super short primer on Data Sciences. There's a lot of hype going on about this topic, from how it will cure cancer all the way to apocalyptic futures where we'll be dominated by robots. But data sciences, in fact, can essentially respond to five questions:

  • Classification. Is this A or B? For example, is that patient likely to develop diabetes in the next 5 years, Yes or No? This is an example of dual-class classification, but if there are more than 2 options, it's generally called multiclass classification.
  • Regression. How much, how many? In this case, you're looking for a number. For example, what is the expected price of the Microsoft stock by the end of the year?
  • Anomaly Detection. Is this weird? A good example is a credit card fraud detection. For example, if an Australian customer that never travels, makes a purchase in Dubai, and immediately after that another one in Sydney, that might raise some flags.
  • Clustering. How is this organized? A good example are recommendation systems, like the ones you see on Netflix or Amazon. By grouping customers that have similar tastes, those systems can easily recommend things that you're likely to watch or buy.
  • Finally, Reinforcement. What's next? That's the realm of reinforcement learning, where machines must make lots of decisions without human intervention.

But if you think about it, being able to respond to these apparently simple questions is a huge breakthrough. Now computers can start to learn, understand, and recognize patterns, without needing to be explicitly coded for that. Hence the terms artificial intelligence, and machine learning. That's pretty amazing.

That being said, Machine Learning models are quite cumbersome to create, test, and improve. For starters, they need a considerable amount of data. The more data you have, the more precise the model tends to be. Also, processing all this information requires a substantial amount of computing power, as well as a lot of time-consuming refining by the data sciences team; both of which are expensive. Once the model is created, though, it gets quite cheap to consume.

And that's the beauty of Cognitive Services:

  • Cognitive Services allow developers to build AI-infused applications without having AI, ML or Data Sciences skills. It can be as simple as calling a REST API, and processing a JSON file back. In a way, you can consider Cognitive Services are MLaaS, Machine Learning as a Service.
  • You get a huge head start by utilizing something that has been refined by Microsoft for years, in technologies such as Skype, Cortana, Xbox and Office; instead of creating a model from scratch.
  • It has a pretty good language support. You have SDKs available for C++, C sharp, Java, Python, and, because most languages nowadays can call REST APIs, it's pretty easy to integrate even if you're using a more exotic one.
  • Cognitive Services do not keep your data after processing, which means it's easier to be compliant with privacy standards such as GDPR. The only exception, though, are the Bing APIs, where the search queries are kept by Microsoft for purposes of improving their models.
  • Finally, although there are variations according to the service, Cognitive Services are generally quite cheap to consume. And several of them have a free tier for you to start with, sometimes without even needing an Azure subscription.

Cognitive Services are, however, based on pre-defined, general-purpose models; even though there's room for customization, as we'll see later on. If your Data Sciences need is not covered by any of these generic models, though, you'll probably have to create your own, using tools such as Azure ML or Databricks. One example is if you want to create a classifier that recognizes different kinds of whistles from a ship.

Because there are so many Cognitive Services, there are over 20 services available at the moment, with a lot more in preview; Microsoft divides them in 5 pillars:

  • The Vision services allow computers to see and recognize patterns in images. Some features of these APIs are recognizing objects, landmarks, celebrities, handwriting and people's faces and emotions.
  • Speech. Speech is about converting back-and-forth between speech and text, as well as translating audio.
  • Language. That allows you to analyze and find meaning in text, such as sentiment, key phrases, and understanding language. It also has a translation component.
  • Decision has, at the moment, just two services. One is Content Moderator, which detects adult content on your websites. The other one is Personalizer, that gives customized content tailored for each user; similar to the Netflix recommenders I mentioned before.
  • And finally, Web Search. These are a set of Bing-related technologies for searching the web. There is some pretty interesting material here, such as the ability to use an image as a search query.

For each one of these pillars, there are also custom solutions that allow you to personalize the models based on your needs. And they generally have their own portals for you to interact with. For example, Computer Vision will probably be able to detect clouds on an image. However, if you want to detect the type of cloud, between Cirrus, Cumulus, and so on, Computer Vision will probably not help you much. You can, instead, use Custom Vision to train the model to recognize those cloud types.

Keep in mind that Cognitive Services are a fast-evolving part of Microsoft AI Strategy. And, therefore, changes quite frequently. We're focusing on this course on the technologies that are generally available at the time of this recording. But just so that you can fully understand the lifecycle, there are technologies that still didn't graduate into a preview level. Customers are still allowed to subscribe to these projects though. There is a very interesting one that gives personality to chatbots. You can see all of these research projects at, Other projects are already in preview, but not yet GA. Examples are Anomaly Detection, Immersive Reader and Speaker Recognition. Then you have the ones generally available, which we'll cover in the next videos. And finally, some services have retired, or mostly merged into other services. For example, the Bing Speech API is now called Speech Services, and the Emotion API merged with the Face API. All right, so now let's take a look in our first demo.

About the Author

Emilio Melo has been involved in IT projects in over 15 countries, with roles ranging across support, consultancy, teaching, project and department management, and sales—mostly focused on Microsoft software. After 15 years of on-premises experience in infrastructure, data, and collaboration, he became fascinated by Cloud technologies and the incredible transformation potential it brings. His passion outside work is to travel and discover the wonderful things this world has to offer.