Another very well-developed category of Cognitive Services is Language. It comprises five services, two of them are Custom, giving you the opportunity to interact with the model.

Let me start with the Translator, since we have already mentioned it in the previous video. With this service, you can perform the following tasks:

• Text and Word Translation. It allows you to translate a word or entire texts to over 70 languages. These models have been used for years not only by Microsoft, but by several other giants of the industry such as HP and eBay.
• Transliteration. Transliteration is the conversion from one alphabet to another, without necessarily doing the translation. You could, for example, convert a sentence in Arabic or Cyrillic to Latin characters.
• Finally, it can do both language and sentence length detection.

Here are a few comments about the Translator Text API. Until recently, in version two, which is now deprecated, this was one of the only Cognitive Services APIs that used XML instead of JSON. However, the current version, V3, is in line with the other services and using JSON instead. Although the quality of the translation is pretty good, it has not yet reached parity with human translation. However, it works much, much faster and at a fraction of the cost. Companies might even consider a mixed workflow, which is translated by a machine and just reviewed by translators. There are two translation modes available, Neural Machine Translation, NMT, is a new standard of translation that offers better translation quality, and is the default on the V3 API for most languages. Statistical Machine Translation or SMT is a more traditional machine translation technique, which doesn't offer the same level of translation as NMT. It should only be used on languages that do not offer NMT support.

Microsoft has also a custom translator, which you can access through https://portal.customtranslator.azure.ai/. It allows you to customize the existing general-purpose translation to your specific industry, such as pharma or automotive. And the way it does it is fascinating!

You just need to go to the portal and create a new project. Custom translator requires a translation Text API to work with. When creating a project, you select a language pair, for example, English and Spanish. You can also select an industry category that most closely is related to your needs.

Then, you upload document pairs, for example, technical manuals with the same content, but in the languages that you have selected during project creation. These documents can be in several formats, including DOCX, PDF, XL, XML, HTML and TXT. For optimal results, you'd need at least 10,000 sentences in the document, but you might upload multiple document pairs to reach this threshold.

When you're ready, you just need to train the model. After a few hours of training, you'll have a custom translation model, specific to your industry! You can make this model available through a REST endpoint and use that instead of the standard translator API. Even the speech API can also use these models.

The model performance is measured by something called bleu score. If your bleu score is higher than the baseline score, that means your custom model is now doing better translations.

Now let's talk about Text analytics, which is another fantastic API of the language category. It can do basically four things:

Language Detection, which allows you to detect which language, or even dialect, the message is in, along with a confidence score for the language detected.

Sentiment Analysis, which returns a sentiment score ranging from zero, very unhappy, to one, very happy. This is absolutely great for customer service situations.

Key Phrase Extraction, which tries to understand the main topics of the text. One practical use of this functionality is to add metadata to text documents.

Finally, Named Entity Recognition, which tries to identify entities within the sentence, for example, people, places, organizations, currencies and so on. That's based on the context. For example, is that Mars, the planet or Mars, the Roman God of War? Entity Recognition used to be a separate Cognitive Service under the knowledge pillar, but that has now been integrated with Text Analytics.

Language Detection supports over 120 languages. For named entity recognition, key phrase extraction and sentiment analysis, support varies according to the language. The only languages that support all three features are English, simplified Chinese, French, German and Spanish.

Now let's talk about LUIS, or Language Understanding Intelligent Service, which you can access through www.luis.ai. LUIS, which is frequently used with Bots, allow you to train computers to understand natural language, so that you don't have to code every single variation of the same sentence.

For example, the sentences, "book me a hotel" and "find me a room" probably mean the same thing that you want to perform a hotel reservation. LUIS figures this out through utterances, intents and entities. Utterances are the sentences that you're sending to the model, for example, "Book me a hotel." Intents are the actions that you want to perform, and they're generally identified by the verb in a sentence, in this case, Reserve. Entities are what you're intending to perform the action upon, and are generally the subject or object of the sentence, in this case, Hotel Room.

When you train the LUIS model and publish the REST endpoint, LUIS sends responses to your queries as JSON files. No surprise here at this point, right? The JSON response will have an array of intents and entities, as well as a confidence level for each possible answer.

Finally, the QnA Maker, which is accessible through www.qnamaker.ai. This API used to be in the knowledge category, which no longer exists. In the new categorization, QnA now belongs to Language.

This API is also perfect for bots. It allows you to extract question-answer pairs from your semi-structured content such as FAQs, product manuals and guidelines, and considerably increase your bots' intelligence. You can also create your own pairs manually, from scratch.

Another great aspect of this API is that the model learns over time, based on the responses from the users which improves the model over time. Now let's see a demo about the Language APIs.