Skill Assessment Technology

Explore the technology that scientifically calculates skill scores

Test your team across multiple cloud platforms and technologies

What is a skill score?

A skill represents the knowledge of a user on a given topic, measured through your ability to do a specific task. Such tasks can be either theoretical or practical, such as answering multiple-choice questions (MCQs) of quizzes and exams or completing the steps of the hand-on labs.

This ability is represented by a numeric value, referred to as skill score, or simply skill. The skill score is a number bounded between 0 and 1000, where 0 delineates a user with no knowledge and 1000 a fully proficient user.

Learn how to build practical skills with labs
What is a skill score?

What does it mean to have a skill score equal to 500?

Not only are users scored in the 0-1000 range, but the tasks that they are required to do are also “scored” with a numeric value in the 0-1000 range. The score of the tasks is typically referred to as task difficulty, and it is one of the properties of a task.

Roughly speaking, a user with a skill of 500 is expected to have a good chance of successfully completing tasks with a difficulty of 500. According to one of the reference theories — Item Response Theory — the chance of success is defined as equal to 50%.

 

Learn about Natural Language Processing (NLP)
Assessments Track Skill Growth

What happens when you complete a task?

Every time you complete a task, the skill assessment algorithm updates your score on the tested skills. The skill score represents an estimate of your actual proficiency, and performance on the given tasks are used to move the estimate as close as possible to your actual knowledge level.

Let’s assume that you correctly answer a multiple-choice question. The algorithm compares your current skill and the question’s difficulty and increases your skill. Such an increase will be considerable if the question’s difficulty is much higher than your skill. This is the case of a beginner user who’s able to answer a very challenging question: the skill score is probably higher than the current estimate, so the algorithm moves it with a consistent increase.

Learn more about assessments
Cloud Academy Skill Profile

How is the task difficulty measured?

The difficulty of the tasks is estimated on the basis of the collected statistics. In a nutshell, easy tasks are expected to be successfully completed by most of the students, while hard ones only by a minority. More precisely, easy tasks are expected to be successfully completed by both beginner and proficient students, while hard tasks are expected to be successfully completed only by skilled students.

Remember that the difficulty is only one of the properties of the tasks — probably the most known and intuitive. Among the other properties, it is worth mentioning discrimination, which represents how well a task is able to differentiate between beginner and proficient students.

In fact, a task can be very difficult (meaning most people fail), but if it has low discrimination, it means that both beginners and very skilled students tend to fail it, reducing its effectiveness in measuring the user’s skills. 

Learn more about assessments with multiple-choice questions
View real-time insights about your organization’s capabilities

How are skills organized?

The content in the Cloud Academy library is tagged with one or more skills, in accordance with the related topics. The number of skills continuously grows as new topics are covered by our training. At the moment, there are several hundred skills.

These skills are organized into a graph (referred to as Knowledge Graph), whose nodes represent the skills and whose edges model their semantic relationships.

In this way, every time a skill is updated, the skills that interconnect are updated accordingly. For instance, when you correctly complete a hands-on lab on Amazon S3, you will see the skill Amazon S3 increased, as well as the skill Storage for AWS, and, in turn, the skill AWS.

Note that the increase in the score of Storage for AWS will be lower than the one on Amazon S3. This is because Storage for AWS aggregates multiple services (e.g., Amazon S3, Amazon Glacier, etc.), so its score must consider all the interconnected subskills. Roughly speaking, the score on Storage for AWS will be the average of its subskills, where the score of the missing subskills is estimated through the existing interconnections.

Assessments Track Skill Growth

How can the skill graph be browsed?

To simplify the user experience and hide the complexity of the skill graph, the skills are shown with the platform as a multi-level tree. This tree can be navigated as a standard menu, starting from the root — referred to as overall — up to a specific service of a certain cloud provider, such as Amazon S3.

Learn how we calibrate new test questions
Asses - Accurately pinpoint baseline skills

Create custom assessments with
Content Engine™