1. Home
  2. Training Library
  3. Microsoft Azure
  4. Courses
  5. Adding Mobile Devices to Your Azure DevOps Strategy

DEMO: App Center CLI UI Testing


Course Introduction
Mobile Build Services
Distribution Groups
Analytics and Diagnostics

The course is part of this learning path

AZ-400 Exam Prep: Microsoft Azure DevOps Solutions
Start course
1h 6m

This course dives into creating a DevOps strategy for mobile applications using the Visual Studio App Center. The App Center gives us a centralized location where we can implement build services, carry out mobile UI testing with multiple devices sets, create public and private distribution groups, and perform release management for our distribution groups.

Build services include tvOS, iOS, Android, Xamarin, and more. UI testing includes UI testing across many popular flagship devices and uses a tier system to gradually cycle out older devices as newer ones become available. Distribution groups allow for deploying new releases to multiple types of users and can be utilized across multiple applications or projects. This course explains many of the options and services available in the App Center and provides guided demonstrations with a mobile app to show how each of these available services works.

By the end of this course, you should know how to utilize App Center to implement a DevOps strategy for integrating and deploying a mobile application. If you have any feedback relating to this course, please contact us at support@cloudacademy.com.

Learning Objectives

  • Create an App Center account and build a new application
  • Perform UI tests in both the App Center CLI and Azure DevOps pipeline
  • Create public and private distribution groups and release apps to them
  • Understand and use the analytics and diagnostics provided by App Center

Intended Audience

This course is intended for DevOps and IT professionals who are looking to implement a DevOps strategy for mobile applications using Visual Studio App Center to build, test, distribute, and deploy their mobile applications. This course is also useful for those preparing to take Microsoft's AZ-400 exam.


To get the most out of this course, you should know how to build and upload mobile applications if you are not using the build tools provided by the App Center. You should know how to work with Node package manager and have NPM installed or know how to install it. You should also be fairly comfortable with using command-line interface tools.


You can find the project files for a sample mobile application in the following GitHub repository: https://github.com/kelsosharp/MobileDevOps



In this demo, we're going to use the App Center command-line interface to upload and run our unit tests from our test project. This requires that you have Node js 6.3 or later, npm and the App Center CLI installed. You also need to make sure that the testing framework you picked when you set up your test run is also installed and configured for the project. In our case, we're using the Xamarin.UITest.

So now that we have a successful build, let's have a look at testing. We will select the test menu option on the left side and that will bring up our test runs page. If you've already had a few test runs, this will show a list of the runs here. If you've set your build configuration to do an initial test on a real device, you will see that test run listed here.

If you do not have any test runs yet, there should be a button that says start testing your app. Clicking this will bring up a dialog, that asks you if you want to start your free trial of device testing. We can click on the Start Trial button and this will bring up a list of devices to choose from.

We can select whichever devices that you'd like to use to test your mobile application on. You can filter the device list by Foreign factor, CPU, memory, operating system and manufacturer. Once you have a list of devices you want to use, just check the checkbox beside each one or check the checkbox at the top of the list to select them all. There's a lot of options here. If you want to see the list of all the devices available, you can check them out at this URL.

When you run the command to upload and execute your UI tests using the App Center CLI, there's a few pieces of information that would be helpful to have handy, or to know when typing out the command. The first is your App Center username or organization name. However, you need to replace the @ symbol in any spaces with a hyphen as these cannot be used in the command-line. The next piece of information that's handy to have, is the path to the APK file that's built by your project. You can specify this path when you build your Android application for the first time in Visual Studio.

The next piece of information that you need to have is the path to the cloud-test.exe file in the Xamarin.UITest NuGet package for our test project. Xamarin no longer uses a NuGet.config file or creates a packages folder as part of the project. So you need to specify the path using the argument, --uitest-tools-dir, to specify the path. This will be different depending on the version of the NuGet package you're using. And if you upgrade your NuGet package, you will need to update this path as well.

This is a fragile and brittle implementation and can cause you hours of frustration if you're not aware of it. The App Center Dev team at Microsoft is aware of this issue and is currently working on providing a better way to handle it.

Finally, you will need the name of the device set in the App Center, that you want to use to test your project. If you create a new series without first creating a device set, there will be a button named Save set, in the top right of the configuration page. If you do not click this and give the set a name, a random ID will be created for the set and the set itself will not be saved.

Once you've chosen your testing series and your testing framework, you will then be given the opportunity to copy the appcenter-cli command to the clipboard. This command has the device set random identifier, and this is what we used for this demo. But if you do not copy to the clipboard now, or you do not click the Save set button and give it a name, you will not be able to find it again once you leave this page. So I recommend that you either create a device set prior to creating a test series or save the device set for later use.

I'm just going to take these paths and paste them along with the command in the notepad here to make it easier for me to edit. You will need to use this every time you want to run the UI tests on this device set in App Center. The identifier given as the name for the device set defines a list of devices being used for the tests. So if you copy these identifiers, you can reuse them in PowerShell or bash so that you can run multiple UI tests against multiple device sets.

Now, I will copy the command and run it in our command window. This, will upload the project files and run the unit tests that are in the Xamarin.UITest project on all the devices in the test series we named Master.

Now as we watch, we can see that these tests are being run in sequence and not in parallel. If we had paid for additional concurrent devices for our apps in our account, we could have run these in parallel and reduce the length of time needed to run our tests. For just a few devices, this is not too much of a problem. But if our device set had hundreds of devices, it could take many hours to get through them all and we would like to run out of device minutes quite quickly.

Once our tests have completed, we can look at the test run page in App Center and see that they've passed successfully. And we can have a look at the results for each of the test as an image on the device it was run on.

That's all there is to having App Center run our UI unit tests on actual physical devices. This gives us the comfort of knowing exactly how our UI will run on each of the devices we used to test with. A small price to pay to know our user experience will be consistent across many devices.

In our next module, we'll be demonstrating how to integrate Apps Center UI testing into an Azure DevOps Pipeline. It's some pretty cool stuff, so let's go check it out.

About the Author
Kelso Sharp

As well being the owner and CTO of Sharp Solutions Group, a software development and IT staffing company based in the Philippines, Kelso is a Microsoft Certified professional and an avid knowledge seeker. His belief is that you need to learn something new each day to stay on top of the constantly changing IT world. He is an avid gamer (both video games and board games) and lives in the Philippines with his wife and soon-to-be-delivered son.