Test Automation Using Atata: Visual Studio Team Services Configuration

    0
    83

    Introduction

    In this article, I would like to show the way how to configure the Visual Studio Team Services to build and run Atata auto-tests using any browser (for Windows platform). I will also show how to view Atata execution log and failed test details.

    Atata Series

    1. Atata – New Test Automation Framework
    2. Test Automation Using Atata: Verification of Web Pages
    3. Test Automation Using Atata: Verification of Validation Messages
    4. Test Automation Using Atata: Visual Studio Team Services Configuration

    Background

    It is recommended to read Atata – New Test Automation Framework article to get familiar with basic Atata Framework concepts. But actually, you may also use this artcile as an instruction even for pure WebDriver .NET test automation, as it moslty describes VS Team Servcies configuration.

    Prepare Team Services Project

    Let’s assume that we have a VS Team Services account with some auto-tests hosted in sources. For testing purposes I used tests from atata-framework/atata-sample-app-tests repository.

    How to sign up for Visual Studio Team Services: https://www.visualstudio.com/en-us/docs/setup-admin/team-services/sign-up-for-visual-studio-team-services.

    Create Build Agent

    If PhantomJS driver is good for you, you may navigate to “Create Build” section (and skip 9th point), as custom build agent is not required in this case. Otherwise you will need a machine for custom build agent with installed browser(s).

    Prepare Build Machine

    1. It is needed to have Visual Studio installed on the server machine.
    2. Install needed browser(s) (e.g. Google Chrome).

    Prepare Account Permissions

    You can find the complete documentation on permissions configuration here: https://www.visualstudio.com/en-us/docs/build/actions/agents/prepare-permissions. Create personal access token and copy it.

    Configure Build Agent

    The process of configuration is described here: https://www.visualstudio.com/en-us/docs/build/actions/agents/v2-windows#download-and-configure-the-agent. But I will try to describe it for our needs shortly.

    1. Go to https://{your-account}.visualstudio.com/_admin/_AgentPool (“Account settings/Agent Pools”) and click “Download agent” button.Download build agent
    2. Unzip the package to the folder where it will run (e.g. D:\TeamServices) and run config.cmd as administrator.
    3. Enter server/account URL: https://{your-account}.visualstudio.com.
    4. Press Enter to use PAT authentication.Configure Agent Authentication
    5. Insert copied personal access token and press Enter.
    6. Press enter to use the default agent pool.
    7. Enter agent name build, e.g. BuildAgent1.
    8. Press enter to use the default work folder name (_work) or specify another.
    9. Enter Y and press Enter to run agent as a service.Configure build agent run
    10. Press Enter to use the default service account or specify another.

    The build agent is created and run as a service. You can go to https://{your-account}.visualstudio.com/_admin/_AgentPool to check the agent.

    Create Build

    1. Navigate to projects page https://{your-account}.visualstudio.com/_projects and select the project.
    2. Using top menu navigate to “Build & Release/Builds”.
    3. Click “New” button.
    4. Select “empty process” on top.
    5. Specify name (e.g. “Build & Run Auto-Tests”).Input build name
    6. Add “NuGet Restore” task.
    7. Add “Visual Studio Build” task. You can change “Visual Studio Version” for appropriate.
    8. Add “Visual Studio Test” task.
    9. Select “Options” tab and change “Default agent queue” drop-down value to “Default”.
    10. You may also enable the triggers on “Triggers” tab to run the build on commits.
    11. Click on top “Save & queue/Save & queue” to add a build to a queue or just “Save & queue/Save”.
    12. Using top menu navigate to “Build & Release/Builds” and then switch to “All Definitions”.
    13. Click on your newly created build (e.g. “Build & Run Auto-Tests”).
    14. Click “Queue new build…” button and then “OK” on popup.

    View Test Results

    After the build is finished you can check the test results

    1. Using top menu navigate to “Test/Runs”.
    2. Click the test run and verify the results.Test run summary
    3. In “Attachments” section there should be TRX file added. You can click it and open with Visual Studio to see detailed (with Atata log output) test results.Test summary TRX file
    4. In “Test Results” tab you can find successful and failed tests. To see failed test details, click it and you’ll see “Error Message”, “Stack trace” and “Attachments (1)” sections.Test SummaryThere is an Atata log in Attachments, it is named “Standard Console Output.log”. Click, and open it using text editor application (better Notepad++ or Visual Studio, as regular Notepad can open the file without line breaks). If you use screenshots, in the log you will see the screenshot file paths (it is path of the file on build agent machine).
    5. Team Services, after the build is completed, also send build results letter with such content:Build results letter

    Here is the sample of the failed test log:

    2017-06-07 10:32:40.0804 INFO Starting test: Home_Fail 2017-06-07 10:32:40.0819 INFO Init WebDriver 2017-06-07 10:32:42.7913 INFO Go to "Home" page 2017-06-07 10:32:42.7938 INFO Go to URL "http://atata-framework.github.io/atata-sample-app/#!/" 2017-06-07 10:32:46.1065 INFO Verify title should equal "Atata Sample App" 2017-06-07 10:32:46.1570 INFO Verify "Atata Sample App" 

    heading content should equal "Atata Sample App" 2017-06-07 10:32:46.3101 INFO Verify content should contain "Lorem ipsum" 2017-06-07 10:32:46.3627 INFO Verify title should equal "Wrong Title" 2017-06-07 10:32:56.5635 ERROR Atata.AssertionException : Invalid "Home" page title. Expected: should equal "Wrong Title" Actual: "Atata Sample App" at Atata.IDataVerificationProviderExtensions.Satisfy[TData,TOwner](IDataVerificationProvider`2 should, Predicate`1 predicate, String message, TData[] args) at Atata.IDataVerificationProviderExtensions.Equal[TData,TOwner](IDataVerificationProvider`2 should, TData expected) at AtataSampleApp.AutoTests.HomeTest.Home_Fail() in D:\TeamServices\_work\1\s\src\AtataSampleApp.AutoTests\HomeTest.cs:line 18 2017-06-07 10:32:56.5640 INFO Take screenshot #01 - Failed 2017-06-07 10:32:56.6390 INFO Screenshot saved to file "D:\TeamServices\_work\1\s\src\AtataSampleApp.AutoTests\bin\Debug\Logs\2017-06-07 10_32_29\Home_Fail\01 - Home page - Failed.png" 2017-06-07 10:32:56.6395 INFO Clean up test context 2017-06-07 10:32:56.7101 INFO Finished test (16.629s) 2017-06-07 10:32:56.7101 INFO Pure test execution time: 13.848s

    Using the screenshot file path you can easily open the screenshot on build machine.

    Video Guide

    [embedded content]

    References

    History

    • June 7, 2017: Initial version posted.

    LEAVE A REPLY