Posting Externally Derived TFS Test Case Results

0
33

Introduction

Whether from the need to test in production environments, or run tests that integrate with tools that cannot run within the Microsoft Build and Development environments, there is still a need to report test case results back to the Team Foundation Server (TFS).  This is especially true for automated tests, requiring an automated posting of Test Case Results.  Here is one way to approach this from an externally run utility that integrates with the Microsoft.TestManagment .NET assemblies.

Caveats

  • Please use at your own risk and [in]convenience.
  • Dependencies on Microsoft’s Team Explorer.  Microsoft’s Team Explorer (TE) is freely available, with multiple versions to match your versions of Visual Studio and TFS.  Because of this variety of environments, as well as respect for redistribution licensing (or lack thereof), the dependent TE assemblies are not included and must be acquired and configured to run independently.
  • Dense, Partial Code Samples.  Apologies in advance for the inappropriately dense coding style.  Please beautify as needed.  For brevity, error checking and variable acquisition has been removed from the code snippets in this article.

Posting Test Case Results

Step One: Find and Add TestManager References

To start, you must download and install Microsoft’s Team Explorer.  Next, map references from your project to Microsoft.TeamFoundation.TestManagement.Client, and Microsoft.TeamFoundation.TestManagement.Common.

You will also need to reference Microsoft.TeamFoundation.Client, Microsoft.TeamFoundation.Common, Microsoft.TeamFoundation.WorkItemTracking.Client, and Microsoft.TeamFoundation.WorkItemTracking.Common assemblies.

Though this will place about 30 assemblies into your output directory, two important files from the Team Explorer assemblies are left out and cannot be referenced directly.  The work around to this is to find the dlls and place them into your project as ‘Build Action = None’ and ‘Copy if newer’ files.

If you add Microsoft.WITDataStore32.dll and Microsoft.WITDataStore64.dll from the Add|Existing Item,… project menu, a static copy of the dlls will be copied into your project source files folder.  My OCD makes me close the project, delete the copied Microsoft dlls, and edit the .csproj file to point back to the original directory, e.g.:

<code>Before:
<ItemGroup>
 <None Include="Microsoft.WITDataStore32.dll">
 <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
 </None>
 <None Include="Microsoft.WITDataStore64.dll">
 <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
 </None>
</ItemGroup>
 
After:
<ItemGroup>
 <None Include="..\..\..\..\..\Program Files (x86)\Microsoft Visual Studio\2017\TestPro\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Microsoft.WITDataStore32.dll">
 <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
 </None>
 <None Include="..\..\..\..\..\Program Files (x86)\Microsoft Visual Studio\2017\TestPro\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Microsoft.WITDataStore64.dll">
 <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
 </None>
</ItemGroup></code>

When you deploy your ‘TCResults’ executable, you will have to enclude the entourage of thirty or so dlls that get stuffed in the project build output directory along with your executable.

Step Two: Secure the TfsTeamProjectCollection Object.

An initial step when integrating with the TFS .net objects is to get the ProjectCollection from a TFS server URI and the name of the project.  This static routine returns the ProjectCollection, if it can be found.

<code>static TfsTeamProjectCollection GetTeamProjectCollection(Uri TFSServer, string ProjectName)
{
 TfsConfigurationServer configServer = TfsConfigurationServerFactory.GetConfigurationServer(TFSServer);
 ReadOnlyCollection<CatalogNode> projects = configServer.CatalogNode.QueryChildren(new[] { CatalogResourceTypes.ProjectCollection }, false, CatalogQueryOptions.None);

 foreach (CatalogNode pc in projects)
 {
 Guid collection_id = new Guid(pc.Resource.Properties["InstanceId"]);
 ReadOnlyCollection<CatalogNode> teamProjects = pc.QueryChildren(new[] { CatalogResourceTypes.TeamProject }, false, CatalogQueryOptions.None);

 foreach (CatalogNode tp in teamProjects)
 {
 if (string.Compare(ProjectName, tp.Resource.DisplayName, CLU.Option.Collection.CaseSensitivity) == 0)
 return configServer.GetTeamProjectCollection(collection_id);
 }
 }
 return null;
}</code>

Step Three: Secure the TestPlan to Post to.

(bring your own bolded variable values)

<code>TfsTeamProjectCollection teamCollection = GetTeamProjectCollection(_tfs_server_uri, _project_name);
ITestManagementService tms = teamCollection.GetService<ITestManagementService>();
ITestManagementTeamProject testProject = tms.GetTeamProject(_project_name);
string wis_command =
 (string.IsNullOrWhiteSpace(_test_plan_name))
 ? " SELECT * FROM TestPlan WHERE [PlanState] = 'Active'"
 : " SELECT * FROM TestPlan WHERE [PlanState] = 'Active' AND [Title] = '" + _test_plan_name + "' ";
ITestPlanCollection testplanCollection = testProject.TestPlans.Query(wis_command);</code>

Explanatory Interlude

Now it gets messy and confusing, muddled with my own conjectures based upon working with a TFS 2013 on-premise server.

Here at fluentbytes.com is an excellent implementation guide, posted by Marcel de Vries, on the consumption of TFS TestCases and TestCaseResults, through the TestManagement interfaces.  From this I got a basic idea of how the .NET objects of the TestManagement assembly were organized, crudely shown below.

But writing new objects into the store is much more constrained than implied by this read-only diagram.  Below is an interface usage diagram of the related TestManagement objects.

What is not discernable from Marcel de Vries‘ article is the relationship and sequencing requirements of TestCaseResults, TestRuns, TestPoints, TestCases, TestSuites, and TestPlans.

The Basic Constraints

  1. TestCaseResults can only be assigned to the TestPoints of a TestRun.
  2. (In my exerience) TestPoints can only be added to a TestRun after it is created, and before it is saved.
  3. TestPoints are a matrix intersection of TestCases with TestSuite configurations.
  4. The TestRun must be created under a valid, preferably “Active” TestPlan.
  5. The TestCaseResult objects are instantiated when the TestRun is initially saved.
  6. The TestCase associated with the TestCaseResult, must also be associated with a targeted TestSuite under the targeted TestPlan.  This connotes a sibling relationship between the new TestRun and the TestSuite, both having the same parent TestPlan.  And a cousin relationship between the TestCase and the TestPoint.

Step Four: Indirectly Create the TestCaseResult Placeholders

To have the placeholder TestCaseResult objects instantiated for the TeamProject‘s TestCase(s) for which there are results, it must be verified that the TestCase(s) are associated with the designated TestSuite.  Then a new TestRun must be created with the appropriate TestPoint(s) added that are associated with the TestCase(s).  When the TestRun is saved, the placeholder TestCaseResult objects are instantiated.  Below the TestCases are queried by Id or by the AutomatedTestName field.

(bring your own bolded variable values)

<code>foreach (ITestPlan testPlan in testplanCollection)
{
 int suite_index = 0;
 ITestSuiteBase testSuite =
 (testPlan.RootSuite.SubSuites.Count > suite_index) 
 ? testPlan.RootSuite.SubSuites[suite_index] : testPlan.RootSuite;
 
 while (testSuite != null)
 {
 if (((ITestSuiteBase2)testSuite).Status.Equals("In Progress")
 &&
 (string.IsNullOrWhiteSpace(_test_suite_name)
 || _test_suite_name.Equals(testSuite.Title))
 &&
 (!string.IsNullOrWhiteSpace(_test_case_id)
 || !string.IsNullOrWhiteSpace(_automated_test_name))
 )
 {
 string testcase_query = null;
 
 if (!string.IsNullOrWhiteSpace(_test_case_id))
 testcase_query = "SELECT * FROM WorkItems WHERE [Work Item Type] = 'Test Case' AND [Id] = '" + _test_case_id + "'";
 else if (!string.IsNullOrWhiteSpace(_automated_test_name))
 testcase_query = "SELECT * FROM WorkItems WHERE [Work Item Type] = 'Test Case' AND [Microsoft.VSTS.TCM.AutomatedTestName] = '" + _automated_test_name + "'";
 
 foreach (ITestCase test_case in testProject.TestCases.Query(testcase_query))
 {
 ITestRun testRun = testPlan.CreateTestRun(true);
 ITestPointCollection testpointCollection = testPlan.QueryTestPoints("SELECT * FROM [TestPoint] WHERE [TestCaseId] = '" + test_case.Id.ToString() + "'");
 if (testpointCollection.Count == 0)
 {
 testSuite.TestCases.Add(test_case);
 testPlan.Save();
 
 testPlan.Refresh();
 testSuite.Refresh(true);
 testpointCollection = testPlan.QueryTestPoints("SELECT * FROM [TestPoint] WHERE [TestCaseId] = '" + test_case.Id.ToString() + "'");
 }
 
 foreach (ITestPoint test_point in testpointCollection)
 {
 if (test_point.ConfigurationName.Equals(_test_configuration_name))
 testRun.AddTestPoint(test_point, null);
 }
 
 testRun.Title = test_case.Title;
 testRun.Save();</code>

Step Five: Post and Save the TestCaseResult

Now that the placeholder TestCaseResut objects have been instantiated, their internal values can be set.

(bring your own bolded variable values)

<code> foreach (ITestCaseResult tcr in testRun.QueryResults(false))
 {
 switch (_tcr_outcome)
 {
 case "NONE":
 tcr.Outcome = TestOutcome.None;
 tcr.State = TestResultState.Completed;
 break;
 
 case "PASS":
 case "PASSED":
 tcr.Outcome = TestOutcome.Passed;
 tcr.State = TestResultState.Completed;
 break;
 
 case "FAIL":
 case "FAILED":
 tcr.Outcome = TestOutcome.Failed;
 tcr.State = TestResultState.Completed;
 break;
 
 case "INCONCLUSIVE":
 tcr.Outcome = TestOutcome.Inconclusive;
 tcr.State = TestResultState.Completed;
 break;
 
 case "TIMEOUT":
 tcr.Outcome = TestOutcome.Timeout;
 tcr.State = TestResultState.Completed;
 break;
 
 case "ABORTED":
 tcr.Outcome = TestOutcome.Aborted;
 tcr.State = TestResultState.Completed;
 break;
 
 case "BLOCK":
 case "BLOCKED":
 tcr.Outcome = TestOutcome.Blocked;
 tcr.State = TestResultState.Pending;
 break;
 
 case "NE":
 case "NOTEXECUTED":
 tcr.Outcome = TestOutcome.NotExecuted;
 tcr.State = TestResultState.Pending;
 break;
 
 case "WARNING":
 tcr.Outcome = TestOutcome.Warning;
 tcr.State = TestResultState.Unspecified;
 break;
 
 case "ERROR":
 tcr.Outcome = TestOutcome.Error;
 tcr.State = TestResultState.Unspecified;
 break;
 
 case "NA":
 case "NOT APPLICABLE":
 tcr.Outcome = TestOutcome.NotApplicable;
 tcr.State = TestResultState.Completed;
 break;
 
 case "PAUSED":
 tcr.Outcome = TestOutcome.Paused;
 tcr.State = TestResultState.Paused;
 break;

 case "INPROGRESS":
 tcr.Outcome = TestOutcome.InProgress;
 tcr.State = TestResultState.InProgress;
 break;
 
 case "UNSPECIFIED":
 default:
 tcr.Outcome = TestOutcome.Unspecified;
 tcr.State = TestResultState.Unspecified;
 break;
 }
 tcr.Comment = _tcr_comment;
 
 tcr.Save();
 }
 testRun.QueryResults(false).Save(false);
 }
 }
 testSuite = (testPlan.RootSuite.SubSuites.Count > ++suite_index) ? testPlan.RootSuite.SubSuites[suite_index] : null;
 }
}</code>

Conclusion

With the ability to post more exhaustive Test Case results from external sources, your projects and teams will have better visibility into your product’s quality metrics.

History

2017.07.27 – Initial Release

LEAVE A REPLY