polyspace.test.TestCaseResult Class

Namespace: polyspace.test

(Python) Review test results for a specific test case in a suite

Since R2024b

Description

This Python^® class contains test results for a specific test case in a C/C++ test suite.

Creation

Syntax

testCaseResult = testSuiteResult.TestCaseResults[testCaseIndex]

Description

testCaseResult = testSuiteResult.TestCaseResults[testCaseIndex] loads the results of test suite number testSuiteIndex:

• From a polyspace.test.TestSuiteResult object testSuiteResult.

• To a polyspace.test.TestCaseResult object testCaseResult.

Here, testCaseIndex is the index of the test case specified as an integer starting from 0.

Properties

Name — Name of test case

string

Name of test case.

Status — Execution status of test case

polyspace.test.DetailedStatusInfo object

Overall execution status of the test case, specified as a polyspace.test.DetailedStatusInfo object with the following properties:

Property	Values	Description
Failed	True or False	The value of this property is True if the test fails or encounters a runt-time error or exception. The value can also be True if the test is expected to fail but passes instead. In all other cases, the value is False.
Incomplete	True or False	The value of this property is True if the test does not run to completion because of an assertion or assumption failure. For more information on assertions and assumptions, see Assessment Macros in Polyspace Test API for C/C++ Code. In all other cases, the value is False.
VerifyFail	True or False	The value of this property is True if the test contains a VERIFY assessment failure, otherwise the value is False. For more information on VERIFY assessments, see Assessment Macros in Polyspace Test API for C/C++ Code.
AssertFail	True or False	The value of this property is True if the test contains an ASSERT assessment failure, otherwise the value is False. For more information on ASSERT assessments, see Assessment Macros in Polyspace Test API for C/C++ Code.
AssumeFail	True or False	The value of this property is True if the test contains an ASSUME assessment failure, otherwise the value is False. For more information on ASSUME assessments, see Assessment Macros in Polyspace Test API for C/C++ Code.
Exception	True or False	The value of this property is True if the test aborted because of an exception being thrown (C++ only), otherwise the value is False.
NotRun	True or False	The value of this property is True if the test did not start. A test might not start for many reasons. For instance, there might be an assertion or assumption failure or runtime error in the suite setup. Otherwise, the value is False.
Crash	True or False	The value of this property is True if the test aborted because of a run-time error, otherwise the value is False.
ExpectFail	True or False	The value of this property is True if the test was expected to fail, otherwise the value is False. For more information on expected failures, see Write C/C++ Tests with Known Errors or Failures to Support Test-Driven Development.

File — File containing test definition

string

File containing test definition.

Line — Line number where test definition begins

integer

Line number where test definition begins. This is the line containing the PST_SIMPLE_TEST, PST_TEST, or PST_TEST_CONFIG macro (depending on whether your test case is a simple test, a test in a named suite without a configuration, or a test in a named suite with a configuration).

Timing — Time of test case execution and timestamp

polyspace.test.TimingInfo object

Timing of suite or test execution and timestamp of execution, specified as a polyspace.test.TimingInfo object with the following properties:

Property	Value	Description
Timestamp	datetime.datetime value	Time stamp of the beginning of test suite execution, for example, 2024-Jul-04 10:38:21.
ExecutionTime	Floating-point value	Time taken for test suite execution in seconds (up to millisecond precision).

AssessmentCount — Total number of assessments

integer

Total number of assessments in the test. This number includes passing and failing assessments.

PassedAssessmentCount — Number of passing assessments

integer

Number of passing assessments in the test case.

FailedAssessmentCount — Number of failing assessments

integer

Number of failing assessments in the test case.

AssessmentResults — Result details per assessment

list of polyspace.test.AssessmentResult objects

Result details for assessments in the test case, returned as a list of polyspace.test.AssessmentResult objects with one object per assessment.

For more information, see polyspace.test.AssessmentResult.

PassedAssessmentResults — Result details per passing assessment

list of polyspace.test.AssessmentResult objects

Result details for passing assessments in the test case, returned as a list of polyspace.test.AssessmentResult objects with one object per passing assessment.

For more information, see polyspace.test.AssessmentResult.

FailedAssessmentResults — Result details per failing assessment

list of polyspace.test.AssessmentResult objects

Result details for failing assessments in the test case, returned as a list of polyspace.test.AssessmentResult objects with one object per failing assessment.

For more information, see polyspace.test.AssessmentResult.

Messages — Messages emitted during test execution

list of polyspace.test.Message objects

Messages emitted during test execution that are not related to assessments. For instance, an issue unrelated to assessments can occur during test execution and a warning message might be emitted The message is specified as a polyspace.test.Message object with the following properties:

Property	Description
Phase	The value can be one of the following: TestPhase.NONE TestPhase.TEST TestPhase.TEST_CONFIG TestPhase.TEST_SETUP TestPhase.TEST_TEARDOWN TestPhase.SUITE_TEST_SETUP TestPhase.SUITE_TEST_TEARDOWN
Type	The value can be one of the following: MessageType.NONE MessageType.WARNING MessageType.ERROR MessageType.FATAL_ERROR MessageType.INFO
Msg	Actual content of the message.

Events — Events that occurred during test execution

list of polyspace.test.Message and polyspace.test.AssessmentResult objects

Events that occurred during test execution specified as a heterogeneous list of polyspace.test.Message and polyspace.test.AssessmentResult objects. The sequence of objects in the list represents the actual sequence in which the events occurred.

For instance, suppose that the body of a test contains one assessment but during test execution, a warning message is emitted before the assessment runs. The Events property contains two objects:

• Events[0] is a polyspace.test.Message object that contains details of the warning message. For more information on this object, see Messages.

• Events[1] is a polyspace.test.AssessmentResult object that contains details of the assessment result. For more information on this object, see AssessmentResults.

The Events property allows you to see the sequence in which the events occurred. If you are not interested in the sequence, use the Messages property to check for warnings and other messages, and the AssessmentResults property to retrieve the results of assessments.

Examples

Print Names of Failing Test Cases

This example shows to print the names of failing test cases after running tests in a project.

In general, you generate and manage Polyspace^® Test™ results by using classes from the polyspace.project and polyspace.test modules. Before starting, make sure you can import these modules on a Python shell or in a Python script without errors. For more information, see Set Up Python API for Polyspace.

1. Import the required modules:

   import polyspace.project
   import polyspace.test
   import os

2. Add source files and xUnit test files to a project. This example uses some example source and test files available with a Polyspace Test installation. Instead, you can use your own sources and tests.

   examples_path = os.path.join(polyspace.__install_path__, "polyspace", 
                               "examples", "pstest", "Getting_Started_Example")
   polyspaceProject = polyspace.project.Project("newProject")
   polyspaceProject.Code.Files.add(os.path.join(examples_path, "sources", "utils.c"))
   polyspaceProject.IncludePaths.add(os.path.join(examples_path, "includes"))
   polyspaceProject.Tests.Files.add(os.path.join(examples_path, "tests", "test.c"))

3. Run the tests added to the project.

   res = polyspace.test.run(polyspaceProject)

4. Print the names of failing test cases (along with their suite names).

   for suiteIndex in range(len(res.TestSuiteResults)):
       # Read result of test suite
       resTestSuite = res.TestSuiteResults[suiteIndex]
       if(resTestSuite.Status.Failed):
           for testIndex in range(len(resTestSuite.TestCaseResults)):
               # Read result of test case
               resTestCase = resTestSuite.TestCaseResults[testIndex]
               if(resTestCase.Status.Failed):
                   print(resTestSuite.Name + "/" + resTestCase.Name)

   Alternatively, you can simply iterate through the test suites and cases as follows:

   for resTestSuite in res.TestSuiteResults:
       if resTestSuite.Status.Failed:
           for resTestCase in resTestSuite.TestCaseResults:
               if resTestCase.Status.Failed:
                   print(f"{resTestSuite.Name}/{resTestCase.Name}")
   

   If you use the example source and test files, you should see the suite and test name updateStateTests/addLatestReading_test_1_fail printed.

Version History

Introduced in R2024b