Author Graphical Tests Using Python API for Polyspace

You can author graphical tests for your C/C++ code base entirely at the command line by using the Python® API for Polyspace®. This example demonstrates the workflow for using the API to:

  • Parse source code and store information about types, functions, and global variables in a polyspace.project.CodeInfo object.

  • Create test suites, test cases, and test data.

  • Create simple test steps. Use information from the polyspace.project.CodeInfo object to populate the test steps with functions to test, inputs, and assessments.

  • Create test steps that use pointer targets.

  • Create complex tests including parametrized tests and scripted tests.

Prerequisites

Make sure you are using a supported Python version and you are able to import the polyspace.project and polyspace.test Python modules without errors. For more information, see Set Up Python API for Polyspace.

Before you use the Polyspace Python API to author tests, it is helpful to have a basic understanding of graphical test authoring within the Polyspace Platform user interface. See Test Authoring in Polyspace Platform User Interface.

Parse Code

Import the modules polyspace.project and polyspace.test, create a project, and add the source files in the folder <polyspaceroot>\polyspace\doc_pstest\getting_started. Then, use the polyspace.project.parseCode function to parse the source code.

# Import polyspace modules.
import polyspace.project
import polyspace.test
import os

# Create project and add files.
proj = polyspace.project.Project("myProject")
examples_path = os.path.join(polyspace.__install_path__, "polyspace", 
                            "examples", "doc_pstest", "python_api_test_authoring", "src")
proj.Code.Files.add(os.path.join(examples_path, "saturate.c"))
proj.Code.Files.add(os.path.join(examples_path, "algo.c"))

# Parse source code.
codeInfo = polyspace.project.parseCode(proj)

The polyspace.project.parseCode function returns an instance of the polyspace.project.CodeInfo class that contains information about functions, types, and globals in the source code. You can use the methods of this class to retrieve this information and use them to author your test cases.

Add Simple Test

Create a simple test for the saturate_value function that supplies an input value 2 to the function and checks the output against the assessment value 2. The test also checks that the value of the global variable minValue remains equal to its initial value of -2.

# Create test suite and test case. 
suiteSaturateValue = proj.TestSuites.create("checkSaturateValueTests")
simpleTest = suiteSaturateValue.TestCases.create("saturate_value_simpleTest")

# Create test step for a simple test.
functionSaturateValue = codeInfo.getFunctionBySignature("int saturate_value(int)")
simpleStep = simpleTest.TestSteps.createTabular("saturate_value_simpleStep", functionSaturateValue)

# Update inputs and assessments that are automatically created.
simpleStep.Inputs["value"].Value = "2"
simpleStep.Assessments["pst_call_out"].Value = "2"

# Create a new assessment to check that the global variable "minValue" is unchanged.
minValue = codeInfo.getGlobalByName("minValue")
simpleStep.Assessments.create(minValue)
simpleStep.Assessments["minValue"].Value = "-2"

Note

You can only assign strings to the Value property of an input, assessment, parameter, test data, or observable. Therefore, put quotes around the values you want to assign. For example:

  • To specify the integer 42, assign the string "42" to the Value property.

  • To specify the double 3.14, assign the string "3.14" to the Value property.

  • To specify the string "My String", assign the string '"My String"' to the Value property.

  • To specify the address of a variable var, assign the string "&var" to the Value property.

When generating code for your tests, Polyspace Test™ uses the content of the string to reconstruct the value of the input, assessment, parameter, test data, or observable.

Add Simple Test with Function Call Count Assessments

Create a simple test for the saturate_and_cache function, and add function call count assessments to verify that the getMinValue function is called at least once and the getMaxValue function is called exactly once.

# Create new test suite and test case. 
suiteSaturateAndCache = proj.TestSuites.create("checkSaturateAndCacheTests")
testCallCount = suiteSaturateAndCache.TestCases.create("saturate_and_cache_call_count_test")

# Create test step for the saturate_and_cache function.
functionSaturateAndCache = codeInfo.getFunctionBySignature("int saturate_and_cache(int)")
callCountStep = testCallCount.TestSteps.createTabular("saturate_and_cache_simpleStep", functionSaturateAndCache)

# Add call count assessments for the getMinValue and getMaxValue functions.
functionGetMinValue = codeInfo.getFunctionBySignature("int getMinValue(void)")
functionGetMaxValue = codeInfo.getFunctionBySignature("int getMaxValue(void)")
callCountStep.Assessments.create(functionGetMinValue) 
callCountStep.Assessments.create(functionGetMaxValue) 

# Modify the getMaxValue call count assessment to verify that the function is called once.
callCountStep.Assessments["getMaxValue()"].Value = "1u"
callCountStep.Assessments["getMaxValue()"].Comparator = "EQUAL"

# Display the test assessments table.
print(callCountStep.Assessments)

# Update the expected value of the assessments that were automatically created for minValue and maxValue.
callCountStep.Assessments["minValue"].Value = "-2"
callCountStep.Assessments["maxValue"].Value = "2"
print(callCountStep.Assessments) 

# (Optional) Run the current tests and generate an HTML report to view results.
res = polyspace.test.run(proj)
res.generateHTMLReport("myCallCountReport")

Add Parameterized Test

Add a test that exercises the saturate_value function for all integer inputs in the range [-50,50] and checks that the output always lies between -2 and 2.

# Create test case and test step for a parameterized test in the existing suite "checkSaturateValueTests".
parameterizedTest = suiteSaturateValue.TestCases.create("saturate_value_parameterizedTest")
parameterizedStep = parameterizedTest.TestSteps.createTabular("saturate_value_parameterizedStep",functionSaturateValue)

# Create integer test parameter that ranges from -50 to 50.
params = parameterizedTest.TestParameters
testParam = params.Parameters.create("myParam", codeInfo.getType("int[101]"))
for k in range(101):
 testParam[k].Value = str(k-50)

# Update the automatically created input and assessment. Check that output <= 2.
parameterizedStep.Inputs["value"].Value = testParam

parameterizedStep.Assessments["pst_call_out"].Value = "2"
parameterizedStep.Assessments["pst_call_out"].Comparator = polyspace.project.AssessmentComparator.LESS_EQUAL

# Create a new assessment to check that output >= -2.
assessment = parameterizedStep.Assessments.create(parameterizedStep.ReturnValue)
assessment.Value = "-2"
assessment.Comparator = polyspace.project.AssessmentComparator.GREATER_EQUAL

Add Test with Pointer Target

Create a simple test for the saturate_and_cache_array function. This function accepts an array of integers values as an input. Therefore, you create two pointer targets — one for the input and the other for the corresponding assessment.

The saturate_and_cache_array function also reads and/or potentially modifies the global variables ErrorMsg, minValue, maxValue, and hasError. However, this example only uses inputs and assessments for the function parameters values and N. Therefore, in this script, you use the pop method to explicitly remove the inputs and assessments that are automatically created for the global variables.

# Add a test case and test step in the existing suite "checkSaturateAndCacheTests".
testPointerTarget = suiteSaturateAndCache.TestCases.create("saturate_and_cache_array_test")
functionSaturateAndCacheArray = codeInfo.getFunctionBySignature("void saturate_and_cache_array(int*, unsigned)")
stepPointerTarget = testPointerTarget.TestSteps.createTabular("saturate_and_cache_array_step",functionSaturateAndCacheArray)

# Remove inputs and assessments for the global variables. This test only involves variables of parameter scope.
stepPointerTarget.Inputs.pop("ErrorMsg")
stepPointerTarget.Inputs.pop("minValue")
stepPointerTarget.Inputs.pop("maxValue")
stepPointerTarget.Assessments.pop("hasError")
stepPointerTarget.Assessments.pop("minValue")
stepPointerTarget.Assessments.pop("maxValue")

# Create input and output pointer targets for "int* values" parameter.
at = codeInfo.getType("int[4]")
pointerTargetInput = testPointerTarget.TestData.create("inputArray",at)
pointerTargetOutput = testPointerTarget.TestData.create("outputArray",at)

for k, v in enumerate([-1, 1, 3, 5]):
    pointerTargetInput[k].Value = str(v)

for k, v in enumerate([-1, 1, 2, 2]):
    pointerTargetOutput[k].Value = str(v)

# Update inputs for "int* values" and "unsigned N" and assessment for "int* values".
stepPointerTarget.Inputs["values"].Value = pointerTargetInput
stepPointerTarget.Inputs["N"].Value = "4"
stepPointerTarget.Assessments["values"].Value = pointerTargetOutput

Add Scripted Test

Create a test with a scripted step that exercises the saturate_and_cache function multiple times. Create inputs and observables for this step. For each observable that you create, an assessment is automatically added to the step.

# Create test case and scripted step in the existing suite "checkSaturateAndCacheTests".
scriptedTest = suiteSaturateAndCache.TestCases.create("saturate_and_cache_test")
scriptedStep = scriptedTest.TestSteps.createScripted("saturate_and_cache_step")

# Add body of scripted step that invokes saturate_and_cache multiple times.
scriptedStep.Body = r"""
out1 = saturate_and_cache(in);
out2 = saturate_and_cache(in+3);
out3 = saturate_and_cache(in+5);
"""
# Create input for the step.
scriptedStep.Inputs.create("in",codeInfo.getType("int"))
scriptedStep.Inputs["in"].Value = "-1"

# Create observables for the step.
scriptedStep.Observables.create("out1",codeInfo.getType("int"))
scriptedStep.Observables.create("out2",codeInfo.getType("int"))
scriptedStep.Observables.create("out3",codeInfo.getType("int"))

# Update assessments that are automatically created from observables.
scriptedStep.Assessments["out1"].Value = "-1"
scriptedStep.Assessments["out2"].Value = "2"
scriptedStep.Assessments["out3"].Value = "2"

Build and Run Tests with Coverage Enabled

Run the tests added to the project with code coverage computation enabled.

res = polyspace.test.run(proj, ProfilingSelection=polyspace.test.ProfilingSelection.COVERAGE)
### Build successful
Build completed.
Starting test executable
Running Graphical tests
Running Graphical test 'checkSaturateValueTests/saturate_value_simpleTest'
--> Passed
Running Graphical test 'checkSaturateValueTests/saturate_value_parameterizedTest'
--> Passed (101/101)
Running Graphical test 'checkSaturateAndCacheTests/saturate_and_cache_call_count_test'
--> Passed
Running Graphical test 'checkSaturateAndCacheTests/saturate_and_cache_array_test'
--> Passed
Running Graphical test 'checkSaturateAndCacheTests/saturate_and_cache_test'
--> Passed
Done running Graphical tests
Tests Summary
|            |      Total |     Passed |     Failed | Incomplete
|------------|------------|------------|------------|------------
|     Suites |          2 |          2 |          0 |          0
|      Tests |        105 |        105 |          0 |          0
Done running tests

Print the percentage decision coverage of the tests.

covRes = res.Profiling.Coverage
decisionCoverageResults = covRes.getCoverageInfo("decision")
relativeDecisionCoverage = decisionCoverageResults.CoveredCount / decisionCoverageResults.TotalCount
print(f"Decision coverage is {relativeDecisionCoverage:.2%}")
Decision coverage is 71.43%

See Also

| | | | | | |

Topics