Unittest: Verify whether the function prints correct error message

Hello all,

I am writing class based unittests, using matlab.unittest.TestCase. In some of my functions, at some conditions, we print error message using the error function. For example:

function ret = myFunc(input1, input2)
    if length(unique(input1)) ~= 2 & S_ntream == 2
        error("Your input doesn't satisfy minimum unique input requirement")
    end
    %Other stuff the function will do if it meets the above condition goes here.
end

Now, in my unittests I want to verify that my program is displaying/returning this error message, when it should. I tried using the verifyError method like following. However, the test failed. (I have designed the emv.input1, emv.input2 such that the function will reach the error branch.)

classdef myFunc_Test < matlab.unittest.TestCase
    properties(TestParameter)
        emv = generateAllInputsForErrorMessageVerification();
    end
    methods(Test, ParameterCombination = 'sequential')
       function testErrorMessages(testCase, emv)
       %I have designed the emv.input1, emv.input2 such that the function
       %will reach the error branch.
            testCase.verifyError(@()myFunc(emv.input1, emv.input2), ... ,
                 "Your input doesn't satisfy minimum unique input requirement");
       end
    end
end

Also, using verifyEqual doesn't make sense as myFunc is not "returning" the error message ("Your input....").

Any suggestions on how this can be tested will be appreciated. Thanks in anticipation.

Answer 1

[Steven Lord]

You can do this in one of two ways.

1. When your function calls error have it call error with an error identifier and use verifyError to Verify that the error with that identifier is thrown when the code you're testing gets Exercised.
2. Check that an error is thrown by calling verifyError with ?MException as the metaClass input.

testcase = matlab.unittest.TestCase.forInteractiveUse;
testcase.verifyError(@() magic(3)+magic(4), 'MATLAB:sizeDimensionsMustMatch', ...
    'Cannot add a 3-by-3 and a 4-by-4')

Verification passed.

testcase.verifyError(@() error('Something went wrong'), ?MException)

Verification passed.

% Show failing test cases
testcase.verifyError(@() magic(3)+magic(4), 'MATLAB:thisIsTheWrongErrorIdentifier', ...
    'Cannot add a 3-by-3 and a 4-by-4')

Verification failed.
    ----------------
    Test Diagnostic:
    ----------------
    Cannot add a 3-by-3 and a 4-by-4
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyError failed.
    --> The function threw the wrong exception.
        
        Actual Exception:
            'MATLAB:sizeDimensionsMustMatch'
        Expected Exception:
            'MATLAB:thisIsTheWrongErrorIdentifier'
    --> Actual Error Report:
            Arrays have incompatible sizes for this operation.
            
            Error in LiveEditorEvaluationHelperEeditorId (line 6)
            testcase.verifyError(@() magic(3)+magic(4), 'MATLAB:thisIsTheWrongErrorIdentifier', ...
    
    Evaluated Function:
      function_handle with value:
    
        @()magic(3)+magic(4)
    ------------------
    Stack Information:
    ------------------
    In /tmp/Editor_lypyr/LiveEditorEvaluationHelperEeditorId.m (LiveEditorEvaluationHelperEeditorId) at 6
    In /MATLAB/toolbox/matlab/connector2/interpreter/+connector/+internal/fevalMatlab.p (fevalMatlab) at 0
    In /MATLAB/toolbox/matlab/connector2/interpreter/+connector/+internal/fevalJSON.p (fevalJSON) at 0

testcase.verifyError(@() 1+1, ?MException) % No exception, test fails

Verification failed.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyError failed.
    --> The function did not throw any exception.
        
        Expected Exception:
            ?MException
    
    Evaluated Function:
      function_handle with value:
    
        @()1+1
    ------------------
    Stack Information:
    ------------------
    In /tmp/Editor_lypyr/LiveEditorEvaluationHelperEeditorId.m (LiveEditorEvaluationHelperEeditorId) at 8
    In /MATLAB/toolbox/matlab/connector2/interpreter/+connector/+internal/fevalMatlab.p (fevalMatlab) at 0
    In /MATLAB/toolbox/matlab/connector2/interpreter/+connector/+internal/fevalJSON.p (fevalJSON) at 0

I prefer the former, as it doesn't assume that if something goes wrong it went wrong in the way that I expected it to go wrong. But if all you care is that whatever operation you're testing didn't succeed that second approach could be useful.