Does this mean that MATLAB is keeping the functions open which created test?
When you create an anonymous function and the body of the function references a variable other than one of the named arguments, then a property named workspace is populated inside the anonymous function object. The workspace property is a 1 x 1 cell array that contains a scalar struct, with one field for each "captured" variable, with the field name the same as the name of the variable, and with the field value being a reference to the variable.
So the first time, i = 1, you had f = @(x) 1 and then when you have i = 2 then the anonymous function @(x) f(x) * x first binds the existing f(x) into the workspace struct . Then because of the assignment, the new anonymous function is bound as f inside the workspace of functionMaker
At this point, the workspace of functionMaker contains a variable named f which is an anonymous function object. The workspace property contains a struct with field f that points to the previous f, the @(x) 1 version. So the @(x) 1 version still has a reference to it, and so the @(x) 1 version is not deleted.
Next cycle, the @(x) f(x) * x anonymous function has previous f bound into it and so there is a retained reference to it. And it still refers to the @(x) 1 version... and so on.
When you test = functionMaker(5); what is returned is the last anonymous function -- the one that has a reference to the previous version of the anonymous function, which in turn has a reference to the one before that, and so on.
If you were to now clear test then the last reference to the most recent anonymous function would be released so the deletion process would crawl through the properties of that anonymous function and clear the properties. Which would release the struct of the top level, which would release the reference to the previous anonymous function, triggering deletion process for it, triggering release of the previous one, and so on all the way down to the @(x) 1 edition which has nothing bound into it and so the chain does not need to be released any further.
After you have assigned test = functionMaker(5) then the previous levels do not live in the workspace of functionMaker and clear functionMaker would not clear those anonymous functions. You would have to clear test (and any copies of test that you had made.)