You would have to convert the filters to dfilt type. Try this:
lp = fir1(100, 0.77, 'low', rectwin(101));
hp = fir1(100, 0.83, 'high', rectwin(101));
h1 = dfilt.df2t(lp,1);
h2 = dfilt.df2t(hp,1);
h = dfilt.cascade(h1,h2);
However, Mathworks documentation states:
"Note that with the DSP System Toolbox installed, one usually does not construct cascade filters explicitly. Instead, one obtains these filters as a result from a design using FDESIGN." You should look into this.