Three state markov chain: plotting graphs

If I have a three state markov chain

T=[2/3,0,1/3;1/4,3/4,0;1/3,0,2/3];

How would I get the quantity in the picture, and is it possible to plot Ri(n) against time n for each of the states i ∈ {1, 2, 3}

Answer 1

[Shantanu Dixit]

Hi Jacob,

For computing against time for each of the states, you can:

• Compute powers of the transition matrix
• Extract the diagonal elements
• Compute the cumulative sum for each state. (Can maintain a row vector corresponding to each state, length = no of steps taken)

% Define the transition matrix
T = [2/3, 0, 1/3; 
     1/4, 3/4, 0; 
     1/3, 0, 2/3];

% Number of steps (define)
n_max = 5; 

% Initialize R_i(n) for each state (1,2,3)
R = zeros(3, n_max);
Tk = T;  % Start with T^1

for n = 1:n_max    
    diag_elements = diag(Tk);  % Extract diagonal elements from (T^n)_ii
    if n == 1
        R(:, n) = diag_elements;
    else
        R(:, n) = R(:, n-1) + diag_elements;
    end
    
    % Update matrix power T^n -> T^(n+1)
    Tk = Tk * T;
end

% Plot
figure;
hold on;
plot(1:n_max, R(1, :), 'r', 'LineWidth', 2);
plot(1:n_max, R(2, :), 'g', 'LineWidth', 2);
plot(1:n_max, R(3, :), 'b', 'LineWidth', 2);
hold off;

xlabel('n (Time Step)');
ylabel('R_i(n)');
title('Return Frequency R_i(n) for Each State');
legend('State 1', 'State 2', 'State 3');
grid on;

Additionally you can refer to the following functionalities which are used for the above computation and plotting

diag: https://www.mathworks.com/help/matlab/ref/diag.html#d126e405561

plot: https://www.mathworks.com/help/matlab/ref/plot.html

Further you may want review the MATLAB onramp course which can be useful for getting good hands-on experience in MATLAB: https://matlabacademy.mathworks.com/details/matlab-onramp/gettingstarted

Hope this helps!