Do you really need the "filter coefficients calculations formula", or would is be satisfying already to call the function butter() to determine the coefficients?
The sampling frequency depends on your data, and therefore this cannot be answered by the forum, but by yourself only. The cut-off frequency depends on the wanted frequency spectrum, such that it is impossible for us to guess it. The cutoff frequency cannot be higher than the half frame rate - this is more or less obvious e.g. trying to filter a signal at 1kHz must fail, when it has been recorded at 1Hz only.
[EDITED] A boiled down 2nd order Butterworth filter algorithm:
function [B, A] = myButter2ndOrder(W)
V = tan(W * 1.5707963267948966);
Sg = V ^ 2;
Sp = V * [-1-1i, -1+1i] / sqrt(2);
% Bilinear transform:
P = (1 + Sp) ./ (1 - Sp);
G = real(Sg / prod(1 - Sp));
% From Zeros, Poles and Gain to numerator and denominator:
B = G * [1, 2, 1];
A = real(poly(P));
W is the cut-off frequency in the range 0 < W < 1, so it is relative to the half sample frequency.
