LASSO with additional non-linear constraint

Hi everyone,

I am really stacked.

I try to find optimal beta coefficients using LASSO technique, but with additional constraints. The problem looks like this.

subject to

Basically, I want to compute beta coefficients using lasso with constraint to be less than or equal to their sum of absolute value differences between them and other coefficients (because there are absolute values in non-linear constraint, I redefined constraints to appropriate form).

beta_b -> I want to compute

beta_a -> I previously computed (there should be parameters for the constraint function).

x -> numeric vector (n x 1)

X -> numeric matrix (n x m)

The lower index L means lagged, we need not consider it.

I tried to use fmincon solver, but I am stacked how should it look like.

I have this objective function

%%Objective function
function f = objfun(x, Y, M, lamb)
  f =  ((x*M - Y)^2 + lamb*abs(x));
end

And nonlinear constraints can look like this

    function [c, ceq] = nonlcon(x, eps, p, betas_ref)

    %%Inequality constraints
    c(1) = x - betas_ref - (eps / p);
    c(2) = betas_ref - x - (eps / p);

    %%Equality constraints
    ceq = [];
    end

But the objective function need to be LASSO, so I do not know how can I put LASSO into this problem.

Please, can anybody help me with this problem?

Answer 1

[Alan Weiss]

For fmincon, or indeed any Optimization Toolbox™ solver, you need to put all of your control variables in one vector, typically called x. I cannot tell from your formulation which are your control variables. You say that you want to compute the beta_b, but I do not see beta_b in your equations. I cannot tell if your x and X are data or are control variables.

In any case, once you figure out which are your control variables, put them all in one vector. Then you can extract them in your objective function, something like this. If one control variable is y = an n-by-m matrix, and another is b = a length t vector, then

 x = [y(:);b(:)]

and x is a nm + t length vector. Inside your objective function you would write something like

 function fun = myobj(x,n,m)
 y = x(1:n*m);
 b = x(n*m+1:end);
 y = reshape(y,n,m);
 % more commands here

Then you would pass

 fun = @(x)myobj(x,n,m)

as the objective function, using a method for passing extra parameters, the parameters n and m in this case, but you probably want to pass more parameters than that.

All that said, I don't know how well fmincon would do at finding an optimum for Lasso. You see, Lasso is an L^1 minimization, which is not smooth.

Good luck,

Alan Weiss

MATLAB mathematical toolbox documentation

[Anton Kovac]

Thank you for your reply.

I tried some options but I did not solve it.

Honestly, it is not clear to me what is my control variables (I have not good background in math). I try to explain

I want to compute betas_b (the regression coefficients of LASSO), so the output of my optimization function should be vector. These betas_b are constrained to the constraint I wrote above.

I have computed betas_a (size(betas_a) = size(betas_b)), and I have the data x_i (vector to be regressed) and X_i (matrix of the regressors). Let's say I have additional parameter lambda (penalty coefficient of the lasso).

So, I do not know how can I build my objective function, if

• I want the output to be a vector (not scalar as it is said in documentation) - but from the documentation I assume, that the output of the fmincon is size of x0.
• I want LASSO type objective function

Further, I tried this

    lamb = 0.001;
    x = [X_i(:); x_i(:)];    % control variables??, X_i matrix, x_i vector
    n = size(X_i, 1);        % rows 
    m = size(X_i, 2);        % columns 

    % Parameters for nonlinear constraints
    eps = 0.05;
    p = size(betas_ref.coefs{i}, 1); % scalar
    b_ref = betas_ref.coefs{i};      % vector -> betas_a
    fun = @(x)objfun(x, n, m);
    mycon = @(x)nonlcon(x, eps, p, b_ref); 

    [x, fval] = fmincon(fun, x0, [], [], [], [], [], [], ...
                        mycon);

My objective function

    %%Objective function
function fun = objfun(x, n, m)
    M = x(1:n*m);
    Y = x(n*m+1:end)';
    M = reshape(M, n, m);
    fun = lasso(M , Y);
end

And constraints

    function [c, ceq] = nonlcon(x, eps, p, betas_ref)

    % betas_ref -> vector of size(x)
    % eps , p -> scalars
    %%Inequality constraints
    c(1) = x - betas_ref - (eps / p); 
    c(2) = betas_ref - x - (eps / p); 

    %%Equality constraints
    ceq = [];
end

But I got an error

Index exceeds matrix dimensions.

Error in objfun (line 4) M = x(1:n*m);

But when I calculate it by hand it fits.

I think my way is not right. So could you please help me with these questions?

• Can I just put to the objective function buit-in function (like lasso)? Something like

    %%Objective function
    function y = myobj(x)
        %%code here
        y = lasso(X, Y)
    end

• If not, how should look my objective function, when it should be the lasso?
• Is my constraints definition right?

Finally, all I want to do is the equation (4) in following article (page 2)

http://www-bcf.usc.edu/~liu32/icdm2012.pdf

[Alan Weiss]

You have so many misconceptions that it is hard to know where to begin.

You say you want your objective function to be a vector. You cannot have a vector-valued objective function in a minimization. Think about it, what should be minimized, the first component of the vector, the second, what? If you want to minimize the norm of the objective function vector, then you want to minimize that scalar value, so that is what you should specify as the objective function. In other words, if you want to minimize a vector-valued objective function, then you have not thought enough about what your problem is.

For another topic: control variables are the variables that can be adjusted to try to find an optimum. If you don't know which variables are the control variables, then you have no business trying to write an objective function. You have to figure that out first, before asking for any more help. As another way to think about it, you want to find some values that optimize something. Those values are your control variables, and the something is your objective function.

When you have figured out your control variables and scalar objective function, we can discuss things further.

Good luck,

Alan Weiss

MATLAB mathematical toolbox documentation