9
$\begingroup$

I'm looking for a Python package that implements multivariate linear regression.

(Terminological note: multivariate regression deals with the case where there are more than one dependent variables while multiple regression deals with the case where there is one dependent variable but more than one independent variables.)

$\endgroup$
  • $\begingroup$ I'm also interested in this but want just the feature vector after the non-linear transform. So on row would be $[1,x_1,x_2,x_1x_2,x_1^2,x_2^2]$ say for degree 2 model with 2 variables. $\endgroup$ – Pinocchio Aug 28 '17 at 22:38
8
$\begingroup$

You can still use sklearn.linear_model.LinearRegression. Simply make the output y a matrix with as many columns as you have dependent variables. If you want something non-linear, you can try different basis functions, use polynomial features, or use a different method for regression (like a NN).

$\endgroup$
  • 1
    $\begingroup$ Are you asking specifically about multivariate logistic regression? As in you want to perform many classifications at once? Multivariate linear regression is certainly implemented. Logistic regression would have to be framed differently to use the sklearn library. $\endgroup$ – jamesmf Oct 29 '15 at 18:34
  • $\begingroup$ Whoops, sorry I misread, I was reading the sklearn.linear_model.LogisticRegression documentation thinking about linear regression. I'll remove my comment to avoid confusing future readers. Thanks! $\endgroup$ – Franck Dernoncourt Oct 30 '15 at 3:30
  • $\begingroup$ wish you would have emphasized how to get the polynomial feature vector... $\endgroup$ – Pinocchio Aug 28 '17 at 23:02
  • $\begingroup$ scikit-learn.org/stable/modules/generated/… $\endgroup$ – jamesmf Aug 29 '17 at 1:15
2
$\begingroup$

Just for fun you can compute the feature by hand by forming tuples $seq =(d_1,...,d_N)$ such that $Sum(seq) = \sum^N_{i=1} \leq D$. Once you form those tuples each entry indicates the power the current raw feature should be raised by. So say $(1,2,3)$ would map to the monomial $x_1 x_2^2 x_3^3$.

The code to get the tuples is:

def generate_all_tuples_for_monomials(N,D):
    if D == 0:
        seq0 = N*[0]
        sequences_degree_0 = [seq0]
        S_0 = {0:sequences_degree_0}
        return S_0
    else:
        # S_all = [ k->S_D ] ~ [ k->[seq0,...,seqK]]
        S_all = generate_all_tuples_for_monomials(N,D-1)# S^* = (S^*_D-1) U S_D
        print(S_all)
        #
        S_D_current = []
        # for every prev set of degree tuples
        #for d in range(len(S_all.items())): # d \in [0,...,D_current]
        d = D-1
        d_new = D - d # get new valid degree number
        # for each sequences, create the new valid degree tuple
        S_all_seq_for_deg_d = S_all[d]
        for seq in S_all[d]:
            for pos in range(N):
                seq_new = seq[:]
                seq_new[pos] = seq_new[pos] + d_new # seq elements dd to D
                if seq_new not in S_D_current:
                    S_D_current.append(seq_new)
        S_all[D] = S_D_current
        return S_all

then it should be easy to do regression if you know linear algebra.

c = pseudo_inverse(X_poly)*y

example. Probably better to do regularized linear regression though if your interested in generalization.


Acknowledgements to Yuval is CS exchange for the help.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.