# Multivariate linear regression in Python

I'm looking for a Python package that implements multivariate linear regression.

(Terminological note: multivariate regression deals with the case where there are more than one dependent variables while multiple regression deals with the case where there is one dependent variable but more than one independent variables.)

• I'm also interested in this but want just the feature vector after the non-linear transform. So on row would be $[1,x_1,x_2,x_1x_2,x_1^2,x_2^2]$ say for degree 2 model with 2 variables. – Pinocchio Aug 28 '17 at 22:38

You can still use sklearn.linear_model.LinearRegression. Simply make the output y a matrix with as many columns as you have dependent variables. If you want something non-linear, you can try different basis functions, use polynomial features, or use a different method for regression (like a NN).

• Are you asking specifically about multivariate logistic regression? As in you want to perform many classifications at once? Multivariate linear regression is certainly implemented. Logistic regression would have to be framed differently to use the sklearn library. – jamesmf Oct 29 '15 at 18:34
• Whoops, sorry I misread, I was reading the sklearn.linear_model.LogisticRegression documentation thinking about linear regression. I'll remove my comment to avoid confusing future readers. Thanks! – Franck Dernoncourt Oct 30 '15 at 3:30
• wish you would have emphasized how to get the polynomial feature vector... – Pinocchio Aug 28 '17 at 23:02
• scikit-learn.org/stable/modules/generated/… – jamesmf Aug 29 '17 at 1:15

Just for fun you can compute the feature by hand by forming tuples $seq =(d_1,...,d_N)$ such that $Sum(seq) = \sum^N_{i=1} \leq D$. Once you form those tuples each entry indicates the power the current raw feature should be raised by. So say $(1,2,3)$ would map to the monomial $x_1 x_2^2 x_3^3$.

The code to get the tuples is:

def generate_all_tuples_for_monomials(N,D):
if D == 0:
seq0 = N*[0]
sequences_degree_0 = [seq0]
S_0 = {0:sequences_degree_0}
return S_0
else:
# S_all = [ k->S_D ] ~ [ k->[seq0,...,seqK]]
S_all = generate_all_tuples_for_monomials(N,D-1)# S^* = (S^*_D-1) U S_D
print(S_all)
#
S_D_current = []
# for every prev set of degree tuples
#for d in range(len(S_all.items())): # d \in [0,...,D_current]
d = D-1
d_new = D - d # get new valid degree number
# for each sequences, create the new valid degree tuple
S_all_seq_for_deg_d = S_all[d]
for seq in S_all[d]:
for pos in range(N):
seq_new = seq[:]
seq_new[pos] = seq_new[pos] + d_new # seq elements dd to D
if seq_new not in S_D_current:
S_D_current.append(seq_new)
S_all[D] = S_D_current
return S_all


then it should be easy to do regression if you know linear algebra.

c = pseudo_inverse(X_poly)*y


example. Probably better to do regularized linear regression though if your interested in generalization.

Acknowledgements to Yuval is CS exchange for the help.