I am completely new to data science and looking to narrow down the search and reduce the learning curve required to solve problems like the one given below

I have a data set with 7 columns , Column A(all positive decimal) is the data point I want to maximize. Column B and C are boolean values remaining columns are a combination of positive and negative decimal numbers. I want to find some relation and insights from all colums such that I can maximize the sum of column A.

  • $\begingroup$ what do you mean by "maximize the sum of column A"? $\endgroup$
    – oW_
    Jan 6 '20 at 20:24
  • $\begingroup$ Column A has a positive number in each row , the end goal is to find a quantifiable relation between all columns such that the sum of all values in column A is maximum $\endgroup$ Jan 6 '20 at 20:29

In R you can run a linear regression. Consider this "academic" minimal example:

df = data.frame(c(3,5,2,7,5,3), c(1,0,1,0,1,0), c(0,1,1,0,1,0))
colnames(df) = c("A", "B", "C")

Take this data as an example:

  A B C
1 3 1 0
2 5 0 1
3 2 1 1
4 7 0 0
5 5 1 1
6 3 0 0

Now we can see how B and C describe A in the best way.

reg = lm(A~B+C, data=df)


            Estimate Std. Error t value Pr(>|t|)  
(Intercept)    4.917      1.322   3.719   0.0338 *
factor(B)1    -1.750      1.774  -0.987   0.3966  
factor(C)1     0.250      1.774   0.141   0.8968  
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.048 on 3 degrees of freedom
Multiple R-squared:  0.2525,    Adjusted R-squared:  -0.2459 
F-statistic: 0.5066 on 2 and 3 DF,  p-value: 0.6463

This tells us that when B, C is 0, A=4.1917 if B=1 we would have A=4.917-1.750 and if C=1 we would have A=4.917+0.25.

So, we can also make predictions:

predict(reg, newdata=df)

Which would be in this case:

       1        2        3        4        5        6 
3.166667 5.166667 3.416667 4.916667 3.416667 4.916667

This is a simple form of ML (linear regression), where the sum of squared residuals is minimized in order to find the coefficients for the intercept as well as B and C which best describe A.

You would write this model like: $A = \beta_0 + \beta_1 B + \beta_2 C + u$, where $u$ is the statistical error term. You would solve this model by minimizing $\sum u^2$ (the sum of squared residuals).

In matrix algebra you could write $y=\beta X + u$, and you would solve this by $(X'X)^{-1}X'y = \hat{\beta}$.

So we do not "maximise" but minimize the statistical error $u$ in order to find the best "fit" for columns B, C given column A.

Have a look at the great book "Introduction to Statistical Learning" to get the main concepts sorted.

  • $\begingroup$ Thank you Peter , this looks like a good place to start. $\endgroup$ Jan 7 '20 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.