1
$\begingroup$

I have a database like:

Site    X    Y
S1      1    1.5
S1      1    1.3
S1      2    1.7
S1      1    1.1
S1      4    5.9
S2      3    4.0
S2      2    2.5
S2      4    9.1
S2      4    9.2
S2      1    2.3

I need to find if $Y$ increases as $X$ increases for every site. In other words, Bigger $X$ corresponds to bigger $Y$.

I know linear regression might suit this problem. But please take a look at the following graph:

enter image description here

Figure 1 is not what I want because small $X$ corresponds to big $Y$. However, Figure 2 is what I want. When I use a linear regression model and RMSE as a measure, it cannot tell the difference between Figure 1 and Figure 2.

Another thing about my database is: $X$ are like levels, which are the same to all the sites. But the $Y\text{'s}$ of every sites are different. For example, for one site, $X=1$ and $Y=20$ means $20$ is a small value because it corresponds the lowest level of $X$. But for another site, $Y=15$ and $X=6$ means $15$ is the highest value because $X$ is the highest level.

So, my problem is: for every site, I need to use a linear model or any other algorithms to judge if $Y$ increases as $X$ does. Then, I need to use a measure to select some sites.

$\endgroup$
1
$\begingroup$

Linear Regression will help you decide whether Y tends to increase with X, but it is not a good tool to prove that Y always increases with X. For this you need to design an algorithm.

To prove that Y always increases with X at each site, first create a table of unique X in ascending order with the corresponding min and max Y for each site:

S1:
X    min(Y)    max(Y)
1    1.1       1.5
2    1.7       1.7    
4    5.9       5.9

S2:
X    min(Y)    max(Y)
1    2.3       2.3
2    2.5       2.5    
3    4.0       4.0
4    9.1       9.2

Now for each site verify the following: For each X check that max(Y) is less than min(Y) for X+1. If this condition ever fails then you have shown that Y does not always increase with X at every site, otherwise you can say that it does.

$\endgroup$
  • $\begingroup$ Thanks a lot for your help. This is a good idea. I just want to ask one more question: There are a lot of sites which do not follow this rule exactly to different extent. So, if there are two sites. One is bad, the other one is very very bad. Could we design a criterion to measure this so that I can compare all the sites with each other? My thinking is to count how many times one site breaks this rule. I just want to know if there is better way to do it. $\endgroup$ – Feng Chen Dec 4 '17 at 22:00
  • $\begingroup$ @FengChen This really depends on the specific nature of the problem. I would have to understand more about what the data represents to help you design the right cost function. I suggest creating a separate question with more details, since this is a different problem. $\endgroup$ – Imran Dec 4 '17 at 23:51
3
$\begingroup$

I don’t think linear model is a good idea for your problem, because it can capture only linear pattern. Furthermore, if I understand you correctly the results for different sites will not be comparable due to different scales/ slopes.

Instead I suggest using the Spearman rank correlation coefficient.

$\endgroup$
  • $\begingroup$ (+1) Maybe it is worth noting that the rank-correlation is to be calculated per site. The closer to one, the more monotone increasing the relation within the site. $\endgroup$ – Michael M Dec 10 '17 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.