7
$\begingroup$

I have two datasets on heart rate of subjects that were recorded in two different places (two different continent to be exact). The two research experiments aimed to find the subjects' emotions based on how much their heart rate change over time. I am using machine learning to predict the subjects' emotions and i am getting acceptable result when tested separately on each dataset. However, i get even better result if i merge the two datasets.

I am not however sure if combining the two datasets is acceptable. As I am combining two somehow different datasets, will it create statistical bias? How should i report my finding in a journal paper?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Welcome to DS SE, The best one can tell.is to go.ahead and try combining and then comparing the scores separately on a fixed Validation DataSet which comes from both the datasets, so that we can benchmark our new findings $\endgroup$
    – Aditya
    Sep 30, 2018 at 16:52
  • $\begingroup$ @Aditya, as i said in my question, i have already combined the two datasets and i get better results. My question is not about if I can combine the datasets. My question is if this will not create statistical bias as i am combining two somehow different datasets $\endgroup$
    – Lapatrie
    Sep 30, 2018 at 17:12

6 Answers 6

Reset to default
3
$\begingroup$

If you add ‘continent’ or ‘location’ as a feature for the model, then you will be able to control for potential bias while getting the results of the additional data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks very much for your helpful suggestion. I will do this and see how it goes $\endgroup$
    – Lapatrie
    Sep 30, 2018 at 22:45
3
$\begingroup$

Adding to what @Super_John said, if adding continents as a Feature, then you can also probably have at-least 2 more features as well,

  • The Latitude
  • The Longitude

Also add another temporary column to indicate the Source (like $1$ to $1st$ df, $2$ to $2nd$ df etc), So that we can add Colors to the k-means

So now we can have a k-means Cluster to see whether values are overlapping or not... (Trying to see it in an Unsupervised Way)

(The analogy is equivalent to the fact that that you can cluster time(24 hours in a day) in a cyclic fashion , like plotting $sin(x)$, $cos(X)$ and then trying to cluster them)

Have a look at this answer, Features Selection, Extraction

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks very much for your answer. I will definitely explore all the suggested methods and will update you what worked best. $\endgroup$
    – Lapatrie
    Oct 1, 2018 at 2:09
  • 1
    $\begingroup$ This is a great suggestion. Fonde - if you do this, you can understand any bias between the datasets and if little or no bias is found (i.e. clusters overlap closely), then you can combine and use the analysis as empirical support for the combined datasets robustness in your modeling task. $\endgroup$
    – Super_John
    Oct 6, 2018 at 6:24
  • $\begingroup$ @Aditya I hope you resume helping others :) $\endgroup$
    – Green Falcon
    Nov 24, 2018 at 21:00
  • 1
    $\begingroup$ I don't have any other option! But you can take them back! It's fine :)) I like this community! @Media $\endgroup$
    – Aditya
    Nov 25, 2018 at 1:14
3
$\begingroup$

Although generally in training a machine learning model, the more data you have the better for training generalised models, that may not be the case here.

Given that the two datasets were collected in completely different environments, they may have completely different distributions. In this case, training a model on the combined dataset may even reduce the performance of the model.

My advice would be, do some statistical analysis on each dataset independently - find the mean and variances of each of the variables for each dataset and compare them for example. If the analysis shows that the two datastes have fairly similar distributions (I’ll leave the definition of fairly similar up to you), then it should be ok to combine the two datasets to train a model on.

Improve this answer
$\endgroup$
2
$\begingroup$

Yes, usually with ML, more data you have, better the results! Of course mixing data from different population is risky, but if it works you are on the right path.

Using more data helps generalise during the training of your model. So, if you are able to test your model over sample from both population and you obtain good result, you can do it.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks. I will try and will update my post after getting the result $\endgroup$
    – Lapatrie
    Sep 30, 2018 at 22:46
1
$\begingroup$

To add to this discussion, a proper evaluation will tell you quite a bit, and can be used to present the work:

  • Create a test set for dataset 1.
  • Create a test set for dataset 2.
  • Train a model using only dataset 1, only dataset 2 and using a combination of dataset 1 and 2 evaluate their performances on both test sets.

If the combined model is significantly better than the separate models, you have something, and I think you can report as such in a possible publication. Of course, you will still have to motivate which machine learning model you use, your performance metric of interest, how you conduct cross-validation, ...

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ You're narrowly focusing on optimizing model performance. The concern is that the data and the experiments are somehow different and that publication should acknowledge any bias between the 2 similar but different experiments. While the datasets can be combined, there must be a level of interpretation that allows for the explanation and measurement of bias between the experiments, which your solutions does not allow for. $\endgroup$
    – Super_John
    Oct 6, 2018 at 6:20
  • 1
    $\begingroup$ By investigating the test error, one can make an estimation of the bias and variance? In addition, I think the test results of the model trained on dataset 1 and evaluated on test set 2 (and the other way around) will tell you a lot? $\endgroup$
    – Archie
    Oct 7, 2018 at 9:22
1
$\begingroup$

Before I could attempt to answer your questions, I will put across what I have understood.

Scenario: Two datasets with heart rate of subjects recorded in two different continents are available.

Aim: Find the subjects' emotions based on how much their heart rate change over time

Objective: Classify subjects' emotions

Noted:

  1. Results are acceptable when trained and tested as separately.

  2. Assume that results would improve upon combining two datasets

Questions:

  1. Is combining the two datasets acceptable?

If the subjects of the two continents are same then there shouldn't be a problem in combining the datasets. Set of Emotions are pretty much the same across same subjects

  1. As you are combining two somehow different datasets, will it create statistical bias?

As long as subjects of two datasets are same then combining will improve your results due to more data.

  1. How should you report your findings in a journal paper?

You could perform hypothesis test(ANOVA) for two samples

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks a lot for your very detailed answer and I am very sorry for the lack of clarity in my original post. The subjects on two different continents are not the same. However, the conditions of recording are somehow similar (but not exactly the same). $\endgroup$
    – Lapatrie
    Oct 4, 2018 at 5:58
  • 1
    $\begingroup$ He mentions "The two research experiments aimed to..." so they are not the same research experiment, although perhaps similar. I think the concern here is around bias in the unknown difference in the experiments, so while they can be defined, they can only be controlled for if the 'continent' or 'experiment id' is included in the model or otherwise tested for independence. $\endgroup$
    – Super_John
    Oct 6, 2018 at 6:15
  • $\begingroup$ @FondeLapatrie Hope you are good. Justification is important while taking certain steps. Since, you can't give more details about this use case. you need to take a decision, is it right to combine two datasets where the subjects are different and experiment condition being similar? If subjects(say Carnivorous mammals) are Tiger and Polar bear of different region then would it be right to combine? This is just an example. Hope you can understand what I am trying to put across. $\endgroup$
    – NRP
    Oct 15, 2018 at 9:23
  • $\begingroup$ @NRP Thanks very much. I understand the pitfalls and the risks associated with merging two datasets. After reading the conversation on this thread, I believe, in my case, it is valid to combine the two datasets. $\endgroup$
    – Lapatrie
    Oct 15, 2018 at 11:37
  • 1
    $\begingroup$ @FondeLapatrie That's good. Wish you success! $\endgroup$
    – NRP
    Oct 16, 2018 at 1:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.