0
$\begingroup$

I'm working on a group project where we have decided to use this Microsoft WIP repository as our starting point. It's a project which compares different frameworks and models and their ability to do NER in order to filter personal data.

We have several out of the box models we're evaluating from frameworks like SpaCy and Flair. We're also training additional models based on data generated with a data generator that is included in the project.

Recently I came to realize that the data generator actually just takes a file which includes templates, for example the line below:

The name in the account is not correct, please change it to [PERSON]

Then the [PERSON] part would be replaced with a name from another file including several names.

I came to realize that there are only 125 templates. These 125 templates are then used to generate 213 training samples, 59 test samples, and 28 validation sets. Thus of the 300 samples used for training our models over half are near duplicates, for example:

The name in the account is not correct, please change it to John Doe
The name in the account is not correct, please change it to Jeff Peters

We then have an additional step where we compare all the models on a different set of data. However this "different" set of data is just 300 more samples generated from the same 125 templates.

I believe this lack of diversity in data, and similarity between training and evaluation data makes all scores obtained so far invalid.

However my project partner does not believe this would have a negative effect. While I feel strongly that the data will create a problem I'm unable to provide strong citation or references to prove this to him.

He is saying we should "deconstruct the algorithm to show that sentence structure isn't as important to a machine as it is to humans".

Can anyone confirm that what I'm describing will be an issue and invalidate our results, and potentially provide references to back the claim?

Our aim is to compare models and their general performance for NER, not just their ability to perform on this specific data-set.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In general, in Machine Learning, if the train set and test set are very similar, then it leads to a small ability to generalization. In other words, the model not performing well on new data. It's called overfitting.

In your case, could be happened more, if some rows are the same in train and data set, then we could say that it's data leakage.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.