1
$\begingroup$

I'm planning on training an NER model, I already do have a large corpus but I did find one more large corpus and I'm quite confident that I can source even more corpora and format its data to my needs.

Assuming all datasets eventually are in the same format and don't have any inconsistencies, does it make sense to concatenate all datasets?

Or does the model converge anyways at some point? Does that make sense at all? What other problems could I encounter?

$\endgroup$
1
$\begingroup$

More data is always better. It makes 100% sense to concatenate all datasets in to a larger one, assuming that they are in the same format. This will most likely improve the performance of your model.

| improve this answer | |
$\endgroup$
0
$\begingroup$

After concatenating, you can always perform feature selection based on their ranking and importance. This way, you can determine to include the new dataset or not.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.