1
$\begingroup$

I have a set of scientific papers of authors who have common research interests from PUBMED and I would like to:

  1. Clustering papers and extracting features from them in order to find other authors with similar research interests in PUBMED
  2. Develop a classification algorithm to determine if a given author has the similar research interest to those in my original set. Really, I am trying find new authors and be able to tell if an author that I discovered, is similar to my original set of authors.

Unfortunately because my data is only one class, I have an extreme data imbalance problem. How should I fix this?

Thus as far as I have looked at the frequency of terms in the papers with mixed results. I have also been using unsupervised learning to extract features with mixed results.

My set of papers and authors are assumed to share similar research interests. Do I need papers and authors that I know aren’t in the set in order to build a classification algorithm?

Should I look at a paper as a whole or should I look at the papers written by each author for classification, instead of looking at the papers as a group?

I suppose I could randomly pick other papers and use those as the examples of other fields. Of course, picking paper at random there is the chance of false negatives.

I am using R with the tm package to do the mining.

$\endgroup$

1 Answer 1

1
$\begingroup$

The question is not very clear but I will give a try anyways.

First of all please note that the major approach to tackle the analysis of co-authorship networks is not what you mentioned, but Network Science techniques as co-authorship is a Social Network.

After this necessary tip let's back to the question itself which is tackling the problem from NLP point of view.

Unfortunately because my data is all of one class already I have an extreme data imbalance problem. How should go about fixing this?

You don't have a class imbalanced problem. Class imbalanced problem means that you have more than one class in which one or more classes are highly larger than others. But you have only one class! This is not a class imbalanced problem. You only need to define your question precisely.

My set of papers and authors are assumed to share similar research interests. To build a classification algorithm do I need papers and authors that I know aren’t in the set?

Depends on what you want to do! Classification of what? You can set your question like classification of authors in the same field then you do not need additional papers. You extract text features according to different papers of all individuals and train your classifier:

  1. Concatenate all sentences of the papers of one individual and call it data. the name of individual is the label. Of course if a paper has two authors, the text will be assigned to both of them.
  2. Extract features from your text data. Unsupervised feature extraction does not care about the labels (TF-IDF is an example) and supervised methods do care.
  3. Choose a classifier and start your learning phase as always.

but if you set your question like classification of authors from different field then sure you need data from other fields! In this case instead of each individual being a class, each field (including all authors in that field) becomes a single label.

Should I look at the paper as a whole or should I look at the papers written by each author for classification and instead of looking at the papers as a group?

It's super difficult to understand what you mean but I guess I answered it above. Depends on your question! If you are going to classify authors inside a community, then all papers of that author is one text data object.

I suppose I could randomly pick other papers and use those at the examples to one that don’t share research interests. Of course, picking paper at random there is the chance of false negatives.

Again difficult to understand but anyways. In this case you better gather data from another field. Randomly mine some papers from other fields and construct a new class of NOT THIS FIELD. Then go for a binary classification(Logistic Regression works well on TF-IDF features). It should be pointed that as your data is text you can not cover every text which is not in your main class so limit your question to a specific border in domain.

Hope I could help :)

PS: The question needs a surgery. I will edit it soon and please check if my edit is conceptually correct.

$\endgroup$
3
  • $\begingroup$ Thanks this helped a lot. It was rather difficult clearly expressing the problem. I am rather new to data science and text mining at don't know the language yet. $\endgroup$ Mar 2, 2018 at 20:17
  • $\begingroup$ To answer one of your questions what I want to do is when be able to determine if a new author belongs to the class that I have data on. For example I have the papers of a authors who all like apples. How do I determine if a new author would also belong to that same class? The answer you gave I think puts me on a good track. $\endgroup$ Mar 2, 2018 at 20:20
  • $\begingroup$ All good my friend. Glad it helpd. upvote/accept of the answer (if it was useful) is appreciated :) $\endgroup$ Mar 2, 2018 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.