I've tried reading the other answers on this topic but I'm unsure if I understand completely.

For my dataset, I have a series of tagged documents, "good" or "bad." Each document belongs to an entity, and each entity has a different number of documents.

Eventually, I'd like to create a classifier to detect whether or not an entity's document is good or bad and to also see what sentences are most similar to the good/bad tag.

All that being said, does it make sense to label my data as following:

train_corpus = []
i = 0
for entity in entities:
    for doc_name in entity:
        for sentence in get_doc(doc_name):
           train_corpus.append(TaggedDocument(sentence, tags = [i, doc_name, entity, doc_name.good_or_bad])

From what I understand, this means that each entity is contextualized by all TaggedDocuments that have that entity's name, whereas each document is contextualized by each sentence that composes it. And the overall good/bad idea is composed of all the sentences that make up either the good or bad documents. Is this a correct interpretation? And if that's the case, could I then do something like:

unlabeled_data = [...]
# return predicted good/bag tag
model.cosine_distance(unlabeled_data[0], "bad")
#get a numerical measure of how far some unlabeled data is from the "bad" tag

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.