I am trying to implement the doc2vec algorithm with a rather small sample size: ca. 120 documents with a total of 25000 unique words. My dataframe looks like this with more indicators:

df = pd.DataFrame({'docid': ['2344', '3553', '6576'],
'country': ['countryA', 'countryA', 'countryB']})

    docid   country     region   text
0   2344    countryA    regionA  documentbody
1   3553    countryA    regionB  documentbody
2   6576    countryB    regionC  documentbody ....

I transform the data to the desired TaggedDocument as follows:

mylist = df['text'].to_list()
docs = []
analyzedDocument = namedtuple('AnalyzedDocument', 'words tags')
for i, text in enumerate(mylist):
    words = text.lower().split()
    tags = [i]
    docs.append(analyzedDocument(words, tags))

This automatically assigns unique tags from 0 onwards.

I am running the following model following the advice to set epochs high and vector size low when working with small data sets.

model = gensim.models.doc2vec.Doc2Vec(vector_size = 60, alpha=0.025, min_alpha=0.025, workers = 4, min_count=2, epochs=400)
model.train(docs, total_examples=model.corpus_count, epochs=model.epochs)

Now, I would like to use different tags from my dataframe. As far as I understood, the tags are basically the key to analyse data with doc2vec. Using the column country as an example, I came up with following solution

data_tagged = df.apply(
    lambda r: TaggedDocument(words=r['text'], tags=[df.loc[r.name].country]), axis=1)

My questions are:

1) If some documents have the same country label, after running doc2vec the number of documents should increase compared to using tags from 0 onwards?

2) How can I assign multiple tags, such as treating every document as an unique instance while also allowing to build groups for country and the other indicators?

3) If 2) is possible, how do I infer to multiple tags?

Thank you very much. Please let me know if I should provide more code.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.