I understand how doc2vec works, but I am unclear the best practice on feeding in data.
Suppose we have a document with multiple sentences
['I really love football. Peyton Manning was a great player']
If we feed this into the algorithm as is, the window for 'Peyton' could include ['love','football','Manning','was']
This doesn't make intuitive sense to me, however, since the words come from different sentences.
Any suggestions?
The aim of Doc2Vec is to produce document level embeddings, thus even if words are sentence-separated if you include them in the same document it has to be considered part of the same semantic source for word similarities. If you don't want such a behavior, you might want to separate your documents in a sentence-level and maybe aggregate the embeddings later in different groups to form your initial paragraphs (depending on your plans)?
With Doc2Vec, each string will be treated as a separate document regardless of any formatting such as sentences or paragraphs. So for an analysis of a book from project Gutenberg, you could have each chapter of the book treated as a document or you could treat each paragraph as a document.
So, for a two sentence paragraph like your example ['I really love football. Peyton Manning was a great player'] and use a sentence tokenizer such as the NLTK package in Python's nltk.tokenize.sent_tokenizer() function which will divide your string into separate sentences. You would then run the tokenized sentences through Doc2Vec and have each sentence as a different document. See this example for an example with a sentence tokenizer in Python
The choice of which level of text to treat as a document should be based on what your research design is and what unit of analysis you want to interpret for the end product. You tagged the question with gensim, so I assume you are trying to do a topic model. You have to weigh whether you are interested on change in topics from sentence to sentence, in paragraph chunks, or some other hierarchy within the text.
