Questions tagged [corpus]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Systematic way of selecting internet texts for a machine translation corpus / dataset?

I am currently working on a neural machine translation project and want to gather a corpus (or dataset) of internet texts that are written in standard and plain language. In theory, it certainly makes ...
tschomacker's user avatar
1 vote
1 answer
140 views

Were any LLMs trained on Google books?

An important limiting factor on the performance of large language models, is the amount of training text available. Of course, using e.g. the Gutenberg archive of public domain books is an obvious ...
rwallace's user avatar
  • 139
1 vote
1 answer
94 views

Quantitatively evaluate similarity between two corpus of texts

I want to assess how similar, or different two corpora are, and if the similarity is statistically significant. Something close to a Kolmogorov–Smirnov test in statistics, but for text data. For ...
Pujan Paudel's user avatar
1 vote
2 answers
47 views

Of all the books ever published in English, what percentage are available for NLP training?

I notice a lot of NLP models are trained on news articles and Wikipedia content and some books. I wonder if NLP models would be better if they were trained on more books. I assume we don't have ...
MattG's user avatar
  • 113
1 vote
0 answers
22 views

Cloud-based visual tool to perform NLP on text corpora

I have some text corpora to share with non-programming clients (~50K documents, ~100M tokens) who would like to perform operations like regex searches, co-locations, Named-entity recognition, and word ...
Strabonio's user avatar
  • 173
0 votes
0 answers
551 views

Can I average the BERT embeddings of multiple instances of the same word to get one vector representation of the word?

In the project I'm working on right now I would like to get one embedding for every unique lemma in a corpus. Could I get this by averaging the embeddings of every instance of a lemma? For example, ...
Hantan G's user avatar
0 votes
1 answer
688 views

What is the difference between sparse and dense corpra?

I didn't got the meaning or the difference between sparse and dense corpra here in this sentence "the reason is that Skip-gram works better over sparse corpora like Twitter and NIPS, while CBOW ...
user's user avatar
  • 1
0 votes
1 answer
34 views

How to program conditional statements for this problem in r

Situation: I'm trying to program the following in r. Task: I am trying to select for words that appear as nouns in my dataset more than they do as adjectives, verbs, or adverbs etc. I have all these ...
n.baes's user avatar
  • 39
0 votes
0 answers
25 views

Building a Model for Time Series Data in R (no forecasting)

Problem: I had planned to use a linear regression model to model time series data in retrospect (i.e., no forecasting). However, I am wondering if this is the best option having come across a few ...
n.baes's user avatar
  • 39
0 votes
2 answers
75 views

Jargon extraction in a text

I have a big text corpus (documentation from a company) and I want to extract the terms that are specific to that area/business. I can do that using TF or TF-IDF and guide myself by the frequency of ...
n.mathfreak's user avatar
1 vote
1 answer
259 views

How to find the probability of a word to belong to a dataset of text

I have two text datasets, one with people that have a certain medical condition and another of random patients. I want to figure out what words are more likely to show up in the dataset with that ...
Kevin's user avatar
  • 261
0 votes
0 answers
20 views

Simple Question: How to Load corpora (.txt files) in R

Context: I have 47 .txt files which are all technically small corpora. They have been lowercased and full-stops have been removed. Question: Would anyone know how to or be able to point me to a ...
n.baes's user avatar
  • 39
0 votes
0 answers
8 views

Passing a word cloud choice to a corpus viewer

Can I pass a selected word (via a click) in a word cloud to the RedExp filter in a corpus viewer? Is: Desired: [email protected]'ll join up next...Thanks for any help
RonDespain's user avatar
0 votes
0 answers
34 views

How to build a speech corpus for continuous speech?

I am developing a speech corpus from audiobooks. I was able to collect recordings with their transcriptions but without timing information. Therefore I am looking for recent techniques to ...
Selma_KA's user avatar
2 votes
1 answer
302 views

Build a corpus for machine translation

I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases ...
Meomeoowww's user avatar
2 votes
1 answer
28 views

Suggestions for improvement? Time series of variation in relative frequency of emotion-related words in academic psychology over time

First time plotting and interpreting time series data and I have used a line plot for ease of use. I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., ...
n.baes's user avatar
  • 39
1 vote
1 answer
493 views

Text classification with Word2Vec on a larger corpus

I am working on a small project and I would like to use the word2vec technique as a text representation method. I need to classify patents but I have only a few of them labelled and to increase the ...
Rob C's user avatar
  • 13
0 votes
1 answer
932 views

Error using corpus widget in Orange v 3.25.0 text mining and no import document option?

I am experimenting with Orange data mining tool. When I use the 'Corpus' widget from the text mining section it gives me the error: Corpus widget error: I have tried many things, but am still unable ...
vipp.k0000's user avatar
1 vote
1 answer
209 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
user's user avatar
  • 31
0 votes
1 answer
53 views

Does building a corpus make sense on a documentation project?

I have zero to experience in data science or machine learning. Because of this I am not able to determine if building a corpus does apply to the problem I am trying to solve. I am trying to build a ...
DevZer0's user avatar
  • 111
2 votes
0 answers
19 views

Tools/tutorials for compiling corpora for NLP experiments? [closed]

I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much ...
Alex S Kinman's user avatar
0 votes
0 answers
109 views

Wikipedia corpus for NLP - Cleaned sentences

I can see many wikipedia dumps out there. I am looking for a wikipedia-made corpus, in which every line is one sentence, without any wikipedia meta tags.
Nathan B's user avatar
  • 241