0
$\begingroup$

I'am using Rstudio to create some codes. the cods works pretty well, without any error and gives the output desired. But when I turn on the PC after period and re-execute the same code, it generes errors especially in reading external files like

(Error in names(out) <- named : 'names' attribute [10] must be the same length as the vector [3]).

Error during wrapup: 'names' attribute [10] must be the same length as the vector [3]

(these 2 different errors are generated from executing the same part of code 2 times. the code want to tokenize a corpus of 10 files and save it in tables of 3 columns)

I'am using the same dataset from the same location and I'am sure of calling the correct libraries

library("tm")

#seting the corpus
corp <- Corpus(DirSource(path), 
               readerControl=list(reader=readPlain, 
                                  language='en_CA',
                                  load=TRUE));


myTokenizer <- function(x, n, n_min) {
  library(tokenizers)
  tok <- unlist(tokenize_ngrams(x, n = n, n_min = n_min))
  M <- matrix(nrow=length(tok), ncol=3, 
              dimnames=list(NULL, c( "gram" , "num.words", "words")))
  M[, 3] <- tok
  M[, 2] <- lengths(strsplit(M[, 3], "\\W+"))  # counts the words
  M[, 1] <- 1:length(tok)
  return(M)
}
  corp<-tm_map( myTokenizer(x=corp, n=6, n_min=1) )
writecorpus(corp)
$\endgroup$
  • $\begingroup$ Did you clean the environment before running your code? Clean the environment before running, or you mat mix variables $\endgroup$ – Mortezaaa Jul 18 '18 at 13:20
  • $\begingroup$ i cleaned it, but it still the same.. it forces the utput to be a vector of 3 with no reason @Mortezaaa $\endgroup$ – Azza ABIDI Jul 18 '18 at 16:13
  • $\begingroup$ The error message says that "out" has 10 names, but "named" has 3. When you tokenize 10 files, the result would be sparse representation with 3 cols. Can you post a minimal example that could be used to reproduce the error? $\endgroup$ – Mortezaaa Jul 19 '18 at 9:27
  • $\begingroup$ @Mortezaaa i updated the code.. the errors appears while executing corpus<-tm_map... $\endgroup$ – Azza ABIDI Jul 19 '18 at 17:03
1
$\begingroup$

You need to convert the elements of corpus to character:

myTokenizer <- function(x, n, n_min) {
  tok <- unlist(tokenize_ngrams(as.character(x), n = n, n_min = n_min))
  M <- matrix(nrow=length(tok), ncol=3, 
              dimnames=list(NULL, c( "gram" , "num.words", "words")))
  M[, 3] <- tok
  M[, 2] <- lengths(strsplit(M[, 3], "\\W+"))  # counts the words
  M[, 1] <- 1:length(tok)
  return(M)
}
res <- tm_map(corp, function (x) myTokenizer(x, n=3, n_min=1))

or apply your function directly without using Corpus:

res <- myTokenizer(x = list("A Sample Text ...", "Lorup ipsum x 12"), n=3, n_min=1))
| improve this answer | |
$\endgroup$
  • $\begingroup$ I can't apply the function directy because i have 400 file and i want that the output has the same name of the input I applied the as.character function to my code. but while extracting corpus this warning appears ""res <- tm_map(corp, function (x) myTokenizer(x, n=3, n_min=1)) if i ignore it and export the corpus, it generes 10.000 text files each one contains anumber (1,2,3....999,1000...) $\endgroup$ – Azza ABIDI Jul 23 '18 at 4:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.