1
$\begingroup$

I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the correlation between nitrate and sulfate, and I am using the cor() function from the stats package for this. The expected result looks different from mine and I don't know why.

If anyone has passed the R Programming course on Coursera, the question is from the programming assignment from week 2

setwd(dirname(rstudioapi::getActiveDocumentContext()$path))
corr <- function(directory, threshold=0){
  csvFiles <-  getFilesById(directory)
  dataFrames <-  readMultipleCsvFiles(csvFiles)
  noNADataFrames <- lapply(dataFrames, na.omit)
  acceptableDataFrames <- filterMoreThanThreshold(noNADataFrames, threshold)
  correlation(acceptableDataFrames)
}
getFilesById <- function(directory = getwd()){
  allFiles = list.files(directory)
  file.path(directory, allFiles)
}
readMultipleCsvFiles <- function(csvFiles) {
  lapply(csvFiles, read.csv)
}
filterMoreThanThreshold <- function(noNADataFrames, threshold){
  Filter(function(x){NROW(x)>threshold}, noNADataFrames)
}
bindMultipleDataFrames <- function(dataFrames) {
  Reduce(function(x, y) rbind(x, y), dataFrames)
}
correlation <- function(acceptableDataFrames){
  sulfate <- lapply(acceptableDataFrames, function(x){x[2]})
  sulfate <- bindMultipleDataFrames(sulfate)
  nitrate <- lapply(acceptableDataFrames, function(x){x[3]})
  nitrate <- bindMultipleDataFrames(nitrate)
  sulnit <-  cbind(sulfate, nitrate)
  cor(sulnit)
}
cr <- corr("specdata", 150)
head(cr)
summary(cr)

I'm supposed to get this:

 > head(cr)
 [1] -0.01895754 -0.14051254 -0.04389737 -0.06815956 -0.12350667 -0.07588814
     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
 > summary(cr)
 -0.21057 -0.04999  0.09463  0.12525  0.26844  0.76313

But instead I get this:

> head(cr)
           sulfate    nitrate
sulfate 1.00000000 0.06088486
nitrate 0.06088486 1.00000000
> summary(cr)
    sulfate           nitrate       
 Min.   :0.06088   Min.   :0.06088  
 1st Qu.:0.29566   1st Qu.:0.29566  
 Median :0.53044   Median :0.53044  
 Mean   :0.53044   Mean   :0.53044  
 3rd Qu.:0.76522   3rd Qu.:0.76522  
 Max.   :1.00000   Max.   :1.00000
$\endgroup$
2
  • $\begingroup$ Welcome to the site. It appears there's probably only one row in your final dataset? Can you add some debug printing to your corr function? What is the dimension of the final data-frame? Can you print a head(final) in that function and check that there are multiple values present sulfate and nitrate? $\endgroup$ – hssay Jul 20 '20 at 12:32
  • $\begingroup$ The cor() function expects two vectors and returns the single correlation coefficient between those two vectors. $\endgroup$ – Ben Norris Jul 22 '20 at 16:57