R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?

Question

I am new to R and usually rely on stata.

I have a considerably large data frame, it contains data from a broad range of surveys and years, each with their own types of classifications and answers. The variables have not been standardised. The data is at an Individual level.

Questions I have are the following:

How can I aggregate the surveys? How can Identify the number of surveys which have provided information on a particular variable i'm interested in like education?

Thank you very much,

raj

Answer 1

If I understand your question correctly you'd like to obtain the proportion of missing values of a specific column in your dataset. This can be obtained with the plyr library

For example:

library(plyr)
df <- data.frame(Study=c('Study_1', 'Study_1', 'Study_2', 'Study_3'), col1=c(NA,NA,5,8), col2=c(8, NA, NA, 7))

column <- 'col1'           # is the column your are interested in

ddply(df, .(Study), function(x){
  length(which(is.na(x[column])))/nrow(x)*100
})

Study  V1
Study_1 100
Study_2   0
Study_3   0

Study_1 has 100% missing values and the other studies none, so except Study_1 all studies contains some information

Answer 2

For the given example:

df <- data.frame(Study=c('Study_1', 'Study_1', 'Study_2', 'Study_3'), col1=c(NA,NA,5,8), col2=c(8, NA, NA, 7))
apply(df,2, function(x) sum(is.na(x)==F))

shows that number of non-NA values per columns are:

Study  col1  col2 
    4     2     2 

That can be interpreted as the number of surveys which have provided information on every variable.

Also could be of your interest:

sum(complete.cases(df))

That show how many surveys provide information for all variables, also known as 'complete cases'.