5
$\begingroup$

I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables and make them consistent?

If there is such thing, that would save me months of tedious work. The data is stored in SPSS format.

$\endgroup$
  • $\begingroup$ openrefine.org $\endgroup$ – Emre Jun 23 '16 at 21:05
  • $\begingroup$ cloud.google.com/dataprep $\endgroup$ – Emre Mar 24 '17 at 16:54
  • $\begingroup$ @Emre I have tried openrefine and it doesn't scale well to lots of columns. Would be curious to try dataprep though. $\endgroup$ – tom Nov 19 '17 at 23:22
1
$\begingroup$

I don't know of an automated tool for this, but as a suggestion, you could look to using feature selection or correlation of all variables in a particular dataset against all other variables.

It depends on the direction that you are wanting to take your analysis later, but the second method will enable you to see variables that are highly correlated, and so might be good candidates for checking that they are duplicates?

Also, not sure if you are aware, but there are a few R packages (memisc is particularly popular) that enable you to read SPSS files directly and then work on the data directly.

Sorry wont save you months of work though ... but might save you days ...

$\endgroup$
0
$\begingroup$

If you are talking about just making the names of the variables consistent across all of the data sets to make them easier to work with, you are dealing with what is known as "Data Integration".

I'm not familiar with SPSS. The base SAS language has data management features/options that lend to performing tricks specific to your data which can semi-automate this kind of process. SAS can also very easily import data from a SPSS data set.

Specialty software exists for this kind of thing as well. So called data integration tools made by SAP and Informatica are the first that come to mind. I'm not sure if IBM makes anything similar. Might be worth your while to explore this route depending on your project budget.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.