Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables and make them consistent?
If there is such thing, that would save me months of tedious work. The data is stored in SPSS format.
I don't know of an automated tool for this, but as a suggestion, you could look to using feature selection or correlation of all variables in a particular dataset against all other variables.
It depends on the direction that you are wanting to take your analysis later, but the second method will enable you to see variables that are highly correlated, and so might be good candidates for checking that they are duplicates?
Also, not sure if you are aware, but there are a few R packages (memisc is particularly popular) that enable you to read SPSS files directly and then work on the data directly.
Sorry wont save you months of work though ... but might save you days ...
If you are talking about just making the names of the variables consistent across all of the data sets to make them easier to work with, you are dealing with what is known as "Data Integration".
I'm not familiar with SPSS. The base SAS language has data management features/options that lend to performing tricks specific to your data which can semi-automate this kind of process. SAS can also very easily import data from a SPSS data set.
Specialty software exists for this kind of thing as well. So called data integration tools made by SAP and Informatica are the first that come to mind. I'm not sure if IBM makes anything similar. Might be worth your while to explore this route depending on your project budget.
