So I have started my master thesis and I have been handed a time series dataset with 2000 rows and nearly 600 columns of data. I have dealt with time series before, but nowhere similar to this level of complexity. Many courses in time series only deals with univariate time series, and now I'm supposed to work on a time series with 600 different factors with extreme loads of nan-values and present it in a simple and illustrating way. Plotting single univariate time series sort of doesn't make sense as there are still 600 other time series to consider. I was just wondering if anyone here have any tips on how to proceed? Any input would be appreciated.
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
You don't give a lot of details about what you want to do so I'm going to say basic things... hopefully this helps:
- Check and clean up the data: if you have columns which contain mostly NaN values they're likely useless, so you can discard them. You can also ditch any feature which always contains the same value.
- Check the correlation between features: you might have some features which are redundant with each other, remove the ones which are less likely to be informative.
- Work with a small subset (rows) first, analyze it and implement a pipeline using this subset. Check from time to time that your pipeline can handle the full dataset, but it's likely that a subset is enough for most of the development. If possible use a few different subsets to cover more cases.
Needless to say, keep a backup of the original data ;)
$\begingroup$
$\endgroup$
Are you constrained to use only R or Python? If you have the option, try using Alteryx. It's a GUI based tool which has greatly helped me explore and analyze large volumes of data. The tool is built on R so plugging in R functions is a breeze too.
