I hope you are aware of the fact that the default type of NumPy is float64 even if it is not required.
In this case you can easily change it to 'float16' without losing information. It can reduce size by 30GB for 10 gestures.
import numpy as np
image_1 = np.ones((240,420))
image_2 = np.ones((240,420))
image_1 = image_1.astype('float16')
Your question is missing some details about your approach. So I will try to answer the question with the information given, and show you what are the missing information.
You have depth images recording different gestures.
Each image has a resolution of 240x420, and you have 200 images per gesture.
I assume each image has one channel (depth). A ...
There are a number of ways to tackle this, I am going to focus on feature selection/extraction, because you mentioned PCA.
Sklearn itself offers a few feature selection/extraction algorithms already, see here, like SelectKBest. This would mean for you to maybe select specific frames, or samples, or even pixels (unlikely).
Further it has not only PCA but a ...
1. It will not change unless you assume a new type for the Feature after Encoding i.e. Label encoding doesn't make a categorical data Continuous
2. Different pairs of feature types require different methods. The defacto Pearson coeff is for Continuous-Continuous feature.Similar read - DS.SE
3. Values coming out from two different types of Correlation method ...
Correlation is for continuous variables, example Pearson co-relation. Using Pearson correlation for categorical or ordinal variables are not recommended. If you are encoding the data, i can imply that it is a categorical variable.
For categorical variables you can find association between the variables (instead of correlation) using Chi-squared test of ...
By definition, if these columns or features contain a constant value and yet the output variables change, then they are not influencing the output and likely can be ignored.
A more formal test is to determine how much of the variance between a model that uses that feature is attributable to that feature.
A simple example to illustrate this principle is to ...
If the Train data(~80%) doesn't have any missing records and you are expecting missing records in test data(~20%).
This can happen in these circumstances(can be other too) -
Only few missing records in the count -
Then these are most probably completely at random, then you can either remove the records or fill with the mean/median of training data
A Good ...
First, we must understand about a common statistical term called population. Given a population say X, a random sample is drawn (in the ideal conditions). Now suppose you are asked to build a predictive model based on this random sample. So, you split the sample into train, test and validation sets. And you start to build the model on the train set. You ...
I agree with Nicholas' answer, a few more thoughts:
you could use a standard English tokenizer (e.g. nltk, Spacy), if only to see how they process hyphenated words. Similarly you could check how it's done in a pre-tokenized dataset, but be aware that the tokenization conventions followed might differ from one dataset to the other.
Imho the choice depends on ...
They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known).
For a tfidf BOW model, you might get good performance from any of the above.
For a model that is sensitive to word order I would certainly go with the first option and might ...
It's important to remember you can always save objects (like dicts or json) into individual cells in Pandas. Especially if you're not sure of how you'd to analyze at the moment.
The Google Analytics Customer Revenue Prediction data uses a lot of JSON
You can see how people analyze the data on the Notebooks section too
If I merge on dates, I'd have multiple repeated rows with fires occurring at the same time in different places, would that be the best way?
Probably not, since you don't want to lose the location information. You should probably find a way to map the latitude/longitude to borough/county between the two datasets, so that you obtain a semantically consistent ...