11

You are right. If someone is using regularization correctly and doing hyperparameter tuning to avoid overfitting, then it should not be a problem theoretically (ie multi-collinearity will not reduce model performance) However it may matter in a number of practical circumstances. Here are two examples: You want to limit the amount of data you need to store ...


3

Let's answer you questions one by one. a) Since there are more than 100 unique products, should I create one hot encoding variables for all my 100 products? There are many ways to encode a categorical variable, a list of them you can find here. Which one you should use depends on your data. Categorical variables can be of many types like ordinal, nominal, ...


3

Linking to the same paper as @scholle but explaining the process differently (book and paper). You do not need to train the model multiple times. The algorithm described in the links above require a trained model to begin with. Given a trained model, compute the metric of interest on some dataset (the book discusses pros/cons of using training set vs test ...


2

I would reccomend you to encode high cardinality categorical variables with Target Encoding methods: Python Library: https://contrib.scikit-learn.org/category_encoders/ Paper: https://link.springer.com/chapter/10.1007%2F978-3-030-85529-1_14 If you want to understand what the model is doing, I would recommend you to look at the Interpretability book https://...


2

The main difference is that a numerical variable has only one dimension for a ML model. Additionally there's always an assumption about the distribution of a numerical variable, for example that it follows a normal distribution. A decision tree would can use a simple condition like "x > 15" to split the data, and if 15 is for instance the median ...


1

The feature selection step is there to guard against model overfitting. The feature selection step may decide that all the variables in the dataset are relevant, or it may decide to remove some. If no feature selection step is performed then no variables are removed and the resulting model may be well-fitted but may be (and likely is) overfitted. The main ...


1

Your question is whether you violate best practice if you use the delta in your HMM, and if there's any pitfalls. I think to answer the first, and prevent the second, you could do what is often done in the case where people want to use a HMM whilst using information from more than one previous state: rewrite the latent chain, such that it becomes a 'standard'...


Only top voted, non community-wiki answers of a minimum length are eligible