spectre
  • Member for 6 months
  • Last seen this week
  • India
How to determine if my data split is appropriate for my data size?
4 votes

Using a 50:50 split is usually not recommended. People usually keep more data for training and less data for testing/validation. The more train data you have, the better the model captures different ...

View answer
Best platform to work with when having millions of rows in dataframe
4 votes

There are 2 things you can do here: 1.) Use libraries like Dask to speed up your data preprocessing. Here is the link 2.) Use cloud computing services like Azure, AWS or GCP. I am not aware of other ...

View answer
Does One-Hot encoding increase the dimensionality and sparsity of dataset?
Accepted answer
4 votes

Which encoding technique to use depends on your data/features. Ordinal encoding is used when there ia a sense of order in your feature. For example you have a feature performance which has values ...

View answer
Categorical to One hot encoding - Big data
Accepted answer
3 votes

Let's answer you questions one by one. a) Since there are more than 100 unique products, should I create one hot encoding variables for all my 100 products? There are many ways to encode a ...

View answer
Creating & handling large matrices in python?
3 votes

I don't know if this is your job work or your personal project but cloud services can help you. Create a free account on Azure (you get 200$ worth free credit for 1 month) and the account is free for1 ...

View answer
Beginning my data science journey
Accepted answer
2 votes

I too have a masters degree in physics so maybe you can relate! The first thing to do would be to get your fundamentals in python strong. Pick up a python beginners tutorial from Youtube and learn all ...

View answer
Test / Train split - is it always necessary (supervised learning)?
2 votes

Yes OOB score can be helpful when data is hard to come by. If your dataset size is small you can use OOB to evaluate the model. But another, much better option would be to split your data and apply ...

View answer
Drop or impute the missing values?
Accepted answer
2 votes

In most cases, dropping data only makes sense when you have a large number of nan values. For example of you have a feature with 98% nan values, it is not going to be of much use to any algorithm. ...

View answer
Do I have to remove features with pairwise correlation even if I am doing a regularized logistic regression?
Accepted answer
2 votes

Yes the L1 regularization will shrink the irrelevant feature coefficients to zero and hence it doesn't require feature selection. In fact it IS a commonly used feature selection technique. So ...

View answer
Dealing with unbalanced training set compared with real world data
2 votes

There are a few things you can do. Keep in mind there are some people who would disagree with one method while others agree on the same. 1.) Since it an imbalanced dataset, you can apply any of the ...

View answer
Removing illogical observations, can I do it?
2 votes

Before removing anything from your dataset (nonsensical values or outliers), it is wise to have an opinion of a subject matter expert. It might be that the person with 11 years and earning 100 dollars ...

View answer
Where can I find study materials?
2 votes

https://youtu.be/rfscVS0vtbw - Python basics https://youtube.com/playlist?list=PLZoTAELRMXVPBTrWtJkn3wWQxZkmTXGwe - Full Machine learning tutorials https://youtu.be/xxpc-HPKN28 - Statistics needed ...

View answer
Which algorithm works well for forecasting sales prediction and the reason to choose particular algorithm?
2 votes

There is no definite answer to this question. Usually all algorithms are tried and the best performing algo is selected. But to answer your question, it depends on the type of data you are working ...

View answer
calibrated classifier ValueError: could not convert string to float
2 votes

For any kind of Machine Learning task or a NLP task (which is what you are doing), you need to convert string/text values to numeric values. The machine cannot uderstand or work with string values. It ...

View answer
Which major should I choose to become a Data Analyst?
Accepted answer
2 votes

Data Science and Data Analysis are 2 different things (though closely related). A Data Scientist can do all the things that a Data Analyst can do but a Data Analyst cannot do all the things a Data ...

View answer
Can data leakage be sometimes acceptable?
2 votes

If you have the whole population and do not want to predict anything, as stated in the question link you gave, then it is fine to use the whole population for preprocessing as it will give better ...

View answer
Two-parametric transformation of Box-Cox vs Yeo–Johnson transformation
Accepted answer
1 votes

Box-Cox transformation cannot work with negative values. You can try feeding negative values to the box-cox transformation and it will give you an error.

View answer
How to improve the learning rate of an MLP for regression when tanh is used with the Adam solver as an activation function?
Accepted answer
1 votes

If your goal is to set the optimum value of hyperparameters (weather it be learning rate, no of layers, activation function etc.) you should look into Keras Tuner. The reason as to why the learning ...

View answer
How to start working on Data Science Projects
1 votes

A good starting point for all newbies is Kaggle. It is a platform for budding and experienced data scientists where you can start learning data science from scratch if you are a newbie (there are ...

View answer
How to remove rows from a data frame that have special character (any character except alphabet and numbers)
1 votes

You can use the regular expression to clean your data. Below I have compiled an almost complete list of functions that one uses frequently when cleaning text data. 1.) Remove URL def remove_URL(...

View answer
Correlating & Combining categorical and numerical features for classification problem
Accepted answer
1 votes

Regarding your first point, you can correlate your features to your target the way you have done (to my knowledge. I'm not 100% sure). This is because you are only converting the categorical classes ...

View answer
Activation and Loss Function not chosen correctly when use Neural Network
Accepted answer
1 votes

There are 2 possible scenarios here. You can use all 3 categories and build a multiclass classification model, where the output layer has 3 neurons, activation function is softmax and loss is sparse ...

View answer
Exact Predictions for Regression problems in Machine Learning
1 votes

Why are you limiting yourself to only 2 models? There are a plethora of different algorithms, both in ML and DL. Try them and see which gives the desired results. Also I noticed you haven't done any ...

View answer
Advantages to combining similarly-named columns for supervised ML?
Accepted answer
1 votes

What you are talking about is called feature engineering. Basically it is done to reduce the dimensionality of the dataset. What we are doing is combining 2 or more features which provide the same ...

View answer
How to predict the winner of a future sports match?
1 votes

I'll answer your second question first! Lets take the example of IPL. For you to predict which team will win a match/tournament, you would need to build a multiclass classification model. The output ...

View answer
Determining if a dataset is balanced
1 votes

You should be looking at the frequency distribution of your dependent variable(output feature) when considering imbalance datasets because we are trying to predict the dependent feature and not the ...

View answer
When Does Feature Selection Takes Place?
1 votes

Normalization is done only for numerical variables and One Hot Encoding only for categorical variables. I would advise split you data into 2 dataframes. One for numerical features and other contains ...

View answer
How do I split correctly split my dataset into train, test and validation?
1 votes

Do not split the test set into half for the second train_test_split. Instead first split your whole data into train and test set. Then split the train set into train and validation sets as shown below....

View answer
improve LinearSVC
Accepted answer
1 votes

There are many ways to increase the accuracy. 1.) Try to get more data. More data usually helps in getting better results. (usually, not always!) 2.) Although you mention you have tried different ...

View answer
Is it acceptable to use label encoding for nominal categorical data when one hot encoding would create too many features?
1 votes

These are some of the things you could try to reduce the dimensionality of your dataset:- 1.) The first thing to do is feature engineering. Try to combine 2 or more features into 1 without losing ...

View answer