All Questions
1,735
questions
200
votes
13
answers
302k
views
K-Means clustering for mixed numeric and categorical data
My data set contains a number of numeric attributes and one categorical.
Say, NumericAttr1, NumericAttr2, ..., NumericAttrN, CategoricalAttr,
where ...
- 5,454
40
votes
6
answers
11k
views
How to set the number of neurons and layers in neural networks
I am a beginner to neural networks and have had trouble grasping two concepts:
How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever.
How does one decide ...
- 573
132
votes
1
answer
376k
views
How to get correlation between two categorical variable and a categorical variable and continuous variable?
I am building a regression model and I need to calculate the below to check for correlations
Correlation between 2 Multi level categorical variables
Correlation between a Multi level categorical ...
- 2,018
260
votes
10
answers
422k
views
How to set class weights for imbalanced classes in Keras?
I know that there is a possibility in Keras with the class_weights parameter dictionary at fitting, but I couldn't find any example. Would somebody so kind to ...
- 8,487
46
votes
5
answers
67k
views
Does gradient descent always converge to an optimum?
I am wondering whether there is any scenario in which gradient descent does not converge to a minimum.
I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am ...
- 563
35
votes
6
answers
16k
views
Why do convolutional neural networks work?
I have often heard people saying that why convolutional neural networks are still poorly understood. Is it known why convolutional neural networks always end up learning increasingly sophisticated ...
- 451
89
votes
7
answers
116k
views
In supervised learning, why is it bad to have correlated features?
I read somewhere that if we have features that are too correlated, we have to remove one, as this may worsen the model. It is clear that correlated features means that they bring the same information, ...
- 1,269
287
votes
8
answers
365k
views
Micro Average vs Macro average Performance in a Multiclass classification setting
I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the ...
- 3,775
282
votes
12
answers
273k
views
What are deconvolutional layers?
I recently read Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, Trevor Darrell. I don't understand what "deconvolutional layers" do / how they work.
The ...
- 18.8k
202
votes
35
answers
33k
views
Publicly Available Datasets
One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
73
votes
7
answers
83k
views
Open source Anomaly Detection in Python
Problem Background:
I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
- 933
195
votes
5
answers
147k
views
What is the "dying ReLU" problem in neural networks?
Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:
"Unfortunately, ReLU units can be fragile during training and can
"die". For ...
- 4,025
240
votes
10
answers
347k
views
What's the difference between fit and fit_transform in scikit-learn models?
I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we ...
- 2,877
55
votes
5
answers
31k
views
Is it always better to use the whole dataset to train the final model?
A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
- 3,920
41
votes
8
answers
9k
views
What would I prefer - an over-fitted model or a less accurate model?
Let's say we have two models trained. And let's say we are looking for good accuracy.
The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted.
The second has an ...
- 519
10
votes
1
answer
3k
views
Why you shouldn't upsample before cross validation
I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross-validate when oversampling data using SMOTE ...
- 417
5
votes
3
answers
8k
views
Obtaining consistent one-hot encoding of train / production data
I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
- 308
30
votes
2
answers
27k
views
How to feed LSTM with different input array sizes?
If I like to write a LSTM network and feed it by different input array sizes, how is it possible?
For example I want to get voice messages or text messages in a ...
- 1,260
21
votes
4
answers
29k
views
Train/Test Split after performing SMOTE
I am dealing with a highly unbalanced dataset so I used SMOTE to resample it.
After SMOTE resampling, I split the resampled dataset into training/test sets using the training set to build a model and ...
- 2,735
3
votes
4
answers
4k
views
Is it possible to get worse model after optimization?
I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong.
in order to ...
- 349
172
votes
4
answers
122k
views
When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?
I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building ...
- 1,843
95
votes
10
answers
437k
views
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
I got ValueError when predicting test data using a RandomForest model.
My code:
...
- 2,735
68
votes
5
answers
49k
views
Adding Features To Time Series Model LSTM
have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to ...
- 1,205
67
votes
4
answers
75k
views
Why mini batch size is better than one single "batch" with all training data?
I often read that in case of Deep Learning models the usual practice is to apply mini batches (generally a small one, 32/64) over several training epochs. I cannot really fathom the reason behind this....
- 8,487
59
votes
6
answers
16k
views
Should I go for a 'balanced' dataset or a 'representative' dataset?
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
- 693
44
votes
2
answers
66k
views
Should we apply normalization to test data as well?
I am doing a project on an author identification problem. I applied the tf-idf normalization to train data and then trained an SVM on that data.
Now when using the classifier, should I normalize test ...
- 665
24
votes
3
answers
6k
views
Starting my career as Data Scientist, is Software Engineering experience required? [closed]
I am an MSc student at the University of Edinburgh, specialized in machine learning and natural language processing. I had some practical courses focused on data mining, and others dealing with ...
- 807
17
votes
5
answers
4k
views
Beginner math books for Machine Learning
I'm a Computer Science engineer with no background in statistics or advanced math.
I'm studying the book Python Machine Learning by Raschka and Mirjalili, but when I tried to understand the math of ...
16
votes
1
answer
20k
views
Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras
I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
- 263
10
votes
5
answers
13k
views
Why decision tree needs categorical variable to be encoded?
As per my intuition, decision trees should work better with categorical variables than with continuous variables. If this is the case, why is encoding needed on categorical variables? Can someone give ...
- 101
5
votes
1
answer
11k
views
How pre-trained BERT model generates word embeddings for out of vocabulary words?
Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
- 2,041
3
votes
1
answer
7k
views
How to interpret Variance Inflation Factor (VIF) results?
From various books and blog posts, I understood that the Variance Inflation Factor (VIF) is used to calculate collinearity. They say that VIF till 10 is good. But I have a question.
As we can see in ...
- 153
151
votes
6
answers
166k
views
The cross-entropy error function in neural networks
In the MNIST For ML Beginners they define cross-entropy as
$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$
$y_i$ is the predicted probability value for class $i$ and $y_i'$ is the true probability ...
- 18.8k
69
votes
11
answers
98k
views
Why should the data be shuffled for machine learning tasks
In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find ...
- 13.9k
63
votes
9
answers
10k
views
Tools and protocol for reproducible data science using Python
I am working on a data science project using Python.
The project has several stages.
Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and ...
- 761
49
votes
3
answers
66k
views
StandardScaler before or after splitting data - which is better?
When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before ...
- 705
40
votes
6
answers
51k
views
Unbalanced multiclass data with XGBoost
I have 3 classes with this distribution:
Class 0: 0.1169
Class 1: 0.7668
Class 2: 0.1163
And I am using xgboost for ...
- 565
35
votes
4
answers
16k
views
Quick guide into training highly imbalanced data sets
I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
- 5,454
26
votes
2
answers
30k
views
Why do we need to discard one dummy variable?
I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable ...
17
votes
4
answers
6k
views
Can a neural network compute $y = x^2$?
In spirit of the famous Tensorflow Fizz Buzz joke and XOr problem I started to think, if it's possible to design a neural network that implements $y = x^2$ function?
Given some representation of a ...
- 317
6
votes
3
answers
2k
views
Correlation vs Multicollinearity
I have been taught to check correlation matrix before going for any algorithm.
I have a few questions around the same:
Pearson Correlation is for numerical variables only.
What if we have to check ...
- 161
4
votes
1
answer
795
views
How to preprocess with NLP a big dataset for text classification
TL;DR
I've never done nlp before and I feel like I'm not doing it in the good way. I'd like to know if I'm really doing things in a bad way since the beginning or ...
- 143
4
votes
2
answers
5k
views
Encode multi-class response variable
In a classification problem when the response variable has multi-class, e.g., "sunny","rainy","cloudy", how should we encode it? I know that for predictors like this, usually we do One Hot Encoding, ...
- 625
3
votes
1
answer
676
views
Difference between PCA and regularisation
Currently, I am confusing about PCA and regularisation.
I wonder what is the difference between PCA and regularisation: particularly lasso (L1) regression?
Seems both of them can do the feature ...
- 133
2
votes
1
answer
436
views
How to efficiently iterate a supervised model over the Cartesian product of very large number of records?
The problem:
Two large databases, with ~1M records each, "old customer data" and "new customer data". The data came from different sources and was ingested at different times, so there are many ...
- 477
124
votes
2
answers
113k
views
Training an RNN with examples of different lengths in Keras
I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
- 1,360
81
votes
5
answers
46k
views
What is the difference between "equivariant to translation" and "invariant to translation"
I'm having trouble understanding the difference between equivariant to translation and invariant to translation.
In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio)...
- 973
59
votes
6
answers
57k
views
Does XGBoost handle multicollinearity by itself?
I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
- 1,783
59
votes
5
answers
77k
views
Number of parameters in an LSTM model
How many parameters does a single stacked LSTM have? The number of parameters imposes a lower bound on the number of training examples required and also influences the training time. Hence knowing the ...
- 1,297
43
votes
5
answers
57k
views
What is the relationship between the accuracy and the loss in deep learning?
I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following:
...
- 1,985