All Questions
1,627
questions
192
votes
13
answers
268k
views
K-Means clustering for mixed numeric and categorical data
My data set contains a number of numeric attributes and one categorical.
Say, NumericAttr1, NumericAttr2, ..., NumericAttrN, CategoricalAttr,
where ...
- 5,204
38
votes
6
answers
9k
views
How to set the number of neurons and layers in neural networks
I am a beginner to neural networks and have had trouble grasping two concepts:
How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever.
How does one decide ...
- 543
120
votes
1
answer
348k
views
How to get correlation between two categorical variable and a categorical variable and continuous variable?
I am building a regression model and I need to calculate the below to check for correlations
Correlation between 2 Multi level categorical variables
Correlation between a Multi level categorical ...
- 1,848
243
votes
10
answers
347k
views
How to set class weights for imbalanced classes in Keras?
I know that there is a possibility in Keras with the class_weights parameter dictionary at fitting, but I couldn't find any example. Would somebody so kind to ...
- 7,807
42
votes
5
answers
56k
views
Does gradient descent always converge to an optimum?
I am wondering whether there is any scenario in which gradient descent does not converge to a minimum.
I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am ...
- 523
33
votes
6
answers
14k
views
Why do convolutional neural networks work?
I have often heard people saying that why convolutional neural networks are still poorly understood. Is it known why convolutional neural networks always end up learning increasingly sophisticated ...
- 431
199
votes
35
answers
32k
views
Publicly Available Datasets
One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
269
votes
11
answers
254k
views
What are deconvolutional layers?
I recently read Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, Trevor Darrell. I don't understand what "deconvolutional layers" do / how they work.
The ...
- 17.4k
44
votes
5
answers
21k
views
Is it always better to use the whole dataset to train the final model?
A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
- 3,720
176
votes
5
answers
124k
views
What is the "dying ReLU" problem in neural networks?
Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:
"Unfortunately, ReLU units can be fragile during training and can
"die". For ...
- 3,765
7
votes
1
answer
2k
views
Why you shouldn't upsample before cross validation
I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross-validate when oversampling data using SMOTE ...
- 323
253
votes
8
answers
302k
views
Micro Average vs Macro average Performance in a Multiclass classification setting
I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the ...
- 3,425
77
votes
7
answers
87k
views
In supervised learning, why is it bad to have correlated features?
I read somewhere that if we have features that are too correlated, we have to remove one, as this may worsen the model. It is clear that correlated features means that they bring the same information, ...
- 1,129
4
votes
3
answers
5k
views
Obtaining consistent one-hot encoding of train / production data
I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination ...
- 298
227
votes
10
answers
318k
views
What's the difference between fit and fit_transform in scikit-learn models?
I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we ...
- 2,737
72
votes
7
answers
82k
views
Open source Anomaly Detection in Python
Problem Background:
I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
- 933
17
votes
5
answers
4k
views
Beginner math books for Machine Learning
I'm a Computer Science engineer with no background in statistics or advanced math.
I'm studying the book Python Machine Learning by Raschka and Mirjalili, but when I tried to understand the math of ...
37
votes
8
answers
7k
views
What would I prefer - an over-fitted model or a less accurate model?
Let's say we have two models trained. And let's say we are looking for good accuracy.
The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted.
The second has an ...
- 479
3
votes
4
answers
2k
views
Is it possible to get worse model after optimization?
I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong.
in order to ...
- 339
167
votes
4
answers
108k
views
When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?
I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building ...
- 1,793
65
votes
5
answers
43k
views
Adding Features To Time Series Model LSTM
have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to ...
- 1,135
64
votes
4
answers
68k
views
Why mini batch size is better than one single "batch" with all training data?
I often read that in case of Deep Learning models the usual practice is to apply mini batches (generally a small one, 32/64) over several training epochs. I cannot really fathom the reason behind this....
- 7,807
55
votes
6
answers
15k
views
Should I go for a 'balanced' dataset or a 'representative' dataset?
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
- 653
90
votes
10
answers
375k
views
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
I got ValueError when predicting test data using a RandomForest model.
My code:
...
- 2,475
28
votes
2
answers
21k
views
How to feed LSTM with different input array sizes?
If I like to write a LSTM network and feed it by different input array sizes, how is it possible?
For example I want to get voice messages or text messages in a ...
- 1,132
37
votes
2
answers
51k
views
Should we apply normalization to test data as well?
I am doing a project on author identification problem. I had applied the tf-idf normalization to train data and then trained a svm on that data.
Now when using the classifier should I normalize test ...
- 595
23
votes
3
answers
6k
views
Starting my career as Data Scientist, is Software Engineering experience required? [closed]
I am an MSc student at the University of Edinburgh, specialized in machine learning and natural language processing. I had some practical courses focused on data mining, and others dealing with ...
- 787
16
votes
1
answer
19k
views
Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras
I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
- 263
2
votes
1
answer
272
views
How to efficiently iterate a supervised model over the Cartesian product of very large number of records?
The problem:
Two large databases, with ~1M records each, "old customer data" and "new customer data". The data came from different sources and was ingested at different times, so there are many ...
- 477
147
votes
6
answers
163k
views
The cross-entropy error function in neural networks
In the MNIST For ML Beginners they define cross-entropy as
$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$
$y_i$ is the predicted probability value for class $i$ and $y_i'$ is the true probability ...
- 17.4k
62
votes
9
answers
10k
views
Tools and protocol for reproducible data science using Python
I am working on a data science project using Python.
The project has several stages.
Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and ...
- 751
57
votes
10
answers
77k
views
Why should the data be shuffled for machine learning tasks
In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find ...
- 13.3k
33
votes
5
answers
38k
views
Unbalanced multiclass data with XGBoost
I have 3 classes with this distribution:
Class 0: 0.1169
Class 1: 0.7668
Class 2: 0.1163
And I am using xgboost for ...
- 495
33
votes
4
answers
15k
views
Quick guide into training highly imbalanced data sets
I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
- 5,204
6
votes
3
answers
1k
views
Correlation vs Multicollinearity
I have been taught to check correlation matrix before going for any algorithm.
I have a few questions around the same:
Pearson Correlation is for numerical variables only.
What if we have to check ...
- 151
3
votes
1
answer
4k
views
How to interpret Variance Inflation Factor (VIF) results?
From various books and blog posts, I understood that the Variance Inflation Factor (VIF) is used to calculate collinearity. They say that VIF till 10 is good. But I have a question.
As we can see in ...
- 153
24
votes
2
answers
24k
views
Why do we need to discard one dummy variable?
I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable ...
4
votes
2
answers
4k
views
Encode multi-class response variable
In a classification problem when the response variable has multi-class, e.g., "sunny","rainy","cloudy", how should we encode it? I know that for predictors like this, usually we do One Hot Encoding, ...
- 515
8
votes
4
answers
8k
views
Why decision tree needs categorical variable to be encoded?
As per my intuition, decision trees should work better with categorical variables than with continuous variables. If this is the case, why is encoding needed on categorical variables? Can someone give ...
- 81
109
votes
2
answers
100k
views
Training an RNN with examples of different lengths in Keras
I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
- 1,210
72
votes
4
answers
32k
views
What is the difference between "equivariant to translation" and "invariant to translation"
I'm having trouble understanding the difference between equivariant to translation and invariant to translation.
In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio)...
- 883
57
votes
5
answers
67k
views
Number of parameters in an LSTM model
How many parameters does a single stacked LSTM have? The number of parameters imposes a lower bound on the number of training examples required and also influences the training time. Hence knowing the ...
- 1,237
27
votes
3
answers
43k
views
Data Science Project Ideas [closed]
I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.
I have just started with Data Science and ...
- 373
33
votes
3
answers
16k
views
What is a good way to transform Cyclic Ordinal attributes?
I am having 'hour' field as my attribute, but it takes a cyclic values. How could I transform the feature to preserve the information like '23' and '0' hour are close not far.
One way I could think ...
- 539
30
votes
4
answers
12k
views
Role derivative of sigmoid function in neural networks
I try to understand role of derivative of sigmoid function in neural networks.
First I plot sigmoid function, and derivative of all points from definition using python. What is the role of this ...
- 447
20
votes
1
answer
34k
views
Why ReLU is better than the other activation functions
Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
- 13.3k
33
votes
2
answers
31k
views
Why use both validation set and test set?
Consider a neural network:
For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the ...
- 1,216
27
votes
1
answer
27k
views
How is a splitting point chosen for continuous variables in decision trees?
I have two questions related to decision trees:
If we have a continuous attribute, how do we choose the splitting value?
Example: Age=(20,29,50,40....)
Imagine that we have a continuous attribute $f$...
- 381
24
votes
3
answers
24k
views
How do you apply SMOTE on text classification?
Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it ...
- 359
12
votes
1
answer
14k
views
How to Predict the future values of time horizon with Keras?
I just built this LSTM neural network with Keras
...
- 263