Skiddles
  • Member for 5 years, 5 months
  • Last seen more than a month ago
Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?
7 votes

Tophat makes some great points. Another thing to consider is that you removed close to 20 percent of your data by removing the "outliers" which leads me to believe that they really aren't outliers, ...

View answer
How to calculate time difference in between rows using loop in panda python
6 votes

I realize that this has already been answered, but I thought I would propose another solution that takes advantage of vectorization. It should perform better if there are a lot of records by avoiding ...

View answer
In a binary classification, should the test dataset be balanced?
Accepted answer
3 votes

The answer to your first question: should the test dataset be balanced as well? is, like many answers in data science, "it depends." And really, it depends on the audience for, and ...

View answer
Logistic regression cost function
3 votes

Not trying to oversimplify the answer, but simply get a calculator to compute these manually and you can see this in action: If t is close to 1, lets just say that is 0.9999 for the example, then: $$ ...

View answer
Choosing between Regression and Classification
2 votes

If you are trying to build a neural network you should stick with a the orientation as a numeric value since neural networks only accept numeric data as input. If you want to use the class values, ...

View answer
cross validation for small dataset
2 votes

Given the size of your data set, the best approach to cross validation is the Leave-One-Out method. You haven't discussed the language or package you used for your model, but generally speaking you ...

View answer
Building predictive model with low correlated data
2 votes

It seems like a challenging problem. If it were my task, I would start with a probabilistic approach like apriori, but you may want to check out Naive Bayes based approach. There are some differences ...

View answer
Nearest Neighbors on mixed data types in high dimensions
2 votes

Conceptually, there's no reason why this 'could' not work, but practically speaking, there are probably better approaches than using KNN. Suitability of KNN One immediate problem with KNN is that ...

View answer
Can ReLU replace a Sigmoid Activation Function in Neural Network
2 votes

To answer your question directly, yes you can replace the sigmoid activation function with the ReLU activation function. I think the real question is "should you?" This is a much harder question to ...

View answer
Does ensemble (bagging, boosting, stacking, etc) always at least increase performance?
2 votes

The short answer is no. I have worked on several projects that evaluated an ensemble of several classifiers versus the classifies themselves. In some cases the precision and recall was better with ...

View answer
Discarding correlation among inputs in a neural network
2 votes

You could try some feature selection techniques if you want to choose 3. That said, have you tested the data for multicollinearity? Maybe you have but I don't think that a1+a2+a3+a4=1 implies a high ...

View answer
KNN with mixed data (feature set)
1 votes

KNN is a classification algorithm where, typically, continuous variables are used to apply a classification. Your problem is a problem of predicting a continuous value using variables that are not ...

View answer
How to classify images Neural Network didn't trained to Understand
Accepted answer
1 votes

Unfortunately, a neural network is only able to compute probabilities on labels that it has been trained to recognize. In your model, you only have three identified labels and presumably trained on a ...

View answer
If Deep learning or Machine learning is blackbox then why companies are still investing?
1 votes

You make a huge assumption that because something is a black box, that it is not valid. Almost every decision made by a human is essentially a black box decision. Most could tell you the important ...

View answer
filter a sequence of coordinates
1 votes

This raises many questions that you may or may not be able to answer, and this may not be helpful but hopefully it is. Regarding the data, what are it's origins? Is it all electronically collected ...

View answer
Hesitate to drop a feature
1 votes

The formal name for your concern is "multicollinearity". This can be a concern in statistically based techniques. There are formal tests that you can perform to assess multicollinearity like ...

View answer
What does the "randomly shuffle training samples" in stochastic gradient descent attain?
1 votes

If you are not training with minibatches, but just one batch per epoch, then random shuffle does nothing. However, if you are training with minibatches, and the data in the first minibatch is related ...

View answer
Possible reasons for word2vec learning context words as most similar rather than words in similar contexts
1 votes

Neural networks are different from other Machine Learning techniques in that they have to learn by repetition. In Neural Network parlance, the repetition is called an epoch. During an epoch, each ...

View answer
Document similarity matching between Doc2Vec documents
1 votes

Based on what you have shared, it seems like Doc2Vec should be suited to your objective. That said, I think the Doc in the package name can lead people astray. It gives the impression that you can ...

View answer
Finding outliers from multiple files
Accepted answer
1 votes

If you are looking to find outlier records based and then identify the file containing the records, K-means may be as good a place to start as any, but perhaps you need to join all the files together ...

View answer
What's a good machine learning model for an univariate data set?
1 votes

Your question is not very specific. I am assuming that you are looking for the full solution (Language, package, function, template). So with that in mind I'll point you to the following question: ...

View answer
During a regression task, I am getting low R^2 values, but elementwise difference between test set and prediction values is huge
1 votes

This line looks wrong to me: diff.append(100*np.abs(y_pred[i]-Y_test.values[i])/Y_test.values[i]) Shouldn't the abs be around the entire calculation? diff.append(100*np.abs((y_pred[i]-Y_test.values[...

View answer
Feasibility: train a model to learn how to extract data from documents
1 votes

Both of the answers from FelixGK and Wargream have merit. With the information you have posted though, I am not sure you are approaching the problem with the right objective, so the expected solution ...

View answer
Multi-Class Text Classification: Doc2Vec performing very bad compared to Hashing Vector
Accepted answer
1 votes

If I read your model correctly, you only performed 5 epochs with the Doc2Vec model. This is probably not be enough for the network to learn the word embeddings. Has your loss leveled out after 5 ...

View answer
Standard Deviation for Z-scores
1 votes

In your example, you should calculate the probability that the value goes up, and use that as your $p$. That said, I think the calculation should be $\sigma = \sqrt{p(1-p)}$ otherwise, your ...

View answer
On a multi lingual sentiment corpus
1 votes

Several questions and thoughts come to mind. What languages are in the corpus? This may impact what services you can leverage. I like the "Sentiment Idea" for languages that are supported natively ...

View answer
Capture pattern in python
1 votes

Try this: import re strings = ["humanresourc-emp-001_id-01_sc-01","itoperation-emp-002_id-02_sc-12","Generalsection-emp-003_id-03_sc-10"] new_list = [] pattern = '[a-zA-Z]+?[-]{1}(?P<empid>emp-[...

View answer
Use text similarity (cosine) instead of machine learning to classify companies into industries
1 votes

This is an interesting approach. For me though, it raises a few questions that might impact the effectiveness of the approach: Were all the descriptions created by the same organization, or ...

View answer
train Neural Network with SGD and see that it overfits data.
1 votes

Of the options you listed, I would try adding a dropout layer first. Another option you did not mention, but may, or may not be feasible, is to get more training data. Often overfitting is a result ...

View answer
What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?
1 votes

You could look at sensitivity and specificity. They can be combined effectively to either provide a basis for a correct classification, or the basis for an exclusionary classification within each ...

View answer