Tasos
  • Member for 7 years, 8 months
  • Last seen this week
How to get accuracy, F1, precision and recall, for a keras model?
Accepted answer
52 votes

Metrics have been removed from Keras core. You need to calculate them manually. They removed them on 2.0 version. Those metrics are all global metrics, but Keras works in batches. As a result, it ...

View answer
How much of data wrangling is a data scientist's job?
40 votes

This is a situation that many blogs, companies and papers acknowledge as something real in many cases. In this paper Data Wrangling for Big Data: Challenges and Opportunities, there is a quote about ...

View answer
How to remove outliers using box-plot?
Accepted answer
15 votes

Seaborn uses inter-quartile range to detect the outliers. What you need to do is to reproduce the same function in the column you want to drop the outliers. It's quite easy to do in Pandas. If we ...

View answer
How to get the number of syllables in a word?
11 votes

You can try another Python library called Pyphen. It's easy to use and supports a lot of languages. import pyphen dic = pyphen.Pyphen(lang='en') print dic.inserted('Rohit') >>'Ro-hit'

View answer
Perform k-means clustering over multiple columns
10 votes

Let's take as an example the Breast Cancer Dataset from the UCI Machine Learning. This is how it looks >> _data.head(5) Age BMI Glucose Insulin HOMA Leptin Adiponectin ...

View answer
Is there any way to define custom entities in Spacy
Accepted answer
6 votes

For pretrained models, spaCy has a few in different languages. You can find them in their official documentation https://spacy.io/models The available models are: English German French Spanish ...

View answer
Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?
Accepted answer
5 votes

You can use the Kolmogorov-Smirnov Test. From Wikipedia In statistics, the Kolmogorov–Smirnov test (K–S test or KS test) is a nonparametric test of the equality of continuous, one-dimensional ...

View answer
pandas - under a column, count the total number of a specific value, instead of using value_counts()
Accepted answer
4 votes

You have plenty of ways to do it. You won't see a big difference in performance. My suggestion is to use whatever feels more convenient for you or your team. import pandas as pd import numpy as np #...

View answer
Small data set in machine learning
4 votes

In general, Machine Learning algorithms handle volumes data. This doesn't mean that you cannot extract information from "small" data. Keep in mind: Overfitting. With only a few data, the risk to ...

View answer
How to Avoid rarely used discrete feature values in a dataset
Accepted answer
3 votes

Working with Categorical Features can raise a few challenges. For instance, you might encounter features with high cardinality in certain values or the other case (your case), features with rare ...

View answer
Seaborn subplots massive whitespace
Accepted answer
3 votes

You can move the title closer to the first figure. Just add the following two lines at the end of the code fig.tight_layout() fig.subplots_adjust(top=0.95)

View answer
read csv file directly from URL / How to Fix a 403 Forbidden Error
Accepted answer
3 votes

The problem is that the url you have doesn't accept "non-browser" requests. The default header of Python requests is 'User-Agent': 'python-requests/2.13.0' You can pass your own headers as an ...

View answer
K-Means visualisation problem 8 numerical feutres
3 votes

It's not clear enough what you try to do. If I understand correctly, you want to train a K-Means clustering and visualize the results. However, you have 8 dimensions in your dataset and obviously, you ...

View answer
How to predict whether or not a customer will renew
3 votes

What you are trying to do is called Churn Prediction. Unfortunately, the dataset you have is not enough to train a model. You need a variety of different features for a proper prediction model. For ...

View answer
Will unnecessary features harm the tree based model?
3 votes

This is not a direct answer to your question, but more like an experiment. I created a simple script in Python where I ran multiple times the Iris dataset with the regular columns and also with 4 ...

View answer
How should I analyze this data from reddit (sample text included)
Accepted answer
3 votes

Data Science is not an algorithm to run on your data. It is the process that helps you answer a specific question. The key to be a data scientist is to ask the right questions. So, first, since you ...

View answer
Choosing best methods for estimating the unknown parameters in a linear regression model
3 votes

After the Data Munging, this is the most difficult task on a prediction model. However, in order to answer it, we need more details. What do you mean by the "best model"? Do you want accuracy and long ...

View answer
What kind of RAM to choose for data analysis?
3 votes

It's not easy to compare two RAMs with different frequency and latency, since both of them affect your performance with not the same way. The short answer is: If you have two RAMs with same capacity ...

View answer
Randomstate and kmeans issues
2 votes

Unfortunately, there isn't a built-in option to do it. Each time you run K-Means, the labels are assigned randomly. Even if you state the same random seed. However, based on this answer in ...

View answer
merge 2 dataframe with Memory Error
2 votes

The problem is that when you merge two dataframes, you need enough memory for both of them, plus the merged one. There is a workaround from a stackoverflow answer What you can do is to read the first ...

View answer
Can we use DecisionTreeClassifier of sklearn for continuous target variable?
Accepted answer
2 votes

In the Wine Dataset you linked, the quality column is not a continues variable but a discrete. It takes integer value between 0 and 10. When you use the DecisionTreeClassifier, you make the ...

View answer
Split the data between the Training Data and Test Data using sklearn
2 votes

What you are looking for is called Stratified sampling From this CrossValidated question, we have a short explanation Stratified sampling aims at splitting one data set so that each split are ...

View answer
How to measure correlation between several categorical features and a numerical label in Python?
2 votes

If you want to do an ANOVA test, you can do it with scipy and stats package. Link to documentation You can do it like that def anova(data): if len(data.groupby(level=1)) <= 2: raise ...

View answer
Why does test data need to be normalized on train data mean and std?
Accepted answer
2 votes

The reason you split your dataset to Training and Test is to simulate real-world cases. What you actually do with the train-split validation is to evaluate your model in unknown data. Imagine now ...

View answer
What is the ideal database that allows fast cosine distance?
2 votes

If you are afraid that the dataset is big that a regular database might not handle it, you could consider an alternative implementation such as SimHash. From Wikipedia, In computer science, ...

View answer
Is there a way to cluster words based on how similarly they sound?
2 votes

I haven't tried it myself, but you could try the IPA (International Phonetic Alphabet) version of your words and then calculate the Levenshtein distance. There is a Python library called panphon. I ...

View answer
Feature engineering from date, mean and standard deviation
2 votes

Date fields are quite interesting data since the limit of what you can "feature engineer" with them is your imagination. However, it is difficult to know a priori if one of them will improve your ...

View answer
Strange Pearson Correlation Coefficient Given DataFrame
2 votes

If two variables are independent, then their correlation will be zero. However, you cannot say the opposite. Zero correlation doesn't necessarily imply independence. It's hard to answer your question ...

View answer
How would you optimize this code?
Accepted answer
2 votes

You can replace the symbol in dataframe without iterating yourself. df = df.replace({'R\$': ''}, regex=True) Then change the type of columns that can be numeric. If you don't know which are those ...

View answer
How would you improve and optimize a manual encoding like this?
2 votes

The easiest and most efficient way to do it, is to use the One Hot Encoding ScikitLearn This is an example: >>> from sklearn.preprocessing import OneHotEncoder >>> enc = ...

View answer