Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,578
questions
7
votes
3
answers
2k
views
0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial
My goal is to classify products pictures into categories such as dress, sandals, etc.
I am using the MNIST fashion dataset, following this official tutorial word-per-word: https://www.tensorflow.org/...
- 335
7
votes
1
answer
4k
views
Simple Q-Table Learning: Understanding Example Code
I'm trying to follow a tutorial for Q-Table learning from this source, and am having difficulty understanding a small piece of the code. Here's the entire block:
...
- 195
7
votes
1
answer
19k
views
What's a good Python HMM library? [duplicate]
I've looked at hmmlearn but I'm not sure if it's the best one.
- 243
7
votes
1
answer
21k
views
Regression in Keras
I was trying to implement a regression model in Keras, but am unable to figure out how to calculate the score of my model, i.e., how well it performed on my dataset.
...
- 3,001
7
votes
2
answers
22k
views
Merging large CSV files in pandas
I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
- 3,001
7
votes
3
answers
23k
views
Anomaly detection on time series
I've just started working on an anomaly detection development in Python.
My data sets are a collection of timeseries. More in details, data are coming from some sensors/meters which record and ...
- 325
7
votes
3
answers
11k
views
Testing independence of random variables in Python
Are there any tools available in Python that allow for testing of independence of two random variables (data columns)? I have two columns of data $X$ and $Y$. They can be both discrete, with values $\{...
- 343
7
votes
2
answers
8k
views
TF-IDF vectorizer doesn't work better than countvectorizer
I am working on a multilabel text classification problem with 10 labels. The dataset is small, +- 7000 items and +-7500 labels in total. I am using python sci-kit learn and something strange came up ...
- 71
7
votes
4
answers
4k
views
Determinate K in K-Means Clustering
I have salary data of several user (Python list).
Now I am using KMeans to cluster them.
Given this data, Is there a way to figure out the best value for 'K' automatically through program?
I tried ...
- 1,890
7
votes
1
answer
16k
views
Simple example of genetic alg minimization
I have been looking for a while for examples of how I could find the points at which a function achieves its minimum using a genetic algorithm approach in Python. I looked at DEAP documentation, but ...
- 275
7
votes
2
answers
20k
views
pandas count values for last 7 days from each date
There are two pd.DataFrame. First is like this:
...
- 73
7
votes
2
answers
6k
views
Is there a way of performing stratified cross validation using xgboost module in python?
I am training and predicting on the same data-set, but I want to perform 10-fold cross-validation and predict on the left out fold and thus predict on the whole data set. How can I do this?
The ...
- 91
7
votes
2
answers
754
views
Coursera ML - Does the choice of optimization algorithm affect the accuracy of multiclass logistic regression?
I recently completed exercise 3 of Andrew Ng's Machine Learning on Coursera using Python.
When initially completing parts 1.4 to 1.4.1 of the exercise, I ran into difficulties ensuring that my ...
- 123
7
votes
2
answers
28k
views
How to find a confidence level given the z value
I have a task from which I would like to find the confidence level given the z value. I have a sample population. From that population, given its distribution, I would like to find the confidence ...
- 173
7
votes
2
answers
441
views
How does real world machine learning production systems run?
Dear Machine Learning/AI Community,
I am just a budding and aspiring Machine Learner who has worked on open online data sets and some POC's built locally for my project. I have built some models and ...
7
votes
3
answers
12k
views
what should be the order of class names in sklearn tree export function (Beginner question on python sklearn)
I am trying a simple example with sklearn decision tree. I am giving "number,is_power2,is_even" as features and the class is "is_even" (of course this is stupid)
Here is the code
...
- 173
7
votes
2
answers
6k
views
Correct interpretation of summary_plot shap graph
While through the various resources online to understand the shap plots, I ended up slightly confused. Find below my interpretation of the overall plot given in examples -
Shap value 0 for a feature ...
- 101
7
votes
1
answer
9k
views
Multivariate Time Series Binary Classification
I have continuous (time series) data. This data is multivariate. Each feature can be represented as time series (they are all calculated on a daily basis). Here is an example:
...
- 175
7
votes
3
answers
2k
views
What is difference between Multi-class One vs All and Multilabel Classification?
Although Multi class is different from Multi label classification, whats difference does adding One vs All make in Multi-class.
Edit 1:
http://scikit-learn.org/stable/modules/multiclass.html#...
- 71
7
votes
3
answers
14k
views
How to predict class label from class probability given by predict_generator for testdata?
While using Keras' flow_from_directory method to train my model on a multi-class image classification problem, the ...
- 243
7
votes
1
answer
1k
views
How to use machine learning to extract product info from the titles of eBay listings
Reposting here because someone correctly pointed out it is better suited for here.
So I have a bunch of titles of eBay listings. I want to extract product information from each title, so I can ...
- 73
7
votes
1
answer
20k
views
Pandas how to fill missing values in one column if the values in another column are equal
I have a dataframe where I need to fill in the missing values in one column (paid_date) by using the values from rows with the same value in a different column (id). There is guaranteed to be no more ...
- 73
7
votes
1
answer
5k
views
Overfitting for minority class after SMOTE w/ random forests
I used SMOTE to make a predictive model, with class 1 having 1800 samples and 35000+ of class 0 samples. Hence, as per SMOTE, synthetic samples were created and the random forest was trained.
However,...
- 423
7
votes
2
answers
183
views
Machine learning model bundled with a library vs. an API
I am thinking to "deploy" a machine learning model (in pickle it is sized 3 megabytes) and after discussing with my developer colleagues, they said it would be better if the model is packed as a ...
- 73
7
votes
1
answer
2k
views
How to decide how many n_neighbors to consider while implementing LocalOutlierFactor?
I have a data set with rows: 134000 and columns: 200. I am trying to identify the outliers in data set using LocalOutlierFactor from scikit-learn. Although I ...
- 121
7
votes
1
answer
79
views
How to decide the shape of input features, when each data file is of different length?
To help me understand the benefits and shortcomings of decision trees, KNN, Neural Networks, ...
- 591
7
votes
1
answer
10k
views
Multiple search/replace on pandas series
I have a pandas dataframe with school names as one of the columns. However, there is quite a bit of misspelling in the school names, for example:
'Abernethy Elem School', 'Abernethy Elementary School'...
- 217
7
votes
1
answer
5k
views
sklearn: SGDClassifier yields lower accuracy than LogisticRegression
I'm participating in the kaggle Iceberg Classifier Challenge, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement ...
- 191
7
votes
2
answers
9k
views
R's mice imputation alternative in Python
What is Python's alternative to missing data imputation with mice in R? Imputation using median/mean seems pretty lame, I'm looking for other methods of imputation, ...
user25935
7
votes
1
answer
10k
views
How to remove columns in Transformer function in Pipeline?
I already used a custom transformation function in a scikit-learn pipeline. In this function I only added features to my data frame. It works great.
Below is a working example:
...
- 121
7
votes
1
answer
2k
views
Reproducing randomForest Proximity Matrix from R package in Python
I am trying to port this little piece of R code to python:
...
- 71
7
votes
2
answers
1k
views
Python distributed machine learning
I occasionally train neural nets for my research, and they usually take quite a long time to run (especially when I'm working on my laptop).
I'm looking for a way to build the model on any computer ...
- 1,071
7
votes
1
answer
398
views
Sensitivity to scaling of features in a multivariate gaussians
I'm using the HMMLearn python package for hidden markov models. That implementation is build on multivariate gaussian distributions.
So I have a string of features. How sensitive are gaussians to ...
- 73
7
votes
1
answer
1k
views
Pyspark: Filter dataframe based on separate specific conditions
How can I select only certain entries that match my condition and from those entries, filter again using regex?
For instance, I have this data frame (df):
col1
col2
col3
col4
A
f
5
g
D
er
2e
sd
F
...
7
votes
1
answer
426
views
On a multi lingual sentiment corpus
I am looking to compile a sentiment corpus for news articles in multiple languages (~100k per lang. for a machine learning experiment) where each article is labeled positive, neutral, or negative. I ...
- 193
7
votes
2
answers
234
views
migrating to python from R: specific questions
I have been using R and RStudio for prototyping and model building and due to some persisting problems (which would only be applicable to the environment that I am using in) we have decided to use ...
- 71
7
votes
1
answer
1k
views
Custom metrics for unbalanced classes problem in RandomForest or SVM
My dataset has highly unbalanced classes ‒ foreground of 30 classes with tens of samples against background set of >100k samples. Classifying foreground class as background is quite OK, while ...
- 71
7
votes
2
answers
1k
views
Time Series Machine Learning Feature Selection Problem
I have to solve a time series model that can take one of two shapes. It can probably take more but here are the two I'm going to ask about. If you have other ideas they are of course welcome.
First ...
- 71
7
votes
0
answers
619
views
Layman's Interpretation of XGBoost Importance [duplicate]
I'm trying to come up with a good way to explain the 3 importance metrics (Gain, Cover, Frequency) to a layman with only a basic understanding of XGBoost and trees in general. How best would you ...
- 121
7
votes
0
answers
2k
views
Fine tuning accuracy lower than Raw Transfer Learning Accuracy
I've used transfer learning on Inception V3 with ImageNet weights on Keras with Tensorflow backend on python 2.7 to create an image classifier. I first extracted and saved the bottleneck features from ...
- 71
7
votes
1
answer
2k
views
In XGBoost, how to change eval function and keeping same objective?
I would like to keep the objective as "reg:linear" and eval_metric as customized RMSE as follows:
...
6
votes
2
answers
22k
views
Delete/Drop only the rows which has all values as NaN in pandas [closed]
I have a Dataframe, i need to drop the rows which has all the values as NaN.
ID Age Gender
601 21 M
501 NaN F
NaN NaN NaN
The ...
- 233
6
votes
4
answers
39k
views
How to plot multiple variables with Pandas and Bokeh
I'm new to Pandas and Bokeh; I'd to create a bar plot that shows two different variables next to each other for comparison.
For instance, with the following Pandas data frame, I'd like to see how ...
- 161
6
votes
2
answers
26k
views
Compare image similarity in Python
I'm using a dataset of movies and would like to group if a movie is the same across different retailers.
Example:
Movie: Beauty and the Beast
Platforms: Google, Netflix, iTunes, Amazon.
I have ...
- 739
6
votes
2
answers
5k
views
How to create interactive plot of thousands of images as output of t-SNE?
I have many images that I want to plot as a result of running t-SNE and I want to be able to interactively explore them. matplotlib does not allow enough interactivity to explore, and plotly is too ...
- 231
6
votes
2
answers
2k
views
Understanding XG Boost Training (Multi class classification)
I have been working with XG boost for classification (multi class classification : 6 classes)
I use 5 fold CV to train and validate my model.
Please refer to the ...
- 165
6
votes
2
answers
27k
views
Value Error: Operands could not be broadcast together with shapes - LSTM
I am trying an LSTM model using tensorflow following this tutorial .
I am having trouble understanding why am I getting an ...
- 375
6
votes
2
answers
34k
views
How to implement Python's MLPClassifier with gridsearchCV?
I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code:
...
- 63
6
votes
3
answers
7k
views
Kaggle notebook Vs Google Colab
What are the major differences between Kaggle notebook and Google Colab notebook?
To work on a dataset my first step is to start a Kaggle notebook but then I cant help thinking what could be the ...
- 61
6
votes
2
answers
3k
views
Checking for stationarity in LSTM
An author in his blog checked for stationarity and removed them in a forecasting problem for using LSTM.I asked others and they said no need to check for it in LSTM.I read some articles and it looked ...
- 163