Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
2,163
questions with no upvoted or accepted answers
9
votes
2
answers
367
views
Are there any graph embedding algorithms like this already?
I wrote an algorithm for generating node embeddings based on the graph's topology. Most of the explanation is done in the readme file and the examples.
The question is:
Am I reinventing the wheel?
...
- 201
9
votes
0
answers
2k
views
Why is my Keras model not learning image segmentation?
Edit: as is turns out, not even the model's initial creator could successfully fine-tune it. This is most likely a problem of implementation, or possibly related to the non-intuitive way in which the ...
- 199
8
votes
1
answer
1k
views
Gensim LDA model: return keywords based on relevance (λ - lambda) value
I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
- 81
8
votes
0
answers
3k
views
Python : Feature Matching + Homography to find Multiple Objects
I'm trying to use OpenCV via Python to find multiple objects in a train image and match it with the key points detected from a query image. For my case, I'm trying to detect the tennis courts in the ...
- 81
7
votes
4
answers
4k
views
How to perform feature selection on dataset with categorical and numerical features?
I am working on a dataset with 30 columns (29 numerical, 1 non-ordinal categorical). I hot-encoded the categorical feature and reached at 35 columns. To improve training efficiency, I want to perform ...
- 71
7
votes
0
answers
2k
views
Using the Python Keras multi_gpu_model with LSTM / GRU to predict Timeseries data
I'm having an issue with python keras LSTM / GRU layers with multi_gpu_model for machine learning.
When I use a single GPU, the predictions work correctly ...
- 171
7
votes
0
answers
2k
views
Fine tuning accuracy lower than Raw Transfer Learning Accuracy
I've used transfer learning on Inception V3 with ImageNet weights on Keras with Tensorflow backend on python 2.7 to create an image classifier. I first extracted and saved the bottleneck features from ...
- 71
6
votes
3
answers
171
views
What Clustering Method Should I Use?
My data is a group of 10 thousand points (each having an node location (x,y)) that are spread across a plane. They are also chromatically-colored based on their weight.
I need to finalize a bayesian ...
6
votes
0
answers
138
views
Unable to transform (greatly performing) Autoencoder into Variational Autoencoder
Following the procedure described in this SO question, I am trying to transform my (greatly performing) convolutional Autoencoder into a Variational version of the same Autoencoder. As explained in ...
- 61
6
votes
1
answer
8k
views
Keras - Implementation of custom loss function with multiple outputs
I am trying to replicate (a way smaller version) the AlphaGo Zero system. However, in the network model, I am having a problem. The loss function I am supposed to implement is the following:
$$l = (z -...
- 205
6
votes
2
answers
3k
views
How to deal with missing data for Bernoulli Naive Bayes?
I am dealing with a dataset of categorical data that looks like this:
...
- 161
6
votes
2
answers
197
views
Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?
I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning.
Should I include the prediction matrix (ie. df containing columns of prediction results ...
- 161
6
votes
0
answers
12k
views
Tuning Gradient Boosted Classifier's hyperparametrs and balancing it
I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated.
Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model:
...
- 286
5
votes
2
answers
3k
views
Triangle Pattern Recognition on Financial Market with Python
I'm working on a personal project to find Triangles on any stock in Python.
I detect the max and min points (shift(-5,+5) because if I consider only shift(-1+1) I have a lot of lines) and write lines ...
5
votes
2
answers
3k
views
Multidimensional scaling producing different results for different seeds
I took the data from here and wanted to play around with multidimensional scaling with this data. The data looks like this:
In particular, I want to plot the cities in a 2D space, and see how much it ...
- 308
4
votes
0
answers
59
views
Does ROC AUC different between crossval and test set indicate overfitting or other problem?
I am training a composite model (XGBoost, Linear Regression, and RandomForest) to predict injured people probability. Well, the results of cross-validation with 5 folds. Well, I can see any problem ...
- 116
4
votes
1
answer
1k
views
SelectKBest and Correlation returns me excatly same feature selection. How?
Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But ...
4
votes
2
answers
7k
views
Saving and loading keras.callbacks.History object with np.save and np.load
I have been saving my training history in keras as follows:
...
- 51
4
votes
1
answer
1k
views
Advice on dealing with very large datasets - HDF5, Python
Recently, I've started working on an application for the visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional ...
- 41
4
votes
1
answer
1k
views
Calculating saliency maps for text classification
I'm following the text classification with movie reviews TensorFlow tutorial, and wanted to extend the project by looking, for a certain input, which words influenced the classification the most.
I ...
- 41
4
votes
1
answer
790
views
Product classification in hierarchical categories based on multiple parameters and non-standard descriptions
I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the ...
- 41
4
votes
1
answer
859
views
Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()
I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
- 41
4
votes
2
answers
235
views
Benefits of using Deep Learning-specific hyperparameter optimization tools vs. sklearn?
There are quite a few library for hyperparameter optimization that are specific to Keras or other Deep Learning libraries, like Hyperas or Talos.
My question is, what's the main benefit of using ...
- 161
4
votes
1
answer
450
views
Multiple activation functions with TensorFlow estimator DNNClassifier
I just want to know if is it possible to use tf.estimator.DNNClassifier with multiple different activation functions. I mean, could I use a DNNClassifier estimator which use different activation ...
- 41
4
votes
1
answer
355
views
Why is my U-matrix visually not separating the classes?
I am visualizing the U-matrix generated using a Self Organizing Map codebook to (visually) identify regions of similarity in the data.
Although I would like to use SOM to identify clusters in ...
- 321
4
votes
1
answer
952
views
What preprocessing steps to be followed before image comparison?
1
down vote
favorite
For example I am trying to find the similarity between two images using skimage - SSIM. The code block will be as follows
...
3
votes
0
answers
63
views
Where can I find implementation of the various improvements of K-nearest neighbors (KNN)?
I have been facing some challenges where traditional KNN algorithm perform well. I'd like to explore more advanced knn solutions. While researching possible solutions, I came across a paper titled
<...
- 2,567
3
votes
1
answer
89
views
Reinforcement Learning applied to Optimisation Problem
Problem Statement: We are given an optimisation problem; with production centres, source airport, destination airports, transfer points and finally delivered to the customers. This is better explained ...
- 31
3
votes
0
answers
1k
views
Identify MCAR, MNAR and MAR in the data
If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely:
MCAR (missing ...
- 1,896
3
votes
0
answers
215
views
Struggling to understand/implement Transformer Decoder
I'm struggling to understand the decoder in a Transformer model, specifically with regards to some aspects of its architecture as well as how it actually handles the data during training.
What I have ...
- 131
3
votes
0
answers
2k
views
Hopfield Network python implementation, Network doesn't converge to one of the learned patterns
I'm trying to implement a Hopfield Network in python using the NumPy library. The network has 2500 nodes (50 height x 50 width). The network learns 10 patterns from images of size 50x50 stored in &...
- 31
3
votes
0
answers
341
views
Is it possible to increase the number of images of one class using data augmentation, which is not applied on the other class, in the same dataset?
I have 2 classes for my image classification problem, say class A and class B, and I am using tensorflow and keras for the same. One of them have around 5K images while the other have just around 2K ...
3
votes
2
answers
849
views
Why does my random forest classifier predicts one class more often?
I have a random forest classifier that predicts 0 class about twice as often as class 1. It also predicts class 0 with higher probabilities than class 1.
It is not a imbalanced dataset.
I tried ...
- 31
3
votes
0
answers
651
views
What exactly negative/positive value of Captum's Integrated Gradient mean?
I use Captum's Integrated Gradient to interprete my PyTorch's neural network. I know that from github and original paper mentioned that ...
Positive attribution score means that the input in that ...
- 31
3
votes
0
answers
850
views
PyTorch: Train without dataloader (loop trough dataframe instead)
I was wondering if it is bad practice to instead of using built in tools such as dataloader just loop trough each row in a pandas df. Lets say I am doing text classification and my training loop looks ...
- 173
3
votes
1
answer
141
views
Issues with self-implemented logistic regression
I am trying to self-implement a logistic regression algorithm to do some self-learning but I am having a bit of trouble with achieving similar accuracy to the logistic regression of sklearn.
Here is ...
- 181
3
votes
0
answers
637
views
Explain FastText model using SHAP values
I have trained fastText model and some fully connected network build on its embeddings. I figured out how to use Lime on it: complete example can be found in Natural Language Processing Is Fun Part 3: ...
- 131
3
votes
1
answer
197
views
Is it possible to solve Rubik's cube using DQN?
I'm trying to solve Rubik's cube using deep learning and I came across with DQN, so I decided to give it a try. I developed all the code and started training but I got this results:
Loss goes up and ...
3
votes
1
answer
956
views
Error when trying Transfer Learning
I'm trying to train a model which is an extension of Google's Inception-V3 for the purpose of recognizing and classifying whether there is any pneumonia using x-ray images.
I've used Tensorflow-Hub ...
- 529
3
votes
1
answer
779
views
Character-level embeddings in python
I'm working on an NLP task that requires the use of character level embeddings, and I've been trying to use Spacy. However, it seems that spacy uses word-level embeddings for the word vectors, and I ...
- 163
3
votes
0
answers
38
views
Serializing a trained classification model into a set of actionable insights
I'm looking for ways to convert a trained classification model into a list of insights based on the resulting parameters of the model.
To make an example, let's assume we trained a decision tree to ...
- 83
3
votes
0
answers
795
views
Balancing the dataset using imblearn undersampling, oversampling and combine?
I have the imbalanced dataset:
data['Class'].value_counts()
Out[22]:
0 137757
1 4905
Name: Class, dtype: int64
...
- 163
3
votes
1
answer
250
views
Transfer Learning Question: Extending the Functionality of a Multipose-Estimation Machine Learning Model?
I have experimented with a number of different machine learning models used for pose estimation. Most of them output a heatmap and offsets for the detected person(s) in the image. I really like the ...
- 31
3
votes
0
answers
43
views
Information Extraction from image / text - approach?
I need assistance with a ML project I am currently trying to create.
I receive a lot of invoices from a lot of different suppliers - all in their own unique layout. I need to extract 3 key elements ...
- 185
3
votes
1
answer
224
views
tensorflow pseudo inverse doesn't work for complex matrices!
The Tensorflow documentation here says that:
tf.linalg.pinv is ''analogous to numpy.linalg.pinv. It differs only in
default value of rcond''.
However, tf.linalg.pinv requires the matrix to ...
- 33
3
votes
0
answers
2k
views
How to apply a groupby rolling function to create multiple columns in the dataframe
I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo and the link to the data is here and the example here. Some Sample of data ...
- 171
3
votes
1
answer
488
views
Error while plotting Logistic Regression Classification
I was trying to plot by using the following code
...
3
votes
2
answers
168
views
Machine learning model with simultaneous function optimization
Consider the following scenario. I am a sculpturer and customers ask me for what price I am willing to provide them with some statues. Their request for sculptures can vary in difficulty, quantity, ...
- 31
3
votes
0
answers
44
views
Improving a simple trig model
I have some data which I know is well approximated as a trig function, and I can fit it with scipy.optimize.curve_fit as follows:
...
- 131
3
votes
1
answer
1k
views
find bigrams in pandas
I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words. I want to find bi-grams using nltk and have ...
- 81