Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).
4,506
questions
2
votes
1answer
211 views
While using reindex method on any dataframe why do original values go missing?
This is the original Dataframe:
What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame :
I managed to do it by this piece of code :
...
3
votes
2answers
673 views
How to install Polynote on Windows?
I've been searching around the Internet for a while but I have not been able to find detailed instructions on how to install Polynote (the polyglot notebook
with first-class Scala support) for Windows ...
1
vote
1answer
27 views
Filling created feature with values
I'm trying to improve accuracy. I created a few new features based on old features. So I need to fill new feature's empty cell with same values in order to equaling shapes.Then, I tried it with median ...
3
votes
2answers
75 views
Is it acceptable not to transform() test data after train data is being fit_transform()-ed
We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
0
votes
1answer
13 views
Machine Learning algorithm for predicting number of cases in pandemic
I’m giving my first steps with AI and Machine Learning so I have the following issue. I’m trying to predict an outcome from COVID-19 number of day vs confirmed cases using scikit-learn library. I mean,...
1
vote
2answers
30 views
Fraud risk propagation in large scale network
What's the best approach to do some graph analytics and risk propagation in a network using python where multiple accounts are connected through a relationship and few of the accounts in the network ...
1
vote
0answers
16 views
Iterative Reweighted Least Squares in python
I am trying to manually implement the irls logistic regression (Chapter 4.3.3 in Bishop - Pattern Recognition And Machine Learning) in python.
For updating the weights, I am using $w' = w-(\Phi^TR\...
2
votes
0answers
29 views
VAE generates bad images. due to unbalanced loss functions?
I'm training a variational autoencoder on CelebA dataset using TensorFlow.keras
The problem I'm facing is that the generated images are not diverse enough and look kinda bad.
example:
What I think:...
2
votes
1answer
58 views
sklearn - SimpleImputer in an empty Pipeline
When building a Pipeline I'm ending up at a scenario that can be simplified like this:
FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps))
Since this is one intermediary step in a ...
1
vote
1answer
46 views
Probabilities in KERAS
I'm struggle with a problem OF class probabilities (binary, 0 and 1). Dont know why but after 100 epoches the probabilities became 0 or 1 (like the class to predict).
Maybe something is not working ...
0
votes
1answer
17 views
Don't know how to preprocess my dataset for image classification
I'm trying to do image classification using CNN. The exact model isn't important but I decided to try use AlexNet and I'm getting abysmal accuracy.
I believe the issue might be with my data ...
0
votes
1answer
13 views
How to preprocess data for image classification from a .txt file?
Basically my issue is that im building an image classification model using AlexNet. I have this pre-split dataset thats already split into training, test, validation. However the issue is that these ...
3
votes
1answer
2k views
ValueError: Numpy arrays that you are passing to your model is not the size the model expected
I am trying to perform concatenation on the Bidirectinal LSTM layer. I have my model defined like this:
...
0
votes
1answer
20 views
Dealing with large data: selecting a sample
I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python.
The data set has 93000 ...
1
vote
1answer
20 views
Which insights a data scientist could derive from text-analysis?
I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging.
Since it is my first ...
0
votes
1answer
76 views
Why linear regression feature coefficients become super large?
Introduction
I've implemented linear regression using sklearn and after all calculations I've got results like this:
...
0
votes
0answers
4 views
What Python algorithms for fitting a curve for a 3D elliptic point cloud to find about its “curvature”?
What Python algorithms for fitting a curve for a 3D elliptic point cloud to find about its "curvature"?
Particularly it's a point cloud of a tree and I want to measure the curvature of its top.
0
votes
4answers
153 views
How to get date columns in pandas?
I need a separate date column(in this format2018-10-08) in this table. please help.
...
0
votes
2answers
138 views
Subtract Rows of Matrix from rows of another matrix numpy
I have two matrix V_r of shape(19, 300) and vecs of shape(100000, 300). I would like to subtract rows of V_r from from rows of vecs. Currently, I am achieving this with the following code. Is there a ...
4
votes
4answers
36k views
print a specific column with a condition using pandas
I have a data set which contains 5 columns, I want to print the content of a column called 'CONTENT' only when the column 'CLASS' equals one.
I know that using ...
0
votes
1answer
24 views
How to save multi-output predicted masks into two different folders after using model.predict_generator
I have a multi output segmentation task, the training process went well, but when Im trying to get the prediction I found difficulties to separate the two output into two different folders, in my ...
1
vote
1answer
39 views
making predictions from detrended data
I am building a machine learning based model (random forest in scikit-learn) to predict maize yields in the U.S. based on data on historical maize yields and temperature and precipitation information. ...
0
votes
0answers
7 views
Example for sentiment analysis
I have one hundred sentences that I would like to study for sentiment analysis. The language is Italian. Could you please provide a small example on how I could approach this problem?
For example, ...
0
votes
1answer
21 views
Ways to increase recall in SVM
I am training an SVM on UCI's Bank Marketing Data Set, the bank additional-full.csv. As the data is skewed I am also interested in recall. I am getting accuracy of about 87.95% but my recall is around ...
13
votes
3answers
14k views
Gradients for bias terms in backpropagation
I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation when we take derivative with respect to bias) and I ...
0
votes
0answers
7 views
How to calculate final AUC for sequential combinations of binary classification models in Python
I am working on developing a binary classification model using GradientBoostingClassifier on a highly imbalanced dataset (100:1) that I plan to implement in 2 steps.
build a model (M1) that will try ...
0
votes
0answers
17 views
MNIST GAN generator loss increasing
I'm trying to train a simple vanilla GAN on MNIST with Tensorflow. I used this github page as a reference and in my process to try and get my GAN to work I've made my code more and more similar to ...
0
votes
1answer
18 views
Make the ML model pass the Load Test
I have made a model to predict some values. In real time, it works perfectly fine and yields prediction within a fraction of a second. The algorithm I am using is PassiveAggresiveClassifier to predict ...
0
votes
1answer
34 views
Table transformation using Numpy, Pandas
I want to transform a table from this :
To This :
Explanation:
basically I'm trying to create a "Sankey chart" for that I need this type of format.
So From Table 1(1st Image): Using Date and ...
0
votes
1answer
61 views
How do I use this depth first search code to obtain a topological sort?
Problem: I need to implement a topological search using the following depth first search code.
Note: The original code comes from here, and this is a problem given in at the end of the chapter.
I'll ...
0
votes
1answer
5 views
How to group by one column and count frequency from other column for each item in the previous column in python?
I am trying to group my data by the 'ID' column. Then I want to count the frequency of 'sequence' for each 'ID'. Here is a sample of the data frame:
...
0
votes
0answers
10 views
SKLearn NearestCentroidClassifier score with predict_proba
I'm using the NearestCentroidClassifier combined with TF-IDF for classification of documents. The are linked to a growing number of document groups. I've set sklearns TfIdfVectorizer and the ...
0
votes
0answers
10 views
Choosing a non-linear regression model and predicting
I'm new to data science and machine learning. I was working on a project and I happened to get this graph.
I want to build a predictive model using this, for each of the boroughs. I do understand ...
1
vote
1answer
1k views
Using LSTM to predict binary classification - accuracy stuck at 50% - how to use statefulness
I am trying to use an LSTM model to make binary classifications; however when I train the model the loss stays around 0.69 (ie. -$\ln(0.5)$) and the accuracy at 0.5, which suggests to me the model is ...
3
votes
5answers
8k views
Different number of features in train vs test
I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).
To be able to train my model I'm converting it ...
0
votes
1answer
23 views
How to impute using simple imputer (custom function)
I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to
remove nans and ...
0
votes
0answers
15 views
1
vote
1answer
6 views
Tuning SVM C parameter
I would like to ask for help regarding my model.
I have a dataset of preprocessed images and I performed a binary classification with SVM on Python.
I tuned the value of the c parameter from 0.001 to ...
0
votes
1answer
11 views
SciKit-Learn: predict values sometimes different from top predict_proba entries?
I noticed that in my text classification problem, I get significantly worse results when I use the max of predict_proba's output as compared to the straight ...
0
votes
0answers
13 views
Is this Tensorflow 1.x network get trained?
I've just started to learn Tensorflow 2.1.0 with Keras 2.3.1 and Python 3.7.7.
I have found the following code from this "Omniglot Character Set Classification Using Prototypical Network":
...
1
vote
1answer
204 views
Data Augmentation Multi Outputs
This question is asked several times here on SE, but I havent been able to find the right answer.
I'm trying to build a network with 1 input and 2 outputs. I don't have a lot of data so I would like ...
0
votes
1answer
115 views
Getting confusion matrix with Keras flow_from_directory
For a homework I have to analyse a set of images. For this I plan to use convolutional neural network.
The images are split onto specific folders :
A test set with 624 photos
dataset/test/normal (...
0
votes
1answer
7 views
Determine how each feature contribute to XGBoost Classification
so for a summary of what I have done:
My dataset has 5 classes and 10 parameters. I used XGBclassifer from sklearn to investigate if I could use those 10 parameters to predict the class of each data ...
1
vote
1answer
144 views
ValueError: The number of classes has to be greater than one; got 1 class in MultiLabel Classification problem
I am working in Python in a Multilabel Classification problem. I have a dataset with texts and around 20k unique labels. I transformed the text to word embeddings and now I use that in a ...
0
votes
3answers
61 views
Problem in implementing CNN
I am using the fashion MNIST dataset to try to work this out:
I am using the data from the links:
Training :
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
...
0
votes
0answers
11 views
Using Keras fit_generator for functional keras models and custom dataset
I have to fit a model that takes three discrete inputs and produces two discrete outputs using a generator made as follows:
...
1
vote
2answers
24 views
XGBoost Objective is Changed
I am trying to use XGBoost in python for logistic regression. I am calling it as follows
...
0
votes
0answers
7 views
Selecting the optimal number of bins in KBinsDiscretizer?
I have Data Frame where are continuous values Features present. I want to bin these Features in category. I am using KBinsDescretizer for this. To find the optimal ...
1
vote
0answers
23 views
Linear Regression not working due to wrong kind of array
I try to deal with my homework. The Job is to take the following Data:
http://www.statsci.org/data/general/uscrime.html and perform a linear regression on it.
The code is published here: https://...
0
votes
0answers
10 views
Error when trying to include class_weight in Keras
I need to use class_weight because my minority class is only 2% of the data set. Here is the code to create and run the model:
...
