Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
2
votes
1answer
211 views

While using reindex method on any dataframe why do original values go missing?

This is the original Dataframe: What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame : I managed to do it by this piece of code : ...
3
votes
2answers
673 views

How to install Polynote on Windows?

I've been searching around the Internet for a while but I have not been able to find detailed instructions on how to install Polynote (the polyglot notebook with first-class Scala support) for Windows ...
1
vote
1answer
27 views

Filling created feature with values

I'm trying to improve accuracy. I created a few new features based on old features. So I need to fill new feature's empty cell with same values in order to equaling shapes.Then, I tried it with median ...
3
votes
2answers
75 views

Is it acceptable not to transform() test data after train data is being fit_transform()-ed

We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
0
votes
1answer
13 views

Machine Learning algorithm for predicting number of cases in pandemic

I’m giving my first steps with AI and Machine Learning so I have the following issue. I’m trying to predict an outcome from COVID-19 number of day vs confirmed cases using scikit-learn library. I mean,...
1
vote
2answers
30 views

Fraud risk propagation in large scale network

What's the best approach to do some graph analytics and risk propagation in a network using python where multiple accounts are connected through a relationship and few of the accounts in the network ...
1
vote
0answers
16 views

Iterative Reweighted Least Squares in python

I am trying to manually implement the irls logistic regression (Chapter 4.3.3 in Bishop - Pattern Recognition And Machine Learning) in python. For updating the weights, I am using $w' = w-(\Phi^TR\...
2
votes
0answers
29 views

VAE generates bad images. due to unbalanced loss functions?

I'm training a variational autoencoder on CelebA dataset using TensorFlow.keras The problem I'm facing is that the generated images are not diverse enough and look kinda bad. example: What I think:...
2
votes
1answer
58 views

sklearn - SimpleImputer in an empty Pipeline

When building a Pipeline I'm ending up at a scenario that can be simplified like this: FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps)) Since this is one intermediary step in a ...
1
vote
1answer
46 views

Probabilities in KERAS

I'm struggle with a problem OF class probabilities (binary, 0 and 1). Dont know why but after 100 epoches the probabilities became 0 or 1 (like the class to predict). Maybe something is not working ...
0
votes
1answer
17 views

Don't know how to preprocess my dataset for image classification

I'm trying to do image classification using CNN. The exact model isn't important but I decided to try use AlexNet and I'm getting abysmal accuracy. I believe the issue might be with my data ...
0
votes
1answer
13 views

How to preprocess data for image classification from a .txt file?

Basically my issue is that im building an image classification model using AlexNet. I have this pre-split dataset thats already split into training, test, validation. However the issue is that these ...
3
votes
1answer
2k views

ValueError: Numpy arrays that you are passing to your model is not the size the model expected

I am trying to perform concatenation on the Bidirectinal LSTM layer. I have my model defined like this: ...
0
votes
1answer
20 views

Dealing with large data: selecting a sample

I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python. The data set has 93000 ...
1
vote
1answer
20 views

Which insights a data scientist could derive from text-analysis?

I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging. Since it is my first ...
0
votes
1answer
76 views

Why linear regression feature coefficients become super large?

Introduction I've implemented linear regression using sklearn and after all calculations I've got results like this: ...
0
votes
0answers
4 views

What Python algorithms for fitting a curve for a 3D elliptic point cloud to find about its “curvature”?

What Python algorithms for fitting a curve for a 3D elliptic point cloud to find about its "curvature"? Particularly it's a point cloud of a tree and I want to measure the curvature of its top.
0
votes
4answers
153 views

How to get date columns in pandas?

I need a separate date column(in this format2018-10-08) in this table. please help. ...
0
votes
2answers
138 views

Subtract Rows of Matrix from rows of another matrix numpy

I have two matrix V_r of shape(19, 300) and vecs of shape(100000, 300). I would like to subtract rows of V_r from from rows of vecs. Currently, I am achieving this with the following code. Is there a ...
4
votes
4answers
36k views

print a specific column with a condition using pandas

I have a data set which contains 5 columns, I want to print the content of a column called 'CONTENT' only when the column 'CLASS' equals one. I know that using ...
0
votes
1answer
24 views

How to save multi-output predicted masks into two different folders after using model.predict_generator

I have a multi output segmentation task, the training process went well, but when Im trying to get the prediction I found difficulties to separate the two output into two different folders, in my ...
1
vote
1answer
39 views

making predictions from detrended data

I am building a machine learning based model (random forest in scikit-learn) to predict maize yields in the U.S. based on data on historical maize yields and temperature and precipitation information. ...
0
votes
0answers
7 views

Example for sentiment analysis

I have one hundred sentences that I would like to study for sentiment analysis. The language is Italian. Could you please provide a small example on how I could approach this problem? For example, ...
0
votes
1answer
21 views

Ways to increase recall in SVM

I am training an SVM on UCI's Bank Marketing Data Set, the bank additional-full.csv. As the data is skewed I am also interested in recall. I am getting accuracy of about 87.95% but my recall is around ...
13
votes
3answers
14k views

Gradients for bias terms in backpropagation

I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation when we take derivative with respect to bias) and I ...
0
votes
0answers
7 views

How to calculate final AUC for sequential combinations of binary classification models in Python

I am working on developing a binary classification model using GradientBoostingClassifier on a highly imbalanced dataset (100:1) that I plan to implement in 2 steps. build a model (M1) that will try ...
0
votes
0answers
17 views

MNIST GAN generator loss increasing

I'm trying to train a simple vanilla GAN on MNIST with Tensorflow. I used this github page as a reference and in my process to try and get my GAN to work I've made my code more and more similar to ...
0
votes
1answer
18 views

Make the ML model pass the Load Test

I have made a model to predict some values. In real time, it works perfectly fine and yields prediction within a fraction of a second. The algorithm I am using is PassiveAggresiveClassifier to predict ...
0
votes
1answer
34 views

Table transformation using Numpy, Pandas

I want to transform a table from this : To This : Explanation: basically I'm trying to create a "Sankey chart" for that I need this type of format. So From Table 1(1st Image): Using Date and ...
0
votes
1answer
61 views

How do I use this depth first search code to obtain a topological sort?

Problem: I need to implement a topological search using the following depth first search code. Note: The original code comes from here, and this is a problem given in at the end of the chapter. I'll ...
0
votes
1answer
5 views

How to group by one column and count frequency from other column for each item in the previous column in python?

I am trying to group my data by the 'ID' column. Then I want to count the frequency of 'sequence' for each 'ID'. Here is a sample of the data frame: ...
0
votes
0answers
10 views

SKLearn NearestCentroidClassifier score with predict_proba

I'm using the NearestCentroidClassifier combined with TF-IDF for classification of documents. The are linked to a growing number of document groups. I've set sklearns TfIdfVectorizer and the ...
0
votes
0answers
10 views

Choosing a non-linear regression model and predicting

I'm new to data science and machine learning. I was working on a project and I happened to get this graph. I want to build a predictive model using this, for each of the boroughs. I do understand ...
1
vote
1answer
1k views

Using LSTM to predict binary classification - accuracy stuck at 50% - how to use statefulness

I am trying to use an LSTM model to make binary classifications; however when I train the model the loss stays around 0.69 (ie. -$\ln(0.5)$) and the accuracy at 0.5, which suggests to me the model is ...
3
votes
5answers
8k views

Different number of features in train vs test

I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100). To be able to train my model I'm converting it ...
0
votes
1answer
23 views

How to impute using simple imputer (custom function)

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and ...
1
vote
1answer
6 views

Tuning SVM C parameter

I would like to ask for help regarding my model. I have a dataset of preprocessed images and I performed a binary classification with SVM on Python. I tuned the value of the c parameter from 0.001 to ...
0
votes
1answer
11 views

SciKit-Learn: predict values sometimes different from top predict_proba entries?

I noticed that in my text classification problem, I get significantly worse results when I use the max of predict_proba's output as compared to the straight ...
0
votes
0answers
13 views

Is this Tensorflow 1.x network get trained?

I've just started to learn Tensorflow 2.1.0 with Keras 2.3.1 and Python 3.7.7. I have found the following code from this "Omniglot Character Set Classification Using Prototypical Network": ...
1
vote
1answer
204 views

Data Augmentation Multi Outputs

This question is asked several times here on SE, but I havent been able to find the right answer. I'm trying to build a network with 1 input and 2 outputs. I don't have a lot of data so I would like ...
0
votes
1answer
115 views

Getting confusion matrix with Keras flow_from_directory

For a homework I have to analyse a set of images. For this I plan to use convolutional neural network. The images are split onto specific folders : A test set with 624 photos dataset/test/normal (...
0
votes
1answer
7 views

Determine how each feature contribute to XGBoost Classification

so for a summary of what I have done: My dataset has 5 classes and 10 parameters. I used XGBclassifer from sklearn to investigate if I could use those 10 parameters to predict the class of each data ...
1
vote
1answer
144 views

ValueError: The number of classes has to be greater than one; got 1 class in MultiLabel Classification problem

I am working in Python in a Multilabel Classification problem. I have a dataset with texts and around 20k unique labels. I transformed the text to word embeddings and now I use that in a ...
0
votes
3answers
61 views

Problem in implementing CNN

I am using the fashion MNIST dataset to try to work this out: I am using the data from the links: Training : http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ...
0
votes
0answers
11 views

Using Keras fit_generator for functional keras models and custom dataset

I have to fit a model that takes three discrete inputs and produces two discrete outputs using a generator made as follows: ...
1
vote
2answers
24 views

XGBoost Objective is Changed

I am trying to use XGBoost in python for logistic regression. I am calling it as follows ...
0
votes
0answers
7 views

Selecting the optimal number of bins in KBinsDiscretizer?

I have Data Frame where are continuous values Features present. I want to bin these Features in category. I am using KBinsDescretizer for this. To find the optimal ...
1
vote
0answers
23 views

Linear Regression not working due to wrong kind of array

I try to deal with my homework. The Job is to take the following Data: http://www.statsci.org/data/general/uscrime.html and perform a linear regression on it. The code is published here: https://...
0
votes
0answers
10 views

Error when trying to include class_weight in Keras

I need to use class_weight because my minority class is only 2% of the data set. Here is the code to create and run the model: ...

1
2 3 4 5
91