Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
3
votes
2answers
75 views

Is it acceptable not to transform() test data after train data is being fit_transform()-ed

We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
1
vote
0answers
16 views

Iterative Reweighted Least Squares in python

I am trying to manually implement the irls logistic regression (Chapter 4.3.3 in Bishop - Pattern Recognition And Machine Learning) in python. For updating the weights, I am using $w' = w-(\Phi^TR\...
0
votes
1answer
13 views

How to preprocess data for image classification from a .txt file?

Basically my issue is that im building an image classification model using AlexNet. I have this pre-split dataset thats already split into training, test, validation. However the issue is that these ...
0
votes
1answer
17 views

Don't know how to preprocess my dataset for image classification

I'm trying to do image classification using CNN. The exact model isn't important but I decided to try use AlexNet and I'm getting abysmal accuracy. I believe the issue might be with my data ...
1
vote
1answer
20 views

Which insights a data scientist could derive from text-analysis?

I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging. Since it is my first ...
0
votes
0answers
4 views

What Python algorithms for fitting a curve for a 3D elliptic point cloud to find about its “curvature”?

What Python algorithms for fitting a curve for a 3D elliptic point cloud to find about its "curvature"? Particularly it's a point cloud of a tree and I want to measure the curvature of its top.
0
votes
0answers
7 views

Example for sentiment analysis

I have one hundred sentences that I would like to study for sentiment analysis. The language is Italian. Could you please provide a small example on how I could approach this problem? For example, ...
0
votes
0answers
7 views

How to calculate final AUC for sequential combinations of binary classification models in Python

I am working on developing a binary classification model using GradientBoostingClassifier on a highly imbalanced dataset (100:1) that I plan to implement in 2 steps. build a model (M1) that will try ...
0
votes
0answers
17 views

MNIST GAN generator loss increasing

I'm trying to train a simple vanilla GAN on MNIST with Tensorflow. I used this github page as a reference and in my process to try and get my GAN to work I've made my code more and more similar to ...
0
votes
1answer
5 views

How to group by one column and count frequency from other column for each item in the previous column in python?

I am trying to group my data by the 'ID' column. Then I want to count the frequency of 'sequence' for each 'ID'. Here is a sample of the data frame: ...
0
votes
0answers
10 views

SKLearn NearestCentroidClassifier score with predict_proba

I'm using the NearestCentroidClassifier combined with TF-IDF for classification of documents. The are linked to a growing number of document groups. I've set sklearns TfIdfVectorizer and the ...
0
votes
0answers
10 views

Choosing a non-linear regression model and predicting

I'm new to data science and machine learning. I was working on a project and I happened to get this graph. I want to build a predictive model using this, for each of the boroughs. I do understand ...
1
vote
1answer
6 views

Tuning SVM C parameter

I would like to ask for help regarding my model. I have a dataset of preprocessed images and I performed a binary classification with SVM on Python. I tuned the value of the c parameter from 0.001 to ...
0
votes
0answers
13 views

Is this Tensorflow 1.x network get trained?

I've just started to learn Tensorflow 2.1.0 with Keras 2.3.1 and Python 3.7.7. I have found the following code from this "Omniglot Character Set Classification Using Prototypical Network": ...
0
votes
1answer
7 views

Determine how each feature contribute to XGBoost Classification

so for a summary of what I have done: My dataset has 5 classes and 10 parameters. I used XGBclassifer from sklearn to investigate if I could use those 10 parameters to predict the class of each data ...
0
votes
1answer
13 views

Machine Learning algorithm for predicting number of cases in pandemic

I’m giving my first steps with AI and Machine Learning so I have the following issue. I’m trying to predict an outcome from COVID-19 number of day vs confirmed cases using scikit-learn library. I mean,...
0
votes
0answers
11 views

Using Keras fit_generator for functional keras models and custom dataset

I have to fit a model that takes three discrete inputs and produces two discrete outputs using a generator made as follows: ...
0
votes
0answers
7 views

Selecting the optimal number of bins in KBinsDiscretizer?

I have Data Frame where are continuous values Features present. I want to bin these Features in category. I am using KBinsDescretizer for this. To find the optimal ...
1
vote
0answers
23 views

Linear Regression not working due to wrong kind of array

I try to deal with my homework. The Job is to take the following Data: http://www.statsci.org/data/general/uscrime.html and perform a linear regression on it. The code is published here: https://...
0
votes
0answers
10 views

Error when trying to include class_weight in Keras

I need to use class_weight because my minority class is only 2% of the data set. Here is the code to create and run the model: ...
0
votes
0answers
9 views

Do we train a feature extractor or we just use a trained network as a feature extractor?

I've just started to learn Tensorflow and Keras. I'm using Tensorflow 2.1.0 and Keras 2.3.7. I'm using this network as a feature extractor. Well, I'm trying to use it: ...
0
votes
0answers
7 views

can i compare output from predict_proba with my y_test value?

all, i am a bit confused at comparing my results from predict_proba (which returns an array) to it's dataframe e.g ...
1
vote
0answers
10 views

clustering more than optimal k and Overfitting in k-means

In my data by using elbow method. i got optimal k to be 3. but , i clustered them into 5 clusters.and the patterns in the cluster are as i wanted them . But, does using k more than optimal k decreases ...
0
votes
0answers
9 views

Using sklearn's make_pipeline output doesn't match between test dataframe and output dataframe

I have a simple sklearn pipeline defined as below and I create a train_test split to fit and test my model. The R2-score looks ...
2
votes
0answers
29 views

VAE generates bad images. due to unbalanced loss functions?

I'm training a variational autoencoder on CelebA dataset using TensorFlow.keras The problem I'm facing is that the generated images are not diverse enough and look kinda bad. example: What I think:...
0
votes
1answer
20 views

Visualization Tools that can show proper branch and merge in Graphs

I want help finding the visualization tool can draw similar architecture as given in the image below Keras visualization produces something similar to this graph. But I'm working in Pytorch. I tried ...
0
votes
1answer
23 views

How to impute using simple imputer (custom function)

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and ...
0
votes
1answer
10 views

Why is the response of my wikidata query not converting to json? [closed]

I am trying to get pictures from wikidata using the following code: ...
1
vote
1answer
27 views

Extract First names from usernames

John10 , michaelscott, James.white , Jr.Jones , James-Anderson ,WhiteWalter10 -- These are some of the different cases of usernames possible(there may be more ). I have about 200K such usernames . I ...
0
votes
0answers
5 views

Which qp solver is best for Support Vector machine implementation?

I'm trying to implement svm from scratch. I have used cvxopt to solve svm dual problem. cvxopt doesn't seem to produce accurate result when compared with other standard library. Are any other good ...
0
votes
2answers
24 views

How to identify and extract patterns from emails

I would like to know if it would be possible to identify some patterns in a text. For example, looking at emails, there is some common words used at the beginning and at the end. ...
0
votes
1answer
14 views

Generating Dinosaur names with Tensorflow RNN

I try to adapt "Text generation with an RNN" tutorial to generate new dinosaur names from a list of the existing ones. For training RNN tutorial text is divided into example character sequences of ...
0
votes
0answers
22 views

Is it ok to get different results in my implementation of svm when compared to other library?

I'm trying to implement svm for learning purpose using cvxopt in python. To check wheather my implementation works or not I compare with results of sklearn. Weights and biases of model where different ...
1
vote
1answer
17 views

Web Scraping: Multiple small files or one large file?

I plan to scrape some forums (Reddit, 4chan) for a research project. We will scrape the newest posts, every 10 minutes for around 3 months. I am wondering how best to store the JSON data from each ...
1
vote
2answers
18 views

Implementing training in PyTorch

I wish to accomplish the following task in PyTorch- I have the COCO dataset, wherein each data sample is used in training YOLO v3. After being processed by the model, the sample is to be deleted if ...
0
votes
0answers
27 views

How to estimate tail distribution for computing Conditional Value at Risk in python?

How to estimate parameteres of generalized pareto distribution in python? 🙏 I solve the task of computing Value at Risk and Conditional Value at Risk. ...
2
votes
1answer
24 views

How to implement my own loss function for Prototype learning using Keras Model

I'm trying to migrate this code, "Omniglot Character Set Classification Using Prototypical Network", into Tensorflow 2.1.0 and Keras 2.3.1. My problem is about how to use euclidean distance between ...
0
votes
1answer
13 views

Clustering based on missing values

I have a dataframe of the form: ...
0
votes
0answers
9 views

How to compute score on this pre-defined accuracy metric? [closed]

I have a dataset where I have to predict the predicted_price, which has a time-series data. I have successfully done a feature engineering on the dataset to extract the DateTime features. Following is ...
0
votes
1answer
15 views

Target in a dataset for text classification

I have a dataset that lacks the target variable. The dataset contains a list of emails and I would need to classify them as spam/not spam. Do you know how I can build a model that can help me to ...
0
votes
0answers
5 views

how to make contour plot centered using netcdf data

hello everyone, I am trying to produce a contour plot using contourf function in python for showing interannual monthly climatology. I do have got my plot but my concern is when I am ...
0
votes
0answers
20 views

How to label data to build a model from scratch

I have different texts, some of them have been manually labelled some others have not. Texts “The modelling dataset were rebuilt yesterday” “This movie is awful” “Trump said that ufo exist” I would ...
1
vote
0answers
17 views

Spam/ham classification

I am exploring the use of lime for spam/ham categorisation. specifically I have a data frame having list of messages. I would need to identify which messages are spam and which ones are ham by using a ...
0
votes
0answers
8 views

error in pandas dataframe when trying to convert dtype object to float

I am trying to aggregate data converted from an xlsx file to a dataframe but it is all dtype object. I tried to convert it to strings and float so I can aggregate it but this is the error I get : <...
1
vote
1answer
25 views

Understand how to simulate a statistics [closed]

This solution describes how to simulate statistics to find a confidence interval. A journalist called 1000 people in town to ask who will they be voting for out of candidates A and B. The observed ...
0
votes
0answers
12 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
0
votes
0answers
22 views

ReLU/Sigmoid Backpropagation Implementation

I've written a 2 layer Neural Network in Python for binary classification. The first layer uses ReLU activation, and the output layer is sigmoid activation. I'm using cross entropy to calculate loss. ...
0
votes
0answers
7 views

Coding advice/tricks to create a Seaborn heatmap for a high number of columns or most relevant only

The number of columns (more than 30) gives the Seaborn heatmap a messy appearance and impossible to glean important information from. What I am wondering is, can I map ONLY variables with the most ...
0
votes
0answers
7 views

Why I have obtained high error ARIMA model

I have a time series data. It has weekly frequency. I want to forecast the data for the nexts weeks with an ARIMA model. This is a chart of my time series data: First I use the method ...

1
2 3 4 5
91