Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

clustering more than optimal k and Overfitting in k-means

In my data by using elbow method. i got optimal k to be 3. but , i clustered them into 5 clusters.and the patterns in the cluster are as i wanted them . But, does using k more than optimal k decreases ...
0
votes
0answers
11 views

Using sklearn's make_pipeline output doesn't match between test dataframe and output dataframe

I have a simple sklearn pipeline defined as below and I create a train_test split to fit and test my model. The R2-score looks ...
3
votes
1answer
91 views

VAE generates bad images. due to unbalanced loss functions?

I'm training a variational autoencoder on CelebA dataset using TensorFlow.keras The problem I'm facing is that the generated images are not diverse enough and look kinda bad. ******(new) Example:****...
0
votes
1answer
26 views

Visualization Tools that can show proper branch and merge in Graphs

I want help finding the visualization tool can draw similar architecture as given in the image below Keras visualization produces something similar to this graph. But I'm working in Pytorch. I tried ...
0
votes
1answer
42 views

How to impute using simple imputer (custom function)

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and ...
1
vote
1answer
27 views

Extract First names from usernames

John10 , michaelscott, James.white , Jr.Jones , James-Anderson ,WhiteWalter10 -- These are some of the different cases of usernames possible(there may be more ). I have about 200K such usernames . I ...
0
votes
0answers
5 views

Which qp solver is best for Support Vector machine implementation?

I'm trying to implement svm from scratch. I have used cvxopt to solve svm dual problem. cvxopt doesn't seem to produce accurate result when compared with other standard library. Are any other good ...
0
votes
2answers
31 views

How to identify and extract patterns from emails

I would like to know if it would be possible to identify some patterns in a text. For example, looking at emails, there is some common words used at the beginning and at the end. ...
0
votes
1answer
16 views

Generating Dinosaur names with Tensorflow RNN

I try to adapt "Text generation with an RNN" tutorial to generate new dinosaur names from a list of the existing ones. For training RNN tutorial text is divided into example character sequences of ...
0
votes
0answers
26 views

Is it ok to get different results in my implementation of svm when compared to other library?

I'm trying to implement svm for learning purpose using cvxopt in python. To check wheather my implementation works or not I compare with results of sklearn. Weights and biases of model where different ...
1
vote
1answer
19 views

Web Scraping: Multiple small files or one large file?

I plan to scrape some forums (Reddit, 4chan) for a research project. We will scrape the newest posts, every 10 minutes for around 3 months. I am wondering how best to store the JSON data from each ...
1
vote
2answers
21 views

Implementing training in PyTorch

I wish to accomplish the following task in PyTorch- I have the COCO dataset, wherein each data sample is used in training YOLO v3. After being processed by the model, the sample is to be deleted if ...
0
votes
0answers
30 views

How to estimate tail distribution for computing Conditional Value at Risk in python?

How to estimate parameteres of generalized pareto distribution in python? 🙏 I solve the task of computing Value at Risk and Conditional Value at Risk. ...
2
votes
1answer
27 views

How to implement my own loss function for Prototype learning using Keras Model

I'm trying to migrate this code, "Omniglot Character Set Classification Using Prototypical Network", into Tensorflow 2.1.0 and Keras 2.3.1. My problem is about how to use euclidean distance between ...
0
votes
2answers
21 views

Clustering based on missing values

I have a dataframe of the form: ...
0
votes
1answer
23 views

Target in a dataset for text classification

I have a dataset that lacks the target variable. The dataset contains a list of emails and I would need to classify them as spam/not spam. Do you know how I can build a model that can help me to ...
0
votes
0answers
6 views

how to make contour plot centered using netcdf data

hello everyone, I am trying to produce a contour plot using contourf function in python for showing interannual monthly climatology. I do have got my plot but my concern is when I am ...
0
votes
0answers
25 views

How to label data to build a model from scratch

I have different texts, some of them have been manually labelled some others have not. Texts “The modelling dataset were rebuilt yesterday” “This movie is awful” “Trump said that ufo exist” I would ...
1
vote
0answers
18 views

Spam/ham classification

I am exploring the use of lime for spam/ham categorisation. specifically I have a data frame having list of messages. I would need to identify which messages are spam and which ones are ham by using a ...
0
votes
0answers
13 views

error in pandas dataframe when trying to convert dtype object to float

I am trying to aggregate data converted from an xlsx file to a dataframe but it is all dtype object. I tried to convert it to strings and float so I can aggregate it but this is the error I get : <...
1
vote
1answer
25 views

Understand how to simulate a statistics [closed]

This solution describes how to simulate statistics to find a confidence interval. A journalist called 1000 people in town to ask who will they be voting for out of candidates A and B. The observed ...
0
votes
1answer
19 views

Efficient Dataframe Transformation

So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase ...
0
votes
0answers
27 views

ReLU/Sigmoid Backpropagation Implementation

I've written a 2 layer Neural Network in Python for binary classification. The first layer uses ReLU activation, and the output layer is sigmoid activation. I'm using cross entropy to calculate loss. ...
0
votes
0answers
8 views

Coding advice/tricks to create a Seaborn heatmap for a high number of columns or most relevant only

The number of columns (more than 30) gives the Seaborn heatmap a messy appearance and impossible to glean important information from. What I am wondering is, can I map ONLY variables with the most ...
0
votes
0answers
8 views

Why I have obtained high error ARIMA model

I have a time series data. It has weekly frequency. I want to forecast the data for the nexts weeks with an ARIMA model. This is a chart of my time series data: First I use the method ...
0
votes
0answers
4 views

Difference in Plots of VAR Plot_Forecast vs Simple Overlay on Pyplot

I am sure, I am doing some thing extremely silly but for the life of me not able to figure our cause of the issue. So here it is: Step 0: Form my training set ...
0
votes
1answer
27 views

How to fix “Expected sequence or array-like”

I am trying to get the accuracy of the model and I am getting this error TypeError: Expected sequence or array-like, got Here's my code sample. ...
1
vote
2answers
56 views

Image Classification low accuracy

I have a dataset that has two folders for training and testing. I am trying to determine whether a patient has an eye disease or not. However, the images I have are hard to work with. I've ran this ...
1
vote
1answer
19 views

Text classification: accuracy [closed]

I would like to understand how to compute the accuracy of cluster analysis. I have hundred of texts. The dataset looks like as follows: ...
0
votes
0answers
15 views

Sentiment Analysis using FastText

I would need to run some sentiment analysis on some texts. I read about the use as FastText for similar purposes. Since the texts are in Italian I would need that the dictionaries can be in Italian. I ...
1
vote
0answers
8 views

Meta Learning: how to train a model with Support Set and Query Set

I've just started to learn Meta Learning reading the book Hands-On Meta Learning with Python. I think I know the answer for my question, but I'm a little confuse about how to implement the algorithm ...
0
votes
1answer
29 views

find monthly climatology for a particular month

I have a 10-year sst monthly data. I want to plot the interannual monthly climatology for the month of October. By using the code I have for python with me I am able to plot the interannual monthly ...
0
votes
1answer
39 views

How to make custom callback in keras to generate sample image in VAE training?

I'm training a simple VAE model on 64*64 images and I would like to see the images generated after every epoch or every couple batches to see the progress. when I train the model I wait until the ...
1
vote
1answer
22 views

Confusion matrix doesn't display properly

I am trying to plot a confusion matrix using the Logistic Regression for a multi-class ...
0
votes
2answers
36 views

Compute Accuracy of k-means [duplicate]

Could you please provide me an example of how I can compute the accuracy for a kmeans clustering? I split my dataset into train and test sets and computed the predicted clusters for the train set. ...
0
votes
1answer
16 views

how to limit datetime in x axis

Hello folks, I am trying to plot three variables in a single XY plot. I am having a problem setting the x-axis limit which is for time. I tried setting limits using x_limit but getting ...
0
votes
1answer
17 views

Text analysis: structure and sentiment

I would need to analyse the structure of texts like this: ...
2
votes
2answers
37 views

High error Arima model - Python

I have a time series data. It has daily frequency. I want to forecast the data for the next week or month with an ARIMA model. This is a chart of my time series data: First I use the method ...
0
votes
0answers
17 views

Save latex equation in Jupyter as image

I have a latex equation in my Jupiter notebook. \begin{equation*} s_{(x,n)} = 2 ^ {-\frac{E(h(x))}{c(n)}} \end{equation*} Can I save the equation as an image? ...
0
votes
0answers
15 views

Unsupervised learning-Feature selection in python

I need to select the most important features from my data frame before starting with nearest neighbours problem. Which methods are the best to do this? My data frame has around 8 categorical features ...
1
vote
0answers
115 views

Pytorch XLA to solve the spawn problems in a Colab Env

As reference only, here is my code It seems that torch.multiprocessing.set_start_method("spawn") can't be used in an Colab Env. Only 'fork' is allowed. I have ...
0
votes
1answer
17 views

spaCy - Text Preprocessing - Keeping “Pronouns” in text

I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text). I have built a custom text classification model. I ...
1
vote
1answer
15 views

How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)

Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0
votes
1answer
26 views

How do I show my pandas dataframe that I have in my jupyter notebook so that I can ask a question on stack exchange/stack overflow?

I've noticed that people produce such an amazing pandas dataframe which allows people to answer the questions. However, whenever I try to copy my table after doing ...
1
vote
1answer
43 views

Plot hurricane track

Hello folks, Recently while going through a research paper I happened to come across a plot showing the hurricane track(figure attached). Can anyone say how it is plotted or anyone has ...
0
votes
0answers
24 views

Why does the LSTM overfit all the time

I have a time series prediction problem from building an LSTM. My code: ...
0
votes
0answers
15 views

Python Pandas, how to select cases for which timestamps should be rounded?

I need to only select the cases for sensor type == air to be rounded to the nearest 5 seconds but do not how I should use a function to make this happen. I do have the following lines: ...
0
votes
1answer
34 views

How to save and load a Q-Learning Agent

I know this may sound nooby, but how do I save a Deep Q-Learning agent's progress? I mean when I close at i.e. episode 500 when my agent is trained and I restart (in my case a pygame) my agent is ...
2
votes
1answer
31 views

Data-preprocessing for Machine Learning model

I am confused about how to preprocess range based category such as age, tumor-size & inv-nodes. Should I take an average of the limits, as in - 14.5, 24.5 and so on or do one hot encoding of the ...
1
vote
1answer
25 views

How do I train a model with pre-saved weights?

I need to train a model but the data after preprocessing is too large to fit in the RAM. Is it possible to preprocess the data in batches and train a model for the small data set and then save the ...

1
3 4
5
6 7
93