Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0 votes
0 answers
25 views

Dealing with near duplicates using NLP

I have a dataframe like as shown below ...
user avatar
  • 2,215
0 votes
0 answers
14 views

Manual garbage collection throwing KeyErrors using Python inside Azure Function

I have a python script that I push up to an Azure Function App (It's specifically a blob storage trigger). My function app is hitting memory caps every so often, and I thought it might be a good idea ...
user avatar
0 votes
1 answer
26 views

Keras Callback to save all the values of each batch per epoch

I would like to gather all the values during training for each epoch. When using the fit function of tensorflow I only receive the last value after run through all ...
user avatar
2 votes
0 answers
70 views

random_state on train_test_split() appears to have large effect in performance metrics?

To summarize the problem: I have a data set with ~1450 samples, 19 features and a binary outcome where classes are fairly balanced (0.51 to 0.49). I split the data into a train set and a test set ...
user avatar
  • 31
0 votes
0 answers
7 views

Constant trend for test error

I'm using keras functional API in order to model a multi-input/multi-output relation in the form (y1,y2) = f(x1,x2,x3). I have a really simple architecture composed by 2 hidden layer of 64 nodes and ...
user avatar
0 votes
0 answers
28 views

Song playlist recommendation system

I want to build a recommender system to suggest similar songs to continue a playlist (similar to what Spotify does by recommending similar songs at the end of a playlist). I want to build two models: ...
user avatar
  • 1
1 vote
3 answers
126 views

How To Develop Cluster Models Where the Clusters Occur Along Subsets of Dimensions in Multidimensional Data?

I have been exploring clustering algorithms (K-Means, K-Medoids, Ward Agglomerative, Gaussian Mixture Modeling, BIRCH, DBSCAN, OPTICS, Common Nearest-Neighbour Clustering) with multidimensional data. ...
user avatar
1 vote
1 answer
84 views

How to use confidence labels?

I have 2 sets of training data in csv files. The training data have class labels, 1 for memorable, and 0 for not memorable. In addition, there is also a confidence label for each sample. The class ...
user avatar
1 vote
0 answers
34 views

Multi Linear Regression on String Values

I'm using datasets which involves mostly of string values. The main outcome of the project is that it should predict success. Now I can use OneHotEncoding to convert string values in numerical format ...
user avatar
0 votes
1 answer
18 views

How to count words in a dataframe?

I would like to count how many Male and female who answer (ex. Biking / Cycling). Below is the sample data:
user avatar
0 votes
0 answers
33 views

pytorch resize and pad image

Does torch have the same functionality as tf.image.resize_with_pad? I need to resize a picture and pad it with zeros to a wanted size. E.g: ...
user avatar
3 votes
2 answers
59 views

Approaches to fit a theoretical model on a physical device

Happy to join this community. Thank you in advance for your kind help! :) Intro I have a physical device characterized by its internal parameters, of which I know the nominal values. I also have the ...
user avatar
0 votes
1 answer
87 views

Python: calculate the weighted average correlation coefficient

I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
user avatar
  • 101
0 votes
0 answers
26 views

write a dataframe using a column that contains a cycle (number of the weekin every year) in python

I want to write a dataframe with a column of the number of week and make it as a row that contains 53 columns (I mean weeks). I really need your help. NOTE THAT: each year has 53 week, and the column ...
user avatar
  • 1
1 vote
0 answers
7 views

Evaluation of the preprocessing to make a dataset anonymous

I have a very huge dataset from the NLP area and I want to make it anonymous. Is there any way to check if my pre-processing is correct? Generaly, is there any way to evaluate how good is the pre-...
user avatar
0 votes
0 answers
18 views

NaNs in predictions with LSTM

I have an LSTM model that I have trained and tested it with a dataset. Now I want to test it to an other dataset and I use the following snippet: ...
user avatar
  • 101
0 votes
1 answer
49 views

Input shape error

I've this item: ['6', '1', '6', '843537', '3', '0', '5', '1006709', '3', '1', '4'] with shape: (11,) but when go to predict with: ...
user avatar
0 votes
0 answers
14 views

use pre-built model to predict audio file

i am using this code to build a model that recognizes emotions in speech, but i can't figure out how to use it after loading it in a new python file. this is what i have but the results are always the ...
user avatar
-1 votes
2 answers
67 views

could not convert string to float: 'YELLOW'

...
user avatar
0 votes
0 answers
16 views

Time series classification using multiples multivariate multi-length timeseries

0 I would like to develop a time series classification algorithm to classify use a of parachute. My data consist of multiple recording files (around 5min at 100hz, length of the recording vary) with a ...
user avatar
0 votes
0 answers
13 views

Federated Learning: some clients with 0% accuracy

Suppose that I am doing a Federated Learning experiment using MNIST. As you know MNIST has 10 classes. Now, Federated Learning is useful especially in cases like hospitals, for collaborations, because ...
user avatar
1 vote
1 answer
25 views

Tensor dot product with rank one tensor from vector

I'm trying to compute an inner product between tensors in numpy. I have a vector $x$ of shape (n,) and a tensor $y$ of shape ...
user avatar
  • 11
1 vote
1 answer
46 views

Imbalanced Binary Dataset in Keras. Finding the best threshold after fit s.t. sensitivity and specificity is maximized?

I have made an ANN in Keras that works on an imbalanced binary dataset. The data is used after fitting the model to predict the binary classes and I want to choose a threshold s.t. sensitivity and ...
user avatar
1 vote
0 answers
62 views

How to draw each ROC curve of an SVM model with cross validation

I would like to make a graph like the following in python: That is, one curve for each fold. I have the following code where I use an SVM model to classify some data ...
user avatar
  • 302
0 votes
0 answers
51 views

Is it possible to implement an rdd version of a for loop having map and reduce using pyspark?

I need to test an algorithm that computes a function on a dataframe where in each execution I drop a column and computes the function. This is a example in python pyspark but without using rdd: ...
user avatar
  • 1
0 votes
0 answers
12 views

What kind of Keras data classification mode should I pick for my kind of machine learning problem?

So I have to come up with a machine learning model for a specific problem that I have. As far as I know, Keras allows for four different types of classification: Binary, Categorical, Multiclass ...
user avatar
0 votes
0 answers
18 views

Which LSTM Training Strategy Performs better?

I would like to use LSTM for predicting multiple time series (Time series about sales per day in multiple countries. In parts, contradicting regional trends are present within the data. Sales is the '...
user avatar
  • 1
0 votes
0 answers
9 views

multi variate time forecasting

I want to forecast in a time serie the 'output'. I have from the past the correlated time series 'output', 'capacity' and 'load'. I also know from the nearby future the time series from the 'capacity' ...
user avatar
0 votes
1 answer
16 views

Using CNNs to detect incorrect label images in dataset

What I want to do is to train a model to identify the images that are incorrectly labeled in my dataset, for example, in a class of dogs, I can find cats images and I want a model that detects all ...
user avatar
2 votes
2 answers
104 views

Genetic Algorithms (Specifically with Keras)

I can't get my deep genetic algorithm snake game to work and I can't figure out why. At this point, I think it must be either the crossover_rate/mutation_rate or the actual crossover code itself is ...
user avatar
0 votes
0 answers
13 views

Evaluate Dendrogram Statistical Significance

I have N=21 objects and each one has about 80 possible not NaN descriptors. I carried out a hierarchical clustering on the objects and I obtained this dendrogram. I want some kind of 'confidence' ...
user avatar
  • 1
0 votes
0 answers
9 views

Naive Bayes TfidfVectorizer predicts everything to one class

I'm trying to run Multinomial Bayes classificator on various balanced data sets and comparing 2 different vectorizers: TfidfVectorizer and CountVectorizer. I have 3 classes: NEG, NEU and POS. I have ...
user avatar
2 votes
1 answer
22 views

The single CSV created by combining a large number of CSV files is too large to process. What options do I have?

The dataset I am currently working on has more than 100 csv files, with each of size more than 250MB. These are files containing time series data captured from different locations and all the files ...
user avatar
0 votes
1 answer
13 views

Question about AMD cpu and if there are any problems with any particular libraries

I am getting a new laptop, the best laptop (or at least on paper) that I saw is using amd cpu and gpu. I heard that amd doesn't work well with tensorflow, curious if this is true, if is true is there ...
user avatar
2 votes
0 answers
28 views

Association between categorical variables with no hierarchy in Python

I have a dataset with over 100 possible variable occurrences across 20 columns. At first glance this problem seemed to fit into hierarchical clustering. I started testing with Agglomerative Clustering,...
user avatar
  • 21
0 votes
0 answers
13 views

Static and dynamic feature for hybrid modle

I would like to make a hybrid model in this way : I have a question about input data. My dynamic input is like [ [ [5],[4],[7] ],[ [7],[5],[7] ],...]. So shape is (...
user avatar
  • 11
0 votes
0 answers
14 views

Python DataScience Library vs SQL

I am working on a project where there is a necessity to store considerable data. I was wondering what is the difference between using SQL and the datascience library in python. I intend to use SQL ...
user avatar
0 votes
1 answer
22 views

How do i generate text from ids in Torchtext's sentencepiece_numericalizer?

The torchtext sentencepiece_numericalizer() outputs a generator with indices SentencePiece model corresponding to token in the input sentence. From the generator, I ...
user avatar
  • 17
1 vote
0 answers
23 views

Discrete wavelet transform - DWT (beginner)

I recently stumbled upon this article : https://www.bportugal.pt/sites/default/files/anexos/papers/wp201612_0.pdf In the paper they use DWT and I am having trouble understanding how to construct them. ...
user avatar
0 votes
0 answers
14 views

How to vectorize this loop process

Hi guys I want to ask if anyone knows how to vectorize this code to make it more optimal and faster. ...
user avatar
0 votes
0 answers
17 views

Can anyone tell me why is my pipeline wrong?

I am trying to build a pipeline in order to perform GridSearchCV to find the best parameters. I already split the data into train and validation and have the following code: ...
user avatar
0 votes
0 answers
22 views

Finding the similarity between two objects, each with multiple features

I have a dataset structured like the below, ID a b c d e f g c001 1 0 0 1 0 18 25 c002 0 1 1 1 0 44 98 c003 1 0 0 1 0 77 45 c004 0 1 1 0 1 39 67 c005 0 0 1 0 1 18 92 Now given a new object with ...
user avatar
0 votes
0 answers
29 views

Training using a dataset on NAS server

I have 2 questions: how would you approach the storage of datasets (with millions of small files) on local network? And how would you take that into account in pytorch code? Hello, I need to store ...
user avatar
0 votes
0 answers
32 views

Calculate days for each years between two dates in a pandas dataframe

I have a usecase wherein the from and to dates are provided. I need to calculate the total_days in each year and the total orders for each year. I'm currently coding in python Below is an example <...
user avatar
  • 1
0 votes
1 answer
48 views

Kmeans clustering in python - Giving original labels to predicted clusters

I have a dataset with 7 labels in the target variable. X = data.drop('target', axis=1) Y = data['target'] Y.unique() array(['Normal_Weight', 'Overweight_Level_I', '...
user avatar
0 votes
0 answers
23 views

How can i merge two datasets with similar words in python?

For instance i have a row value on the dataset_1 : "Entity" = Apple dataset_2 : "Entity" = iCloud Apple (Entity is the column) I need to merge one dataset to the other by the ...
user avatar
0 votes
1 answer
22 views

Why fourier transform extrapolation goes to extreme on edges but not in the middle, how to fix it

Why fourier transform extrapolation goes to extreme on edges but not in the middle, how to fix it with python ...
user avatar
0 votes
1 answer
10 views

python - How do i remove duplicates but keep row values in other column

i have a small df with: ID and full Name and i want to remove duplicates in ID but keep all names in new columns (kind of explode) so if i have 3 same ids, i would have new df with columns: ID, name, ...
user avatar
  • 1
0 votes
0 answers
16 views

how do i add a dropdown menu display GOES image on a tkinter popup window

im trying to add a dropdown menu on my tkinter popup window but when ever i run it on my visual studio code ide nothing displays but when i run the code by it self on jupyter everything work fine so ...
user avatar
0 votes
0 answers
7 views

the mean and standard deviation aren't the same as those of the input data i provided after sampling

I have a log-normal mean and a standard deviation. after i converted them to the underlying normal distribution's parameters mu and sigma, I sampled from the log-normal distribution however when i ...
user avatar