Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

python kfda - methods fit and score

I'm struggling with my project, based on kfda package. The Code is here. ...
1
vote
2answers
16 views

How to properly use oversampling without inflating results?

I am using with a tiny private dataset (over 192 samples) with 4 classes. A preprocessing step is trivial in order to do any classification. Among feature selection and extraction techniques, i ...
0
votes
1answer
11 views

Can I cluster an aggregated data-set (grouped by) and apply dimensionality reduction?

I have data of sales, however it is in the millions, about 500M rows. I aggregate this data by factors such as location, shoptype, country_of_shop, cardtype, and then the aggregated statistic is: ...
-2
votes
1answer
11 views

What is this PIP syntax: “azureml-dataset-runtime[pandas,fuse]” [closed]

I came recently across this environment file: ...
0
votes
1answer
16 views

Are there any tree-based models that use a genetic algorithm to generate the trees?

I have a large dataset (195 features x 20m samples) that I have trained using XGBoost. I would like to see if a genetic algorithm can beat XGBoost since the data has so much noise it is prone to ...
2
votes
0answers
24 views

Storing multiple time series in a python pandas dataframe efficiently

I got different time series. Each time series contains different attributes, which have data entries at different points of time (timestamps). Each time series is labeled and has a label attribute. I ...
0
votes
1answer
37 views

Image classification with CNN Python

I'm working on image classification using CNN, my dataset contains more than 50 classes (50 folders) which represent the types of car parts, and in each folder we have vehicle brands, each vehicle ...
0
votes
0answers
10 views

aggregating multi output regression outputs vs single output (=top down) approach: is it worth it?

basically I have this problem where I need to forecast the sales of some stores. such stores have multiple product lines of which I have the split data (say Y1,Y2,Y3 where Y1+Y2+Y3=Y). I have also ...
1
vote
1answer
37 views

How to plot multiple bar charts with different infill and outline?

I have the CSV's of data. I want to plot the distribution of every attribute or a single one to compare the distribution from other CSV's, Since all attributes are the same but the distribution is ...
0
votes
1answer
47 views

How to plot the bar charts of precision , recall and f-measure?

I have used 4 machine learning models on a task and now I am struggling to plot their bar charts just like shown below in the image . I am printing classification report to get precision , recall etc ....
0
votes
1answer
33 views

How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
0
votes
0answers
25 views

2D Z-score/Mahalanobis distance that includes a penalty for uncertainty

I have some 2D points and I want to assess their performance against the target point. When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
1
vote
1answer
17 views

Incorrect visualisation using Plotly

...
0
votes
0answers
24 views

Save and Load Simple Transformer Model

I have trained Text classifier using simpleTranformer.ai I am struggling to save and load the model in docker container. Please let me know how can I save the trained model and then load it into ...
0
votes
2answers
17 views

Warning when plotting confusion matrix with all sample of one class

I have two arrays: the first one with all the correct labels (they are all set to zero since each sample belong to the same class) and another one with all the labels predicted by my neural network. ...
0
votes
0answers
11 views

Visualizing post-hoc results of many days

I'm looking for ideas how to plot many Tukey HSD results. My origianl dataframe has 3 treatments. The values of each patient were measured every day for one month. One-way ANOVA has shown that there ...
0
votes
1answer
29 views

Why my classification accuracy is high for both training and testing data?

I have a dataset with 10 features and 1 binary classification target. I tested this dataset with decision tree classifier. I did some basic check like missing values but the data looks clean. My ...
0
votes
0answers
10 views

How can I intuitively calculate the accuracy of my financial prediction model?

I've built a SARIMAX model based on my personal spendings record as a college thesis and have reached a point where I'm pretty content with how it turned out and am getting ready to start writing the ...
1
vote
3answers
34 views

Are there ML Libs in Python robust to missing data?

So I was searching on how to handle missing data and came across this post from Machine Learning Mastery. This article states that some algorithms can be made robust to missing data, such as Naive ...
1
vote
0answers
26 views

Why don't we set initial hidden states in RNN to random small numbers like we do to the weights?

I'm following a couple of tutorials on RNNs and the instructor said that we should always set the initial hidden state in our RNNs to a tensor of all zeros and I couldn't really understand why. Even ...
0
votes
0answers
15 views

Balancing on the particular imbalanced classes of image dataset

I have a dataset that has 12 classes in the base directory. However, these 12 classes consist of several amounts of Images. The number of images of 12 classes is inconsistent therefore its impacts the ...
0
votes
0answers
15 views

How can i interpret cluster attributes from clusters in hierarchal clustering?

I have data with numerical columns for products bought e.g. price, etc, many are numerical but it is also crucial i incude the industry classification of product e.g/ fashion, gardenware etc. If i use ...
1
vote
0answers
16 views

How to calculate MAE and threshold in a multivariate time series

I'm trying to understand how to calculate the MAE in my time series and then the thresholds to understand which of my data in the test set are anomalies. I'm following this tutorial, which is based on ...
0
votes
1answer
12 views

How to build a comunication interface between a reinforcement learning and a videogame?

I want to try to build a reinforcement learning model in order to play old arcade videogame, I know about projects in order to build AI for playing videogames, but I don't know how could I build an ...
0
votes
0answers
7 views

How can i get all the values related to the first column in a direct or indirect relationship?

I wanna write a function using python that allows me to compare each value from the second column of my DataFrame with the first column and if they don't matche compare the second column in the same ...
0
votes
0answers
15 views

Weighted accuracy, sensitivity and specificity

I have a confusion matrix TN= 27 FP=20 FN =11 TP=6 I want to calculate the weighted average for accuracy, sensitivity and specificity. I know the equation but unsure how to do the weighted averages.
0
votes
0answers
7 views

How to decide the right granularity in time series?

I am wondering how to decide right granularity I should use for time series if I have data for hourly as well as daily and given that I don't have business constraint which asks me to do forecasting ...
0
votes
0answers
6 views

loss is neither increasing nor decreasing in siamese network

I am making a model which differentiate between two fingerprint(dataset) using simple siamese network but even after 400 epochs loss doesn't decrease. Loss is stuck at 6000 and accuracy is also not ...
0
votes
2answers
31 views

How to group every data point with HDBSCAN to some group to have no noise?

TASK I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits) I use HDBSCAN to do it GOAL The goal is when users come on our site and I can ...
-2
votes
1answer
14 views

Apache Beam - Python Inquiry [closed]

I have created a python pipeline to be run in Dataflow that takes a column from a csv file and counts the number of listings and returns them to a dictionary key value pair. I am very new to Apache ...
0
votes
0answers
21 views

Error in Entity-Embedding of Categorical Variables

Code versions: python = 3.8.0 tensorflow = 2.2.0 scikitlearn = 0.23.2 I'm training a neural network on an all categorical dataset. Because of the sparsity caused ...
-1
votes
0answers
12 views

How to figure out how variables affect undulating sequences in a plot?

Let's say I have a file named accepted_D.dat that looks like this. This file gives 7247 and variables that describe them: header: "model ID", "Z"...
0
votes
1answer
19 views
0
votes
0answers
10 views

speech emotion recongtion cnn or rnn?

I want to build a speech emotion classifier and I labeled my data into 3 emotions {negative, neutral, positive} the speech files I have are different of length, thus my audio features (mfcc,zcr, etc.) ...
0
votes
1answer
16 views

Neural network for a data with 5 inputs and 1 output of 3 different types

I am trying to create a neural network for my data which is as follows ...
1
vote
1answer
12 views

how to get the number of columns that have the same value between 2 rows [closed]

I want to compare two records and get the number of columns that have the same value , for example I have two rows, the first row is : rank rank1 rank2 1 1 3 and the second row is rank rank1 rank2 ...
1
vote
1answer
15 views
0
votes
1answer
13 views

Organizing a csv file of multiple datasets into a list of Pandas dataframes

I have a csv file, containing results from a Computational Fluid Dynamics (CFD) simulation (a sample of my csv file is attached as a google drive link; file size: 1,392KB). In particular, the csv file ...
0
votes
0answers
18 views

How to repeat classification experiment with different set of data by loop

I want to repeat my experiment pipeline to validate my feature extraction. My goal is with different set of data which randomly picked as train test and validation, I want to check if the result are ...
1
vote
0answers
27 views

How do you perform basic statistical analysis on a Binary Dataset?

I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of statistical tests I should run to draw up simple conclusions. I'm doing a data science project and the ...
1
vote
0answers
18 views

Pre-processing packet data [closed]

I am confused about how to pre-processed packet data. Does anybody have an idea of how to encode network ports and IP addresses? My goal is to model the data in a such way that I will get a good ...
0
votes
0answers
9 views

Using Wasserstein loss function for image-to-image-regression

The context I have a 3D array (representing a grayscale 3D image) and want to turn this into another 3D array of the same size. In this output array the value of each pixel is a number that measures ...
0
votes
1answer
24 views

Transitioning from a python script for data transformation to BigQuery

So I have a dataset spread over multiple and ever-growing excel files all of which looks like: email order_ID order_date user1@email.com 1234 23-Mar-2021 user2@email.com 1235 23-Mar-2021 user2@...
0
votes
0answers
1 view

Machine learning model to predict performance degradation over period of time

I am new in ML space and working on the task to build a model that would predict performance degradation of integration bus in real time or "almost real" time. The idea is fairly ...
0
votes
1answer
18 views

How to perform tokenization for tweets in xlnet?

X_train has only one column that contains all tweets. ...
0
votes
0answers
8 views

Is it mandatory to have a DateTime index for a pandas dataframe to run ARIMA models on it?

I'm trying to learn about Arima models. In every tutorial, the Dataframe considered has a DateTime index. When I looked upon the mathematical formula for Arima model, it only depends on the past ...
0
votes
0answers
27 views

How to Determine Alpha values in Random Forest Classifier

I am working on a classification problem where I am applying various machine learning models. I have used DecisionTreeClassifier from Sklearn on my dataset using the following steps: Calculated alpha ...
1
vote
1answer
37 views

Eigenvalues of covariance matrix are negative

I'm working on the PCA of the mnist dataset, and I get a very strange result, I created a matrix whose rows are flattened mnist images, When I try to compute the eigenvalues of the covariance matrix, ...
0
votes
0answers
23 views

How to programmatically differentiate between extractive and abstractive summarization in NLP?

We are using different pre-trained models in python transformers library, to generate summaries(both extractive and abstractive). So is there a programmatic way, based on the output summary, we can ...