Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,591
questions
0
votes
0
answers
28
views
Forecast n-steps into the future with LightGBM
I have learnt to create the LightGBM models with multiple features and target, but in ALL the examples I have seen thus far they only show how to do a "predict" on the x_test_df. What I am ...
- 101
0
votes
0
answers
21
views
What does KDE plot tell to me?
What the KDE plot tells to me? How can I evaluate if my model is good by looking at the graph? For example I have this KDE plot of the residuals(it's x_pred-y_pred) of a machine learning evaluation of ...
- 41
1
vote
1
answer
187
views
How can I write a Python program, that recognizes clapping sounds in an audio file, and saves them to disk?
I would like to write a Python program that takes an audio file as input, recognizes clapping sounds in it, then extracts these sounds into a file.
My idea is the following:
Get (or create) a dataset ...
- 113
0
votes
1
answer
19
views
How to correctly compute the correlation index of a column value from table in Python 3?
I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
0
votes
0
answers
43
views
Scikit-learn and TensorFlow with very different MLP models
I'm using Multilayer Perceptron ANNs at the very beginning of my project (it's a binary classification problem). Because it's simpler, I started with Scikit-learn. I got a magic result, with my model ...
- 101
0
votes
0
answers
11
views
Distribution check - different results using plt.hist and seaborn.displot
I have an apparently very simple task: checking the distribution of values contained in an array data. Only thing is, I get two completely different results by ...
- 101
0
votes
2
answers
28
views
What ML Algo should I use to plot & predict customer location vs. amount spent?
Im using real customer csv files from my ecommerce store. I was initially going to use k cluster with 2 values. I want the two values to be Province Code, and amount spend on my store. This will ...
0
votes
1
answer
33
views
What can be the reasons for 95% of samples belong to one cluster when there is 5 clusters?
'''I used the k-means algorithm to clustering set of documents which are textual data only.
The document has 2lack records.
Surprisingly the result for the clustering is
90% of records is storing in 1 ...
- 1
1
vote
1
answer
50
views
Best image format for face detection and face recognition with the DeepFace library
I'm using the DeepFace library for face recognition and detection.
I was wondering if there is a better format (png, jpg, etc) than others to get better results.
Is there a preferred image format for ...
- 113
0
votes
1
answer
22
views
Quasi complete separation problem
I have some question related to quasi complete seperation problem on logistic regression algorithm.
So i run the model to predict credit risk and turns out it gave me good prediction score (AUC around ...
- 138
1
vote
2
answers
73
views
How to interpret differences between 2D and 3D T-SNE visualization of similar words from Word2Vec embedding?
I have created a Word2Vec model based on the transcript of the Office. I am now trying to visualize the embedding space for the top similar words of an input word with t-SNE in 2D and 3D. I ...
- 35
6
votes
1
answer
238
views
Logistic Regression Modeling & Interpretation [closed]
I'm building a logistic regression model to predict the credit risk of lending company customers.
I'm using dataset from kaggle : https://www.kaggle.com/datasets/ranadeep/credit-risk-dataset/code
...
- 138
2
votes
0
answers
170
views
Adding punctuation for a long text
I want to add punctuation to a long text (youtube transcript) before using a Transformer pipeline for summarization.
I have found this answer here:
original answer
thus I have tried:
...
- 21
2
votes
2
answers
375
views
How to arrange multiple multivariate time series of different length before passing it to Keras LSTM layer
I have a number of multivariate time series that are produced by the same kind of process but:
are of significantly different lengths;
each time series is an independent instance, and the ...
- 182
0
votes
0
answers
24
views
How to improve the AUC scores of my link prediction algorithm?
I am exploring a link prediction algorithm, but my AUC scores change every time I run it and it is always very low. It varies from 50 to around 72. I was wondering if there is a way to improve their ...
0
votes
1
answer
39
views
How do I interpret results from scipy pdf() function in concerns with the probability density function
I am working on a personal project and want to understand what the output of Scipy's pdf() function. I understand that the PDF is for one singular line under a distribution curve so finding a point ...
0
votes
0
answers
10
views
Transfer learning (or fine-tuning) pre-trained model on non-text data (PyTorch)
I am currently fine-tuning a sentiment analysis bert-based model using PyTorch Trainer from hugging face. So far, so good.
I have easily managed to fine-tune the model on my text data. However, I'd ...
- 101
0
votes
0
answers
15
views
How to find a unique feature vector for retail items?
I'm working on this problem where we get images of SKUs (Stock Keeping Unit) or in other words retail products and my job is to classify which product is it.
I want to find a unique feature vector ...
- 101
1
vote
2
answers
60
views
How to vectorize and speed-up double for-loop for pandas dataframe when doing text similarity scoring
I have the following dataframe:
...
- 306
0
votes
0
answers
14
views
GluonTS with small datasets
I am working with a small dataset which is approx. 60 observations. Is it possible to build models from GluonTS which provides probabilistic forecasting also which makes forecasting more reliable but ...
0
votes
1
answer
38
views
SARIMAX model for predictiong the next 100 days
I have daily weather data for almost 50 years for a weather station, and I want to predict the next 100 days' weather. I use python and all of the tools I've used so far (pmdarima.auto_arima, ...
- 1
0
votes
1
answer
36
views
numpy power function output for 2 to the power 63
I am running some test code and observed something strange. the numpy power function giving strange output post 2 to the power 62
...
- 111
0
votes
1
answer
284
views
numpy.ndarray' object has no attribute 'batch' [closed]
I am building an image classifier using cnn using the following code-
I am getting this error, despite making several changes-
Kindly help me fix this error.
0
votes
0
answers
30
views
SNS Clustermap with correlation with target value
I am trying to replicate a graph similar to this.
https://link.springer.com/article/10.1186/s12864-017-4353-7/figures/1
I am trying to identify if combination of 2 features (30 features, 400000 rows/...
- 101
1
vote
1
answer
200
views
Grid_search (RandomizedSearchCV) extremely slow with SVM (SVC)
I'm testing hyperparameters for an SVM, however, when I resort to Gridsearch or RandomizedSearchCV, I haven't been able to get a resolution, because the processing time is exceeding hours.
My dataset ...
1
vote
0
answers
113
views
How do I build a multi-input and multi-output neural network?
I want to build a neural network classifier which has as inputs array of dimension (24, 2, 20001) and as outputs array of dimension ...
0
votes
0
answers
14
views
Why is PACF/ACF graphs not working?
Below I have tried to graph PACF and ACF for my time series data.
However, for some reason the graphs do not properly show the critical blue region that would help me try to choose what p and q values ...
- 11
1
vote
1
answer
2k
views
0
votes
1
answer
82
views
Reading a model stored as binary
I'm using a s3 bucket to store a model I trained in python. Since I'm using an s3 bucket I convert the file to binary first and then store it on the bucket.
...
0
votes
0
answers
28
views
What to use Multi-level Classification for Machine Learning predicting target Y1 and Y2
I have the data set where different signals are there. where it may be single or double or in groups so there are two outcomes where one outcome is dependent on the other.
...
0
votes
1
answer
26
views
detecting abnormality in a specific feature with respect to others (unsupervised?)
I have a large dataset with a feature y which is dependent in part on features x1 and x2. All features are noisy, and y is also dependent on other parameters not captured in the dataset. I would like ...
0
votes
1
answer
12
views
How to demonstrate two variables are orthogonal with respect to the output in a 3-D Python dataset?
I have a Python dataset with 300 samples and 3 columns: 2 independent integer variables X,Y and the dependent continuous variable ...
0
votes
1
answer
19
views
Are there any error functions with imbalanced negative/positive impact
I have a regression task, where positive error should be much worse than negative one. It means the importance of positive error bigger. For example, If real value is less than predicted one weights ...
- 23
0
votes
1
answer
59
views
What is the math behind the Keras Tokenizer() function?
I am doing an essay on the mathematics behind a text classifier with NLP and neural networks and I would like to know how exactly the TOKENIZER function of Keras works. Whether cosine similarity is ...
- 3
0
votes
0
answers
196
views
Problems with implementing LIME
Hope you are holding on tight, Christmas soon!
**Updates:
I'm only able to get the results in probability but I want it in pure binary (0 or 1). I tried using .predict instead of predict_proba(x_test) ...
- 1
0
votes
0
answers
33
views
How to use my resampled data (through SMOTE) for my models?
I'm quite new to Python coding so maybe this is an obvious thing.
I am dealing with an imbalanced data set. I have encoded the categorical features and transformed them into numpy.array. After this, I ...
- 1
1
vote
1
answer
43
views
How to find correlation between features and labels when features are unordered words?
I have a data set with ~ 85 labels and a target (house price)
I wish to explore the correlation between the labels and the effect it has on the target.
I have some features that number and others that ...
- 11
0
votes
0
answers
15
views
how to get the extracted feature vector from transfer learning models python?
I am trying to implement a classification model with ResNet50. I know, CNN or transfer learning models extract the features themselves. How can I get the extracted feature vector from the image ...
- 172
0
votes
0
answers
499
views
Yolo confusion matrix - Background Class
I am training the yoloV5 model, where, I have two class[object = A and B].
In the dataset, the class B has more samples than class ...
0
votes
1
answer
14
views
Generating a numpy array using information from two other arrays
I have the two following numpy arrays:
initial_array = np.array ([2, -10, 3, 5, 4])
indicator array = np.array([1, 3, 2, 3])
I want to generate a new numpy array ...
- 1
0
votes
0
answers
33
views
In ensembles combining models, does it make sense for a model to have negative weight?
I have 13 models ranging from simple models like Seasonal Naïve Average to complex models like Random Forests, The weights of the models is calculated based on the LPMinimize of the error during the ...
- 103
0
votes
0
answers
16
views
What is the best way to acess the data in a pandas Window object?
I have a pandas DataFrame object, and I call its rolling method to get a Window object. Now I want to modify the data in each of the windows to get a new set of windows.
Reading the source code, I ...
0
votes
0
answers
38
views
How to extract all timestamps of badminton shot sound in an audio clip using Neural Networks?
I am trying to find the instances in a source audio file taken from a badminton match where a shot was hit by either of the players. For the same purpose, I have marked the timestamps with positive (...
- 141
0
votes
0
answers
9
views
I'm having issues with my python program. It's supposed to read a sheet of an excel file and predict the missing values on another sheet. I get error
Error:Found input variables with inconsistent numbers of samples: [104, 52]
...
- 1
1
vote
1
answer
27
views
The Sklearn train_test_split function is create training data and test data which are not similar
I am working on loan default data and my model is not able to make accurate predictions on the test set because the the default percentage on the test set is very different from that of the training ...
- 13
0
votes
0
answers
17
views
Is it correct to generate similar rows by reducing the time-frame of an instance?
I'm participating in a People Analytics project with a small historic dataset that includes event variables. The aim is to predict employee's attrition.
I have variables like area, dept., company, etc....
- 61
0
votes
0
answers
49
views
Can I do prompt-learning on HuggingFace Transformers?
I'm trying to solve a Prompt Learning task, but I'm not finding information on how to do it with HuggingFace Transformers. However, I've been thinking about it, and doing prompt-learning is basically ...
0
votes
0
answers
15
views
Deep learning models for time series forecasting changing the input of the experiment
For my project I received the following pandas dataframe which contains the dissipated heat from an experiment. Each column represents the power provided to the experiment. The distance between each ...
- 1
0
votes
1
answer
34
views
General question regarding n_components in UMAP and the resulting plot
Lets say we define n_components in UMAP to be 5 (or really any number greater than 2). When we plot the result in 2D, what features are being displayed? Does UMAP identify the features with the most ...
0
votes
1
answer
27
views
