Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,667
questions
0
votes
0
answers
12
views
How to solve MemoryError problem?
I have audio signals which I need to convert them to melspectrogram. I am using Deepfilternet to remove background noise. when I use the output of Deepfilternet for next phase like padding, it showed ...
- 1
0
votes
0
answers
23
views
Python Darts API
I'm using the darts API for some time series prediction in python, while looking at the documentation for the RNN model I saw this parameters called training length. The description is not very clear ...
1
vote
0
answers
7
views
Huge variance for RandomForestRegressor models
The experiment is the following:
train a RFR with a 15k train rows
get predictions on 8k test rows, save predictions as y_hat0
remove 1 random row from the training set and retrain the RFR
save ...
- 11
0
votes
0
answers
7
views
YOLOV4 model training
I can't start my model tutorial. I ran it correctly, I tried it before, but now it doesn't even start. why could it be? Did the update come?
0
votes
0
answers
22
views
Wavelet choice for wav file analysis
Context
I am working on the Maestro v3.0 dataset https://magenta.tensorflow.org/datasets/maestro#v300 to familiarize myself with signal processing. Years ago, I mainly worked on image processing with ...
- 101
1
vote
1
answer
51
views
Time series prediction on test set produces multiple cycles instead of following trend
I have a non-stationary time series where I am trying to build a model for forecasting. So far on test set it produces multiple cycles no matter which technique I use.
There's just one feature ...
0
votes
1
answer
30
views
How can I solve this kind of problem about predicting the sequence of once in life events?
So let's imagine I have a dataset of children. For each of them a have a bunch of characteristics (generation, gender, race, class, urban/rural, religion, bmi, number of siblings etc..) and plus the ...
- 1
0
votes
3
answers
161
views
Clusterise a group of answers taking the question as relevant information
I am solving a problem where I group answers to a given question into clusters using k-means algorithm. The steps I follow are:
For every answer I get the corresponding vector. Reduce the vector ...
- 51
0
votes
0
answers
38
views
Perform k-means clustering over multiple columns using python but without library sklearn.kmeans
I'm on a project with the topic "clustering" using the KMeans method.
so I have a data set with 10 columns and 1000 rows divided into 5 clusters (in this case there are 5 robot movements). I'...
0
votes
1
answer
75
views
Workflow for reinforcement learning in Python
I'm new to reinforcement learning but (I believe) I have certain experience with machine learning. I know if you work on, say, a classification problem, your workflow will be (in a very high level) ...
0
votes
0
answers
29
views
Imputation of multi-peaked dataset
I am a beginner to model fitting, and I have been working on generating a model for a CO2 emissions data set. The distribution of the data points in a number of these columns are very markedly dual-...
0
votes
0
answers
423
views
Aggregate a DataFrame using multiple conditions in groupby
I have a dataframe with many duplicates. I have determined rules for duplicate identification, for example :
If street_name AND ...
- 829
0
votes
1
answer
39
views
9-dimensional data classification
I have a dataset with 9 columns and 20 million rows. I have now split the dataset into two parts using IQR.
One part is my inliers, the other part is my outliers.
I now want to build a categorizer ...
0
votes
1
answer
46
views
What data am I supposed to store into the "input_data" variable?
I'm supposed to pass preprocessed input to a TFLite model to make inferences. I have an excel file containing data and a tensorflowlite model file. I received instructions on how to do this; here is ...
- 1
0
votes
0
answers
46
views
Matrix Factorization with SVD/Gradient Descent - Dealing with new users
I'm working on implementing matrix factorization for a recommender system. I implemented matrix factorization, once with stochastic gradient descent and once with scipy's svds() function. The thing ...
- 1
0
votes
0
answers
24
views
Use ML.NET trained model in Python
How would i use a ML.NET trained (with VS ML Model Builder) model in Python?
It's a .zip file btw.
ZIP file structure:
...
0
votes
1
answer
89
views
`ValueError: There must be exactly two hue levels to use `split`.', while two levels?
The following plotting issue occurs in preparation for a model build:
...
- 5
0
votes
1
answer
53
views
Are there any R packages that support Deep RNNs?
I recently found an interesting paper on what it really means for a recurrent neural network (RNN) to be deep here. Depth can be added in several different ways (state to state, input to state, etc.) ...
- 41
0
votes
0
answers
15
views
How do I reconstruct irregularly spaced lat-lon data (in the form of a matrix) using spherical harmonic fitting?
Introductory Links to spherical harmonics give a mildly rigorous introduction to the concept; but very few links describe how to actually use them. I have latitude-longitude data arranged the ...
- 11
0
votes
2
answers
44
views
Data Science Project Data Workflow Structure
I'm in the middle of a project of marketing regarding the sales prediction with promotions. The client has very complex business processes and so the data needs a lot of preprocessing (joins, filters, ...
- 63
-1
votes
1
answer
88
views
Falcon 7B LLM Evaluation using TruLens
The problem I am facing is after defining the prompt template, creating a chain using Langchain, defining the huggingface evaluation module from trulens_eval to check the toxicity of the response and ...
0
votes
0
answers
49
views
Correlation Matrix for Very Large Data Set
I am tasked to create a correlation matrix on a large portion of an MRI imaging data set (all numeric entries). The dimensions of the data is around 28,000x1,800. I have already set a correlation ...
- 101
0
votes
0
answers
27
views
Multivariate Time Series Forecasting of only 1 of the variables
I have a dataset that has time series data in the following format -
1 column for the date,
5 dependent variables,
1 independent variable (need to predict)
I want to see if the dependent variables ...
- 1
0
votes
0
answers
22
views
Can somebody explain to me, how the Hyperparameters in this expamle work? I only uderstood the meaning for gamma
I am quite new to this topic, but I want to understand how Reinforcement Learning (RL) works in this example (https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/). I have ...
- 1
0
votes
1
answer
38
views
Stick to R or switch to Python as a newby in Reinforcement Learning?
Background
I would consider myself to have a decent level in R and virtually 0 skills in any other programming language - something quite common in my field. Note also that my background is not in ...
- 3
0
votes
0
answers
34
views
How to solve error with 2D labels in partial fit regression problems?
I would like to use continual learning (incremental learning) in my code. The dataset is composed of cartesian coordinates in which X and y is the same. I've solved the fact of using a 2D in the y ...
0
votes
0
answers
26
views
Why do I get such a low accuracy despite having good metrics?
I'm working on an image segmentation problem. I'm fine-tuning the network, and when displaying the metrics I obtain, I have some doubts.
I am going to provide some details about the network:
Solver: ...
- 304
0
votes
2
answers
69
views
How to apply supervised machine learning when the target variable depends on multiple rows (with varying number of rows)?
Here is a example dataset that resembles the one I am using.
Let's call one row = one experiment.
I'm trying to predict Y based on features X1 through X7
The problem I am facing is very similar to ...
- 1
1
vote
1
answer
206
views
What is the most straightforward way to run python code in a chrome extension?
I am currently trying to develop a chrome extension for a project. My knowledge of Javascript is quite limited and this is my first using it, so I am trying to reduce as much as possible its use. The ...
- 11
0
votes
0
answers
47
views
Build a model with cross-validation on entire dataset to learn insights?
Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed.
Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
1
vote
1
answer
30
views
Patterns binary classification - model doesn't overfit
I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.
I have written a model in Keras with ...
- 121
0
votes
0
answers
55
views
Implicit feedback based recommendation - User - User recommendation
My dataset involves customer_id, item_id and Count of purchases. My datset size is small...we have 15 users and 317 items..
Currently, am trying to build a recommendation system based on user based ...
- 2,555
0
votes
1
answer
27
views
How should I approach generating recommendations for fish stock management?
I've been asked to "Model and study the size and age composition of fish populations as dynamical systems for a given area and generate recommendations for seasonal fish stock management based on ...
0
votes
0
answers
43
views
Mean Average Precision with 11 points interpolation method Python libs
I want to calculate mAP with 11 points interpolation method for object detection, as described here:
https://learnopencv.com/mean-average-precision-map-object-detection-model-evaluation-metric/
What ...
- 61
1
vote
0
answers
25
views
What approaches can I take to reconstruct missing climate data?
I have climate datasets consisting of tables where each row has four elements: latitude, longitude, rainfall, and timestamp. This dataset covers the entirety of the planet. However this dataset is ...
- 11
0
votes
0
answers
19
views
How to feed patient data to tflite file model?
So I have the patient data I need in an excel file, and I need to feed that data to a tflite file. Could someone point me in the right direction to do this using python, please (preferably a code ...
- 1
0
votes
2
answers
93
views
Should I call inverse_transform?
I built a K-Nearest Neighbors Regressor GridSearchCV which calls a sklearn pipeline as the estimator.
Training appears to be working fine. My confusion is after calling predict and checking the RMSE ...
- 3
0
votes
0
answers
25
views
How to check for correlations using ANOVA between the selected feature columns and the target feature column
Suppose I have a set of feature columns that I select to check for correlations with the target feature 'is_Star'; which is of binary values and the former consists of numeric values
Why do i end up ...
- 1
0
votes
1
answer
27
views
Do prediction intervals for Random Forest predictions are average of prediction intervals of trees being Random Forest estimators?
I am working on adding prediction intervals for each prediction value of new input samples.
Do prediction intervals for Random Forest predictions are average of prediction intervals of trees being ...
- 15
0
votes
3
answers
88
views
Heatmap not respecting the color bounds
For the following dataframe of a chi2 correlation study, i started to plot a heatmap:
...
0
votes
1
answer
96
views
Splitting CSV data with missing target variable using train_test_split in Python
I have a dataset in CSV format that consists of a training dataset with around 300 instances and a test dataset with around 100 instances. The issue is that the target variable (the column we want to ...
0
votes
0
answers
16
views
Error in seq2seq translation when passing predicted output to rnn due to input shape not always being the same
I'm working on a language translator and I'm getting an error I'm unsure about.
During the decoding process when using argmax on the predicted output I am sometimes getting an RuntimeError ...
0
votes
0
answers
24
views
How to build a NN for classifying custom timeseries data?
I'm trying to learn a little bit of machine learning and it occurred to me that my own household produces a lot of interesting data. Specifically, the ModBus-enabled digital power meter installed a ...
- 125
0
votes
0
answers
11
views
Model outputting repeated result in Flask back-end
I am trying to implement this working audio classification model into a Flask app for use on a website. However, anytime I upload an audio file, even one that was used on the test/training of the ...
0
votes
1
answer
28
views
How can I find what kind of args and kargs a pandas function takes?
I was wondering how do I find what are valid kargs and args for a given function? For example in pandas there is this function cumin DataFrame.cummin(axis=None, skipna=True, *args, **kwargs); how do I ...
- 111
0
votes
1
answer
49
views
Training and validation loss sometimes not decreasing in Keras dense layer with the same data and random seed
I want to use a Multi-Layer-Perceptron in Keras (Dense layer) to map 6 inputs to 1 output. For doing this I use the following code:
...
- 81
0
votes
1
answer
96
views
How can I compare the accuracy of imputation models if there is already missing dataset in the file?
Let's say I have a dataset of 50,000 where about 2% were already missing from the beginning. From what I have learned, we need to use indicators to compare the imputation model with the ground truth ...
1
vote
0
answers
49
views
Highlight specific paragraphs from documents
I have a bunch of documents in which I want to highlight certain paragraphs/keyphrases. I have a list of the most frequently appearing sentences and I want to search for these paragraphs/keyphrases in ...
- 1,896
0
votes
0
answers
22
views
Should I remove noise by fitting a curve in my case?
I have an object that is moving towards target point and I want to determine when it will reach or pass through it. I receive points in real time, which represent the current location of the object.
...
- 141
0
votes
0
answers
24
views
s3 object as corpus_file for gensim Doc2Vec
Is it possible to use a txt or jsonl file in an s3 bucket as the corpus_file input for a gensim Doc2Vec model? I am looking for something of the form:
...
- 101