Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,667
questions
0
votes
0
answers
33
views
heavy underfitting of keras LSTM regression
I moved the question from stackoverflow to here.
I used keras LSTM to do the standard regression project of ...
- 127
0
votes
1
answer
27
views
calculate the predicted value based on coefficient and constant in python
i have the coefficients and the constant (alpha). i want to multiply and add the values together like this example. (it has to be done for 300000 rows)
Prediction = constant + (valOfRow1 * col1) + (-...
- 101
0
votes
0
answers
23
views
ROC curve for multiclassification - results sound not correct
I'm working on a multiclassification task using LSTM algorithm, i generated my roc curve plots but they give scores like 1 , 0.99, 0.97 however i have an accuracy of 0.97, Precision 0.65, Sensitivity/...
- 11
0
votes
0
answers
47
views
ValueError Found array with 0 sample shape= (0,2) while a minimum of 1 is required by KMeans
I am new to logistic regression and cam across this error. Please help.
import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
column_names =
["GENDER","...
0
votes
0
answers
111
views
0
votes
0
answers
7
views
What are the paprements needed and how should be filled using LIME with NLP in Python?
it is my first time using LIME and I have never used any interpretation technique before.
most likely I am doing something wrong but I cannot figure out what is it.
I tried googling and going through ...
- 539
0
votes
0
answers
25
views
How to get best result by using hyperparameter tuning and stacking regression?
I want to use regression ML models to get the best possible R2 score. So I decided to do both hyperparameter tuning and combining models with stacking regressor. I'm wondering if I should do ...
- 1
0
votes
0
answers
28
views
Memory / Segfault doing auto_arima
I am trying to do a day based SARIMA modelling using the pmdarima package like so:
...
- 101
1
vote
1
answer
141
views
Given historical states and action/reward data, is reinforcement learning a reasonable approach?
To summarize my problem:
I want to maximize my total reward over all timesteps
I have 3 discrete actions at each time step.
The state vector for each time step has 5 features. The features are ...
- 251
0
votes
1
answer
26
views
Is it possible to optimize ML models further and get better R2 score than the one suggested by lazypredict?
I'm using lazypredict in Jupyter Notebook. It automatically run different ML models and sort the R^2 score result of them in a form of table so that one can realize which models perform the best. I'm ...
- 1
0
votes
0
answers
75
views
Get graph execution error for BiLSTM and LSTM on keras
I want to use BiLSTM mode for text classification tasks. I use a data generator to get already batched and embedded files that have been split into 64 different files(for training) and 4 files each ...
- 1
1
vote
1
answer
48
views
causal inference vs sensitivity analysis to create acurate ML predictions?
I recently discovered the concept of "causal inference" and I'm trying to understand if I've grasped its essence correctly. If I understand it correctly, it involves finding what influences ...
- 195
0
votes
1
answer
76
views
Is there a way to automatically split large clusters that are greater than some maximum number of points?
I ran HDBSCAN on these coordinates and got some clusters but some are too large. HDBSCAN has a minimum cluster size parameter, but no maximum size. All I want is to intuitively divide larger clusters ...
0
votes
0
answers
10
views
Updating Predictive Model to Forecast from Today’s Date
I have a historical dataset with four numerical columns: c1, c2, c3, and c4. Column c4 is calculated and represents the number of days between the active date and the closed date. I have created a ...
- 1
2
votes
0
answers
44
views
LSTM output capped at a maximum
I am using a LSTM built using to forecast a single-value (solar irradiance) by using weather data as my input.
When predicting my validation test, I get a weird results as it looks like all my ...
- 273
0
votes
0
answers
40
views
Data Cleanup for Regression
I have a simple dataset of 1 output and 1 input and want to fit a linear regression to the dataset.
The data has a certain level of noise to it (potentially driven by another input, which I will ...
0
votes
0
answers
17
views
Why does the first call to a TensorFlow function execute much slower than the second call?
I was doing an Image Classification problem using TensorFlow. I was generating the mean images for two image datasets having the same size. The dataset was generated using the tf.data API. Thereafter ...
- 77
0
votes
0
answers
39
views
0
votes
0
answers
28
views
How to remove augmented data from test data
I am working with this dataset https://data.mendeley.com/datasets/hxsnvwty3r/1 for object classification model like CNN. In the description of the dataset, I see in the description there are "...
- 1
0
votes
0
answers
26
views
Multi input model tensorflow with fixed data as input
I am tring to implement the following architecture.
alpha and beta are fixed matrices, they are matrices I want to input on every forward pass. Meaning for every batch they should be the same My ...
0
votes
0
answers
13
views
Is it possible okay to use regression MLP for ordinal classification problem when target variable is numerical?
I have a target variable of 1-10 that represent difficulty level. These are individual classes represented by integers with 1 being the easiest and 10 most difficult. I have decided to use regression ...
0
votes
0
answers
21
views
Interpolation techniques predicting negative values in a column with positive values
I tried different interpolation techniques & for some numeric columns with only positive values the interpolation techniques: Polynomial, Cubic Spline & Radial Basis are predicting negative ...
- 275
0
votes
0
answers
11
views
clustering droplet data along trajectories
I have regular t,x,y,z-datapoints of droplets, that move continuously from top to bottom on different but similar paths. The droplets also have a volume that is approximately measured, they can appear ...
0
votes
1
answer
454
views
chainlit test failed
I am using windows OS. I created a folder named ABC under C. In command prompt, I typed cd C:\ABC and then typed the following commands:
...
0
votes
0
answers
18
views
Seeking guidance on data analysis techniques to estimate project LOE based on numeric counts of outputs produced
I’m looking for some basic guidance on where to focus my research in support of some data analysis I’m looking to perform.
The problem space is identifying a methodology for estimating Level of Effort ...
- 109
0
votes
0
answers
18
views
How does TensorFlow method tf.data.Dataset.reduce() work?
I was trying to compute the mean of the images that I fetched from a GCS bucket using TensorFlow's tf.data input pipeline. For this, I came across two methods: ...
- 77
0
votes
0
answers
25
views
how to find a repeated event in time series?
I need to find the repeated events in time series. if one event always has the similar value, similar description, and happens at similar date, then it is a repeated pattern. Let me make an example ...
- 207
0
votes
1
answer
23
views
I cannot run MNIST MWE (hello world for DL)
I have installed Anaconda and want tor run MWE for MNIST but I'me getting this error:
D:\STAZENE_last\Anaconda2\Lib\site-packages\torch\cuda\__init__.py:107:
...
- 101
0
votes
1
answer
364
views
Training embeddings on own dataset
In my project I follow the retrieval augmented generation (RAG) approach.
I want to create embeddings for my own dataset and use it in combination with llama-2.
In the dataset are german annual ...
- 101
0
votes
0
answers
29
views
How to train Keras Sequential model in batches?
newbie here. I have a large dataset in many local CSV files. It's impossible to fit it in memory within budget. How can I train a Sequential model in batches? I've found ...
- 105
0
votes
0
answers
18
views
How to detect a kick signal
Background:
I am measuring the reflection signal from the pipe. First strong kick is called firing kick. The later small bumps ...
- 111
0
votes
1
answer
44
views
How is the backward propagation is done in pytroch? When to use torch.no_grad, also when and where is the gradinte calcuated?
I have this training loop in pytorch. the loss_fn = nn.CrossEntropyLoss() and optim = torch.optim.Adam(net.parameters(), lr=lr)
<...
0
votes
1
answer
24
views
A model for recognizing suspicious user activity on a forum
I have a log of user actions on my website. Some users were blocked for spamming manually. This is also shown in the log.
Here is an example of the dataset:
...
- 13
1
vote
0
answers
14
views
Clustering Similar Articles Using Mixed Data: Seeking Advice and Validation
Question:
I'm working on a project where I need to cluster a dataset of articles based on various features, including text, numeric values, and categorical data. I've implemented a clustering approach ...
- 11
0
votes
0
answers
13
views
Macro Averaging vs. Samples Averaging in multilabel classification problems
I am currently working on a multilabel classification problem and I have developed some models to solve it using the SciKit-Learn framework.
I wish now to evaluated the models by producing scores for ...
- 11
1
vote
0
answers
98
views
How to split graph data into train and test sets for link prediction problem using node emebbdings and cosine similarity
I would like to predict new links using node embeddings and cosine similarity, but I am unsure how to split the data set into training and testing, and how to evaluate new links.
This is my code ...
0
votes
0
answers
20
views
Need help in Webscraping T20 World Cup 2022 data
I have a list of links of match scorecards and i have stored the respective html content of each link in a list. I need to get the csv files I am attaching as the output. Any help would be ...
0
votes
1
answer
24
views
Why most keras examples use shape=(None,) instead of shape=(None)?
I'm reading several keras tutorials and found that many examples are written like this:
keras.Input(shape=(None,), dtype="int64", name="english")...
- 185
0
votes
1
answer
57
views
Likelihood function of a Beta-Normal distribution
I'm going over a paper regarding the calculation of credit scoring By Kalkbrener and Onwunta. In the article, they derive the following Likelihood function, in order to find the MLE for $R^2$:
(...
0
votes
0
answers
34
views
Tensorflow model wont train when using validation data
Im trying to train a tensorflow neural network. Since the training data is too big for my computer RAM I have divided it into sub-datsets and sequentially use them for training. I have a problem ...
0
votes
0
answers
17
views
Detection of noise and signal components with deep learning
There are thousands of datasets from the signal in the bellow image. With these datasets and machine learning, I want to first detect the jumping points (blue in the image) which are noises, and after ...
- 1
0
votes
1
answer
65
views
Combine two separate models created via Transfer Learning?
Suppose have two 'image classification' models created by transfer learning on the same base model[1], each producing a different set of labels/classes. Trained at different times, with different ...
- 101
1
vote
1
answer
44
views
How to know the confidence of a classification on unlabeled data generated after model training?
I have created (in python) the code for a Random Forest classification model for a labeled dataset using sklearn. The model works very well.
...
0
votes
0
answers
31
views
What is numpy.dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m]) in numpy.dot() documentation?
I was trying to understand difference between np.dot and np.matmul and in the docs of np.dot() there is this code
dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m]). ...
- 1
0
votes
0
answers
10
views
How to solve MemoryError problem?
I have audio signals which I need to convert them to melspectrogram. I am using Deepfilternet to remove background noise. when I use the output of Deepfilternet for next phase like padding, it showed ...
- 1
0
votes
0
answers
21
views
Python Darts API
I'm using the darts API for some time series prediction in python, while looking at the documentation for the RNN model I saw this parameters called training length. The description is not very clear ...
1
vote
0
answers
7
views
Huge variance for RandomForestRegressor models
The experiment is the following:
train a RFR with a 15k train rows
get predictions on 8k test rows, save predictions as y_hat0
remove 1 random row from the training set and retrain the RFR
save ...
- 11
0
votes
0
answers
7
views
YOLOV4 model training
I can't start my model tutorial. I ran it correctly, I tried it before, but now it doesn't even start. why could it be? Did the update come?
0
votes
0
answers
22
views
Wavelet choice for wav file analysis
Context
I am working on the Maestro v3.0 dataset https://magenta.tensorflow.org/datasets/maestro#v300 to familiarize myself with signal processing. Years ago, I mainly worked on image processing with ...
- 101
1
vote
1
answer
51
views
Time series prediction on test set produces multiple cycles instead of following trend
I have a non-stationary time series where I am trying to build a model for forecasting. So far on test set it produces multiple cycles no matter which technique I use.
There's just one feature ...
