Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
0 votes
1 answer
20 views

Predict coordinates from input of coordinates

I'm a newbie at data science and I want to ask how can I predict a set of coordinates from a set of input coordinates? That is (x1, y1) -> (x2, y2). To give a ...
1 vote
1 answer
36 views

Interpreting Learning Curves of models

I need some help to understand if the models are overfitting and which of these we can consider "the best". On the internet i only find simple examples with learning curves but in these ...
  • 41
0 votes
1 answer
21 views

How can I take the cross-correlation between two discrete signal with slightly different discrete time points?

I have two discrete signals x1 and x2 with corresponding time points t1 and ...
0 votes
0 answers
29 views

How to dynamically add columns in Dataframe Spark by using a config.ini file parameters

I am learning to use Pyspark and at work I have been assigned a task that I can not solve for now. There is a config.ini file where there are a series of ...
2 votes
3 answers
67 views

99% accuracy in train and 96% in test is too much overfitting?

I have a binary classification problem, the classes are quite balanced (57%-43%), with a GridSearch with Random Forest Classifier I obtained the best hyperparameters and I applied the model to train ...
  • 41
0 votes
0 answers
18 views

pytesseract performance for digit recognition

The sequence in the following image is correctly recognized by pytesseract via the usual function "pytesseract.image_to_string". But the following image with a similar sequence is not ...
0 votes
0 answers
22 views

Fuzzy Inference System IF-THEN Rules

I am currently doing a fuzzy inference system project in Python, however, a question came up. I have to do it the hard way, that is, do it step by step, as done here: https://pythonhosted.org/scikit-...
  • 1
0 votes
0 answers
12 views

How to tokenize custom dataset?

I have generated a dataset of quadratic equations and I want to train T5 model to solve them. Dataset looks like this: Could somebody please give a hint on how to tokenize this correctly? What to do ...
1 vote
2 answers
48 views

Are Chi-square and ANOVA (f_classif) to select best features?

I have a binary classification problem (target 0 o 1), I have both variables continuous and categorical as features. I understood that about Chi-square i can use only categorical features to evaluate ...
  • 41
0 votes
1 answer
34 views

NLP Question: Where can I find a list of all open compound words (with spaces in them), like "peanut butter" or "high school"? [close

I already have a list of "1-gram" words, which include closed compound words like "skyscraper" or "weatherman." However, I'm also interested in compiling a list of "...
  • 3
0 votes
1 answer
29 views

Trying to fill null values with sub-grouped mean value using pandas fillna() and groupby().transform() is doing nothing with the null values

I'm doing EDA on the US police shooting dataset where I'm stuck with filling the 482 null values for the age column. Since no strategy was mentioned on how to fill ...
  • 101
0 votes
0 answers
11 views

How to create synthetic data with custom feature values?

Currently I create a synthetic dataset using the python sdv package and an existing dataset as below: ...
  • 21
0 votes
1 answer
33 views

How to translate text automatically using Google Translate API (or any other approach) in python

I have a dataset of reviews from TripAdvisor and I would like to translate non-English reviews into English. The reviews are in many different languages: my dataset contains reviews in 44 different ...
0 votes
0 answers
24 views

Machine Learning | Learning and ROC Curves interpretation

Not a problem but rather a matter of understanding, my goal is to evaluate which is the best classification model for my dataset among k-nearest neighbors, naive bayes classifier and decision trees, ...
0 votes
0 answers
5 views

Identifying and removing repeated text from json files post inference

Good morning: We index and summarize medical records using a proprietary algorithm. Being medical records they repeat the same information almost on every single page, like facility name, medical ...
0 votes
0 answers
7 views

sequential rule mining

What are the available libraries today for this problem in python I have a schedule plan based on the location with a time series and in a different order I want to predict the schedule plan giving ...
  • 1
0 votes
0 answers
27 views

extract ORC file to csv using python

I am extracting large ORC files (58G) into csv files in python jupyter. I could not generate csv file after run this command. ...
  • 1
1 vote
0 answers
46 views

How to convert an ONNX model with dynamic input shape to Tensorflow model

I am trying to convert an ONNX model with a dynamic input shape to TensorFlow format using the onnx_tf package. I am using TensorFlow 2.11.0, ONNX 1.13.0, and onnx_tf 1.10.0. The input to the model ...
0 votes
1 answer
26 views

How do I fill the null ages of a person according to the average age of males or females in a particular race

I got this US police shootings dataset where many age are null. Each person's race and gender...
  • 101
0 votes
1 answer
26 views

Creating new features as linear combination of others as part of a scikit-learn pipeline?

I have a number of raw features that go into a scikit-learn model. I've already got a number of preprocessing steps (such as PolynomialFeatures) that creates additional features as part of my pipeline....
0 votes
1 answer
22 views

Differences in OvO SVC decision_function and predict methods in Scikit learn package

I have observed that there are differences in the implementation of the decision_function and the predict methods the one versus one multi class implementation of SVC in the sci-kit learn package. Is ...
0 votes
0 answers
19 views

Forecast n-steps into the future with LightGBM

I have learnt to create the LightGBM models with multiple features and target, but in ALL the examples I have seen thus far they only show how to do a "predict" on the x_test_df. What I am ...
  • 101
0 votes
0 answers
17 views

What does KDE plot tell to me?

What the KDE plot tells to me? How can I evaluate if my model is good by looking at the graph? For example I have this KDE plot of the residuals(it's x_pred-y_pred) of a machine learning evaluation of ...
  • 41
1 vote
1 answer
61 views

How can I write a Python program, that recognizes clapping sounds in an audio file, and saves them to disk?

I would like to write a Python program that takes an audio file as input, recognizes clapping sounds in it, then extracts these sounds into a file. My idea is the following: Get (or create) a dataset ...
  • 113
0 votes
1 answer
15 views

How to correctly compute the correlation index of a column value from table in Python 3?

I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
0 votes
0 answers
15 views

Scikit-learn and TensorFlow with very different MLP models

I'm using Multilayer Perceptron ANNs at the very beginning of my project (it's a binary classification problem). Because it's simpler, I started with Scikit-learn. I got a magic result, with my model ...
0 votes
0 answers
7 views

Distribution check - different results using plt.hist and seaborn.displot

I have an apparently very simple task: checking the distribution of values contained in an array data. Only thing is, I get two completely different results by ...
  • 101
0 votes
2 answers
27 views

What ML Algo should I use to plot & predict customer location vs. amount spent?

Im using real customer csv files from my ecommerce store. I was initially going to use k cluster with 2 values. I want the two values to be Province Code, and amount spend on my store. This will ...
0 votes
1 answer
28 views

What can be the reasons for 95% of samples belong to one cluster when there is 5 clusters?

'''I used the k-means algorithm to clustering set of documents which are textual data only. The document has 2lack records. Surprisingly the result for the clustering is 90% of records is storing in 1 ...
1 vote
1 answer
31 views

Best image format for face detection and face recognition with the DeepFace library

I'm using the DeepFace library for face recognition and detection. I was wondering if there is a better format (png, jpg, etc) than others to get better results. Is there a preferred image format for ...
  • 113
0 votes
1 answer
19 views

Quasi complete separation problem

I have some question related to quasi complete seperation problem on logistic regression algorithm. So i run the model to predict credit risk and turns out it gave me good prediction score (AUC around ...
1 vote
2 answers
54 views

How to interpret differences between 2D and 3D T-SNE visualization of similar words from Word2Vec embedding?

I have created a Word2Vec model based on the transcript of the Office. I am now trying to visualize the embedding space for the top similar words of an input word with t-SNE in 2D and 3D. I ...
  • 25
6 votes
1 answer
226 views

Logistic Regression Modeling & Interpretation [closed]

I'm building a logistic regression model to predict the credit risk of lending company customers. I'm using dataset from kaggle : https://www.kaggle.com/datasets/ranadeep/credit-risk-dataset/code ...
0 votes
0 answers
10 views

I want to copy some specific images (validation images) from my training dataset to a different folder

I have used tf.keras.preprocessing.image_dataset_from_directory(validation_split=.20) now I want to copy those 20% validation dataset images to another folder I am using this file_paths = valid_ds....
2 votes
0 answers
64 views

Adding punctuation for a long text

I want to add punctuation to a long text (youtube transcript) before using a Transformer pipeline for summarization. I have found this answer here: original answer thus I have tried: ...
  • 21
1 vote
2 answers
106 views

How to arrange multiple multivariate time series of different length before passing it to Keras LSTM layer

I have a number of multivariate time series that are produced by the same kind of process but: are of significantly different lengths; each time series is an independent instance, and the ...
  • 172
0 votes
0 answers
19 views

How to improve the AUC scores of my link prediction algorithm?

I am exploring a link prediction algorithm, but my AUC scores change every time I run it and it is always very low. It varies from 50 to around 72. I was wondering if there is a way to improve their ...
0 votes
1 answer
21 views

How do I interpret results from scipy pdf() function in concerns with the probability density function

I am working on a personal project and want to understand what the output of Scipy's pdf() function. I understand that the PDF is for one singular line under a distribution curve so finding a point ...
0 votes
0 answers
9 views

Transfer learning (or fine-tuning) pre-trained model on non-text data (PyTorch)

I am currently fine-tuning a sentiment analysis bert-based model using PyTorch Trainer from hugging face. So far, so good. I have easily managed to fine-tune the model on my text data. However, I'd ...
0 votes
0 answers
12 views

How to find a unique feature vector for retail items?

I'm working on this problem where we get images of SKUs (Stock Keeping Unit) or in other words retail products and my job is to classify which product is it. I want to find a unique feature vector ...
1 vote
2 answers
35 views

How to vectorize and speed-up double for-loop for pandas dataframe when doing text similarity scoring

I have the following dataframe: ...
0 votes
0 answers
12 views

GluonTS with small datasets

I am working with a small dataset which is approx. 60 observations. Is it possible to build models from GluonTS which provides probabilistic forecasting also which makes forecasting more reliable but ...
0 votes
1 answer
23 views

SARIMAX model for predictiong the next 100 days

I have daily weather data for almost 50 years for a weather station, and I want to predict the next 100 days' weather. I use python and all of the tools I've used so far (pmdarima.auto_arima, ...
0 votes
1 answer
26 views

numpy power function output for 2 to the power 63

I am running some test code and observed something strange. the numpy power function giving strange output post 2 to the power 62 ...
  • 111
0 votes
1 answer
115 views

numpy.ndarray' object has no attribute 'batch' [closed]

I am building an image classifier using cnn using the following code- I am getting this error, despite making several changes- Kindly help me fix this error.
0 votes
0 answers
14 views

SNS Clustermap with correlation with target value

I am trying to replicate a graph similar to this. https://link.springer.com/article/10.1186/s12864-017-4353-7/figures/1 I am trying to identify if combination of 2 features (30 features, 400000 rows/...
1 vote
1 answer
71 views

Grid_search (RandomizedSearchCV) extremely slow with SVM (SVC)

I'm testing hyperparameters for an SVM, however, when I resort to Gridsearch or RandomizedSearchCV, I haven't been able to get a resolution, because the processing time is exceeding hours. My dataset ...
1 vote
0 answers
27 views

How do I build a multi-input and multi-output neural network?

I want to build a neural network classifier which has as inputs array of dimension (24, 2, 20001) and as outputs array of dimension ...
0 votes
0 answers
10 views

Why is PACF/ACF graphs not working?

Below I have tried to graph PACF and ACF for my time series data. However, for some reason the graphs do not properly show the critical blue region that would help me try to choose what p and q values ...
1 vote
1 answer
1k views

Input shape of dense layer in keras

I'm trying to build a new model ...
  • 11