Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,590
questions
1
vote
0
answers
38
views
Modified kmeans algorithm returns the wrong answer
I am trying to create a kmeans algorithm that is based on the Earth Movers Distance instead of the Euclidean distance. However, when I run it, it just returns the same value for all data points.
The ...
- 11
0
votes
0
answers
22
views
how to visualise a distribution over time?
Background
I'm analysing data where data recency and an attribute of the data, in this case person's age, could serve to identify a simply defined cohort that can be considered defunct. I'd like to ...
1
vote
1
answer
43
views
Train test split before or train test split inside cross validation
I want to do cross validation. So should i split my data into train and test with sklearn train_test_split and use cross validation like this:
...
- 138
0
votes
0
answers
26
views
What is the difference between nmf.fit() nmf.fit_transform() in a easy way?
I am reading several questions on this topic.
It seems quite clear to me for TFIDF why we have .fit_transform() and .transform()
...
- 123
0
votes
1
answer
34
views
Model with mae gives nan predictions but gives reasonable predictions with mse. Why would this be?
Same exact model in keras but i just changed the loss from mse to mae. First one works, but the second just gives nans as predictions. Huber also gives predictions, but mae does not.
0
votes
0
answers
27
views
How do I implement recurrent activations for LSTM/GRU cells in Pytorch?
Although Tensorflow has simple parameters with which I can initialize the recurrent activation of a GRU or LSTM cell, Pytorch does not.
Could someone provide me the best way to add recurrent ...
0
votes
0
answers
17
views
Determine unusual occurrence of words in classes
I am working on a project where I have 20+ classes/groups. Each of these groups perform certain text searches. I am looking for specific keywords example 'code' which is an anomaly. The challenge is ...
- 11
1
vote
1
answer
19
views
Which pre-trained model to select to generate embeddings from shop names written in English?
Good afternoon!
I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
- 11
1
vote
0
answers
16
views
How can I detect errors/irregularity in a dataset
I've been working on a pathfinding project using topographic data for my software development course, even though I've never really had calculus lessons.
The algorithm works fine by itself, however, ...
- 11
0
votes
0
answers
31
views
How are fp32 weights converted to fp16 post training in PyTorch?
Can the weights of a model trained in full precision be converted to half precision post-training, with or without loss of accuracy? If so, what is the maths involved behind the conversion?
Thanks
- 1
0
votes
0
answers
22
views
How to replicate MMCV conv layer with integrated batch norm using PyTorch
Question
How do I replicate the following convolution layer from MMCV using PyTorch? I cannot find any reference in the MMCV docs on how norm_cfg works.
...
- 46
0
votes
0
answers
89
views
NetworkX: Subgraph matching
Trying to match a Query subgraph to a Target graph, where:
Query:
and Target:
As I understand matching in this case should return tuples of matching nodes:
...
- 275
0
votes
0
answers
36
views
Time series prediction of individual customer (transaction count and gross value) using LSTM/NARX
Problem:
I'm trying to rewrite Matlab code that goal is prediction of gross value using historical data of 2 features: gross value of next week. The data is about customer transaction in retail where ...
0
votes
0
answers
17
views
Best Tool for NN with Convolutional Layers
I am working on creating a NN with the following architecture:
Input layer (180 neurons)
Hidden Layer 1 (18 neurons)
Hidden Layer 2 (4 neurons)
Output Layer (1 neuron)
I am trying to figure out the ...
- 101
0
votes
1
answer
22
views
Python dataset normalization for convolutional autoencoder
I have a csv files which contain pixel neighboorhood information. Here an example of the dataset:
...
- 137
0
votes
1
answer
54
views
Can I have 0 loss in the validation set and still have bad accuracy?
I am starting in the world of deep neural networks and doing a series of tests with a convolutional model, I have found the following case:
The accuracy in the training set is much better (around 0.85)...
- 302
0
votes
1
answer
20
views
How to differentiate Bitmojis gender?
I have seen this Project Larry-zx's Githubproject
It is used to create a .pth file which can differenciate between a bitmojis gender.
Now I have created that .pth file but no idea how to use it, ...
1
vote
1
answer
68
views
Random Forest Classifier is giving me an array of zeroes
I used VGG16 as feature extractor on a dataset with 9 classes and trained the Random Forest Classifier on the feature vector. I tried to make prediction on the test feature vector but the prediction ...
- 23
0
votes
0
answers
18
views
Train AI to recognize a Snapchat's bitmoji's gender
I have found this https://www.kaggle.com/competitions/bitmoji-faces-gender-recognition/overview/description competition. I need the file module that returns either male or female but cant find it. How ...
0
votes
1
answer
27
views
What kind of model can be suitable for predicting 100 values from 25 values by using a Neural Network in TensorFlow?
Previously I used Sequential model for this problem, but later I read from the TensorFlow website that it is not suitable to use it. I'd like to have a NN model to predict 100+ values from the given ...
- 3
2
votes
1
answer
36
views
Trying to average node values over local neighborhoods in a graph using a GCN
I'm new to Graph Convolutional Networks (and pytorch in general) so I'm trying to verify that the message passing layer is working as expected before I go on to adding layers to the network.
But when ...
- 21
0
votes
0
answers
14
views
Is this Time Series Non-Stationary? Or is the ADF test wrong?
I have two time series I want to prove if they are stationary or not. The first time series is the total hours from a YouTube channel over time and the second one is the organic hours (total hours - ...
1
vote
1
answer
193
views
ChatGPT and my PhD research [closed]
I am currently a PhD student in the field of NLP and I can see a way how ChatGPT can solve my current research question. My research question is related to reasoning based on text. What can I possibly ...
- 19
0
votes
0
answers
13
views
How to convert multidimensional list to ploatly graph
I read the text from below html table in array such that each row is a list and all these rows are stored in list.
Now I need to convert this list of to dataframe and plot a graph using ploatly.
...
- 101
0
votes
1
answer
80
views
LSTM model is producing really bad results for multiclass text classification for imbalanced small dataset
I am training a LSTM model on my current dataset to predict the multiclass categories - there are 18 mutually exclusive categories and the dataset has ...
- 163
0
votes
0
answers
17
views
Generate new sample with higher variance, covariance from empirical distribution
I am analyzing the joint distribution of three continuous variables. I would like to sample from the joint distribution of these three variables, which is straight forward. However, I also want to ...
1
vote
0
answers
22
views
Python train convolutional on numerical values shape issue
I want to train a convolutional neural network autoencoder on a csv file which contains values pixel neighborhood position of an original image of 1024x1024. When I try to train it, I have the ...
- 137
1
vote
0
answers
9
views
Drawing transparent bounding boxes with Torchvision
I am using Torchvision in a Python script to draw bounding boxes and then crop images based on the bounding boxes drawn. The script works well, except that the bounding boxes obscure the subjects in ...
- 11
0
votes
1
answer
39
views
Predict coordinates from input of coordinates
I'm a newbie at data science and I want to ask how can I predict a set of coordinates from a set of input coordinates? That is (x1, y1) -> (x2, y2).
To give a ...
1
vote
1
answer
40
views
Interpreting Learning Curves of models
I need some help to understand if the models are overfitting and which of these we can consider "the best". On the internet i only find simple examples with learning curves but in these ...
- 41
0
votes
1
answer
65
views
How can I take the cross-correlation between two discrete signal with slightly different discrete time points?
I have two discrete signals x1 and x2 with corresponding time points t1 and ...
- 1
0
votes
0
answers
255
views
How to dynamically add columns in Dataframe Spark by using a config.ini file parameters
I am learning to use Pyspark and at work I have been assigned a task that I can not solve for now.
There is a config.ini file where there are a series of ...
- 101
2
votes
4
answers
178
views
99% accuracy in train and 96% in test is too much overfitting?
I have a binary classification problem, the classes are quite balanced (57%-43%), with a GridSearch with Random Forest Classifier I obtained the best hyperparameters and I applied the model to train ...
- 41
0
votes
0
answers
43
views
pytesseract performance for digit recognition
The sequence in the following image
is correctly recognized by pytesseract via the usual function "pytesseract.image_to_string". But the following image with a similar sequence is not ...
0
votes
0
answers
33
views
Fuzzy Inference System IF-THEN Rules
I am currently doing a fuzzy inference system project in Python, however, a question came up.
I have to do it the hard way, that is, do it step by step, as done here: https://pythonhosted.org/scikit-...
- 1
0
votes
0
answers
19
views
How to tokenize custom dataset?
I have generated a dataset of quadratic equations and I want to train T5 model to solve them.
Dataset looks like this:
Could somebody please give a hint on how to tokenize this correctly? What to do ...
1
vote
2
answers
88
views
Are Chi-square and ANOVA (f_classif) to select best features?
I have a binary classification problem (target 0 o 1), I have both variables continuous and categorical as features. I understood that about Chi-square i can use only categorical features to evaluate ...
- 41
1
vote
1
answer
91
views
NLP Question: Where can I find a list of all open compound words (with spaces in them), like "peanut butter" or "high school"? [close
I already have a list of "1-gram" words, which include closed compound words like "skyscraper" or "weatherman."
However, I'm also interested in compiling a list of "...
- 13
0
votes
1
answer
95
views
Trying to fill null values with sub-grouped mean value using pandas fillna() and groupby().transform() is doing nothing with the null values
I'm doing EDA on the US police shooting dataset where I'm stuck with filling the 482 null values for the age column. Since no strategy was mentioned on how to fill ...
- 101
0
votes
0
answers
17
views
How to create synthetic data with custom feature values?
Currently I create a synthetic dataset using the python sdv package and an existing dataset as below:
...
- 21
0
votes
1
answer
127
views
How to translate text automatically using Google Translate API (or any other approach) in python
I have a dataset of reviews from TripAdvisor and I would like to translate non-English reviews into English.
The reviews are in many different languages: my dataset contains reviews in 44 different ...
0
votes
0
answers
31
views
Machine Learning | Learning and ROC Curves interpretation
Not a problem but rather a matter of understanding, my goal is to evaluate which is the best classification model for my dataset among k-nearest neighbors, naive bayes classifier and decision trees, ...
0
votes
0
answers
8
views
Identifying and removing repeated text from json files post inference
Good morning: We index and summarize medical records using a proprietary algorithm. Being medical records they repeat the same information almost on every single page, like facility name, medical ...
0
votes
0
answers
14
views
sequential rule mining
What are the available libraries today for this problem in python
I have a schedule plan based on the location with a time series and in a different order
I want to predict the schedule plan giving ...
- 1
0
votes
0
answers
96
views
extract ORC file to csv using python
I am extracting large ORC files (58G) into csv files in python jupyter.
I could not generate csv file after run this command.
...
- 1
1
vote
0
answers
171
views
How to convert an ONNX model with dynamic input shape to Tensorflow model
I am trying to convert an ONNX model with a dynamic input shape to TensorFlow format using the onnx_tf package. I am using TensorFlow 2.11.0, ONNX 1.13.0, and onnx_tf 1.10.0. The input to the model ...
0
votes
1
answer
53
views
How do I fill the null ages of a person according to the average age of males or females in a particular race
I got this US police shootings dataset where many age are null. Each person's race and gender...
- 101
0
votes
1
answer
42
views
Creating new features as linear combination of others as part of a scikit-learn pipeline?
I have a number of raw features that go into a scikit-learn model. I've already got a number of preprocessing steps (such as PolynomialFeatures) that creates additional features as part of my pipeline....
- 151
0
votes
1
answer
26
views
Differences in OvO SVC decision_function and predict methods in Scikit learn package
I have observed that there are differences in the implementation of the decision_function and the predict methods the one versus one multi class implementation of SVC in the sci-kit learn package. Is ...
0
votes
0
answers
28
views
Forecast n-steps into the future with LightGBM
I have learnt to create the LightGBM models with multiple features and target, but in ALL the examples I have seen thus far they only show how to do a "predict" on the x_test_df. What I am ...
- 101