Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,579
questions
0
votes
0
answers
29
views
When using sqldf for join in python, is there a format to be followed for column names inside sql query?
I'm trying to use sqldf to do a left join on python. Here's my code:
...
- 1
3
votes
2
answers
273
views
is there a deep learning model that handle 47800+ classes for classification?
I am trying to build a text classifier with 47893 classes and 1.3 billion (1,302,687,947) data samples.
What would be the best classifier to build with such kind of data? Each data label will contain ...
- 239
0
votes
0
answers
14
views
LIME visualization for resnet50
I am implementing LIME on my resnet50 mode. There are 4 classes in the dataset.
the code snippet of LIME:
...
- 172
0
votes
0
answers
14
views
Reinforcement Learning - What environment and algorithms should I use?
I have to do a project on Reinforcement Learning.
Environment
First, I need to choose an environment to use. It should meet one of two assumptions:
it should be stochastic OR
it should require ...
- 101
0
votes
0
answers
22
views
How to structure dataframe combinations for regression, without corruption/loss?
I have a data set, redacted sample below. My goal is linear regression. My question is: Have I created unintended results, due to how I structured the df, using concat and/or div?
For example, ...
- 11
0
votes
0
answers
57
views
Getting error "Failed to find data adapter that can handle input" even after converting list to array
I am getting this error : ValueError: Failed to find data adapter that can handle input' I even changed the list to arrays but still the error keeps pooping up.
This is the code:
...
0
votes
1
answer
18
views
Is there a way to embed curves into smaller vectors?
I know about embeddings for words, but I would like to know if it is possible to do something similar for curves. What I mean by curves is a curve of a function. Say I have 1000 points corresponding ...
- 1
0
votes
0
answers
7
views
Plus sign inside circle, there seem to be many meaning. Are they all related?
In Linear algebra plus sign inside a circle refers to direct sum.
In a paper "Towards Neural Mixture Recommender for Long Range Dependent User Sequences" it seems to refer to concatenation.
...
- 385
0
votes
1
answer
25
views
ML Predicted Model for 2 values
I have a data set with 96 rows. It contains date, source, spend and number of customers.
I have 4 different sources that generate customers and you can see in the dataset how much I spend and how many ...
0
votes
0
answers
12
views
0
votes
0
answers
23
views
What are the advantages of keeping NAN values instead of handling/replacing them? Pandas
In Pandas, what are the benefits of keeping the nan values in the DataFrame/Series instead of replacing/handling them with another value? Can we use them to indicate an explicit state or flag? Are ...
- 101
0
votes
0
answers
17
views
Grad cam outputs for all the images are the same
I am using grad cam to see which regions of the test images are most important for the prediction of resnet50. The output I got has some errors.
Code Snippets:
<...
- 172
0
votes
0
answers
8
views
ETS model in statsmodels library
I have several questions regarding the ETS model in statsmodels library. The description of the model can be found here.
The default initialization_method is ...
- 101
0
votes
0
answers
22
views
Incompatible shape in SimpleRNN layer
I'm reading the "Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow" book and got to the exercise telling the readers to create a model able to predict bach chorale notes. For ...
- 113
0
votes
0
answers
20
views
Can't understand grad cam output
I've implemented Grad Cam. The colors in a Grad-CAM heatmap usually use a red-to-blue color scale, where red signifies the highest importance and blue signifies the lowest importance. The intensity of ...
- 172
2
votes
1
answer
82
views
How to present a statistical justification for the choice of models with approximate accuracies?
In an experiment involving the comparison of classification algorithms, how can I assess whether there are statistically significant differences between the analyzed models? For example, the following ...
0
votes
0
answers
27
views
How to handle imbalance in input variables?
Currently working on a finance dataset which has more than 20 input variables with high imbalance.
[Apparently, the target variable is also imbalanced (for this I am currently considering to handle it ...
- 1
0
votes
0
answers
20
views
Can early stopping be used with stratified k-fold validation to help avoid overfitting in neural networks?
I am using stratified cross validation and using transfer learning for the classification. I have 4 classes. I am training my model with fold=10 and epoch=20.
code snippets:
...
- 172
0
votes
0
answers
17
views
How to rank terms with bm25 and bo1 pipeline
In pyTerrier I have list of single terms. For example (I choose those tokens to be as relevant as possible and as irrelevant as possible to enhance the effect):
...
- 101
0
votes
0
answers
26
views
Keras, dealing with a non-binary problem
Hello I have a dataset that lets me know if player 1 has won:
BMI, Temperature, Weight, Player1Win
33.6,17,50.0
26.6,19,31.0
23.3,16,32.1
28.1,20,21.0
43.1,17,33.1
...
1
vote
1
answer
32
views
Validation acc is very high in each fold but Test acc is very low
I am trying to implement a neural network. I am using CNN model for classifying. First I split the dataset into train and test.
Code Snippet:
...
- 172
1
vote
0
answers
48
views
scikit learn logistic regression to annotate single cell RNA seq data
I want to use scikit learn logistic regression to train a model on a labelled single cell RNA sample and subsequently apply this model on new unlabelled single cell RNA seq samples to annotate the ...
- 11
0
votes
0
answers
15
views
How to scale a vector into a smaller vector?
I have a vector A of 6 integers [1000,900,800,700,600,500] and each number corresponds to another vector B [0,1,2,3,4,5].
How can I get a vector A_new of 5 integers if I scale the vector B to [0,0.25,...
- 11
0
votes
0
answers
10
views
Machine learning predictions way different than actual values
My data set is a dependent variable (salary) and 140 binary columns for training.
I am using linear regression to predict the salary.
I split the data and tried to predict but the values I am getting ...
0
votes
0
answers
36
views
Which level of skills a junior data scientist with an year of experience should have?
I’ve been working as a junior data scientist for about a year. I decided to join a Kaggle competition but struggle with understanding how the train and test are created. In particular, I spent about ...
- 1
1
vote
1
answer
39
views
How to curve fit, Z variable dependent on X and Y?
I'm trying to find the function for this visualization:
I would like to get feedback if I'm taking the right approach. My approach:
These data points are created by a person. They are two ...
- 111
0
votes
2
answers
38
views
LSTMs how to forecast out N steps
I have about 3 weeks of 15 minute building electricity power data and curious to know how can I predict an entire days worth of electricity into the future? 96 Future values that makes up 24 hours......
- 381
0
votes
0
answers
10
views
Scipy WrappedCauchy isn't wrapping when loc != 0
I want to plot the PDF of a wrapped Cauchy distribution, but when I set a value for the location, the entire distribution gets shifted, rather than wrapped.
When I run this:
...
- 53
0
votes
0
answers
59
views
In CNN, When do we increase, or decrease, the number of filters/neurons?
Good morning,
I would like to understand how do we choose between increasing or decreasing the number of filters applied in a CNN.
My logic response to this, would be to take Autoencoder as an example ...
- 1
0
votes
0
answers
12
views
Varying output size in recurrent neural network
My dataset has an varying size in the output data (meaning can be a vector of 2 or 5 integers). I read that recurrent neural network (RNN) can be a solution to that but did not find an easy example of ...
- 11
1
vote
1
answer
43
views
Is it possible to reverse the layers of a convolutional neural network?
From my understanding typically a convolutional neural network has a matrix (e.g. an image) as input and output is either an integer or a vector of integers in regression and in classification a ...
- 11
1
vote
0
answers
58
views
Short Text Topic Modelling in Python
I have a large dataset of short reviews and I would like to find the most recurring themes. For this reason, I got into topic modeling.
I am looking for some good tutorials and references for short ...
0
votes
0
answers
19
views
time series predict into future
I am using Python and stats models SARIMA and I can predict out 24 samples like shown below.
Can I do the same thing with machine learning but (not LSTM) and not have to feed an array of data through ...
- 381
0
votes
0
answers
8
views
Tensorflow text classification with subject for each text
I want to classify texts with additional input 'text' subject. I acquire these subjects from wikidata 'instance of' properties. I designed a neural net model as below. Network takes texts and subjects ...
- 121
0
votes
0
answers
35
views
Understanding the stochastic average gradient (SAG) algorithm used in sklearn
For pedagogical purposes I've been trying to create my own implementation of the stochastic average gradient (SAG) algorithm in a logistic regression framework. Page 10 of the associated paper ...
1
vote
1
answer
32
views
Confusion matrix and precision problem
I'm trying to calculate the precision of a trained model. I have generated the right values for the true positive rate and the false positive rate. And I know that the formula should be TP/TP + FP.
...
- 11
1
vote
0
answers
38
views
Modified kmeans algorithm returns the wrong answer
I am trying to create a kmeans algorithm that is based on the Earth Movers Distance instead of the Euclidean distance. However, when I run it, it just returns the same value for all data points.
The ...
- 11
0
votes
0
answers
21
views
how to visualise a distribution over time?
Background
I'm analysing data where data recency and an attribute of the data, in this case person's age, could serve to identify a simply defined cohort that can be considered defunct. I'd like to ...
1
vote
1
answer
35
views
Train test split before or train test split inside cross validation
I want to do cross validation. So should i split my data into train and test with sklearn train_test_split and use cross validation like this:
...
- 138
0
votes
0
answers
23
views
What is the difference between nmf.fit() nmf.fit_transform() in a easy way?
I am reading several questions on this topic.
It seems quite clear to me for TFIDF why we have .fit_transform() and .transform()
...
- 123
0
votes
1
answer
25
views
Model with mae gives nan predictions but gives reasonable predictions with mse. Why would this be?
Same exact model in keras but i just changed the loss from mse to mae. First one works, but the second just gives nans as predictions. Huber also gives predictions, but mae does not.
0
votes
0
answers
25
views
How do I implement recurrent activations for LSTM/GRU cells in Pytorch?
Although Tensorflow has simple parameters with which I can initialize the recurrent activation of a GRU or LSTM cell, Pytorch does not.
Could someone provide me the best way to add recurrent ...
0
votes
0
answers
17
views
Determine unusual occurrence of words in classes
I am working on a project where I have 20+ classes/groups. Each of these groups perform certain text searches. I am looking for specific keywords example 'code' which is an anomaly. The challenge is ...
- 11
1
vote
1
answer
19
views
Which pre-trained model to select to generate embeddings from shop names written in English?
Good afternoon!
I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
- 11
1
vote
0
answers
16
views
How can I detect errors/irregularity in a dataset
I've been working on a pathfinding project using topographic data for my software development course, even though I've never really had calculus lessons.
The algorithm works fine by itself, however, ...
- 11
0
votes
0
answers
24
views
How are fp32 weights converted to fp16 post training in PyTorch?
Can the weights of a model trained in full precision be converted to half precision post-training, with or without loss of accuracy? If so, what is the maths involved behind the conversion?
Thanks
- 1
0
votes
0
answers
18
views
How to replicate MMCV conv layer with integrated batch norm using PyTorch
Question
How do I replicate the following convolution layer from MMCV using PyTorch? I cannot find any reference in the MMCV docs on how norm_cfg works.
...
- 46
0
votes
0
answers
72
views
NetworkX: Subgraph matching
Trying to match a Query subgraph to a Target graph, where:
Query:
and Target:
As I understand matching in this case should return tuples of matching nodes:
...
- 275
0
votes
0
answers
32
views
Time series prediction of individual customer (transaction count and gross value) using LSTM/NARX
Problem:
I'm trying to rewrite Matlab code that goal is prediction of gross value using historical data of 2 features: gross value of next week. The data is about customer transaction in retail where ...
0
votes
0
answers
17
views
Best Tool for NN with Convolutional Layers
I am working on creating a NN with the following architecture:
Input layer (180 neurons)
Hidden Layer 1 (18 neurons)
Hidden Layer 2 (4 neurons)
Output Layer (1 neuron)
I am trying to figure out the ...
- 101