Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
0 votes
0 answers
29 views

When using sqldf for join in python, is there a format to be followed for column names inside sql query?

I'm trying to use sqldf to do a left join on python. Here's my code: ...
  • 1
3 votes
2 answers
273 views

is there a deep learning model that handle 47800+ classes for classification?

I am trying to build a text classifier with 47893 classes and 1.3 billion (1,302,687,947) data samples. What would be the best classifier to build with such kind of data? Each data label will contain ...
  • 239
0 votes
0 answers
14 views

LIME visualization for resnet50

I am implementing LIME on my resnet50 mode. There are 4 classes in the dataset. the code snippet of LIME: ...
0 votes
0 answers
14 views

Reinforcement Learning - What environment and algorithms should I use?

I have to do a project on Reinforcement Learning. Environment First, I need to choose an environment to use. It should meet one of two assumptions: it should be stochastic OR it should require ...
0 votes
0 answers
22 views

How to structure dataframe combinations for regression, without corruption/loss?

I have a data set, redacted sample below. My goal is linear regression. My question is: Have I created unintended results, due to how I structured the df, using concat and/or div? For example, ...
0 votes
0 answers
57 views

Getting error "Failed to find data adapter that can handle input" even after converting list to array

I am getting this error : ValueError: Failed to find data adapter that can handle input' I even changed the list to arrays but still the error keeps pooping up. This is the code: ...
0 votes
1 answer
18 views

Is there a way to embed curves into smaller vectors?

I know about embeddings for words, but I would like to know if it is possible to do something similar for curves. What I mean by curves is a curve of a function. Say I have 1000 points corresponding ...
  • 1
0 votes
0 answers
7 views

Plus sign inside circle, there seem to be many meaning. Are they all related?

In Linear algebra plus sign inside a circle refers to direct sum. In a paper "Towards Neural Mixture Recommender for Long Range Dependent User Sequences" it seems to refer to concatenation. ...
  • 385
0 votes
1 answer
25 views

ML Predicted Model for 2 values

I have a data set with 96 rows. It contains date, source, spend and number of customers. I have 4 different sources that generate customers and you can see in the dataset how much I spend and how many ...
0 votes
0 answers
12 views

how can i get precision and recall on ANN

I received this error, how can I resolve it? ...
  • 1
0 votes
0 answers
23 views

What are the advantages of keeping NAN values instead of handling/replacing them? Pandas

In Pandas, what are the benefits of keeping the nan values in the DataFrame/Series instead of replacing/handling them with another value? Can we use them to indicate an explicit state or flag? Are ...
0 votes
0 answers
17 views

Grad cam outputs for all the images are the same

I am using grad cam to see which regions of the test images are most important for the prediction of resnet50. The output I got has some errors. Code Snippets: <...
0 votes
0 answers
8 views

ETS model in statsmodels library

I have several questions regarding the ETS model in statsmodels library. The description of the model can be found here. The default initialization_method is ...
  • 101
0 votes
0 answers
22 views

Incompatible shape in SimpleRNN layer

I'm reading the "Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow" book and got to the exercise telling the readers to create a model able to predict bach chorale notes. For ...
  • 113
0 votes
0 answers
20 views

Can't understand grad cam output

I've implemented Grad Cam. The colors in a Grad-CAM heatmap usually use a red-to-blue color scale, where red signifies the highest importance and blue signifies the lowest importance. The intensity of ...
2 votes
1 answer
82 views

How to present a statistical justification for the choice of models with approximate accuracies?

In an experiment involving the comparison of classification algorithms, how can I assess whether there are statistically significant differences between the analyzed models? For example, the following ...
0 votes
0 answers
27 views

How to handle imbalance in input variables?

Currently working on a finance dataset which has more than 20 input variables with high imbalance. [Apparently, the target variable is also imbalanced (for this I am currently considering to handle it ...
  • 1
0 votes
0 answers
20 views

Can early stopping be used with stratified k-fold validation to help avoid overfitting in neural networks?

I am using stratified cross validation and using transfer learning for the classification. I have 4 classes. I am training my model with fold=10 and epoch=20. code snippets: ...
0 votes
0 answers
17 views

How to rank terms with bm25 and bo1 pipeline

In pyTerrier I have list of single terms. For example (I choose those tokens to be as relevant as possible and as irrelevant as possible to enhance the effect): ...
  • 101
0 votes
0 answers
26 views

Keras, dealing with a non-binary problem

Hello I have a dataset that lets me know if player 1 has won: BMI, Temperature, Weight, Player1Win 33.6,17,50.0 26.6,19,31.0 23.3,16,32.1 28.1,20,21.0 43.1,17,33.1 ...
1 vote
1 answer
32 views

Validation acc is very high in each fold but Test acc is very low

I am trying to implement a neural network. I am using CNN model for classifying. First I split the dataset into train and test. Code Snippet: ...
1 vote
0 answers
48 views

scikit learn logistic regression to annotate single cell RNA seq data

I want to use scikit learn logistic regression to train a model on a labelled single cell RNA sample and subsequently apply this model on new unlabelled single cell RNA seq samples to annotate the ...
0 votes
0 answers
15 views

How to scale a vector into a smaller vector?

I have a vector A of 6 integers [1000,900,800,700,600,500] and each number corresponds to another vector B [0,1,2,3,4,5]. How can I get a vector A_new of 5 integers if I scale the vector B to [0,0.25,...
  • 11
0 votes
0 answers
10 views

Machine learning predictions way different than actual values

My data set is a dependent variable (salary) and 140 binary columns for training. I am using linear regression to predict the salary. I split the data and tried to predict but the values I am getting ...
0 votes
0 answers
36 views

Which level of skills a junior data scientist with an year of experience should have?

I’ve been working as a junior data scientist for about a year. I decided to join a Kaggle competition but struggle with understanding how the train and test are created. In particular, I spent about ...
1 vote
1 answer
39 views

How to curve fit, Z variable dependent on X and Y?

I'm trying to find the function for this visualization: I would like to get feedback if I'm taking the right approach. My approach: These data points are created by a person. They are two ...
  • 111
0 votes
2 answers
38 views

LSTMs how to forecast out N steps

I have about 3 weeks of 15 minute building electricity power data and curious to know how can I predict an entire days worth of electricity into the future? 96 Future values that makes up 24 hours......
  • 381
0 votes
0 answers
10 views

Scipy WrappedCauchy isn't wrapping when loc != 0

I want to plot the PDF of a wrapped Cauchy distribution, but when I set a value for the location, the entire distribution gets shifted, rather than wrapped. When I run this: ...
0 votes
0 answers
59 views

In CNN, When do we increase, or decrease, the number of filters/neurons?

Good morning, I would like to understand how do we choose between increasing or decreasing the number of filters applied in a CNN. My logic response to this, would be to take Autoencoder as an example ...
0 votes
0 answers
12 views

Varying output size in recurrent neural network

My dataset has an varying size in the output data (meaning can be a vector of 2 or 5 integers). I read that recurrent neural network (RNN) can be a solution to that but did not find an easy example of ...
  • 11
1 vote
1 answer
43 views

Is it possible to reverse the layers of a convolutional neural network?

From my understanding typically a convolutional neural network has a matrix (e.g. an image) as input and output is either an integer or a vector of integers in regression and in classification a ...
  • 11
1 vote
0 answers
58 views

Short Text Topic Modelling in Python

I have a large dataset of short reviews and I would like to find the most recurring themes. For this reason, I got into topic modeling. I am looking for some good tutorials and references for short ...
0 votes
0 answers
19 views

time series predict into future

I am using Python and stats models SARIMA and I can predict out 24 samples like shown below. Can I do the same thing with machine learning but (not LSTM) and not have to feed an array of data through ...
  • 381
0 votes
0 answers
8 views

Tensorflow text classification with subject for each text

I want to classify texts with additional input 'text' subject. I acquire these subjects from wikidata 'instance of' properties. I designed a neural net model as below. Network takes texts and subjects ...
0 votes
0 answers
35 views

Understanding the stochastic average gradient (SAG) algorithm used in sklearn

For pedagogical purposes I've been trying to create my own implementation of the stochastic average gradient (SAG) algorithm in a logistic regression framework. Page 10 of the associated paper ...
1 vote
1 answer
32 views

Confusion matrix and precision problem

I'm trying to calculate the precision of a trained model. I have generated the right values for the true positive rate and the false positive rate. And I know that the formula should be TP/TP + FP. ...
  • 11
1 vote
0 answers
38 views

Modified kmeans algorithm returns the wrong answer

I am trying to create a kmeans algorithm that is based on the Earth Movers Distance instead of the Euclidean distance. However, when I run it, it just returns the same value for all data points. The ...
  • 11
0 votes
0 answers
21 views

how to visualise a distribution over time?

Background I'm analysing data where data recency and an attribute of the data, in this case person's age, could serve to identify a simply defined cohort that can be considered defunct. I'd like to ...
1 vote
1 answer
35 views

Train test split before or train test split inside cross validation

I want to do cross validation. So should i split my data into train and test with sklearn train_test_split and use cross validation like this: ...
0 votes
0 answers
23 views

What is the difference between nmf.fit() nmf.fit_transform() in a easy way?

I am reading several questions on this topic. It seems quite clear to me for TFIDF why we have .fit_transform() and .transform() ...
0 votes
1 answer
25 views

Model with mae gives nan predictions but gives reasonable predictions with mse. Why would this be?

Same exact model in keras but i just changed the loss from mse to mae. First one works, but the second just gives nans as predictions. Huber also gives predictions, but mae does not.
0 votes
0 answers
25 views

How do I implement recurrent activations for LSTM/GRU cells in Pytorch?

Although Tensorflow has simple parameters with which I can initialize the recurrent activation of a GRU or LSTM cell, Pytorch does not. Could someone provide me the best way to add recurrent ...
0 votes
0 answers
17 views

Determine unusual occurrence of words in classes

I am working on a project where I have 20+ classes/groups. Each of these groups perform certain text searches. I am looking for specific keywords example 'code' which is an anomaly. The challenge is ...
1 vote
1 answer
19 views

Which pre-trained model to select to generate embeddings from shop names written in English?

Good afternoon! I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
  • 11
1 vote
0 answers
16 views

How can I detect errors/irregularity in a dataset

I've been working on a pathfinding project using topographic data for my software development course, even though I've never really had calculus lessons. The algorithm works fine by itself, however, ...
  • 11
0 votes
0 answers
24 views

How are fp32 weights converted to fp16 post training in PyTorch?

Can the weights of a model trained in full precision be converted to half precision post-training, with or without loss of accuracy? If so, what is the maths involved behind the conversion? Thanks
0 votes
0 answers
18 views

How to replicate MMCV conv layer with integrated batch norm using PyTorch

Question How do I replicate the following convolution layer from MMCV using PyTorch? I cannot find any reference in the MMCV docs on how norm_cfg works. ...
  • 46
0 votes
0 answers
72 views

NetworkX: Subgraph matching

Trying to match a Query subgraph to a Target graph, where: Query: and Target: As I understand matching in this case should return tuples of matching nodes: ...
  • 275
0 votes
0 answers
32 views

Time series prediction of individual customer (transaction count and gross value) using LSTM/NARX

Problem: I'm trying to rewrite Matlab code that goal is prediction of gross value using historical data of 2 features: gross value of next week. The data is about customer transaction in retail where ...
0 votes
0 answers
17 views

Best Tool for NN with Convolutional Layers

I am working on creating a NN with the following architecture: Input layer (180 neurons) Hidden Layer 1 (18 neurons) Hidden Layer 2 (4 neurons) Output Layer (1 neuron) I am trying to figure out the ...
  • 101