Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,588
questions
8
votes
2
answers
6k
views
Applying dimensionality reduction on OneHotEncoded array
I have a really large data set with mixed variables. I have converted categorical variables to numerical using OneHotEncoding and it has resulted in more than a ...
- 231
8
votes
1
answer
10k
views
KL-divergence returns infinity
Given an original probability distribution P, I want to measure how much an approximation Q differs from the initial distribution. For that I calculate the KL-divergence via ...
user17988
8
votes
4
answers
12k
views
Exploratory Data Analysis with Image Datset
In Machine Learning Kernels on Kaggle I often see EDAs with structured data. So, I was wondering, if there are any recommended/standard procedures for EDA with image datasets. What kind of statistical ...
8
votes
3
answers
17k
views
Validation loss much higher than training loss
I am training a CNN on some text data. The sentences are padded and embedded and fed to a CNN. The model architecture is:
...
- 205
8
votes
3
answers
8k
views
Isolation forest sklearn contamination param
I am working on an unsupervised anomaly detection task on time series data using an isolation forest algorithm.
I am developing it in Python, more in detail using ...
- 325
8
votes
1
answer
46k
views
Am I doing a log transformation of data correctly?
I'm doing some exploratory data analysis on some data and I get these histograms:
That looks like a candidate for a log transformation on the data, so I run the following Python code to transform the ...
- 183
8
votes
1
answer
6k
views
Prepending Input layer to pre-trained model
I'm trying to input numpy arrays of shape (1036800,) - originally images of shape (480, 720, 3) - into a pre-trained VGG16 model to predict continuous values.
I've tried several variations of the ...
- 491
8
votes
3
answers
2k
views
Convolutional Neural Network not learning EEG data
I have trained a simple CNN (using Python + Lasagne) for a 2-class EEG classification problem, however, the network doesn't seem to learn. loss does not drop over epochs and classification accuracy ...
- 1,071
8
votes
1
answer
7k
views
Recognition human in images through HOG descriptor and SVM classifier performs poorly
I'm using a HOG descriptor, coupled with a SVM classifier, to recognise humans in pictures. I'm using the Python wrappers for OpenCV.
I've used the excellent tutorial at pymagesearch, which explains ...
- 255
8
votes
1
answer
4k
views
What's the default Scorer in Sci-kit learn's GridSearchCV?
Even if I don't define the scoring parameter, it scores and makes a decision for best estimator, but documentation says the default value for scoring is "None", so what is it using to score ...
- 81
8
votes
2
answers
3k
views
Data anonymization in Python
I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
- 2,407
8
votes
5
answers
4k
views
How can we extract fields from images?
I am making an document parser which extracts data fields from the documents and store them in a structured way. Each field in my dataset is horizontal which is easy to extract.
But the model fails ...
- 81
8
votes
5
answers
302k
views
I got the following error : 'DataFrame' object has no attribute 'data'
I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result:
Program:
...
- 81
8
votes
2
answers
7k
views
Audio Analysis : Segment audio based on speaker recognition
I have audio clips of people being interviewed and am trying to split the audio clips using python such that all speech segments of the interviewee are outputted in one audio file (eg .wav format) &...
- 181
8
votes
1
answer
2k
views
How does Tensorflow compute gradients of reduce_min operation?
With a non-differentiable operation, such as a minimization, how does Tensorflow compute the gradients? Some kind of soft-minimum approximation?
If so, can I retrieve the analytical computation for a ...
- 480
8
votes
1
answer
11k
views
Imbalanced dataset in MLP classifier in python
I am dealing with imbalanced dataset and I try to make a predictive model using MLP classifier. Unfortunately the algorithm classifies all the observations from test set to class "1" and hence the f1 ...
- 203
8
votes
2
answers
126k
views
ValueError: could not convert string to float: '���'
I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
- 231
8
votes
1
answer
453
views
How far can one go with excel? [closed]
in my business we handle all analytics through Excel. This includes mostly scheduling, production planning and accounting operations. We currently are looking into adding a bit of predictive modelling ...
- 340
8
votes
2
answers
5k
views
LSTM: How to deal with nonstationarity when predicting a time series
I want to do one-step-ahead predictions for time series with LSTM.
To understand the algorithm, I built myself a toy example: A simple autocorrelated process.
...
- 1,636
8
votes
1
answer
4k
views
Complex Chunking with NLTK
I am trying to figure out how to use NLTK's cascading chunker as per Chapter 7 of the NLTK book. Unfortunately, I'm running into a few issues when performing non-trivial chunking measures.
Let's ...
- 234
8
votes
1
answer
3k
views
Bag of Visual Words
What I am trying to do:
I am trying to classify some images using local and global features.
What I have done so far:
I have extracted sift descriptors for each image and I am using this as my ...
- 261
8
votes
1
answer
6k
views
how to compare different sets of time series data
I am trying to do some anomaly detection between time#series using Python and sklearn (but other package suggestions are definitely welcome!).
I have a set of 10 time-series; each time-series ...
8
votes
1
answer
9k
views
Multiple output for multi step ahead prediction using LSTM with keras
I am new to deep learning and LSTM (with keras). I am trying to solve a multi-step ahead time series prediction. I have 3 time series: A, B and C and I want to predict the values of C. I am training ...
- 211
8
votes
1
answer
2k
views
Recurrent neural network multiple types of input Keras
For a project I want to use recurrent neural networks, however my knowledge on this subject is still somewhat limited. I do have some experience with convolutional nets and traditional neural networks....
- 9,258
8
votes
1
answer
891
views
Gensim LDA model: return keywords based on relevance (λ - lambda) value
I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
- 81
8
votes
0
answers
2k
views
Python : Feature Matching + Homography to find Multiple Objects
I'm trying to use OpenCV via Python to find multiple objects in a train image and match it with the key points detected from a query image. For my case, I'm trying to detect the tennis courts in the ...
- 81
7
votes
4
answers
47k
views
How to download a Jupyter Notebook from GitHub?
This is a fairly basic question.
I am working on a data science project inside of a Pandas tutorial. I can access my Jupyter notebooks through my Anaconda installation. The only problem is that the ...
- 1,595
7
votes
4
answers
4k
views
Convert Numpy array with 'n' and 'y' into integer array of 0 and 1
I have a NumPy array of strings: 'n', 'y', wanna convert it into integer array of 0, 1, how to convert it?
...
- 81
7
votes
6
answers
3k
views
Is there any way to explicitly measure the complexity of a Machine Learning Model in Python
I'm interested in model debugging and one of the points that it mentions is to compare your model with a "less complex" one to check if the performance is substantially better on the most ...
- 2,371
7
votes
1
answer
12k
views
Summary statistics by category using Python
I have a datset with Scores and Categories and I would like to calculate the summary statistics for each of these categories. The data look something like this:
...
- 488
7
votes
2
answers
3k
views
Why Scikit and statsmodel provide different Coefficient of determination?
First of all, I know there is a similar question, however, I didn't find it so much helpful.
My issue is concerning simple Linear regression and the outcome of R-Squared. I founded that results can ...
- 173
7
votes
3
answers
2k
views
Are there any projects to work on the Internet to fight against COVID?
Are there any open source projects that a novice data analyst and mathematician can do, to fight against covid-19 epidemic? I mean, I know that the best I can do is to stay away from people and now I ...
7
votes
2
answers
5k
views
Struggling to integrate sklearn and pandas in simple Kaggle task
I'm trying to use the sklearn_pandas module to extend the work I do in pandas and dip a toe into machine learning but I'm struggling with an error I don't really understand how to fix.
I was working ...
- 233
7
votes
2
answers
32k
views
calculate distance between each data point of a cluster to their respective cluster centroids
I have a dataset of some keywords in some text files. Using the append feature I have access each text file and I append all of the keywords to token_dict like this
...
- 231
7
votes
2
answers
7k
views
Recommender system based on purchase history, not ratings
I'm exploring options for recommender systems optimized for the insurance industry, which would take into account
i) product holdings
ii) user characteristics (segment, age, affluence, etc.).
I ...
- 193
7
votes
5
answers
18k
views
Training Dataset for Sentiment Analysis of Movie Reviews
I am currently working on sentiment analysis using Python. I wanted to find whether reviews given for a movie is positive or negative based on sentiment analysis. I have found a training dataset as ...
- 1,045
7
votes
2
answers
19k
views
Trying to understand Logistic Regression Implementation
I'm currently using the following code as a starting point to deepen my understanding of regularized logistic regression. As a first pass I'm just trying to do a binary classification on part of the ...
- 73
7
votes
2
answers
37k
views
How to determine input shape in keras?
I am having difficulty finding where my error is while building deep learning models, but I typically have issues when setting the input layer input shape.
This is my model:
...
- 173
7
votes
1
answer
47k
views
How to add date column in python pandas dataframe
I want to add a date column (from 1/1/1979 upto the data is) in pandas data frame.
Currently, my data frame looks like this:
...
- 173
7
votes
3
answers
8k
views
How to convert nested list into a single list in python? [closed]
I have a list that contain id number. Some elements of the list is an another list. To convert nested list into a single list, I write a recursive function using collections module.
My codes are ...
- 79
7
votes
2
answers
19k
views
Which convolution should I use? Conv2d or Conv1d
I've dataset which contains dlib landmark points of the faces. I'm using keras to train a model.
The dataset shape is (length_of_dataset,68,2). I know that I've two ...
- 195
7
votes
2
answers
34k
views
7
votes
1
answer
28k
views
Validation loss increases and validation accuracy decreases
I have an issue with my model. I'm trying to use the most basic Conv1D model to analyze review data and output a rating of 1-5 class, therefore the loss is categorical_crossentropy. Model structure is ...
- 71
7
votes
2
answers
2k
views
Memory error when using more layers in CNN model
On my dell core i7 - 16GB RAM - 4gb 960m GPU laptop, I am working on a project to the classify lung CT images using 3d CNN. I'm using the CPU version of tensorflow. The images are prepared as numpy ...
- 1,127
7
votes
1
answer
3k
views
Feature agglomeration: Is it testing interactions?
I have been looking at feature agglomeration in Python's scikit-learn. According to the user guide, feature agglomeration "applies Hierarchical clustering to group together features that behave ...
- 95
7
votes
3
answers
7k
views
How to train a xgboost model on data that is too big for the memory?
What are the best practices to train xgboost (eXtreme gradient boosting) models on data that is to big to hold it in memory at once? Splitting the data and train multiple models? Are there more ...
- 694
7
votes
1
answer
12k
views
How to generate training data for OCR
I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to ...
7
votes
2
answers
13k
views
Plotting different values in pandas histogram with different colors
I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used <...
- 3,001
7
votes
1
answer
1k
views
How to numerically estimate MLE estimators in python when gradients are very small far from the optimal solution?
I am exploring how to model a data set using normal distributions with both mean and variance defined as linear functions of independent variables.
Something like N ~ (f(x), g(x)).
I generate a ...
- 171
7
votes
1
answer
693
views
Reticulate vs Python
I have gotten very used to coding in R and especially in RStudio. I like that interface. Nonetheless, I have some work that I ought to do in Python. I know that I can run Python code in RStudio if I ...
- 3,744