Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
7
votes
2answers
1k views

Memory error when using more layers in CNN model

On my dell core i7 - 16GB RAM - 4gb 960m GPU laptop, I am working on a project to the classify lung CT images using 3d CNN. I'm using the CPU version of tensorflow. The images are prepared as numpy ...
7
votes
1answer
16k views

What's a good Python HMM library? [duplicate]

I've looked at hmmlearn but I'm not sure if it's the best one.
7
votes
1answer
10k views

How to generate training data for OCR

I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to ...
7
votes
2answers
4k views

Is there a library that would perform segmented linear regression in python?

There is a package named segmented in R. Is there a similar package in python?
7
votes
1answer
1k views

How to numerically estimate MLE estimators in python when gradients are very small far from the optimal solution?

I am exploring how to model a data set using normal distributions with both mean and variance defined as linear functions of independent variables. Something like N ~ (f(x), g(x)). I generate a ...
7
votes
1answer
14k views

Simple example of genetic alg minimization

I have been looking for a while for examples of how I could find the points at which a function achieves its minimum using a genetic algorithm approach in Python. I looked at DEAP documentation, but ...
7
votes
2answers
3k views

Applying dimensionality reduction on OneHotEncoded array

I have a really large data set with mixed variables. I have converted categorical variables to numerical using OneHotEncoding and it has resulted in more than a ...
7
votes
1answer
5k views

Prepending Input layer to pre-trained model

I'm trying to input numpy arrays of shape (1036800,) - originally images of shape (480, 720, 3) - into a pre-trained VGG16 model to predict continuous values. I've tried several variations of the ...
7
votes
2answers
621 views

Coursera ML - Does the choice of optimization algorithm affect the accuracy of multiclass logistic regression?

I recently completed exercise 3 of Andrew Ng's Machine Learning on Coursera using Python. When initially completing parts 1.4 to 1.4.1 of the exercise, I ran into difficulties ensuring that my ...
7
votes
3answers
2k views

Convolutional Neural Network not learning EEG data

I have trained a simple CNN (using Python + Lasagne) for a 2-class EEG classification problem, however, the network doesn't seem to learn. loss does not drop over epochs and classification accuracy ...
7
votes
2answers
1k views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
7
votes
2answers
344 views

How does real world machine learning production systems run?

Dear Machine Learning/AI Community, I am just a budding and aspiring Machine Learner who has worked on open online data sets and some POC's built locally for my project. I have built some models and ...
7
votes
1answer
5k views

Which type auto encoder gives best results for text

I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to ...
7
votes
1answer
302 views

How to decide how many n_neighbors to consider while implementing LocalOutlierFactor?

I have a data set with rows: 134000 and columns: 200. I am trying to identify the outliers in data set using LocalOutlierFactor from scikit-learn. Although I ...
7
votes
1answer
64 views

How to decide the shape of input features, when each data file is of different length?

To help me understand the benefits and shortcomings of decision trees, KNN, Neural Networks, ...
7
votes
2answers
62k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
7
votes
1answer
241 views

On a multi lingual sentiment corpus

I am looking to compile a sentiment corpus for news articles in multiple languages (~100k per lang. for a machine learning experiment) where each article is labeled positive, neutral, or negative. I ...
7
votes
1answer
2k views

Bag of Visual Words

What I am trying to do: I am trying to classify some images using local and global features. What I have done so far: I have extracted sift descriptors for each image and I am using this as my ...
7
votes
2answers
163 views

migrating to python from R: specific questions

I have been using R and RStudio for prototyping and model building and due to some persisting problems (which would only be applicable to the environment that I am using in) we have decided to use ...
7
votes
1answer
7k views

Multiple output for multi step ahead prediction using LSTM with keras

I am new to deep learning and LSTM (with keras). I am trying to solve a multi-step ahead time series prediction. I have 3 time series: A, B and C and I want to predict the values of C. I am training ...
7
votes
1answer
896 views

Custom metrics for unbalanced classes problem in RandomForest or SVM

My dataset has highly unbalanced classes ‒ foreground of 30 classes with tens of samples against background set of >100k samples. Classifying foreground class as background is quite OK, while ...
7
votes
0answers
553 views

Layman's Interpretation of XGBoost Importance [duplicate]

I'm trying to come up with a good way to explain the 3 importance metrics (Gain, Cover, Frequency) to a layman with only a basic understanding of XGBoost and trees in general. How best would you ...
6
votes
4answers
32k views

How to plot multiple variables with Pandas and Bokeh

I'm new to Pandas and Bokeh; I'd to create a bar plot that shows two different variables next to each other for comparison. For instance, with the following Pandas data frame, I'd like to see how ...
6
votes
1answer
9k views

Summary statistics by category using Python

I have a datset with Scores and Categories and I would like to calculate the summary statistics for each of these categories. The data look something like this: ...
6
votes
3answers
21k views

Xgboost - How to use feature_importances_ with XGBRegressor()?

How could we get feature_importances when we are performing regression with XGBRegressor()? There is something like ...
6
votes
5answers
677 views

Decision tree with final decision being a linear regression

Question: I want to implement a decision tree with each leaf being a linear regression, does such a model exist (preferable in sklearn)? Example case 1: Mockup data is generated using the formula: <...
6
votes
3answers
22k views

Split a list of values into columns of a dataframe?

I am new to python and stuck at a particular problem involving dataframes. The image has a sample column, however the data is not consistent. There are also some floats and NAN. I need these to be ...
6
votes
3answers
89k views

Dataframe has no column names. How to add a header?

I am using a dataset to practice for building a decision tree classifier. Here is my code: ...
6
votes
3answers
8k views

Train new data to pre-trained model

Let's say I've trained my model and made my predictions. My question is... How can I append some new data to my pre-trained model without retrain the model from the beginning.
6
votes
4answers
12k views

Tool to Label Images for Supervised Classification

I have a couple thousand photos of whales taken from drones and I'm planning to build a simple binary classifier to run on these and future images to see if they contain a whale. I'd like to label ...
6
votes
2answers
13k views

Python or R for implementing machine learning algorithms for fraud detection [closed]

I was wondering which language can I use: R or Python, for my internship in fraud detection in an online banking system: I have to build machine learning algorithms (NN, etc.) that predict transaction ...
6
votes
4answers
18k views

How to use SimpleImputer Class to replace missing values with mean values using Python?

This is my code ...
6
votes
3answers
3k views

What is normalization for?

I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do? ...
6
votes
2answers
16k views

Compare image similarity in Python

I'm using a dataset of movies and would like to group if a movie is the same across different retailers. Example: Movie: Beauty and the Beast Platforms: Google, Netflix, iTunes, Amazon. I have ...
6
votes
4answers
17k views

Training Dataset for Sentiment Analysis of Movie Reviews

I am currently working on sentiment analysis using Python. I wanted to find whether reviews given for a movie is positive or negative based on sentiment analysis. I have found a training dataset as ...
6
votes
1answer
7k views

Cosine Distance > 1 in scipy

I am working on a recommendation engine, and I have chosen to use SciPy's cosine distance as a way of comparing items. I have two vectors: ...
6
votes
2answers
6k views

Using cross-validation technique for a CNN model?

I am working on the CNN model, as always I use batches with epochs to train my model, for my model, when it completed training and validation, finally I use a test set to measure the model performance ...
6
votes
2answers
96 views

Can we specify the number of data generated(minority class) using SMOTE?

I am trying to improve classification of imbalanced dataset creditcard fraud using SMOTE imbalanced_learn. But, in this it generates the data to 50%, can we give a specific number for the data to be ...
6
votes
1answer
26k views

Number of features of the model must match the input. Model n_features is `N` and input n_features is `X`.

I am new to data science and trying get some results. I'm applying Decision Tree Classifier. When my train and test datasets' size are not equal I get an error `...
6
votes
2answers
22k views

Keras - no prediction probability for multiple output models?

I have built the following model: ...
6
votes
2answers
6k views

Predicting probability from scikit-learn SVC decision_function with decision_function_shape='ovo'

I have a multiclass SVM classifier with labels 'A', 'B', 'C', 'D'. This is the code I'm running: ...
6
votes
1answer
6k views

How to implement global contrast normalization in python?

I am trying to implement global contrast normalization in python from Yoshua Bengio's Deep Learning book (section 12.2.1.1 pg. 442). From the book, to get a normalized image using global contrast ...
6
votes
2answers
4k views

Image clustering by similarity measurement (CW-SSIM)

I'm trying to use scikit-learn and pyssim for clustering a set of images - less than 100. The end goal is to place the images into several buckets (clusters) according to the calculated similarity ...
6
votes
1answer
1k views

In elbow curve how to find the point from where the curve starts to rise?

I am computing a distance metric on my data. The result is then being sorted in ascending order. The samples having distance more than a specific threshold are to be marked as outliers and will be ...
6
votes
1answer
2k views

When does decision tree perform better than the neural network?

I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
6
votes
3answers
6k views

Clarification on the Keras Recurrent Unit Cell

I paste below the Keras documentation on Recurrent layer ...
6
votes
2answers
50k views

How can I fill NaN values in a Pandas DataFrame in Python?

I am trying to learn data analysis and machine learning by trying out some problems. I found a competition "House prices" which is actually a playground competition. Since I am very new to this ...
6
votes
1answer
2k views

Why after adding categorical data the Linear Regression fails?

Based on a training set we applied a simple Linear Regression on some attributes that all were numeric. Now we have more attributes in terms of categories and of course we applied one-hot-encoding to ...
6
votes
1answer
151 views

Reticulate vs Python

I have gotten very used to coding in R and especially in RStudio. I like that interface. Nonetheless, I have some work that I ought to do in Python. I know that I can run Python code in RStudio if I ...
6
votes
3answers
129 views

What is the best algorithm/solution for predicting the following?

I have a dataset that comprises 76 countries, and 6 columns of distinct quantitative variables, which are the mean values of that variable relative to each country: If I were to take a random sample ...

1
3 4
5
6 7
93