Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
8
votes
2answers
60k views

How to delete entire row if values in a column are NaN [closed]

I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this: ...
8
votes
3answers
968 views

What to consider before learning a new language for data analysis

I'm currently in the very early stages of preparing a new research-project (still at the funding-application stage), and expect that data-analysis and especially visualisation tools will play a role ...
8
votes
3answers
8k views

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (GridSearchCV) finds the best ...
8
votes
2answers
9k views

Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set?

As I increase the number of trees in scikit learn's GradientBoostingRegressor, I get more negative predictions, even though there are no negative values in my ...
8
votes
4answers
11k views

How do we standardize arrays with NaN?

I used StandardScaler() to standardize data so far, but this doesn't work with NaNs. None of the other methods I know of (MinMaxScaler, RobustScaler, MaxAbsScaler) work with NaNs either. Are there ...
8
votes
4answers
2k views

How to replace NaN values for image data?

My data set has a total of 200 columns, where each column corresponds to the same pixel in all of my images. In total, I have 48,500 rows. The labels for the data range from 0-9. The data looks ...
8
votes
3answers
23k views

Best frequent itemset package in python

Could anyone please recommend a good frequent itemset package in python? I only need to find frequent itemset, no need of finding the association rules. Thanks!
8
votes
3answers
10k views

How to get the probability of belonging to clusters for k-means?

I need to get the probability for each point in my data set. The idea is to compute distance matrix (first column contsins distances to first cluster, second column conteins distances to second ...
8
votes
2answers
54k views

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. ...
8
votes
3answers
7k views

Word2Vec how to choose the embedding size parameter

I'm running word2vec over collection of documents. I understand that the size of the model is the number of dimensions of the vector space that the word is embedded into. And that different dimensions ...
8
votes
2answers
11k views

Using TF-IDF with other features in SKLearn

What is the best/correct way to combine text analysis with other features? For example, I have a dataset with some text but also other features/categories. SKlearn's TF-IDF vectoriser transforms text ...
8
votes
2answers
810 views

Pylearn2 vs TensorFlow

I am about to dive into a long NN research project and wanted a push in the direction of Pylearn2 or TensorFlow? As of Dec 2015 has the community started to lean one direction or another? This link ...
8
votes
1answer
436 views

Theano in deep learning research

How widely is Theano used in deep learning research? Is Theano a good start to learn the implementation of machine learning algorithms? Will learning the implementation of something like a feed ...
8
votes
1answer
7k views

How to train data by batch from disk?

I am working on a convolutional neural network for image classification. The training dataset is too large to be loaded on my computer memory (4gb), on top of that I also need to try some augmentation ...
8
votes
2answers
18k views

Counting indexes in pandas

I feel like this is a rudimentary question but I'm very new to this and just haven't been able to crack it / find the answer. Ultimately what I'm trying to do here is to count unique values on a ...
8
votes
1answer
8k views

Multiple Categorical values for a single feature how to convert them to binary using python

I have a data set of movies which has 28 columns. One of them is genres. For each row in this data set, the value for column genres is of the form "Action|Animation|Comedy|Family|Fantasy". I want to ...
8
votes
3answers
2k views

How the embedding layer is trained in Keras Embedding layer

How is the embedding layer trained in Keras Embedding layer? (say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext) Assume we do not use a pretrained embedding.
8
votes
5answers
19k views

DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20

I was watching Machine Learning A- Z from SuperDataScience but when I was doing below code sample: ...
8
votes
1answer
34k views

Am I doing a log transformation of data correctly?

I'm doing some exploratory data analysis on some data and I get these histograms: That looks like a candidate for a log transformation on the data, so I run the following Python code to transform the ...
8
votes
2answers
2k views

Why does Q Learning diverge?

My Q-Learning algorithm's state values keep on diverging to infinity, which means my weights are diverging too. I use a neural network for my value-mapping. I've tried: Clipping the "reward + ...
8
votes
1answer
7k views

Recognition human in images through HOG descriptor and SVM classifier performs poorly

I'm using a HOG descriptor, coupled with a SVM classifier, to recognise humans in pictures. I'm using the Python wrappers for OpenCV. I've used the excellent tutorial at pymagesearch, which explains ...
8
votes
4answers
9k views

How to run a pyspark application in windows 8 command prompt

I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] ...
8
votes
3answers
4k views

ReLU vs sigmoid in mnist example

PLEASE NOTE: I am not trying to improve on the following example. I know you can get over 99% accuracy. The whole code is in the question. When I tried this simple code I get around 95% accuracy, if I ...
8
votes
2answers
9k views

Why we should not feed LDA with tfidf

Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
8
votes
1answer
420 views

Questions When Advancing from Vanilla Neural Network to Recurrent Neural Network

I've recently learned how a vanilla neural network would work, with given number of inputs, hidden nodes, and the same number of outputs as inputs. I've been looking at various posts now related to ...
8
votes
1answer
374 views

How far can one go with excel? [closed]

in my business we handle all analytics through Excel. This includes mostly scheduling, production planning and accounting operations. We currently are looking into adding a bit of predictive modelling ...
8
votes
2answers
3k views

LSTM: How to deal with nonstationarity when predicting a time series

I want to do one-step-ahead predictions for time series with LSTM. To understand the algorithm, I built myself a toy example: A simple autocorrelated process. ...
8
votes
1answer
4k views

Complex Chunking with NLTK

I am trying to figure out how to use NLTK's cascading chunker as per Chapter 7 of the NLTK book. Unfortunately, I'm running into a few issues when performing non-trivial chunking measures. Let's ...
8
votes
1answer
4k views

how to compare different sets of time series data

I am trying to do some anomaly detection between time#series using Python and sklearn (but other package suggestions are definitely welcome!). I have a set of 10 time-series; each time-series ...
8
votes
1answer
1k views

sklearn - overfitting problem

I'm looking for recommendations as to the best way forward for my current machine learning problem The outline of the problem and what I've done is as follows: I have 900+ trials of EEG data, where ...
7
votes
2answers
35k views

Cross Validation in Keras

Suppose I would like to train and test the MNIST dataset in Keras. The required data can be loaded as follows: ...
7
votes
3answers
2k views

Are there any projects to work on the Internet to fight against COVID?

Are there any open source projects that a novice data analyst and mathematician can do, to fight against covid-19 epidemic? I mean, I know that the best I can do is to stay away from people and now I ...
7
votes
2answers
4k views

Understanding scipy sparse matrix types

I am trying to select the best scipy sparse matrix type to use in my algorithm. In that, I should initialize data in a vij way, then I should use it to perform matrix vector multiplication. Eventually ...
7
votes
3answers
15k views

Algorithms and techniques for spell checking

Can anyone suggest any algorithm and technique for spell checking? After some googling, I found some interesting ones such as this one from Peter Norvig, http://norvig.com/spell-correct.html and few ...
7
votes
2answers
5k views

Struggling to integrate sklearn and pandas in simple Kaggle task

I'm trying to use the sklearn_pandas module to extend the work I do in pandas and dip a toe into machine learning but I'm struggling with an error I don't really understand how to fix. I was working ...
7
votes
4answers
17k views

How to use SimpleImputer Class to replace missing values with mean values using Python?

This is my code ...
7
votes
2answers
736 views

Transform an Autoencoder to a Variational Autoencoder?

I would like to compare the training by an Autoencoder and a variational autoencoder. I have already run the traing using AE. I would like to know if it's possible to transform this AE into a VAE and ...
7
votes
1answer
18k views

Resume Parsing - extracting skills from resume using Machine Learning

I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?...
7
votes
1answer
19k views

cosine_similarity returns matrix instead of single value

I am using below code to compute cosine similarity between the 2 vectors. It returns a matrix instead of a single value 0.8660254. [[ 1. 0.8660254] [...
7
votes
2answers
5k views

Recommender system based on purchase history, not ratings

I'm exploring options for recommender systems optimized for the insurance industry, which would take into account i) product holdings ii) user characteristics (segment, age, affluence, etc.). I ...
7
votes
2answers
18k views

Trying to understand Logistic Regression Implementation

I'm currently using the following code as a starting point to deepen my understanding of regularized logistic regression. As a first pass I'm just trying to do a binary classification on part of the ...
7
votes
3answers
13k views

Does Tensorflow support a Decision Tree Classifier?

I am trying to implement decision tree classifier to classify my data set. I am using Python. Now it is easy to implement in scikit learn, but how can I implement this in tensorflow.
7
votes
1answer
20k views

How does class_weights work in RandomForestClassifier

I'm facing a problem with unbalanced classes, and have tried out a couple of methods like over and under sampling. However, my cross validation mean comes out to be only 0.4 and my confusion matrix ...
7
votes
3answers
6k views

How to convert nested list into a single list in python? [closed]

I have a list that contain id number. Some elements of the list is an another list. To convert nested list into a single list, I write a recursive function using collections module. My codes are ...
7
votes
2answers
10k views

Which convolution should I use? Conv2d or Conv1d

I've dataset which contains dlib landmark points of the faces. I'm using keras to train a model. The dataset shape is (length_of_dataset,68,2). I know that I've two options. The first is using ...
7
votes
2answers
6k views

Why `max_features=n_features` does not make the Random Forest independent of number of trees?

Consider the following simple classification problem (Python, scikit-learn) ...
7
votes
3answers
608 views

Find if the two datasets are close to each other

I have the following three datasets. ...
7
votes
2answers
815 views

What are the disadvantages of having a left skewed distribution?

I'm currently working on a classification problem and I've a numerical column which is left skewed. i've read many posts where people are recommending to take log transformation or boxcox ...
7
votes
2answers
3k views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
7
votes
2answers
1k views

Memory error when using more layers in CNN model

On my dell core i7 - 16GB RAM - 4gb 960m GPU laptop, I am working on a project to the classify lung CT images using 3d CNN. I'm using the CPU version of tensorflow. The images are prepared as numpy ...

1 2 3
4
5
93