Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
14 votes
2 answers
4k views

Efficient dimensionality reduction for large dataset

I have a dataset with ~1M rows and ~500K sparse features. I want to reduce the dimensionality to somewhere in the order of 1K-5K dense features. ...
  • 3,850
14 votes
2 answers
8k views

How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries

Experts in my field are capable of predicting the likelyhood an event (binary spike in yellow) 30 minutes before it occurs. Frequency here is 1 sec, this view represents a few hours worth of data, i ...
  • 143
14 votes
1 answer
38k views

Heatmap on a map in Python

Mode Analytics has a nice heatmap feature, but it is not conducive to comparing maps (only one per report). What they do allow is data to be pulled easily into a wrapped python notebook. And then ...
  • 323
14 votes
2 answers
35k views

Reshaping of data for deep learning using Keras

I am a beginner to Keras and I have started with the MNIST example to understand how the library actually works. The code snippet of the MNIST problem in the Keras example folder is given as : ...
  • 3,001
14 votes
5 answers
47k views

Feature importance with scikit-learn Random Forest shows very high Standard Deviation

I am using scikit-learn Random Forest Classifier and I want to plot the feature importance such as in this example. However my result is completely different, in the sense that feature importance ...
  • 829
14 votes
2 answers
21k views

Why we should not feed LDA with tfidf

Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
  • 141
13 votes
2 answers
72k views

Perform k-means clustering over multiple columns

I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not ...
  • 141
13 votes
3 answers
3k views

What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not?

I spent the past year learning Python. As a person who thought coding was impossible to learn for those outside of the CS/IT sphere, I was obviously gobsmacked by the power of a few lines of Python ...
  • 143
13 votes
3 answers
29k views

Best frequent itemset package in python

Could anyone please recommend a good frequent itemset package in python? I only need to find frequent itemset, no need of finding the association rules.
  • 2,635
13 votes
3 answers
56k views

Replace all numeric values in a pyspark dataframe by a constant value

Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the ...
  • 131
13 votes
4 answers
3k views

Extract information from sentence

I'm creating a simple chatbot. I want to obtain the information from the user response. An example scenario: Bot : Hi, what is your name? User: My name is Edwin. ...
  • 333
13 votes
3 answers
2k views

An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs

I would like to use ANNs for my problem, but the issue is my inputs and outputs node numbers are not fixed. I did some google searches before asking my question and found that the RNN may help me with ...
  • 303
13 votes
4 answers
7k views

How do I make an interactive PCA scatterplot in Python?

The matplotlib library is very capable but lacks interactiveness, especially inside Jupyter Notebook. I would like a good offline plotting tool like plot.ly.
13 votes
1 answer
64k views

Expected 2D array, got scalar array instead

Can anyone help me with this error. I did the following code but it does not work and I am getting the following error: ...
  • 131
13 votes
2 answers
37k views

Validation loss and accuracy remain constant

I am trying to implement this paper on a set of medical images. I am doing it in Keras. The network essentially consists of 4 conv and max-pool layers followed by a fully connected layer and soft max ...
  • 1,022
13 votes
2 answers
7k views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a Word2Vec or Glove,...
  • 4,156
13 votes
2 answers
12k views

K-fold cross validation when using fit_generator and flow_from_directory() in Keras

I am using flow_from_directory() and fit_generator in my deep learning model, and I want to use cross validation method to train ...
  • 758
12 votes
5 answers
31k views

How to scrape imdb webpage?

I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage. I am using BeautifulSoup module. Following is the code I am ...
  • 1,057
12 votes
1 answer
14k views

XGBoost Linear Regression output incorrect

I am a newbie to XGBoost so pardon my ignorance. Here is the python code : ...
  • 223
12 votes
3 answers
20k views

Find the consecutive zeros in a DataFrame and do a conditional replacement

I have a dataset like this: Sample Dataframe ...
  • 533
12 votes
4 answers
52k views

Export pandas to dictionary by combining multiple row values

I have a pandas dataframe df that looks like this ...
  • 223
12 votes
1 answer
14k views

Tensorflow neural network TypeError: Fetch argument has invalid type

I am making a simple neural network using tensorflow, with data I collected myself, however, it's not cooperating :P I have run into an error that I can't fix or find the fix for and I would love your ...
12 votes
3 answers
6k views

Issue with IPython/Jupyter on Spark (Unrecognized alias)

I am working on setting up a set of VMs to experiment with Spark before I spend go out and spend money on building up a cluster with some hardware. Quick note: I am an academic with a background in ...
  • 121
12 votes
3 answers
18k views

Train new data to pre-trained model

Let's say I've trained my model and made my predictions. My question is... How can I append some new data to my pre-trained model without retrain the model from the beginning.
  • 163
12 votes
2 answers
34k views

cosine_similarity returns matrix instead of single value

I am using below code to compute cosine similarity between the 2 vectors. It returns a matrix instead of a single value 0.8660254. [[ 1. 0.8660254] [...
12 votes
4 answers
7k views

Prohibitive size of random forest when saved to disk

When saved to disk using cPickle: https://stackoverflow.com/questions/20662023/save-python-random-forest-model-to-file, my random forest is 6.57 GB. ...
  • 263
12 votes
2 answers
90k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
  • 251
12 votes
2 answers
7k views

Why is training take so long on my GPU?

Details: GPU: GTX 1080 Training: ~1.1 Million images belonging to 10 classes Validation: ~150 Thousand images belonging to 10 classes Time per Epoch: ~10 hours I've setup CUDA, cuDNN and Tensorflow( ...
  • 121
12 votes
1 answer
4k views

Can HDF5 be reliably written to and read from simultaneously by separate python processes?

I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using ...
  • 297
12 votes
4 answers
10k views

Neural networks - Find most similar images

I am working with Python, scikit-learn and keras. I have 3000 thousands images of front-faced watches like the following ones: Watch_1, Watch_2, Watch_3. I want to write a program which receives as ...
  • 1,027
12 votes
3 answers
6k views

Help regarding NER in NLTK

I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ...
  • 195
12 votes
2 answers
6k views

Python Machine Learning/Data Science Project Structure

I'm looking for information on how should a Python Machine Learning project be organized. For Python usual projects there is Cookiecutter and for R ProjectTemplate. This is my current folder ...
12 votes
1 answer
14k views

How to Predict the future values of time horizon with Keras?

I just built this LSTM neural network with Keras ...
  • 263
12 votes
1 answer
6k views

Train on batches in Tensorflow

I'm currently trying to train a model on a large csv file (>70GB with more than 60 million rows). To do so I'm using tf.contrib.learn.read_batch_examples. I'm struggling in understanding how this ...
  • 121
11 votes
2 answers
46k views

Pandas merge column duplicate and sum value [closed]

How to merge duplicate column and sum their value? What I have A 30 A 40 B 50 What I need A 70 B 50 DF for this ...
11 votes
3 answers
50k views

Keras Sequential model returns loss 'nan'

I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation ...
  • 365
11 votes
5 answers
5k views

Share Jupyter Notebook with a non programmer

I would like to share a Jupyter Notebook with somebody who is not a programmer (via a URL). I understand that I must publish my work to GitHub and then use http://nbviewer.jupyter.org/ to render the ...
  • 111
11 votes
3 answers
37k views

Xgboost - How to use feature_importances_ with XGBRegressor()?

How could we get feature_importances when we are performing regression with XGBRegressor()? There is something like ...
  • 683
11 votes
2 answers
8k views

Understanding scipy sparse matrix types

I am trying to select the best scipy sparse matrix type to use in my algorithm. In that, I should initialize data in a vij way, then I should use it to perform matrix vector multiplication. Eventually ...
11 votes
3 answers
92k views

How to group identical values and count their frequency in Python?

Newbie to analytics with Python so please be gentle :-) I couldn't find the answer to this question - apologies if it is already answered elsewhere in a different format. I have a dataset of ...
11 votes
3 answers
20k views

How to get the probability of belonging to clusters for k-means?

I need to get the probability for each point in my data set. The idea is to compute distance matrix (first column contsins distances to first cluster, second column conteins distances to second ...
11 votes
1 answer
33k views

Difference between interpolate() and fillna() in pandas

Since interpolate and fillna method does the same work of filling na values. What is the basic difference between the two. What is the significance of having these two different methods?? Can anyone ...
11 votes
3 answers
17k views

Increasing SpaCy max NLP limit

I'm getting this error: ...
  • 211
11 votes
1 answer
6k views

How does Naive Bayes classifier work for continuous variables?

I know that for categorical features we just calculate the prior and likelihood probability assuming conditional independence between the features. How does it work for continuous variables? How can ...
  • 901
11 votes
2 answers
9k views

Why is learning rate causing my neural network's weights to skyrocket?

I am using tensorflow to write simple neural networks for a bit of research and I have had many problems with 'nan' weights while training. I tried many different solutions like changing the optimizer,...
  • 113
11 votes
1 answer
16k views

Multiple Categorical values for a single feature how to convert them to binary using python

I have a data set of movies which has 28 columns. One of them is genres. For each row in this data set, the value for column genres is of the form "Action|Animation|Comedy|Family|Fantasy". I want to ...
  • 111
11 votes
1 answer
20k views

Keras LSTM with 1D time series

I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
11 votes
2 answers
12k views

How to perform Logistic Regression with a large number of features?

I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression. According to the "rule if ten" I need at least 10 events for each feature to be ...
11 votes
6 answers
28k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
  • 1,045
11 votes
1 answer
3k views

Why could my DDQN get significantly worse after beating the game repeatedly?

I've been trying to train a DDQN to play OpenAI Gym's CartPole-v1, but found that although it starts off well and starts getting full score (500) repeatedly (at around 600 episodes in the pic below), ...