Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
1
vote
1answer
2k views

Segmenting pandas dataframe with lists as elements

I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. ...
1
vote
1answer
498 views

Why is patsy used to prepare data for logistic regression?

I'm pretty new to both ML & scikit-learn. I've noticed that some example tutorials & codes online use patsy's dmatrices to prepare data for logistic regression. I don't understand why this is ...
1
vote
1answer
974 views

How to visualize convolutional network focus on autoencoder?

I'm trying to create some cool (and at least somewhat meaningful) visualizations of my CNN that I am using to autoencode images, as I use the encoded middle layer to do similarity comparisons. I.e. I ...
5
votes
1answer
84 views

How to Deploy your trained ML model in client VM without they getting access to code

I am new to deployment and have a basic doubt about deploying my ML code on client's vm. So I have built a python project which collects data from client site, processing, predicts and displays the ...
1
vote
1answer
699 views

How to determine best parameters after grid search?

I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can ...
6
votes
1answer
2k views

How to make k-means distributed?

After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code: ...
6
votes
1answer
529 views

Why could my DDQN get significantly worse after beating the game repeatedly?

I've been trying to train a DDQN to play OpenAI Gym's CartPole-v1, but found that although it starts off well and starts getting full score (500) repeatedly (at around 600 episodes in the pic below), ...
6
votes
2answers
2k views

What are the alternatives to Python + Spark (pyspark)?

I like Python, and I like Spark, but they don't fit very well together. In particular, it is very hard to use python functions in spark (have to create JVM binding for function in python) it is hard ...
6
votes
5answers
969 views

How can we extract fields from images?

I am making an document parser which extracts data fields from the documents and store them in a structured way. Each field in my dataset is horizontal which is easy to extract. But the model fails ...
2
votes
2answers
27 views

Resources for text classification algorithms

I'm mining raw Facebook comments (irrelevant) and i am looking for an algorithm that can classify their context as negative/positive/neutral. So you can think of the output in the form of two columns. ...
1
vote
0answers
1k views

SOM initial values for learning rate and neighborhood sigma

I am using SOM (Self-Organizing Maps) of Kohonen, or more specifically, the MiniSom, found here to cluster and visualize my data. As you can see in the above site, the example given is: ...
0
votes
1answer
29 views

Bootstrap clarification for datasets with multiple columns

How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns? For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with ...
1
vote
0answers
100 views

Keras Error: An operation has `None` for gradient

Lets assume, that you want to build a GAN in keras. The generator transforms an image from ...
-2
votes
0answers
7 views
0
votes
0answers
11 views

Why is scipy.optimize.curve_fit so complicated to use? [closed]

I would like to do some nonlinear curve-fitting on 2D data using scipy.optimize.curve_fit following this tutorial and find the steps required in order to "feed&...
3
votes
3answers
23k views

SMOTE and multi class oversampling

I have read that the SMOTE package is implemented for binary classification. In the case of n classes, it creates additional examples for the smallest class. Can I balance all the classes by running ...
0
votes
1answer
3k views

how to export the tables into a csv file pandas

The following is a piece of code I wrote to create a pivot table for categorical vs continuous variable. ...
1
vote
1answer
26 views

Multiple linear regression for multi-dimensional input and output?

Assume that I have $N$ points $x_i,i=1,...,N$ in some $A>1$-dimensional space $\mathbb{R}^A$ with pointwise evaluations of some function $f:\mathbb{R}^A \rightarrow \mathbb{R}^B$, i.e. $f(x_i),i=1,...
3
votes
0answers
21 views

Why Does XGBoost Keep One Feature at High Importance?

I am training an XGboost model for binary classification on around 60 sparse numeric features. After training, the feature importance distribution has one feature with importance > 0.6, and all the ...
0
votes
0answers
14 views

Feature-Target correlation graphs [closed]

I have a set of features and labels with the following dimensions: where axis=0 corresponds to the number of pixels and axis=1 corresponds to the features and time, for example, 17=117 where 17 is the ...
-2
votes
0answers
16 views

how much mean and standard deviation is required in order to get signal-to-noise ratio of 10 decibel [closed]

i need to add Gaussian noise on a given time series in order to make the signal-to-noise ratio of noisy time-series to be 10decibel.So in general, for adding the Gaussian noise to a time series we ...
0
votes
0answers
13 views

Folder of images in Google Drive to array in Google Colab [closed]

How do I get an array of images in Google Colab using a folder of images I have in Google Drive? I have already mounted Google Drive locally.
1
vote
0answers
498 views

Input format of label images (ImageDataGenerator) for multi-class semantic image segmentation in keras

I wondered if you could help me (and hopefully others too) to understand how to use keras' ImageDataGenerator to load in my ...
0
votes
3answers
6k views

Sklearn ValueError: X has 2 features per sample; expecting 11

I try to visualizing multiple logistic regression but I get the above error. I'm practicing on red wine quality data set from kaggle. Here is a full traceback: ...
1
vote
0answers
258 views

Ensure class balanced batches while hyperparameter tuning keras models with grid search

Ensuring class balanced batches while training keras models is possible using fit_generator method. I used imblearn.keras.BalancedBatchGenerator for that and it works fine! But i wanted to do that ...
1
vote
3answers
62 views

Is it a best practice to exclude retweets from the data set?

I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is ...
3
votes
1answer
46 views

Find non overlapping area between two kde plots in python

I was attempting to determine whether a feature is important or not base on its kde distribution for target variable. I am aware how to plot the kde plot and guess after looking at the plots, but is ...
13
votes
4answers
17k views

seaborn heatmap not displaying correctly

for some reason, my heatmap is not displaying correctly anymore! It was working just fine even with 6 classes. Since the last time I used it, I've installed many packages ( including plotly), I don't ...
2
votes
1answer
5k views

Stacked time series plot in python

In pandas I can set the date as index, and then run df.plot() to see a line chart. How do I make that line chart stacked as in the picture below?
1
vote
1answer
406 views

How should I construct a binary classifier for thousands of positive data and millions of unlabeled data?

So far, I have stumbled upon many advices and papers on PU Learning and Unary classification. TLDR: Does anyone have suggestions for specific algorithm or implementation for labeled data of only one ...
0
votes
1answer
161 views

Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data

I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with ...
0
votes
1answer
31 views

activation function for binarized neural networks

I am trying to implement a binarized neural network using keras and tensorflow. Here is my current code: ...
0
votes
0answers
5 views

Why it is reshaped the last layers of VGG_UNet segmentation model?

I want to solve a multiclass segmentation task using deep learning (in python). Here, is a summary of vgg_unet model that is mainly collected from GitHub. So, in my dataset 8 labels are available. So, ...
0
votes
1answer
30 views

Support Vector Machine (SVM) for classification problem based on Earth Mover's Distance (EMD)

I would like to run SVM for my classification problem using the Earth Mover's Distance (EMD) as a distance measurement. As I understood the documentation for Python scikit-learn (https://scikit-learn....
0
votes
0answers
14 views

Building a content-based recommendation system using products' metadata as features?

I am currently working on an apparel recommendation system, where I have tabulated data containing a list of products with their respective metadata (brand, category, color etc.) I have an additional ...
0
votes
0answers
14 views

How to apply an algorithm for text classification?

I would like to know more about algorithms to apply for text classifications. I know that steps include also remove of punctuation and stop words and tokenisation to clean the text. If I already ...
0
votes
2answers
101 views

1D convolution for uni-variate data

Every one. I have EEG dataset with 80 subjects, 3072 data points and 100 trials. This a univariate data, it mean there is only one channel. I am confused how to feed this data to convolution neural ...
-2
votes
1answer
22 views

NotFittedError says this StandardScaler instance is not fitted yet while using inverse_transform() [closed]

I have a dataset and i have used Support Vector Regression.So i needed to use StandardScaler module from sklearn.preprocessing fro Feature Scaling. After training my model when i came to predict it ...
0
votes
1answer
15 views

Model for predicting duration based on categorical data

I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all ...
0
votes
0answers
6 views

Can I use LSI (Latent Semantic indexing) to get similar docs for several documents at the same time?

I'm working on a Recommander system in which I'm using LSI to get similarities between videos. I wonder if I can provide to LSI matrix more than one document and get similar docs for all those. In the ...
9
votes
3answers
2k views

How the embedding layer is trained in Keras Embedding layer

How is the embedding layer trained in Keras Embedding layer? (say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext) Assume we do not use a pretrained embedding.
9
votes
3answers
4k views

ReLU vs sigmoid in mnist example

PLEASE NOTE: I am not trying to improve on the following example. I know you can get over 99% accuracy. The whole code is in the question. When I tried this simple code I get around 95% accuracy, if I ...
1
vote
1answer
24 views

Extracting information with corresponding fields

I have large pool of scanned county documents. I need to extract information like document title, borrower name&address, lender name&address etc. The text is like this Eg: the deed of trust,...
11
votes
3answers
1k views

An Artificial Neuron Network (ANN) with an arbitrary number of inputs and outputs

I would like to use ANNs for my problem, but the issue is my inputs and outputs node numbers are not fixed. I did some google search before asking my question and found that the RNN may help me ...
3
votes
1answer
540 views

SVR is giving same prediction for all features

I'm creating a basic application to predict the 'Closing' value of a stock for day n+1, given features of stock n using Python and Scikit-learn A sample row in my dataframe looks like this (2000 rows) ...
2
votes
1answer
346 views

memory problem due to large file

I'm new in Python and definitely, I'm sure I do some mistake. Here is my problem and thanks all to your help in advance. I have 2 files (one is Hive) and the other is csv and merging them. I have ...
2
votes
1answer
605 views

Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Reference I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning? Also, what is learning ...
2
votes
1answer
23 views
3
votes
1answer
804 views

Python: Fast indexing of strings in nested list without loop

I have an array dataset from which I need to build D whose each element is a list of strings (Third column of dataset). The dataset looks like: ...

1
2
3 4 5
95