Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
4
votes
2answers
16k views

calculate distance between each data point of a cluster to their respective cluster centroids

I have a dataset of some keywords in some text files. Using the append feature I have access each text file and I append all of the keywords to token_dict like this ...
10
votes
2answers
3k views

Why is training take so long on my GPU?

Details: GPU: GTX 1080 Training: ~1.1 Million images belonging to 10 classes Validation: ~150 Thousand images belonging to 10 classes Time per Epoch: ~10 hours I've setup CUDA, cuDNN and Tensorflow( ...
3
votes
2answers
6k views

Clustering Data to learned cluster

I am new to data science, I have clustered some data using Scipy agglomerative clustering. how can I fit new data into the learned clusters? ...
2
votes
1answer
58 views

Unknown return value in CNTK 103A

I came across this statement from CNTK 103A: gzfname, h = urlretrieve(src, './delete.me') I understand all the rest but that ...
5
votes
1answer
653 views

Custom objective function in xgboost for Regression

I am working on a regression problem, where I want to modify the loss function in xgboost library such that my predictions should never be lesser than the actual value. I have written this code: <...
2
votes
1answer
255 views

Facing a difficult regular expression issue in cleaning text data

I am trying to substitute a sequence of words with some symbols from a long string appearing in multiple documents. As an example, suppose I want to remove: ...
1
vote
1answer
159 views

How do i use the Gaussian function with a Naive Bayes Classifier?

I would like some help with the maths and to check I have understood the algorithm correctly. So I’ve been learning off of a video and I tried my own example. At the end of it I got dodgy results; I ...
0
votes
2answers
53 views

Classify text labels in to a similar category [closed]

I'm trying to classify same kind of text labels in to one category. For example, if I have labels like qty, quantity, qty_no all of them should direct to Quantity. ...
4
votes
1answer
853 views

Accuracy of word and sent tokenize versus custom tokenizers in nltk

The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and ...
1
vote
1answer
1k views

Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set. I am trying to build a ...
32
votes
2answers
39k views

Merging two different models in Keras

I am trying to merge two Keras models into a single model and I am unable to accomplish this. For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use ...
0
votes
1answer
520 views

Replacing the feature variables with encoded variables

I have several nominal variables which I've encoded using the LabelEncoder() function. Now I want to replace the encoded values in the place of the raw datas of the ...
3
votes
3answers
7k views

sklearn : missing pruning for decision trees

Why pruning is not currently supported in scikit-learn? How can we tune the decision trees to make a workaround?
1
vote
0answers
81 views

statistical significance test between binary label features

I have 667 features and I want to find features that have a significant boundary between a binary class label before I apply a classification model (e.g Naive Bayes/ SVM) to improve classification ...
1
vote
1answer
2k views

How to Replace an object in Pandas array using replace with dictionary from excel file?

suppose i have a data like this and i want to change the value of name and gender into an integer, and i have a dictionary like this i'm understand that i can use replace function in pandas, but i ...
3
votes
1answer
2k views

How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]

I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS. I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
2
votes
1answer
118 views

When using numerical duplicates for categorical data, new columns should be added or values be converted?

If this is the dataframe, in pandas ...
3
votes
1answer
1k views

python sklearn decision tree classifier feature_importances_ with feature names when using continuous values [closed]

I'm using sklearn Decision Tree Classifier with some continuous features. When I run export_graphviz I see the same features in more than one nodes and with different values. Example: I would like to ...
1
vote
1answer
1k views

Simple way to visualise word2vec vector space

I want to visualize my word2vec vector space (with zoom in and zoom out). I found a really interesting GitHub project named word2vec explorer. However, it does not seem to work in windows and mac. ...
0
votes
1answer
8k views

How to print feature names in conjunction with feature Importance using Imbalanced-learn library?

I used BalancedBaggingClassifier from imblearn library to do an unbalanced classification task. How can I get feature improtance of the estimator in conjunction ...
0
votes
1answer
2k views

Agglomerative Hierarchial Clustering in python using DTW distance

I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (...
1
vote
0answers
32 views

I want to learn how to construct data science packages on top of core packages. Is there a list of excellent data science packages I can learn from? [closed]

Short question I want to learn how to construct data science packages on top of core packages. Is there a list of excellent data science packages I can learn from? Long question I recently came ...
0
votes
1answer
4k views

Keras Neural Network training is stuck (gets stuck around epoch 6) [closed]

I have a training dataset with 8000 rows, and I am trying to train a Keras Neural Network on it, using 100 epochs. However, the training process gets stuck around epoch 6 every time, as shown below. I'...
0
votes
1answer
153 views

Why there is two output in Titanic case in tflearn quickstart?

I am stuck in the tflearn quickstart guide. I've tried tit and it works perfectly with accuracy around 78%, but the problem is I don't understand why the "labels" consist of two "survived" columns ...
9
votes
1answer
27k views

Difference between interpolate() and fillna() in pandas

Since interpolate and fillna method does the same work of filling na values. What is the basic difference between the two. What is the significance of having these two different methods?? Can anyone ...
0
votes
1answer
59 views

allocation to customers [closed]

My data set contains the pickup latitude, pickup longitude of the customer and drivers current location latitude and longitude. My aim is to distribute a set of M or M-x (in the case where shortage ...
5
votes
1answer
2k views

Feature agglomeration: Is it testing interactions?

I have been looking at feature agglomeration in Python's scikit-learn. According to the user guide, feature agglomeration "applies Hierarchical clustering to group together features that behave ...
1
vote
1answer
165 views

Find threshold in large dataset

I have a problem. I have a data set with some users and their ratings in several movies. The movies are separated into 19 genres. I want to cluster the users by their preferences(ratings in the ...
1
vote
1answer
635 views

What affect will replacing words with bigrams have on TfIDF?

Say I have a corpus of text documents on which I have calculated each documents TfIDF vector. With this sparse matrix representation of the corpus, I can calculate similarities between documents by ...
1
vote
0answers
253 views

How to tune the hyper-parameters of an estimator in Orange Tool

Hi I want to tune/search hyper-parameters of SVM in Orange tool. How can I do? I want to apply Exhaustive Grid Search to SVM http://scikit-learn.org/stable/modules/grid_search.html
6
votes
2answers
22k views

Keras - no prediction probability for multiple output models?

I have built the following model: ...
1
vote
2answers
365 views

Text classification problem using Python or R

I am a novice in machine learning and new to NLP. I am looking for ideas on how to solve the below two problems. I have a dataset with two columns, "Titles" and "Description". Titles column has names ...
2
votes
1answer
125 views

How can I extract numerical information conditional on other information?

Given text data, I'd like to extract numerical information based on the presence of other information. For instance, "There are 119 people in the gym right now. 100 people in the weight room, 19 on ...
4
votes
1answer
41k views

How to create an array from the list of arrays in python [closed]

I was trying to write a python code that can set some neural network channels or neurons to zero at the inference; and I wrote the code below. The code generates 10 different arrays for different ...
2
votes
1answer
311 views

Customized function for Agglomerative Clustering

I need to use a customized linkage function for my project. For this function, on every step I should calculate the objective containing similarity (distance) between subclusters to be merged ...
0
votes
1answer
2k views

How to correctly infer vectors in Gensim doc2vec?

I would like to know which is the correct procedure for inferring vectors in Gensim doc2vec. I have a dataframe df with a feature, called ...
1
vote
3answers
5k views

Grouping of similar looking text

I have a data frame which has two columns, "Title" and "Description". The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same ...
3
votes
1answer
370 views

How is that possible that a reward function depends both on the next state and an action from current state?

There's the concept of "expected value of the next reward", often denoted as $\mathcal{R}^a_{ss'}$, and defined as $$ \mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1} \mid s_t = s, a_t = a, s_{t+1} = s'...
4
votes
1answer
59k views

AttributeError: 'numpy.ndarray' object has no attribute 'predict'

I have trained and saved a model : ...
1
vote
0answers
45 views

PCA Reduction resulted in an elliptical form

I have a dataset with 19 features (columns). I normalized them using sklearn.preprocessing.normalize then I used PCA to reduce them to 2 components for plotting ...
1
vote
1answer
335 views

Is there a quick way to speed up ICP in python using a cached KD-tree

I am currently using ICP to match 2 point clouds. These point clouds evolve in time, so I have to repeat this process many times. I am using a standard KD tree from scipy for my nearest neighbor ...
11
votes
1answer
86k views

How to plot two columns of single DataFrame on Y axis

i have two DataFrames(Action,Comedy). Action contains two columns(year,rating) ratings columns contains average rating with respect to year. Comedy Dataframe contains same two columns with different ...
1
vote
1answer
4k views

How to select features based on feature importance using SelectFromModel?

I would appreciate if you could let me know how to select features based on feature importance using SelectFromModel. I wrote: ...
1
vote
0answers
267 views

How Do I Perform Gradient Descent for Discrete Predictions [closed]

I have a dataframe, df, that is of the format: Dimension, 1, 2, ..., 55 Dim1, 0.22, 0.152, ..., 0.10 ... Dim20, 0.52, 0.89, ..., 0.09 And I have a training set: participant_1 :{Dimension: Dim12, ...
2
votes
1answer
1k views

Transfer learning within Tensorflow's inception model

I want to freeze the layers, except the first three layers, in the Inception v3 model with TensorFlow in Python 3, and modify the weights of these three layers to be able to re-initialize and re-train ...
1
vote
1answer
2k views

scikit-learn classifier reset in loop

I'm trying to evaluate classifiers comparison by running the sample script that can be found at the following address. http://scikit-learn.org/stable/auto_examples/classification/...
0
votes
1answer
222 views

What is the best way to normalize histogram vectors to get distribution?

l have the following sample of 4 vectors of dimension 5 . They are sparse vectors, in a way that each value in a vector represent the frequency (number of occurrence of values). For instance v_1=[0,4,...
2
votes
1answer
3k views

What is the dimensionality of the bias term in neural networks?

I am trying to build a neural network (3 layers, 1 hidden) in Python on the classic Titanic dataset. I want to include a bias term following Siraj's examples, and the 3Blue1Brown tutorials to update ...
0
votes
1answer
844 views

How to code an SVM's equation including kernels?

So I need to code an SVM from bottom up in Python, and I cannot use stuff like libSVM or scikit-learn, for reasons of my own. Referring to Andrew Ng's excellent notes on Support Vector Machines, I ...

1
72 73
74
75 76
93