Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
2
votes
1answer
251 views

Writing custom data analysis program

I have a number of large datasets (10GBs) each with data fetched from a NoSQL database that I have remotely downloaded on my desktop. I would like to write a Python program to run some custom data ...
1
vote
1answer
1k views

Pandas: access fields within field in a DataFrame

Suppose I have such a JSON file: ...
7
votes
1answer
15k views

Simple example of genetic alg minimization

I have been looking for a while for examples of how I could find the points at which a function achieves its minimum using a genetic algorithm approach in Python. I looked at DEAP documentation, but ...
4
votes
1answer
2k views

Information Extraction from Free-form text to create Transactions

I'm working on a use case where the user will be provided a text box to enter the details of the transaction application. For Example user might enter the below text and I have to parse the data and ...
3
votes
1answer
410 views

Assigning values to missing target vector values in scikit-learn

I have a dataset containing data on temperature, precipitation, and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...
10
votes
3answers
4k views

Building a machine learning model to predict crop yields based on environmental data

I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data. Please note that ...
3
votes
2answers
607 views

Using R and Python together

I'm new in this field and I started working with data by using R. Because of that, I find R much easier to approach a data project. However, apparently an employer wants you to know an object-oriented ...
0
votes
1answer
688 views

Using sklearn DictVectorizer in real-time systems

Any binary one-hot encoding is aware of only values seen in training, so features not encountered during fitting will be silently ignored. For real time, where you have millions of records in a second,...
11
votes
1answer
13k views

XGBoost Linear Regression output incorrect

I am a newbie to XGBoost so pardon my ignorance. Here is the python code : ...
2
votes
1answer
610 views

Which text clustering algorithms should I use to group free text fields in Python?

I have a data set in which users have input free text (think "APPLE", "Apple", "apple", "appl") and I'm trying to group the input by response type (e.g. organize all of the preceding examples by "...
8
votes
1answer
7k views

Recognition human in images through HOG descriptor and SVM classifier performs poorly

I'm using a HOG descriptor, coupled with a SVM classifier, to recognise humans in pictures. I'm using the Python wrappers for OpenCV. I've used the excellent tutorial at pymagesearch, which explains ...
6
votes
1answer
6k views

How to customise cost function in Scikit learn's model?

For example, when I have a problem that false negative should be penalised more, how can I incorporate that requirement in the algorithm such as SVM?
0
votes
1answer
142 views

Why in deep learning or another AI technique python is used rather than other languages? [closed]

I want to know, why python? Instead of C#, Java, C\C++ and so on?
2
votes
1answer
112 views

How to analyse move kindness in python (Logistic regression, neural network, etc.)?

With a team of researchers we were given the assignment to make a scale for the move kindness (how inviting a room or place is for exercise--e.g., a gym). In order to get objective results we were ...
0
votes
2answers
79 views

Logistic regression program does not give statistically correct result [closed]

I wrote a very simple and compact logistic regression program using theano. I am initialising my data randomly, and I have restricted the number of training steps to 1, so since the weights are also ...
3
votes
1answer
2k views

Handling categorical features in Factorization Machines algorithm - Feature Hashing vs. One-Hot encoding

For solving a prediction problem I'm willing to use the Factorization Machines, a model that in addition to learning linear weights on features, learn a vector space for each feature to learn pairing ...
30
votes
3answers
42k views

Hypertuning XGBoost parameters

XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem? This is ...
9
votes
7answers
11k views

Python library that can compute the confusion matrix for multi-label classification

I'm looking for a Python library that can compute the confusion matrix for multi-label classification. FYI: scikit-learn doesn't support multi-label for confusion matrix) What is the difference ...
2
votes
1answer
161 views

Theano logistic regression understanding code

I am used to C/Java like programming, and sometimes I am getting a headache on understanding the Python notation. On the logistic regression code available online, I am trying to understand this line ...
3
votes
2answers
2k views

Scikit Learn's RandomForestRegressor is not giving results on large data set

I have a Pandas dataframe X(20346, 4116). All independent columns have binary variable as 0 or 1. Whereas dependent column has continuous variable. When I execute below code using the scikit-learn ...
3
votes
2answers
3k views

Is $R^2$ an appropriate evaluation metric for k-Nearest Neighbors?

I found a source that stated that $R^2$ is the ”percentage of the response variable variation that is explained by a linear model.” (Source) Since kNN is not a linear model (it is nonparametric), is ...
3
votes
1answer
709 views

Theano logistic regression example

I am trying to understand some simple neural net case using theano. The deeplearning.net site gives the following simple code for implementing a logistic regression application to a simple case: <...
32
votes
5answers
55k views

Calculating KL Divergence in Python

I am rather new to this and can't say I have a complete understanding of the theoretical concepts behind this. I am trying to calculate the KL Divergence between several lists of points in Python. I ...
10
votes
2answers
85k views

Creating new columns by iterating over rows in pandas dataframe

I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99 ...
8
votes
2answers
824 views

Pylearn2 vs TensorFlow

I am about to dive into a long NN research project and wanted a push in the direction of Pylearn2 or TensorFlow? As of Dec 2015 has the community started to lean one direction or another? This link ...
4
votes
4answers
14k views

Library to count number of objects in image?

I'm reading a paper on counting cells, humans, etc. Is there an off-the-shelf library for Python/Theano or TensorFlow I could use?
7
votes
2answers
14k views

pandas count values for last 7 days from each date

There are two pd.DataFrame. First is like this: ...
6
votes
1answer
1k views

How to place XGBoost in a full stack for ML?

Is XGBoost complete by itself for prod-strength machine learning? If not, with which other tools or libs is it typically combined, and how? (I recently read a description of a stack that included ca ...
3
votes
1answer
692 views

Content based recommendation on Mahout

Is it possible to get recommendation on similar product using Mahout ? eg : I have data set of movies with following attributes Movie_name, Actor_1, Actor_2, Actress_1, Actress_2, Director, Theme, ...
7
votes
2answers
1k views

Python distributed machine learning

I occasionally train neural nets for my research, and they usually take quite a long time to run (especially when I'm working on my laptop). I'm looking for a way to build the model on any computer ...
19
votes
3answers
42k views

Feature extraction of images in Python

In my class I have to create an application using two classifiers to decide whether an object in an image is an example of phylum porifera (seasponge) or some other object. However, I am completely ...
4
votes
3answers
94 views

Make use of relationships on recommendation systems

I have a data set of user rating for movie as user_name, product_name, user_rating and I am using this data to recommend new movie to user (collaborative ...
1
vote
0answers
1k views

Canopy clustering implementation in Python

As far as I understand, canopy clustering preprocessing procedure is not implemented in scikit-learn. Is there a python library for this?
1
vote
0answers
35 views

Making sense of blocks python package output

I've modified the Blocks tutorial, to train an MLP neural net with a dataset provided for an assignment in my ML course. I'd like to evaluate the accuracy of the ...
4
votes
4answers
1k views

How to visualise multidimensional categorical data with additional time dimension

I am trying to visualize a dataset that registers the career of people in an organisation and their backgrounds before starting in that particular organisation. I 'd like to show how individuals are ...
2
votes
2answers
1k views

Building Recommendation engine with Python

Which all are the equivalent or advanced libraries in Python for building recommendation systems like Mahout for Collaborative Filtering and Content Based Filtering ? Also is there a way to ...
7
votes
2answers
18k views

Trying to understand Logistic Regression Implementation

I'm currently using the following code as a starting point to deepen my understanding of regularized logistic regression. As a first pass I'm just trying to do a binary classification on part of the ...
5
votes
1answer
1k views

Implementation of Tree Kernels in Python

I'm looking to use a tree kernel (e.g. Zelenko et al., 2003, Moschitti 2006, etc.) for text classification in Python. However, I'm still a little unclear on the implementation. Is there an existing ...
10
votes
1answer
10k views

Python Seaborn: how are error bars computed in barplots?

I'm using seaborn library to generate bar plots in python. I'm wondering what statistics are used to compute the error bars, but can't find any reference to this in the seaborn's barplot documentation....
16
votes
2answers
30k views

Multivariate linear regression in Python

I'm looking for a Python package that implements multivariate linear regression. (Terminological note: multivariate regression deals with the case where there are more than one dependent variables ...
3
votes
5answers
3k views

What are some nice algorithms/techniques for optimizing and predicting Click Through Rates (CTR)?

Definition of Click Through Rates: CTR is the number of clicks that your ad receives divided by the number of times your ad is shown expressed as a percentage (clicks ÷ impressions = CTR).
2
votes
2answers
165 views

Find most representative image

I have 1,000 photoshoot-quality images of pianos on white background with very little noise (people in the background, etc). How would I go about finding the piano image that looks most like the ...
2
votes
1answer
491 views

One multilabel classifier or one for each type of label?

Let's say I need to classify addresses with scikit-learn, so if I want my classifier to be able to classify addresses by the street name, and post/zip code, should I do a OneVsRest classifier, or ...
1
vote
3answers
2k views

Reduce dimension, then apply SVM

Just out of curiousity, is it generally a good idea to reduce the dimension of training set before using it to train SVM classifier? I have a collection of documents, each of them is represented by a ...
6
votes
1answer
633 views

Python + Neural Nets + Large dimension Large Dataset

I am building a NLP application. My dataset has 0.6M datapoints each of 0.15M dimensions. My feature vector is highly sparse - mostly 0s, atmost 20 1s in any feature vector. I am using a 32 GB machine ...
17
votes
5answers
14k views

Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more ...
19
votes
4answers
21k views

Python library for segmented regression (a.k.a. piecewise regression)

I am looking for a Python library that can perform segmented regression (a.k.a. piecewise regression). Example:
3
votes
1answer
3k views

Extract the “path” of a data point through a decision tree in sklearn

I'm working with decision trees in python's scikit learn. Unlike many use cases for this, I'm not so much interested in the accuracy of the classifier at this point so much as I am extracting the ...
7
votes
1answer
8k views

Cosine Distance > 1 in scipy

I am working on a recommendation engine, and I have chosen to use SciPy's cosine distance as a way of comparing items. I have two vectors: ...
9
votes
2answers
5k views

Prohibitive size of random forest when saved to disk

When saved to disk using cPickle: https://stackoverflow.com/questions/20662023/save-python-random-forest-model-to-file, my random forest is 6.57 GB. ...