Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
2
votes
3answers
2k views

Merging repeating data cells in csv

I have a CSV file with around 1 Million rows. Let say its have details like ...
1
vote
2answers
367 views

Random forest model gives same result for all test data, Next step?

I am teaching myself some data science and have started a Kaggle project. I have fitted a random forest classification model (using sci-kit learn) with a few millions rows of data. There are two ...
23
votes
4answers
21k views

Is it necessary to standardize your data before clustering?

Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line: ...
24
votes
4answers
32k views

Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to. What I would like to do is take a scikit-learn's SGDClassifier and have it ...
4
votes
2answers
316 views

Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred?

I have one corpus of documents on diabetes, another on Leonardo da Vinci, and another on animation and computer graphics. I combined all of these documents into a LDA and got a topic like the one ...
4
votes
1answer
2k views

Testing bimodality of data

I am trying to see if my data is multimodal (in fact, I am more interested in bimodality of the data). I performed dip test and it does evidence against unmodal data. However, I want to see, in ...
2
votes
1answer
2k views

What kind of RAM to choose for data analysis? [closed]

I'm planning to upgrade my computer and also want to buy some new DDR3 RAM modules. Making some research I realized that there are 2 main characteristics of RAMs (beside capacity): frequency and ...
11
votes
3answers
5k views

Issue with IPython/Jupyter on Spark (Unrecognized alias)

I am working on setting up a set of VMs to experiment with Spark before I spend go out and spend money on building up a cluster with some hardware. Quick note: I am an academic with a background in ...
62
votes
8answers
74k views

Open source Anomaly Detection in Python

Problem Background: I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
-1
votes
1answer
428 views

K MEANS Big dataset [closed]

Can you please recommend me for a big dataset for k means? It would be cool if it will integrate easily with python, but any thing will be good.
2
votes
2answers
7k views

How to convert unstructured texts to structured data?

I would like to write an algorithm to convert unstructured texts (with contest descriptions) to structured data with the following fields: contest start date (optional) contest end date main prize ...
1
vote
1answer
396 views

Newton-Raphson or EM Algorithm in Python [closed]

Is there any implementation of Newton-Raphson or EM Algorithm? Can I get the source code of it? I tried googling, but didn't come across any. So asking here. Thanks!
3
votes
2answers
489 views

IPython notebook shortcut to run

I am currious if there is a shortcut for runing the selected cell in IPython notebook such as alt+F5 or ctrl+f5 or something simmiliar that is present in Visual Studio (I don't recall exactly now ) ,...
2
votes
0answers
51 views

Can I conclude my finding with just one linear regression result? [closed]

I'm trying to learn Linear Regression and I'd like someone to look at my notebook to see if I miss out anything at all. How how else I can tune my model better. Here's my notebook. I get the ...
2
votes
1answer
531 views

Modules on Python which are useful for missing Word/Letter prediction in text paragraphs from a coprpus

Can someone please recommend me some Python packages which I can use for missing word prediction in a paragraph of text. The missing words are a part of existing semantic list of words.( List of ...
3
votes
2answers
3k views

Question about (Python/Orange) Apriori associative algorithm

I'm trying to wrap my head around Association rules and frequent itemsets. So I threw my data in, instead of the samples one and sometime it works, sometimes it doesn't. ...
2
votes
1answer
87 views

Spatial clustering based on response to inputs and building a reduced model

Stats version: I have a few measurements of a function that takes three inputs and produces a few 2D fields of outputs: $f_i(a,b,c;x,y)$, with $f$ being a vector of several quantities. I would like to ...
1
vote
2answers
502 views

matplotlib geostrophic wind plot

I am working on an assignment that is teaching how to plot and label using matplotlib using Python. Science or math is not my background. I have been given the formula for calculating the geostrophic ...
3
votes
1answer
271 views

python - Will this data mining approach work? Is it a good idea?

I need to extract fields like the document number, date, and invoice amount from a bunch of .csv files, which I believe are referred to as "unstructured text." I have some labeled input files and will ...
3
votes
2answers
326 views

Denormalise data in Neural Networks

I wrote a Neural Networks prediction model in Python. My data has a few inputs and two outputs. In order to make it work, I have to normalise every column on data for good prediction results. ...
10
votes
2answers
16k views

How to convert categorical data to numerical data in Pyspark

I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to ...
4
votes
1answer
262 views

Can you use clustering to pick out signals in noisy data?

As my first project into data science, I would like to pick out the main clusters in noisy data. I think a good example would be trying to pick out certain links on a given StackExchange question that ...
2
votes
0answers
1k views

GATE with Python

I am working on named entity recognition project and find GATE very useful. But I couldn't find a Python connector for GATE yet. Is it possible or is there a way to integrate GATE work flows into ...
3
votes
1answer
66 views

Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster

On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script: dumbo ...
4
votes
2answers
111 views

Bigdata cluster compatible distributed predictive model

I might be asking a dumb question but my question is can I write a python program (lets say a classifier) using some library that scales in hadoop (not only using a simple parallel processing).The ...
5
votes
4answers
264 views

What is the best way to propose an item from a set based on previous choices?

The goal of this question is to be able to propose a user further choices based on his past experiences: like Amazon's book advices. From a set of mp3 files, I assume that a set of mp3 tags data is ...
4
votes
2answers
77 views

Finding aggregated information of data

I am new to data science. I have a dataset of around 200,000 records, having 5 columns. There is a field called, class. For each ...
8
votes
4answers
8k views

How to run a pyspark application in windows 8 command prompt

I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] ...
9
votes
2answers
1k views

Implementing Complementary Naive Bayes in python?

Problem I have tried using Naive bayes on a labeled data set of crime data but got really poor results (7% accuracy). Naive Bayes runs much faster than other alogorithms I've been using so I wanted ...
3
votes
2answers
932 views

NER on Twitter data

What are the best method/library/data available to extract named entities [Names and Location] from Twitter data ? [Other than dictionary lookup] I tried with Python-Stanford NER, But it seems to ...
6
votes
1answer
326 views

Sensitivity to scaling of features in a multivariate gaussians

I'm using the HMMLearn python package for hidden markov models. That implementation is build on multivariate gaussian distributions. So I have a string of features. How sensitive are gaussians to ...
2
votes
1answer
2k views

Implementing sklearn.linear_model.SGDClassifier using python

I have an excel file that contains details related to determining the quality of a wine and I want to implement the linear model concept using the function sklearn.linear_model.SGDClassifier(SVM => ...
2
votes
0answers
120 views

Probability tree in SPSS/ R/ others?

Similar questions have been asked in this context but this one seems a bit different. [Aim] We would like to find out what the probability is of a person purchasing a 4USD column D (see below) price ...
2
votes
0answers
1k views

Caffe net.predict() outputs random results (GoogleNet)

I used pretrained GoogleNet from https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet and finetuned it with my own data (~ 100k images, 101 classes). After one day training I achieved 62% ...
3
votes
0answers
421 views

Complete a Hungarian stem to a real word

I'm quite new to the NLTK package of Python and to NLP too (I usually work in R but for NLP purposes and scraping maybe Python is more able). I scrap articles from Hungarian newsportals and want to ...
0
votes
2answers
614 views

Sampling from a multivariate von Mises-Fisher distribution in Python

I am looking for a simple way to sample from a multivariate von Mises-Fisher distribution in Python. I have looked in the stats module in scipy and the numpy module but only found the univariate von ...
4
votes
3answers
673 views

How to recognize a two part term when the space is removed? (“bigdata” and “big data”)

I'm not a NLP guy and I have this question. I have a text dataset containing terms which go like, "big data" and "bigdata". For my purpose both of them are the same. How can I detect them in ...
8
votes
1answer
426 views

Theano in deep learning research

How widely is Theano used in deep learning research? Is Theano a good start to learn the implementation of machine learning algorithms? Will learning the implementation of something like a feed ...
0
votes
1answer
464 views

Support vector regression and paremeters [closed]

I am doing load forecasting using SVR(kernel='rbf').How can I understand which is the best value for parameters C, epsilon and gamma?Thanks.
4
votes
1answer
248 views

Minimize correlation between input and output of black box system

I am not sure if "minimize correlation" is the right title for this issue but I could not find a better sentence to describe what I would like to achieve. Let's say that I have a black box with ...
4
votes
1answer
233 views

What are available methods for modeling startup survival rates?

I am interested in modeling startup companies failure and success rates to describe what is the representative startup. I have 40 companies in a dataset. Each company is represented as a list of all ...
3
votes
1answer
113 views

Looking for smallest set of rows that form a natural key in a data set

I have several sets of text files on hdfs that are exports from relations. Unfortunately I do not know the structure of the table is, but I do know that each has a multi-part key that defines a row ...
8
votes
1answer
4k views

Complex Chunking with NLTK

I am trying to figure out how to use NLTK's cascading chunker as per Chapter 7 of the NLTK book. Unfortunately, I'm running into a few issues when performing non-trivial chunking measures. Let's ...
2
votes
1answer
2k views

Multi-label text classification with minimum confidence threshold

Building out a system that tries to apply zero or more predefined labels to text. For each label, we've: built out a reasonably good vocabulary of high-value words/features developed a corpus ...
4
votes
2answers
4k views

Add Custom Labels to NLTK Information Extractor

I am working on an information extractor specifically purposed with parsing relationships between entities such as movies, directors, and actors. NLTK appears to provide the necessary tools to ...
2
votes
1answer
3k views

Where is the error in the following code?

I am trying to analyze the movie database using python, downloaded from imdb. While trying to generate some plots, I am running into errors which confuses me. I am trying to generate a matrix of small ...
1
vote
1answer
256 views

Question on decision tree in the book Programming Collective Intelligence

I'm currently studying Chapter 7 ("Modeling with Decision Trees") of the book "Programming Collective intelligence". I find the output of the function mdclassify()...
1
vote
3answers
2k views

Training Dataset required for Classifier

I am currently trying to develop a classifier in python using Naive Bayes technique. I need a dataset so that I can train it. My classifier would classify a new document given to it into one of the ...
1
vote
1answer
1k views

How to classify whether text answer is relevant to an initial text question

I have a text classification problem in which i need to classify an answer to a message as either relevant or not. In the first phase of my calculations, I have already used a SVM to determine if ...
1
vote
1answer
719 views

Basic ML based Prediction model

I had this basic query on ML and would like to get basic ideas on modelling prediction models using ML and Python. Say I have a training data of 1000 items as Item_name, Attrib_1, Attrib_2, Attrib_3,...