# All Questions

35,961 questions
Filter by
Sorted by
Tagged with
734 views

### Automomatic typographical error correction

Does anyone know where I might be able to find a list of the most common typing errors and their corrections? This is separate from more complicated considerations concerning general spelling checking ...
• 163
9k views

### Fraud detection use text mining

I would like to find different patterns recognition algorithm to detect different type of fraud. I have 1 million unstructured text documents about the clients' information with metadata about the ...
1 vote
31 views

### large database operation. check for relatedness between entities

I have one small list of entities, such as: Russia Vladimir Moscow Then I have a massive database of JSON indices. For each entry there are multiple alpha-...
71 views

### Predicting earthquakes using disturbances in DTH TV transmission

It is said that before an earthquake happens, a viewer experiences disturbances in DTH TV transmission in the form of distorted images on the screen which automatically correct after a few seconds. ...
1 vote
751 views

### How do I access data on my EBS Volume from R-Studio Server on Ubuntu EC2 Instance

I have setup R-Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client. How do I set path in r-studio server to ...
• 121
1 vote
61 views

### Which classifier is efficient in dealing with test query which belongs to no trained class?

Suppose classifier trained with 5 class, and input query content does not belong to any of the trained class data. Naive bayes provides and random class as a result here. Which classifier deals best ...
• 111
1k views

### Correlation between time to event data and continuous data

I want to measure the correlation between the survival time which is a time to event data and the patient's activity count which is measured on continuous scale. What type of correlation coefficient ...
• 21
5k views

### Gradient Descent Step for word2vec negative sampling

For word2vec with negative sampling, the cost function for a single word is the following according to word2vec:  E = - log(\sigma(v_{w_{O}}^{'}.u_{w_{I}})) - \sum_{k=1}^K log(\sigma(-v_{w_{k}}^{'}....
365 views

### Tidying Time Intervals for Plotting a Histogram in R

I'm doing some cluster analysis on the MLTobs from the LifeTables package and have come across a tricky problem plotting ...
• 111
1 vote
207 views

### What are Hybrid Classifiers used in Sentiment Analysis?

What are Hybrid classifiers used for sentiment analysis? How are they built? Please suggest good tutorial/book/link for reference. Also how are they different from other classifiers like SVM and Naive ...
301 views

### Algorithm to find common sequence

Assume that "1,2,3" are the ids of users, active means that person visited the stackoverflow in last one month (0=passive, 1=active), and there are positive and negative votes. ...
• 83
291 views

### Question on decision tree in the book Programming Collective Intelligence

I'm currently studying Chapter 7 ("Modeling with Decision Trees") of the book "Programming Collective intelligence". I find the output of the function mdclassify()...
• 45
2k views

### Training Dataset required for Classifier

I am currently trying to develop a classifier in python using Naive Bayes technique. I need a dataset so that I can train it. My classifier would classify a new document given to it into one of the ...
1 vote
1k views

### How to classify whether text answer is relevant to an initial text question

I have a text classification problem in which i need to classify an answer to a message as either relevant or not. In the first phase of my calculations, I have already used a SVM to determine if ...
• 117
869 views

### Using clustering and Lasso with cv

I used clustering on my dataset. Now when I'm trying to use a LASSO with cv to predict a response, one of the variables it takes into consideration is which cluster a new point is classified into.(I ...
1 vote
237 views

### non-linear optimization for a linear classifier? (scikit-learn)

Using scikit-learn, why would you use bfgs optimization which is non-linear for a linear classifier as logistic regression? I am confused. Does the optimization method finds the optimum of the chosen ...
• 807
6k views

### What is the best way to scale a numerical dataset [closed]

I have a dataset with differents attributes which don't have the same range on their values which is a problem when we need to compute distance beetween objects. After some research i found that i can ...
• 410
23k views

Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL? Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
• 153
1k views

### Optimizing Weka for large data sets

First of all, I hope I'm in the right StackExchange here. If not, apologies! I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million ...
• 374
1 vote
778 views

### Basic ML based Prediction model

I had this basic query on ML and would like to get basic ideas on modelling prediction models using ML and Python. Say I have a training data of 1000 items as Item_name, Attrib_1, Attrib_2, Attrib_3,...
• 1,890
1 vote
146 views

### what can be done with the following data? [closed]

I have scraped the rating that a customer gave in the following categories namely overall rating score, value for money, seat comfort,staff service, catering and entertainment from an airline forum. I ...
1 vote
428 views

### Use of Nash-Equilibrium in big data environments

Has there been a successful implementation of Nash-Equilibrium in big data problems like suggesting a best buy in a stock market, in traffic monitoring systems or crowd control systems. All the above ...
• 111
448 views

### Finding parameters with extreme values (classification with scikit-learn)

I am currently working with the forest cover type prediction from Kaggle, using classification models with scikit-learn. My main purpose is learning about the different models, so I don't pretend to ...
• 807
1 vote
86 views

### Prove Reccurrent Neural Network can exhibit oscillatory behavior

I understand how recurrent neural networks work, however I'm trying to build a deep intuitive understanding of their behavior which is difficult for me because they exhibit such complex behaviors. ...
• 111
377 views

### How can I use Data Science to profoundly contribute to Humanity?

Undergraduate researcher here. I worked at many traditional scientific research labs, ranging from cancer biology, to radiation medicine, and to supercapacitors. I'm thinking of switching to ...
• 141
2k views

### Using EM (Expectation Maximization) algorithm for Training Logistic Regression

Is it possible to learn the weights for a logistic regression classifier using EM (Expectation Maximization)algorithm? Is there any instance reference?
• 171
1 vote
2k views

### Can someone explain the following error in my python code?

I am analyzing a dataset in Python for strictly learning purpose. In the code below that I wrote, I am getting some errors which I cannot get rid off. Here is the code first: ...
• 1,067
130 views

### Finding frequencies in a noisy, "uneven" dataset

I'm working on a problem where frequency analysis applies (decomposition of a signal into frequencies, that is), but it's noisy and the samples are unevenly spaced. Specifically: given a list of ...
34 views

### percentage of confidance on desion trees results

I am looking towards a solution where classification algorithms produce output with some confidence value. but I am confused whether classification algorithms are able to produce results with ...
• 23
1 vote
46 views

### What algorithmic approach for selecting similar, relevancy based documents

I have an application that tracks people making mentions of various topics. We've used a Bayes algorithm to do some simple classification (users give a thumbs up/thumbs down) to pick the people that ...
1k views

### What are some good sources to learn fraud/anomaly detection in normal/time-series data?

I would like to know more on fraud/anomaly detection. I am looking for good source or survey article/book etc out there which will give me some preliminary idea of the area. Any suggestion is ...
• 1,067
130 views

### Algorithms to find average distance between data maxima

I'm having trouble figuring out a way to analyze some simple data. When graphed, the data I have make a somewhat sinusoidal curve. What I want to do is to find the x-values of the maximum peaks of the ...
1 vote
3k views

### Theano/Lasagne/Nolearn Neural Network Image Input

I am working on image classification tasks and decided to use Lasagne + Nolearn for neural networks prototype. All standard examples like MNIST numbers classification run well, but problems appear ...
• 31
1 vote
739 views

### Difference between OLS(statsmodel) and Scikit Linear Regression

I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model. Here is the code which I using statsmodel library with ...
• 226
1 vote
717 views

### R lm(log(y)~x,data) models and predict, need to remember the exp. R2 differences

I noticed a difference between R and Excel when it comes to trend lines. Basically I am automating a simple process that someone does. He/she takes web traffic and finds the best trend line by ...
• 221
423 views

### Finding predictions using biglm without finding errors

I'm using the biglm R package for linear regression. click/impression is the output required. But the test data does not contain click and impression. The predict...
• 69
31k views

### How to scrape imdb webpage?

I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage. I am using BeautifulSoup module. Following is the code I am ...
• 1,067
1 vote
155 views

### Abstract data type?

I don't know, if I can ask this here, but I'm interested to know what kind of abstract data type (ADT) does Twitter use to model relations between profiles and why. I'm just starting to learn ADTs, so ...
• 11
7k views

### Pandas time series optimization problem: add year

I have a pandas DataFrame containing a time series column. The years are shifted in the past, so that I have to add a constant number of years to every element of that column. The best way I found is ...
1 vote
349 views

I'm conjecturing that with Complete-linkage clustering two elements from the same cluster will always be closer to each other some other element from another cluster. In more formal terms: Let $C$ ...
• 83
2k views

### Understanding of minbucket function in CART model using R

Assume the training data is fruit, which I am going to use for prediction in a CART model in R: ...
• 1
357 views

### about predict the class for a new datapoint

I have a new data point and want to classify it into the existing classes.I can calculate pairwise distance for the new point to all existing points(in the existing classes). I know using KNN would be ...
101 views

### SAS PROC means (two variants together)

PROC means data=d mean; var a; class b; var a; run; I want to perform the "PROC means" for continuous "var a": 1) in general and 2) by classes. But it performed by the classes only. How to make ...
573 views

### Data mining- Clustering techniques

I have a project for comparison between clustering techniques using the data set of SSA for birth names from 1910-2013 years for the different states. I have finished applying my clustering ...
• 119
1k views

### Neural Network Golf: smallest network for a certain level of performance

I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance. By small I mean few parameters, not few ...
• 3,132
79 views

### Possibility of working on KDDCup data in local system

I'm trying to apply classification algorithms to KDD Cup 2012 track2 data using R http://www.kddcup2012.org/c/kddcup2012-track2 It seems not possible to work with this 10GB training data on my local ...
• 69
431 views

### Post processing with Random Forest

I understand that by filtering out the instances with labels that Random Forest trees are uncertain upon with their decisions, and model these with another classifier could give a better overall ...
• 967
1 vote
1k views

### Predictive Model for Customer Payment Pattern

I am working for a logistics firm and there are approx. 750+ customers who avail our services. I am in the process of building and generating some insight for the business based on the payments made ...
• 31
11k views

### Extracting model equation and other data from 'glm' function in R

I've made a logistic regression to combine two independent variables in R, using pROC package and I obtain this: ...
• 21
1 vote