All Questions
35,961
questions
0
votes
1
answer
734
views
Automomatic typographical error correction
Does anyone know where I might be able to find a list of the most common typing errors and their corrections? This is separate from more complicated considerations concerning general spelling checking ...
- 163
6
votes
3
answers
9k
views
Fraud detection use text mining
I would like to find different patterns recognition algorithm to detect different type of fraud. I have 1 million unstructured text documents about the clients' information with metadata about the ...
- 69
1
vote
0
answers
31
views
large database operation. check for relatedness between entities
I have one small list of entities, such as:
Russia
Vladimir
Moscow
Then I have a massive database of JSON indices. For each entry there are multiple alpha-...
- 191
3
votes
1
answer
71
views
Predicting earthquakes using disturbances in DTH TV transmission
It is said that before an earthquake happens, a viewer experiences disturbances in DTH TV transmission in the form of distorted images on the screen which automatically correct after a few seconds.
...
- 33
1
vote
1
answer
751
views
How do I access data on my EBS Volume from R-Studio Server on Ubuntu EC2 Instance
I have setup R-Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client. How do I set path in r-studio server to ...
- 121
1
vote
1
answer
61
views
Which classifier is efficient in dealing with test query which belongs to no trained class?
Suppose classifier trained with 5 class, and input query content does not belong to any of the trained class data.
Naive bayes provides and random class as a result here. Which classifier deals best ...
- 111
2
votes
2
answers
1k
views
Correlation between time to event data and continuous data
I want to measure the correlation between the survival time which is a time to event data and the patient's activity count which is measured on continuous scale. What type of correlation coefficient ...
- 21
3
votes
1
answer
5k
views
Gradient Descent Step for word2vec negative sampling
For word2vec with negative sampling, the cost function for a single word is the following according to word2vec:
$$
E = - log(\sigma(v_{w_{O}}^{'}.u_{w_{I}})) - \sum_{k=1}^K log(\sigma(-v_{w_{k}}^{'}....
- 63
0
votes
2
answers
365
views
Tidying Time Intervals for Plotting a Histogram in R
I'm doing some cluster analysis on the MLTobs from the LifeTables package and have come across a tricky problem plotting ...
- 111
1
vote
1
answer
207
views
What are Hybrid Classifiers used in Sentiment Analysis?
What are Hybrid classifiers used for sentiment analysis? How are they built? Please suggest good tutorial/book/link for reference. Also how are they different from other classifiers like SVM and Naive ...
- 165
6
votes
2
answers
301
views
Algorithm to find common sequence
Assume that "1,2,3" are the ids of users, active means that person visited the stackoverflow in last one month (0=passive, 1=active), and there are positive and negative votes.
...
- 83
2
votes
1
answer
291
views
Question on decision tree in the book Programming Collective Intelligence
I'm currently studying Chapter 7 ("Modeling with Decision Trees") of the book "Programming Collective intelligence".
I find the output of the function mdclassify()...
- 45
2
votes
3
answers
2k
views
Training Dataset required for Classifier
I am currently trying to develop a classifier in python using Naive Bayes technique. I need a dataset so that I can train it. My classifier would classify a new document given to it into one of the ...
- 23
1
vote
1
answer
1k
views
How to classify whether text answer is relevant to an initial text question
I have a text classification problem in which i need to classify an answer to a message as either relevant or not.
In the first phase of my calculations, I have already used a SVM to determine if ...
- 117
3
votes
2
answers
869
views
Using clustering and Lasso with cv
I used clustering on my dataset. Now when I'm trying to use a LASSO with cv to predict a response, one of the variables it takes into consideration is which cluster a new point is classified into.(I ...
- 31
1
vote
0
answers
237
views
non-linear optimization for a linear classifier? (scikit-learn)
Using scikit-learn, why would you use bfgs optimization which is non-linear for a linear classifier as logistic regression? I am confused. Does the optimization method finds the optimum of the chosen ...
- 807
2
votes
1
answer
6k
views
What is the best way to scale a numerical dataset [closed]
I have a dataset with differents attributes which don't have the same range on their values which is a problem when we need to compute distance beetween objects. After some research i found that i can ...
- 410
15
votes
5
answers
23k
views
Downloading a large dataset on the web directly into AWS S3
Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?
Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
- 153
2
votes
1
answer
1k
views
Optimizing Weka for large data sets
First of all, I hope I'm in the right StackExchange here. If not, apologies!
I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million ...
- 374
1
vote
1
answer
778
views
Basic ML based Prediction model
I had this basic query on ML and would like to get basic ideas on modelling prediction models using ML and Python.
Say I have a training data of 1000 items as
Item_name, Attrib_1, Attrib_2, Attrib_3,...
- 1,890
1
vote
1
answer
146
views
what can be done with the following data? [closed]
I have scraped the rating that a customer gave in the following categories namely overall rating score, value for money, seat comfort,staff service, catering and entertainment from an airline forum. I ...
- 165
1
vote
2
answers
428
views
Use of Nash-Equilibrium in big data environments
Has there been a successful implementation of Nash-Equilibrium in big data problems like suggesting a best buy in a stock market, in traffic monitoring systems or crowd control systems.
All the above ...
- 111
2
votes
1
answer
448
views
Finding parameters with extreme values (classification with scikit-learn)
I am currently working with the forest cover type prediction from Kaggle, using classification models with scikit-learn. My main purpose is learning about the different models, so I don't pretend to ...
- 807
1
vote
0
answers
86
views
Prove Reccurrent Neural Network can exhibit oscillatory behavior
I understand how recurrent neural networks work, however I'm trying to build a deep intuitive understanding of their behavior which is difficult for me because they exhibit such complex behaviors.
...
- 111
3
votes
3
answers
377
views
How can I use Data Science to profoundly contribute to Humanity?
Undergraduate researcher here. I worked at many traditional scientific research labs, ranging from cancer biology, to radiation medicine, and to supercapacitors. I'm thinking of switching to ...
- 141
3
votes
1
answer
2k
views
Using EM (Expectation Maximization) algorithm for Training Logistic Regression
Is it possible to learn the weights for a logistic regression classifier using EM (Expectation Maximization)algorithm? Is there any instance reference?
- 171
1
vote
2
answers
2k
views
Can someone explain the following error in my python code?
I am analyzing a dataset in Python for strictly learning purpose.
In the code below that I wrote, I am getting some errors which I cannot get rid off. Here is the code first:
...
- 1,067
0
votes
1
answer
130
views
Finding frequencies in a noisy, "uneven" dataset
I'm working on a problem where frequency analysis applies (decomposition of a signal into frequencies, that is), but it's noisy and the samples are unevenly spaced.
Specifically: given a list of ...
- 3
0
votes
1
answer
34
views
percentage of confidance on desion trees results
I am looking towards a solution where classification algorithms produce output with some confidence value. but I am confused whether classification algorithms are able to produce results with ...
- 23
1
vote
1
answer
46
views
What algorithmic approach for selecting similar, relevancy based documents
I have an application that tracks people making mentions of various topics. We've used a Bayes algorithm to do some simple classification (users give a thumbs up/thumbs down) to pick the people that ...
3
votes
2
answers
1k
views
What are some good sources to learn fraud/anomaly detection in normal/time-series data?
I would like to know more on fraud/anomaly detection. I am looking for good source or survey article/book etc out there which will give me some preliminary idea of the area.
Any suggestion is ...
- 1,067
3
votes
1
answer
130
views
Algorithms to find average distance between data maxima
I'm having trouble figuring out a way to analyze some simple data. When graphed, the data I have make a somewhat sinusoidal curve. What I want to do is to find the x-values of the maximum peaks of the ...
- 31
1
vote
1
answer
3k
views
Theano/Lasagne/Nolearn Neural Network Image Input
I am working on image classification tasks and decided to use Lasagne + Nolearn for neural networks prototype.
All standard examples like MNIST numbers classification run well, but problems appear ...
- 31
1
vote
1
answer
739
views
Difference between OLS(statsmodel) and Scikit Linear Regression
I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model.
Here is the code which I using statsmodel library with ...
- 226
1
vote
0
answers
717
views
R lm(log(y)~x,data) models and predict, need to remember the exp. R2 differences
I noticed a difference between R and Excel when it comes to trend lines. Basically I am automating a simple process that someone does. He/she takes web traffic and finds the best trend line by ...
- 221
0
votes
1
answer
423
views
Finding predictions using biglm without finding errors
I'm using the biglm R package for linear regression.
click/impression is the output required. But the test data does not contain click and impression.
The predict...
- 69
12
votes
5
answers
31k
views
How to scrape imdb webpage?
I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage.
I am using BeautifulSoup module. Following is the code I am ...
- 1,067
1
vote
1
answer
155
views
Abstract data type?
I don't know, if I can ask this here, but I'm interested to know what kind of abstract data type (ADT) does Twitter use to model relations between profiles and why. I'm just starting to learn ADTs, so ...
- 11
6
votes
3
answers
7k
views
Pandas time series optimization problem: add year
I have a pandas DataFrame containing a time series column. The years are shifted in the past, so that I have to add a constant number of years to every element of that column.
The best way I found is ...
- 773
1
vote
2
answers
349
views
Complete link clustering
I'm conjecturing that with Complete-linkage clustering two elements from the same cluster will always be closer to each other some other element from another cluster.
In more formal terms:
Let $C$ ...
- 83
-2
votes
1
answer
2k
views
Understanding of minbucket function in CART model using R
Assume the training data is fruit, which I am going to use for prediction in a CART model in R:
...
- 1
0
votes
2
answers
357
views
about predict the class for a new datapoint
I have a new data point and want to classify it into the existing classes.I can calculate pairwise distance for the new point to all existing points(in the existing classes). I know using KNN would be ...
0
votes
2
answers
101
views
SAS PROC means (two variants together)
PROC means data=d mean;
var a;
class b; var a;
run;
I want to perform the "PROC means" for continuous "var a":
1) in general and
2) by classes.
But it performed by the classes only.
How to make ...
- 5
2
votes
3
answers
573
views
Data mining- Clustering techniques
I have a project for comparison between clustering techniques using the data set of SSA for birth names from 1910-2013 years for the different states.
I have finished applying my clustering ...
- 119
6
votes
1
answer
1k
views
Neural Network Golf: smallest network for a certain level of performance
I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance. By small I mean few parameters, not few ...
- 3,132
0
votes
1
answer
79
views
Possibility of working on KDDCup data in local system
I'm trying to apply classification algorithms to KDD Cup 2012 track2 data using R
http://www.kddcup2012.org/c/kddcup2012-track2
It seems not possible to work with this 10GB training data on my local ...
- 69
2
votes
1
answer
431
views
Post processing with Random Forest
I understand that by filtering out the instances with labels that Random Forest trees are uncertain upon with their decisions, and model these with another classifier could give a better overall ...
- 967
1
vote
1
answer
1k
views
Predictive Model for Customer Payment Pattern
I am working for a logistics firm and there are approx. 750+ customers who avail our services. I am in the process of building and generating some insight for the business based on the payments made ...
- 31
2
votes
1
answer
11k
views
Extracting model equation and other data from 'glm' function in R
I've made a logistic regression to combine two independent variables in R, using pROC package and I obtain this:
...
- 21
1
vote
2
answers
112
views
Correlations - Get values in the way we want
I have :
a matrix X with N lines
a vector Y
I've computed the Euclidean distance with Y for each line of X.
What I get is a vector of distances.
What I want is a vector of scores between 0 and 1, ...
- 113