# All Questions

35,901 questions
Filter by
Sorted by
Tagged with
0 votes
1 answer
91 views

### new and fresh ideas with stack exchange dataset [closed]

I'm a new bee for data analysis . i need to work on a research project in big data analysis. first of all i search for a dataset and i found interest in stack exchange data dump. however i browse for ...
1 vote
1 answer
77 views

### computational time and precision trade off

I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical ...
5 votes
3 answers
2k views

### How do you compare term counts between two different periods, with different underlying corpus sizes, without bias?

I'll set the question up with an example. You are analysing news coverage text data from 2014, and find that a term appears less often in the third quarter of 2014 than the final quarter (let's ...
1 vote
2 answers
65 views

### Determine what data, does certain column have in common

I have an Excel sheet with n columns, these columns contain info about the students. For admissions we have the score of test scores in multiple subject areas, ...
• 11
14 votes
2 answers
3k views

### Fast k-means like algorithm for $10^{10}$ points?

I am looking to do k-means clustering on a set of 10-dimensional points. The catch: there are $10^{10}$ points. I am looking for just the center and size of the largest clusters (let's say 10 to 100 ...
• 3,132
3 votes
1 answer
78 views

### When to Perform non-linear dimension reduction

When do we feel need to go through non-linear transformation like kernel PCA ? Please share an example
• 161
5 votes
5 answers
2k views

### What kind of research can be done with an email data set?

I found a data set called Enron Email Dataset. It is possibly the only substantial collection of "real" email that is public. I found some prior analysis of this work: A paper describing the Enron ...
• 287
2 votes
2 answers
78 views

### Library/package/tool for geographical data visualizations

I am looking for a good free existing tool which visualizes geographical data (let's say in the form of coordinates) by plotting them on a map. It can be a library (see this question on ...
• 255
0 votes
1 answer
20 views

### Creating Strings corresponding to Location Co-ordinates [closed]

What are some Python libraries which can convert a (X,Y) tuple to strings? (1.23,4.56) yields strings “1_4”, “12_45”, “123_456”.
3 votes
1 answer
456 views

### Finding user similarities within informal data sets

I'm new to all this and am putting together a learning project. I've decided on finding similarities between users in a data set such as http://en.wikipedia.org/wiki/Enron_Corpus. After doing a bit of ...
3 votes
3 answers
178 views

### Datasource for regression model prediction : Machine Learning

I am trying to work using Amazon machine learning, but the data set that I have is small. The model I want to build is for regression based predictions and the domain I am aiming for the data set to ...
3 votes
1 answer
2k views

### In Weka, how to draw learning curve evaluated on both test and training set?

This is just for finding overfitting gap. After initial research, I can only find method to draw learning curve using evaluation of test set. However, I could not evaluate on training set and over ...
• 183
5 votes
2 answers
4k views

### Add Custom Labels to NLTK Information Extractor

I am working on an information extractor specifically purposed with parsing relationships between entities such as movies, directors, and actors. NLTK appears to provide the necessary tools to ...
• 234
5 votes
1 answer
5k views

### Where is the cost parameter C in the RBF kernel in SVM?

RBF kernel using SVM depends on two parameters C and gamma. If the equation of the kernel RBF as the following: $K(X,X')= \exp(\gamma||X-X'||^2)$ In the equation I can see where can I use gamma, but ...
• 51
4 votes
0 answers
1k views

### Performance profiling and tuning in Apache Spark [closed]

Beyond the immediate suspects defined in the spark documentation, what are some ways to profile, tune and boost performance of an Apache Spark application?
• 141
7 votes
1 answer
4k views

### Why does logistic regression in Spark and R return different models for the same data?

I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of ...
• 113
8 votes
1 answer
3k views

### Which classification algorithms to try for classifying text data into 300 categories

I have 40000 rows of text data of health care domain. Data has one column for text (2-5 sentences) and one column for its category. I want to classify that into 300 categories. Some categories are ...
• 191
2 votes
1 answer
247 views

### Time Complexity notation in Big Data platforms

I am redesigning some of the classical algorithms for Hadoop/MapReduce framework. I was wondering if there any established approach for denoting Big(O) kind of expressions to measure time complexity? ...
• 141
184 votes
5 answers
136k views

### What is the "dying ReLU" problem in neural networks?

Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says: "Unfortunately, ReLU units can be fragile during training and can "die". For ...
• 3,895
13 votes
3 answers
4k views

### Are ontologies and the Semantic Web dead? [closed]

Is the Semantic Web dead? Are ontologies dead? I am developing a work plan for my thesis about "A knowledge base through a set ontology for interest groups around wetlands". I have been researching ...
7 votes
1 answer
757 views

### Lightweight data provenance tool

One of the problems I often encounter is that of poor data provenance. When I do research I continuously make modifications to my code and rerun experiments. Each time I'm faced with a number of ...
• 245
1 vote
2 answers
4k views

### Neural networks with non-negative weights [closed]

Could you tell me, are there any techniques for building neural networks with non-negative weights?
• 11
22 votes
4 answers
16k views

### Dimensionality and Manifold

A commonly heard sentence in unsupervised Machine learning is High dimensional inputs typically live on or near a low dimensional manifold What is a dimension? What is a manifold? What is the ...
• 2,482
4 votes
1 answer
745 views

### NoSQL engine/service recommendation for geolocation data

First of all, I am new in this field we call big data, so my questions may be naive. In order to build an application, which deals with geolocation data, which could be : latitude and longitude ...
• 373
1 vote
1 answer
60 views

### Classify the coordinate data

I'd like to classify the data on coordinate. Here are 2 example data. data1 = [(1,1), (2,2), (3, 3), (4, 2), (5, 3), (6, 0)] data2 = [(1,1), (2,2), (3, 10), (4, 9), (5, 10), (6, 0)] The bold part ...
• 111
0 votes
1 answer
2k views

### Why is the Naive Bayes classifier of sklearn faster than sklearns SVM?

I've used scikit-learn in Python to compare results of naive Bayes and SVM. I've found that naive Bayes is quicker than SVM. Could anyone shed some light on reasons ...
3 votes
1 answer
204 views

### How to approach automated text writing?

What are the tools, practices and algorithms used in automated text writing? For example, lets assume that I have access to wikipedia/wikinews and similar websites API and I would like to produce ...
4 votes
2 answers
5k views

### How Mllib in Spark select variables in logistic regression

I have a question about MLlib in Spark.(with Scala) I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but ...
• 113
0 votes
2 answers
570 views

### Understanding text conversion into SVM input [closed]

In Support Vector Machines, when used for sentiment analysis, text gets converted into a set of data points. How does this happen, usually?
2 votes
2 answers
271 views

### How should I create a single score with two values as input?

I have two series of values, a and b as inputs and I want to create a score, c, which reflects both of them equally. The distribution of a and b are below In both cases, the x-axis is just an index. ...
• 123
10 votes
1 answer
15k views

### Spark, optimally splitting a single RDD into two

I have a large dataset that I need to split into groups according to specific parameters. I want the job to process as efficiently as possible. I can envision two ways of doing so Option 1 - Create ...
• 1,205
2 votes
1 answer
1k views

### Analyzing survey data for predictions

I've got survey data that resembles: ...
• 23
4 votes
4 answers
554 views

### Reference about social network data-mining

I am not in the data science field, but I would like to examine in depth this field and, particularly, I would like to start from the analysis of the social networks data. I am trying to find some ...
• 279
20 votes
4 answers
23k views

### K-means: What are some good ways to choose an efficient set of initial centroids?

When a random initialization of centroids is used, different runs of K-means produce different total SSEs. And it is crucial in the performance of the algorithm. What are some effective approaches ...
• 333
4 votes
1 answer
921 views

### What types of features are used in a large-scale click-through rate prediction problem?

Something that I often see in papers (example) about large-scale learning is that click-through rate (CTR) problems can have up to a billion of features for each example. In this Google paper the ...
• 143
-1 votes
1 answer
47 views

### International Journal [closed]

I would like to post a paper in International Conference on Soft Computing. I want to know whether the journal is a reputed journal.
1 vote
1 answer
191 views

### Gradient descent parameter estimation Package for R

I am looking for a package that does gradient descent parameter estimation in R, maybe with some bootstrapping to get confidence intervals. I wonder if people call it something different here as I get ...
• 233
2 votes
2 answers
62 views

### Pound notation in Summation

I was going through a paper comparing glove and word2vec. I came across the pound notation shown below. What does it mean when used like this? The link for paper is here
2 votes
1 answer
4k views

### Where is the error in the following code?

I am trying to analyze the movie database using python, downloaded from imdb. While trying to generate some plots, I am running into errors which confuses me. I am trying to generate a matrix of small ...
• 1,067
-1 votes
1 answer
2k views

### How is frequent itemsets compared with item-based collaborative filtering in recommender systems?

What is the difference between data mining approaches: frequent itemsets and item-based collaborative filtering in the area of recommender systems?
0 votes
1 answer
732 views

### Automomatic typographical error correction

Does anyone know where I might be able to find a list of the most common typing errors and their corrections? This is separate from more complicated considerations concerning general spelling checking ...
• 163
6 votes
3 answers
9k views

### Fraud detection use text mining

I would like to find different patterns recognition algorithm to detect different type of fraud. I have 1 million unstructured text documents about the clients' information with metadata about the ...
1 vote
0 answers
31 views

### large database operation. check for relatedness between entities

I have one small list of entities, such as: Russia Vladimir Moscow Then I have a massive database of JSON indices. For each entry there are multiple alpha-...
3 votes
1 answer
71 views

### Predicting earthquakes using disturbances in DTH TV transmission

It is said that before an earthquake happens, a viewer experiences disturbances in DTH TV transmission in the form of distorted images on the screen which automatically correct after a few seconds. ...
1 vote
1 answer
750 views

### How do I access data on my EBS Volume from R-Studio Server on Ubuntu EC2 Instance

I have setup R-Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client. How do I set path in r-studio server to ...
• 121
1 vote
1 answer
61 views

### Which classifier is efficient in dealing with test query which belongs to no trained class?

Suppose classifier trained with 5 class, and input query content does not belong to any of the trained class data. Naive bayes provides and random class as a result here. Which classifier deals best ...
• 111
2 votes
2 answers
1k views

### Correlation between time to event data and continuous data

I want to measure the correlation between the survival time which is a time to event data and the patient's activity count which is measured on continuous scale. What type of correlation coefficient ...
• 21
3 votes
1 answer
5k views

### Gradient Descent Step for word2vec negative sampling

For word2vec with negative sampling, the cost function for a single word is the following according to word2vec:  E = - log(\sigma(v_{w_{O}}^{'}.u_{w_{I}})) - \sum_{k=1}^K log(\sigma(-v_{w_{k}}^{'}....
0 votes
2 answers
364 views

### Tidying Time Intervals for Plotting a Histogram in R

I'm doing some cluster analysis on the MLTobs from the LifeTables package and have come across a tricky problem plotting ...
• 111
1 vote
1 answer
207 views

### What are Hybrid Classifiers used in Sentiment Analysis?

What are Hybrid classifiers used for sentiment analysis? How are they built? Please suggest good tutorial/book/link for reference. Also how are they different from other classifiers like SVM and Naive ...

15 30 50 per page