All Questions
35,901
questions
0
votes
1
answer
91
views
new and fresh ideas with stack exchange dataset [closed]
I'm a new bee for data analysis . i need to work on a research project in big data analysis. first of all i search for a dataset and i found interest in stack exchange data dump. however i browse for ...
- 109
1
vote
1
answer
77
views
computational time and precision trade off
I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical ...
5
votes
3
answers
2k
views
How do you compare term counts between two different periods, with different underlying corpus sizes, without bias?
I'll set the question up with an example. You are analysing news coverage text data from 2014, and find that a term appears less often in the third quarter of 2014 than the final quarter (let's ...
1
vote
2
answers
65
views
Determine what data, does certain column have in common
I have an Excel sheet with n columns, these columns contain info about the students. For admissions we have the score of test scores in multiple subject areas, ...
- 11
14
votes
2
answers
3k
views
Fast k-means like algorithm for $10^{10}$ points?
I am looking to do k-means clustering on a set of 10-dimensional points. The catch: there are $10^{10}$ points.
I am looking for just the center and size of the largest clusters (let's say 10 to 100 ...
- 3,132
3
votes
1
answer
78
views
When to Perform non-linear dimension reduction
When do we feel need to go through non-linear transformation like kernel PCA ? Please share an example
- 161
5
votes
5
answers
2k
views
What kind of research can be done with an email data set?
I found a data set called Enron Email Dataset. It is possibly the only substantial collection of "real" email that is public. I found some prior analysis of this work:
A paper describing the Enron ...
- 287
2
votes
2
answers
78
views
Library/package/tool for geographical data visualizations
I am looking for a good free existing tool which visualizes geographical data (let's say in the form of coordinates) by plotting them on a map.
It can be a library (see this question on ...
- 255
0
votes
1
answer
20
views
Creating Strings corresponding to Location Co-ordinates [closed]
What are some Python libraries which can convert a (X,Y) tuple to strings?
(1.23,4.56) yields strings “1_4”, “12_45”, “123_456”.
3
votes
1
answer
456
views
Finding user similarities within informal data sets
I'm new to all this and am putting together a learning project. I've decided on finding similarities between users in a data set such as http://en.wikipedia.org/wiki/Enron_Corpus. After doing a bit of ...
- 131
3
votes
3
answers
178
views
Datasource for regression model prediction : Machine Learning
I am trying to work using Amazon machine learning, but the data set that I have is small. The model I want to build is for regression based predictions and the domain I am aiming for the data set to ...
- 189
3
votes
1
answer
2k
views
In Weka, how to draw learning curve evaluated on both test and training set?
This is just for finding overfitting gap.
After initial research, I can only find method to draw learning curve using evaluation of test set. However, I could not evaluate on training set and over ...
- 183
5
votes
2
answers
4k
views
Add Custom Labels to NLTK Information Extractor
I am working on an information extractor specifically purposed with parsing relationships between entities such as movies, directors, and actors. NLTK appears to provide the necessary tools to ...
- 234
5
votes
1
answer
5k
views
Where is the cost parameter C in the RBF kernel in SVM?
RBF kernel using SVM depends on two parameters C and gamma. If the equation of the kernel RBF as the following:
$K(X,X')= \exp(\gamma||X-X'||^2)$
In the equation I can see where can I use gamma, but ...
- 51
4
votes
0
answers
1k
views
Performance profiling and tuning in Apache Spark [closed]
Beyond the immediate suspects defined in the spark documentation, what are some ways to profile, tune and boost performance of an Apache Spark application?
- 141
7
votes
1
answer
4k
views
Why does logistic regression in Spark and R return different models for the same data?
I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of ...
- 113
8
votes
1
answer
3k
views
Which classification algorithms to try for classifying text data into 300 categories
I have 40000 rows of text data of health care domain. Data has one column for text (2-5 sentences) and one column for its category.
I want to classify that into 300 categories. Some categories are ...
- 191
2
votes
1
answer
247
views
Time Complexity notation in Big Data platforms
I am redesigning some of the classical algorithms for Hadoop/MapReduce framework. I was wondering if there any established approach for denoting Big(O) kind of expressions to measure time complexity?
...
- 141
184
votes
5
answers
136k
views
What is the "dying ReLU" problem in neural networks?
Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:
"Unfortunately, ReLU units can be fragile during training and can
"die". For ...
- 3,895
13
votes
3
answers
4k
views
Are ontologies and the Semantic Web dead? [closed]
Is the Semantic Web dead? Are ontologies dead?
I am developing a work plan for my thesis about "A knowledge base through a set ontology for interest groups around wetlands". I have been researching ...
7
votes
1
answer
757
views
Lightweight data provenance tool
One of the problems I often encounter is that of poor data provenance.
When I do research I continuously make modifications to my code and rerun experiments. Each time I'm faced with a number of ...
- 245
1
vote
2
answers
4k
views
Neural networks with non-negative weights [closed]
Could you tell me, are there any techniques for building neural networks with non-negative weights?
- 11
22
votes
4
answers
16k
views
Dimensionality and Manifold
A commonly heard sentence in unsupervised Machine learning is
High dimensional inputs typically live on or near a low dimensional
manifold
What is a dimension? What is a manifold? What is the ...
- 2,482
4
votes
1
answer
745
views
NoSQL engine/service recommendation for geolocation data
First of all, I am new in this field we call big data, so my questions may be naive.
In order to build an application, which deals with geolocation data, which could be : latitude and longitude ...
- 373
1
vote
1
answer
60
views
Classify the coordinate data
I'd like to classify the data on coordinate.
Here are 2 example data.
data1 = [(1,1), (2,2), (3, 3), (4, 2), (5, 3), (6, 0)]
data2 = [(1,1), (2,2), (3, 10), (4, 9), (5, 10), (6, 0)]
The bold part ...
- 111
0
votes
1
answer
2k
views
Why is the Naive Bayes classifier of sklearn faster than sklearns SVM?
I've used scikit-learn in Python to compare results of naive Bayes and SVM. I've found that naive Bayes is quicker than SVM. Could anyone shed some light on reasons ...
- 29
3
votes
1
answer
204
views
How to approach automated text writing?
What are the tools, practices and algorithms used in automated text writing?
For example, lets assume that I have access to wikipedia/wikinews and similar websites API and I would like to produce ...
- 599
4
votes
2
answers
5k
views
How Mllib in Spark select variables in logistic regression
I have a question about MLlib in Spark.(with Scala)
I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but ...
- 113
0
votes
2
answers
570
views
Understanding text conversion into SVM input [closed]
In Support Vector Machines, when used for sentiment analysis, text gets converted into a set of data points. How does this happen, usually?
2
votes
2
answers
271
views
How should I create a single score with two values as input?
I have two series of values, a and b as inputs and I want to create a score, c, which reflects both of them equally. The distribution of a and b are below
In both cases, the x-axis is just an index.
...
- 123
10
votes
1
answer
15k
views
Spark, optimally splitting a single RDD into two
I have a large dataset that I need to split into groups according to specific parameters. I want the job to process as efficiently as possible. I can envision two ways of doing so
Option 1 - Create ...
- 1,205
2
votes
1
answer
1k
views
4
votes
4
answers
554
views
Reference about social network data-mining
I am not in the data science field, but I would like to examine in depth this field and, particularly, I would like to start from the analysis of the social networks data.
I am trying to find some ...
- 279
20
votes
4
answers
23k
views
K-means: What are some good ways to choose an efficient set of initial centroids?
When a random initialization of centroids is used, different runs of K-means produce different total SSEs. And it is crucial in the performance of the algorithm.
What are some effective approaches ...
- 333
4
votes
1
answer
921
views
What types of features are used in a large-scale click-through rate prediction problem?
Something that I often see in papers (example) about large-scale learning is that click-through rate (CTR) problems can have up to a billion of features for each example. In this Google paper the ...
- 143
-1
votes
1
answer
47
views
International Journal [closed]
I would like to post a paper in International Conference on Soft Computing. I want to know whether the journal is a reputed journal.
1
vote
1
answer
191
views
Gradient descent parameter estimation Package for R
I am looking for a package that does gradient descent parameter estimation in R, maybe with some bootstrapping to get confidence intervals. I wonder if people call it something different here as I get ...
- 233
2
votes
2
answers
62
views
Pound notation in Summation
I was going through a paper comparing glove and word2vec. I came across the pound notation shown below. What does it mean when used like this?
The link for paper is here
- 63
2
votes
1
answer
4k
views
Where is the error in the following code?
I am trying to analyze the movie database using python, downloaded from imdb. While trying to generate some plots, I am running into errors which confuses me.
I am trying to generate a matrix of small ...
- 1,067
-1
votes
1
answer
2k
views
How is frequent itemsets compared with item-based collaborative filtering in recommender systems?
What is the difference between data mining approaches: frequent itemsets and item-based collaborative filtering in the area of recommender systems?
0
votes
1
answer
732
views
Automomatic typographical error correction
Does anyone know where I might be able to find a list of the most common typing errors and their corrections? This is separate from more complicated considerations concerning general spelling checking ...
- 163
6
votes
3
answers
9k
views
Fraud detection use text mining
I would like to find different patterns recognition algorithm to detect different type of fraud. I have 1 million unstructured text documents about the clients' information with metadata about the ...
- 69
1
vote
0
answers
31
views
large database operation. check for relatedness between entities
I have one small list of entities, such as:
Russia
Vladimir
Moscow
Then I have a massive database of JSON indices. For each entry there are multiple alpha-...
- 191
3
votes
1
answer
71
views
Predicting earthquakes using disturbances in DTH TV transmission
It is said that before an earthquake happens, a viewer experiences disturbances in DTH TV transmission in the form of distorted images on the screen which automatically correct after a few seconds.
...
- 33
1
vote
1
answer
750
views
How do I access data on my EBS Volume from R-Studio Server on Ubuntu EC2 Instance
I have setup R-Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client. How do I set path in r-studio server to ...
- 121
1
vote
1
answer
61
views
Which classifier is efficient in dealing with test query which belongs to no trained class?
Suppose classifier trained with 5 class, and input query content does not belong to any of the trained class data.
Naive bayes provides and random class as a result here. Which classifier deals best ...
- 111
2
votes
2
answers
1k
views
Correlation between time to event data and continuous data
I want to measure the correlation between the survival time which is a time to event data and the patient's activity count which is measured on continuous scale. What type of correlation coefficient ...
- 21
3
votes
1
answer
5k
views
Gradient Descent Step for word2vec negative sampling
For word2vec with negative sampling, the cost function for a single word is the following according to word2vec:
$$
E = - log(\sigma(v_{w_{O}}^{'}.u_{w_{I}})) - \sum_{k=1}^K log(\sigma(-v_{w_{k}}^{'}....
- 63
0
votes
2
answers
364
views
Tidying Time Intervals for Plotting a Histogram in R
I'm doing some cluster analysis on the MLTobs from the LifeTables package and have come across a tricky problem plotting ...
- 111
1
vote
1
answer
207
views
What are Hybrid Classifiers used in Sentiment Analysis?
What are Hybrid classifiers used for sentiment analysis? How are they built? Please suggest good tutorial/book/link for reference. Also how are they different from other classifiers like SVM and Naive ...
- 165