All Questions

Filter by
Sorted by
Tagged with
4
votes
2answers
204 views

Where did this NY Times op-ed get his Google Search data?

I hope this is a question appropriate for SO. The article in question: http://www.nytimes.com/2015/01/25/opinion/sunday/seth-stephens-davidowitz-searching-for-sex.html As far as I can tell, the only ...
3
votes
4answers
2k views

How to learn a classifier from a dataset with high imbalance

What are the most useful techniques for learning a binary classifier from a dataset with a high degree of imbalance (i.e., a dataset with the "target" class being much rarer than the "background" ...
14
votes
1answer
30k views

Intuition for the regularization parameter in SVM

How does varying the regularization parameter in an SVM change the decision boundary for a non-separable dataset? A visual answer and/or some commentary on the limiting behaviors (for large and small ...
16
votes
2answers
10k views

High-dimensional data: What are useful techniques to know?

Due to various curses of dimensionality, the accuracy and speed of many of the common predictive techniques degrade on high dimensional data. What are some of the most useful techniques/tricks/...
12
votes
1answer
4k views

What happens when we train a linear SVM on non-linearly separable data?

What happens when we train a basic support vector machine (linear kernel and no soft-margin) on non-linearly separable data? The optimisation problem is not feasible, so what does the minimisation ...
0
votes
1answer
90 views

Questions on “Active Archive”

In the big data world there is lot of talk about implementing an "active archive". I see cloudera talk about it a lot. The value prop is that you move the low value (and less used) data from EDW to ...
26
votes
5answers
3k views

VM image for data science projects

As there are numerous tools available for data science tasks, and it's cumbersome to install everything and build up a perfect system. Is there a Linux/Mac OS image with Python, R and other open-...
2
votes
1answer
611 views

Bag of Words creation in image

In SIFT feature extraction how the key points will be generated and how the features will be stored in the database. In image will the bag of visual words be images or text words?
2
votes
2answers
153 views

Could someone please offer me some guidance on some kind of particular, SPECIFIC project that I could attemp, to “get my feet wet, so to speak” [closed]

I am COMPLETELY new to the field of Data Science, mainly because every employer I have worked for, simply COULDN'T sell any customers anything that would use techniques learned in this field. Of ...
0
votes
1answer
168 views

Visualize performance, % of goal implementation

I want to visualize goal achievment progress. This is my first idea: use area chart to show progress in current metric use horizontal band to show the goal value colorize areas under/above the band ...
39
votes
3answers
3k views

When to use what - Machine Learning [closed]

Recently in a Machine Learning class from professor Oriol Pujol at UPC/Barcelona he described the most common algorithms, principles and concepts to use for a wide range of machine learning related ...
15
votes
1answer
16k views

What is the difference between feature generation and feature extraction?

Can anybody tell me what the purpose of feature generation is? And why feature space enrichment is needed before classifying an image? Is it a necessary step? Is there any method to enrich feature ...
1
vote
1answer
180 views

Does high error rate in regression imply the data set is unpredictable?

I have a data set of video watching records in a 3G network. In this data set, 2 different kind of features are included: user-side information, e.g., age, gender, data plan and etc; Video watching ...
0
votes
1answer
202 views

What are the current killing machine learning methods? [closed]

I was wondering whether we could list machine learning winning methods to apply in many fields of interest: NLP, image, vision, medical, deep package inspection, etc. I mean, if someone will get ...
3
votes
2answers
113 views

What kind of data is not appropriate using CF to do recommendation?

I am currently working on a recommendation system for daily news. At first, I evaluated all the recommender algorithms and their corresponding settings (e.g., similarities, factorizers, ...etc) ...
1
vote
2answers
2k views

Probability distribution in input-output pairs

This question might sound silly. But I have been wondering why do we assume that there is a hidden probability distribution between input-output pairs in machine learning setup ? For example, if we ...
2
votes
2answers
518 views

Generate tags for live chat transcripts

I'm wondering if there's a way to automatically generate a list of tags for live chat transcripts without domain knowledge. I've tried applying NLP chunking to the chat transcripts and keep only the ...
0
votes
2answers
44 views

Training a function that maps n-dim to n-dim

As an example, say the input is an array of numbers representing an audio snippet and the output is a transformed/filtered version of it. What would be the proper name for that? Which are examples of ...
1
vote
2answers
770 views

Appropriate way to store data in R

I have data, which looks like this: These data are only for one subject. I will have a lot more. These data will be analyzed in R. Now I'm storing them like this: ...
5
votes
4answers
920 views

Determine highly correlated segments

Given a dataset that has a binary (0/1) dependent variable and a large collection of continuous and categorical independent variables, is there a process and ideally a R package that can find ...
3
votes
1answer
13k views

Querying DBpedia from Python [closed]

How can I get information about an entity from DBpedia using Python? Eg: I need to get all DBpedia information about USA. So I need to write the query from python (SPARQL) and need to get all ...
0
votes
1answer
953 views

What is the underlying pattern, or algorithm(s) used in these numbers?

I have 11 lottery tickets (used) and I have discovered that in each ticket, the 3rd digit's value is +1 of the value of the 6th digit. I have 11 tickets, each ticket is composed of 16 digits. Would ...
3
votes
2answers
4k views

Aspect based sentiment analysis using machine learning approach

I am very new in machine learning. I have annotated data with category, aspect, opinion word and sentiment. for example, for the bellow text "The apple was really tasty" I have category->food, ...
1
vote
1answer
29 views

Shifting dataPoints up by a constant (Is there an issue with too many 0's for features?)

I am currently collecting second by second data regarding buyer vs seller initiated trades for different financial instruments (securities mostly). If there are more buyer initiated trades in a given ...
0
votes
1answer
5k views

Storing Big Matrix in DataBase

I have a mysql database with the following format: id string 1 foo1... 2 foo2... .. ... There are >100k entries in this db. What I want ...
2
votes
2answers
919 views

Approaches to Bag-Of-Words Information Retrieval

I'm interested in an overview of the modern/state-of-the-art approaches to bag-of-words information retrieval, where you have a single query $q$ and a set of documents which you hope to rank by ...
1
vote
1answer
8k views

Hardware requirements for Linux server to run R & RStudio [closed]

I want to build a home server/workstation to run my R projects. Based on what I have gathered, it should probably be Linux based. I want to buy the hardware now, but I am confused with the many ...
0
votes
1answer
441 views

Learning cost function for linear regression

(Me: Never learned calculus or advanced math and I started Stanford openclasses for machine learning. I know basic matrix calculations.) One chapter of my course is about cost function. I have been ...
9
votes
5answers
11k views

Tutorials on topic models and LDA

I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with ...
1
vote
1answer
29 views

Correcting Datasets with artificially low starting values

I am working on a project where we would like to take the ratio of two measurements A/B and subject these ratios to a ranking algorithm. The ratio is normalized prior to ranking (though the ranking/...
8
votes
1answer
4k views

Similarity measure based on multiple classes from a hierarchical taxonomy?

Could anyone recommend a good similarity measure for objects which have multiple classes, where each class is part of a hierarchy? For example, let's say the classes look like: ...
1
vote
1answer
272 views

Can we use package “dplyr” on R base 3.0.2?

Is there any way to use package dplyr on RStudio having R base 3.0.2 ? I am not interested in plyr package.
2
votes
1answer
170 views

Correlation threshold for Neural Network features selection

I'm trying to do a correlation analysis between inputs and outputs inspecting the data in order to understand which input variables to include. What could be a threshold in the correlation value to ...
1
vote
1answer
75 views

Mimic a Mahout like system

I have a data set, in excel format, with account names, reported symptoms, a determined root cause and a date in month year format for each row. I am trying to implement a mahout like system with a ...
7
votes
1answer
2k views

Finding unpredictability or uncertainty in a time series

I am interested in finding a statistic that tracks the unpredictability of a time series. For simplicity sake, assume that each value in the time series is either 1 or 0. So for example, the following ...
1
vote
1answer
80 views

Communicating clearly about “samples”

It is often pointed out that sample is an overloaded term in statistics and the sciences being supported by statistics. In my field (geological sciences) as in most other sciences, the process of ...
1
vote
1answer
87 views

normalize identification values properly

I'm building a neural network to analyze a business' sales. I'm normalizing all input values to the range {0,1}. I'm struggling with the day of the week column. ...
1
vote
1answer
118 views

Compare Neural Network generalization results

I'm trying to develop my neural network with both early stopping and bayesian regularization (matlab implementation, lm algorithm is used for both). Since in bayesian regularization I have not the ...
1
vote
1answer
214 views

How i can generate the probabilistic graph for my dataset?

Im doing my academic project. im having the base paper for reference the paper is IEEE paper "effective and efficient clustering methods for correlated probabilistic graph". i wish to do this in R ...
2
votes
4answers
3k views

Format for storing textual data

For an upcoming project, I'm mining textual posts from an online forum, using Scrapy. What is the best way to store this text data? I'm thinking of simply exporting it into a JSON file, but is there a ...
7
votes
2answers
3k views

Lambda Architecture - How to implement the Merge Layer / Query Layer

I am reading up about lambda architecture. It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data. So we have our ...
0
votes
1answer
468 views

Are Correlithm Objects used for anything in the industry?

This refers to a system described in a book by Nick Lawrence titled "Correlithm Object Technology". The author coined the term "correlithm" as a combination of "correlation" and "algorithm". The ...
2
votes
1answer
131 views

Object Recognition for classification, is it being used in industry?

I'm wondering if e-commerce companies where products are offered by users, such as EBay, are using Object Recognition to ensure that an uploaded image corresponds to an specific type of object (...
2
votes
3answers
378 views

Creating a Graph from Address Data

What's the best / easiest way to create a graph from address data? For example, if I have 100 houses from all across a city is there any easy way to determine the shortest distance between two houses ...
1
vote
0answers
564 views

How to create separate nodes for each column in a spreadsheet using Neo4J?

I have an excel file containing lot of columns. I want to create a graph database in Neo4J with all these columns as a separate node so that I can establish relationship between them and play around ...
2
votes
1answer
55 views

Does a NB wrapper consider feature subset size?

while comparing two different algorithms to feature selection I stumbled upon the follwing question: For a given dataset with a discrete class variable we want to train a naive bayes classifier. We ...
0
votes
1answer
159 views

How to model this “un predicatability” problem?

Imagine modeling the "input(plaintext) - output(ciphertext)" pairs of an encryption algorithm as a data science problem. Very informally, the strength of an encryption scheme is measured by the ...
4
votes
2answers
5k views

Scikit Learn: KMeans Clustering 3D data over a time period (dimentionality reduction?)

I have a dataset of xyz coordinates with a date component in a pandas dataframe ex: date1: $[x_1,y_1,z_1]$, date2: $[x_2,y_2,z_2]$, date3: $[x_3,y_3,z_3]$, .. I would like to classify a sample of ...
1
vote
0answers
42 views

How to transform one graph to a spectrum?

Recently, I studied a paper called "What Does Your Chair Know About Your Stress Level? It can be download at the link below. http://www.barnrich.ch/wiki/data/media/pub/...
2
votes
1answer
217 views

Which Graph Properties are Useful for Predictive Analytics?

Let's assume I'm building a content recommendation engine for online content. I have web log data which I can import into a graph, containing a user ID, the page they viewed, and a timestamp for when ...

15 30 50 per page
1
572 573
574
575 576
586