# All Questions

29,251 questions
Filter by
Sorted by
Tagged with
2answers
204 views

### Where did this NY Times op-ed get his Google Search data?

I hope this is a question appropriate for SO. The article in question: http://www.nytimes.com/2015/01/25/opinion/sunday/seth-stephens-davidowitz-searching-for-sex.html As far as I can tell, the only ...
4answers
2k views

### How to learn a classifier from a dataset with high imbalance

What are the most useful techniques for learning a binary classifier from a dataset with a high degree of imbalance (i.e., a dataset with the "target" class being much rarer than the "background" ...
1answer
30k views

### Intuition for the regularization parameter in SVM

How does varying the regularization parameter in an SVM change the decision boundary for a non-separable dataset? A visual answer and/or some commentary on the limiting behaviors (for large and small ...
2answers
10k views

### High-dimensional data: What are useful techniques to know?

Due to various curses of dimensionality, the accuracy and speed of many of the common predictive techniques degrade on high dimensional data. What are some of the most useful techniques/tricks/...
1answer
4k views

### What happens when we train a linear SVM on non-linearly separable data?

What happens when we train a basic support vector machine (linear kernel and no soft-margin) on non-linearly separable data? The optimisation problem is not feasible, so what does the minimisation ...
1answer
90 views

### Questions on “Active Archive”

In the big data world there is lot of talk about implementing an "active archive". I see cloudera talk about it a lot. The value prop is that you move the low value (and less used) data from EDW to ...
5answers
3k views

### VM image for data science projects

As there are numerous tools available for data science tasks, and it's cumbersome to install everything and build up a perfect system. Is there a Linux/Mac OS image with Python, R and other open-...
1answer
611 views

### Bag of Words creation in image

In SIFT feature extraction how the key points will be generated and how the features will be stored in the database. In image will the bag of visual words be images or text words?
2answers
153 views

### Could someone please offer me some guidance on some kind of particular, SPECIFIC project that I could attemp, to “get my feet wet, so to speak” [closed]

I am COMPLETELY new to the field of Data Science, mainly because every employer I have worked for, simply COULDN'T sell any customers anything that would use techniques learned in this field. Of ...
1answer
168 views

### Visualize performance, % of goal implementation

I want to visualize goal achievment progress. This is my first idea: use area chart to show progress in current metric use horizontal band to show the goal value colorize areas under/above the band ...
3answers
3k views

### When to use what - Machine Learning [closed]

Recently in a Machine Learning class from professor Oriol Pujol at UPC/Barcelona he described the most common algorithms, principles and concepts to use for a wide range of machine learning related ...
1answer
16k views

### What is the difference between feature generation and feature extraction?

Can anybody tell me what the purpose of feature generation is? And why feature space enrichment is needed before classifying an image? Is it a necessary step? Is there any method to enrich feature ...
1answer
180 views

### Does high error rate in regression imply the data set is unpredictable?

I have a data set of video watching records in a 3G network. In this data set, 2 different kind of features are included: user-side information, e.g., age, gender, data plan and etc; Video watching ...
1answer
202 views

### What are the current killing machine learning methods? [closed]

I was wondering whether we could list machine learning winning methods to apply in many fields of interest: NLP, image, vision, medical, deep package inspection, etc. I mean, if someone will get ...
2answers
113 views

### What kind of data is not appropriate using CF to do recommendation?

I am currently working on a recommendation system for daily news. At first, I evaluated all the recommender algorithms and their corresponding settings (e.g., similarities, factorizers, ...etc) ...
2answers
2k views

### Probability distribution in input-output pairs

This question might sound silly. But I have been wondering why do we assume that there is a hidden probability distribution between input-output pairs in machine learning setup ? For example, if we ...
2answers
518 views

### Generate tags for live chat transcripts

I'm wondering if there's a way to automatically generate a list of tags for live chat transcripts without domain knowledge. I've tried applying NLP chunking to the chat transcripts and keep only the ...
2answers
44 views

### Training a function that maps n-dim to n-dim

As an example, say the input is an array of numbers representing an audio snippet and the output is a transformed/filtered version of it. What would be the proper name for that? Which are examples of ...
2answers
770 views

### Appropriate way to store data in R

I have data, which looks like this: These data are only for one subject. I will have a lot more. These data will be analyzed in R. Now I'm storing them like this: ...
4answers
920 views

### Determine highly correlated segments

Given a dataset that has a binary (0/1) dependent variable and a large collection of continuous and categorical independent variables, is there a process and ideally a R package that can find ...
1answer
13k views

### Querying DBpedia from Python [closed]

How can I get information about an entity from DBpedia using Python? Eg: I need to get all DBpedia information about USA. So I need to write the query from python (SPARQL) and need to get all ...
1answer
953 views

### What is the underlying pattern, or algorithm(s) used in these numbers?

I have 11 lottery tickets (used) and I have discovered that in each ticket, the 3rd digit's value is +1 of the value of the 6th digit. I have 11 tickets, each ticket is composed of 16 digits. Would ...
2answers
4k views

### Aspect based sentiment analysis using machine learning approach

I am very new in machine learning. I have annotated data with category, aspect, opinion word and sentiment. for example, for the bellow text "The apple was really tasty" I have category->food, ...
1answer
29 views

### Shifting dataPoints up by a constant (Is there an issue with too many 0's for features?)

I am currently collecting second by second data regarding buyer vs seller initiated trades for different financial instruments (securities mostly). If there are more buyer initiated trades in a given ...
1answer
5k views

### Storing Big Matrix in DataBase

I have a mysql database with the following format: id string 1 foo1... 2 foo2... .. ... There are >100k entries in this db. What I want ...
2answers
919 views

### Approaches to Bag-Of-Words Information Retrieval

I'm interested in an overview of the modern/state-of-the-art approaches to bag-of-words information retrieval, where you have a single query $q$ and a set of documents which you hope to rank by ...
1answer
8k views

### Hardware requirements for Linux server to run R & RStudio [closed]

I want to build a home server/workstation to run my R projects. Based on what I have gathered, it should probably be Linux based. I want to buy the hardware now, but I am confused with the many ...
1answer
441 views

### Learning cost function for linear regression

(Me: Never learned calculus or advanced math and I started Stanford openclasses for machine learning. I know basic matrix calculations.) One chapter of my course is about cost function. I have been ...
5answers
11k views

### Tutorials on topic models and LDA

I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with ...
1answer
29 views

### Correcting Datasets with artificially low starting values

I am working on a project where we would like to take the ratio of two measurements A/B and subject these ratios to a ranking algorithm. The ratio is normalized prior to ranking (though the ranking/...
1answer
4k views

### Similarity measure based on multiple classes from a hierarchical taxonomy?

Could anyone recommend a good similarity measure for objects which have multiple classes, where each class is part of a hierarchy? For example, let's say the classes look like: ...
1answer
272 views

### Can we use package “dplyr” on R base 3.0.2?

Is there any way to use package dplyr on RStudio having R base 3.0.2 ? I am not interested in plyr package.
1answer
170 views

### Correlation threshold for Neural Network features selection

I'm trying to do a correlation analysis between inputs and outputs inspecting the data in order to understand which input variables to include. What could be a threshold in the correlation value to ...
1answer
75 views

### Mimic a Mahout like system

I have a data set, in excel format, with account names, reported symptoms, a determined root cause and a date in month year format for each row. I am trying to implement a mahout like system with a ...
1answer
2k views

### Finding unpredictability or uncertainty in a time series

I am interested in finding a statistic that tracks the unpredictability of a time series. For simplicity sake, assume that each value in the time series is either 1 or 0. So for example, the following ...
1answer
80 views

### Communicating clearly about “samples”

It is often pointed out that sample is an overloaded term in statistics and the sciences being supported by statistics. In my field (geological sciences) as in most other sciences, the process of ...
1answer
87 views

### normalize identification values properly

I'm building a neural network to analyze a business' sales. I'm normalizing all input values to the range {0,1}. I'm struggling with the day of the week column. ...
1answer
118 views

### Compare Neural Network generalization results

I'm trying to develop my neural network with both early stopping and bayesian regularization (matlab implementation, lm algorithm is used for both). Since in bayesian regularization I have not the ...
1answer
214 views

### How i can generate the probabilistic graph for my dataset?

Im doing my academic project. im having the base paper for reference the paper is IEEE paper "effective and efficient clustering methods for correlated probabilistic graph". i wish to do this in R ...
4answers
3k views

### Format for storing textual data

For an upcoming project, I'm mining textual posts from an online forum, using Scrapy. What is the best way to store this text data? I'm thinking of simply exporting it into a JSON file, but is there a ...
2answers
3k views

### Lambda Architecture - How to implement the Merge Layer / Query Layer

I am reading up about lambda architecture. It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data. So we have our ...
1answer
468 views

### Are Correlithm Objects used for anything in the industry?

This refers to a system described in a book by Nick Lawrence titled "Correlithm Object Technology". The author coined the term "correlithm" as a combination of "correlation" and "algorithm". The ...
1answer
131 views

### Object Recognition for classification, is it being used in industry?

I'm wondering if e-commerce companies where products are offered by users, such as EBay, are using Object Recognition to ensure that an uploaded image corresponds to an specific type of object (...
3answers
378 views

### Creating a Graph from Address Data

What's the best / easiest way to create a graph from address data? For example, if I have 100 houses from all across a city is there any easy way to determine the shortest distance between two houses ...
0answers
564 views

### How to create separate nodes for each column in a spreadsheet using Neo4J?

I have an excel file containing lot of columns. I want to create a graph database in Neo4J with all these columns as a separate node so that I can establish relationship between them and play around ...
1answer
55 views

### Does a NB wrapper consider feature subset size?

while comparing two different algorithms to feature selection I stumbled upon the follwing question: For a given dataset with a discrete class variable we want to train a naive bayes classifier. We ...
1answer
159 views

### How to model this “un predicatability” problem?

Imagine modeling the "input(plaintext) - output(ciphertext)" pairs of an encryption algorithm as a data science problem. Very informally, the strength of an encryption scheme is measured by the ...
2answers
5k views

### Scikit Learn: KMeans Clustering 3D data over a time period (dimentionality reduction?)

I have a dataset of xyz coordinates with a date component in a pandas dataframe ex: date1: $[x_1,y_1,z_1]$, date2: $[x_2,y_2,z_2]$, date3: $[x_3,y_3,z_3]$, .. I would like to classify a sample of ...
0answers
42 views

### How to transform one graph to a spectrum?

Recently, I studied a paper called "What Does Your Chair Know About Your Stress Level? It can be download at the link below. http://www.barnrich.ch/wiki/data/media/pub/...
1answer
217 views

### Which Graph Properties are Useful for Predictive Analytics?

Let's assume I'm building a content recommendation engine for online content. I have web log data which I can import into a graph, containing a user ID, the page they viewed, and a timestamp for when ...

15 30 50 per page