All Questions
35,961
questions
4
votes
1
answer
90
views
Data store for testing data products?
Is there a recommended approach for storing processed data for testing new data products?
Basically, I'd like to have a system where a data scientist or an analyst could think of a new data product ...
- 141
1
vote
1
answer
753
views
LFR as synthesized data for dynamic community detection methods
In recent methods for community detection in dynamic networks, LFR benchmark is used as dynamic dataset generator, but I thought it is for static community based data generation. For example in paper ...
- 359
2
votes
1
answer
965
views
Building a static local website using Rmarkdown: step by step procedure [closed]
I am trying to understand the procedure of building a static local website using R and Rmarkdown. I am aware of a Rmarkdown website where the procedure is outlined, ...
- 123
1
vote
1
answer
151
views
Is it ethical to claim experience with big data when the data isn't a part of the new advertising/social media/retail fad?
Obviously most employers, when hiring a data scientist, would prefer experience with big data and/or data science. But what can one safely assume they will acknowledge as experience?
Let's say ...
- 322
1
vote
0
answers
88
views
Correct tagging but wrong parse tree
If I parse following sentence:
He is playing cricket in ground with grass.
with stanford parser, the result is:
...
- 111
0
votes
1
answer
160
views
SAS. How to write "OR" [closed]
How to write "OR" in this example?
DATA a1;
set a;
if var1=1 OR 2;
run;
P.S. va1 is the categorial (with categories: 1, 2, 3)
- 5
1
vote
2
answers
1k
views
Classification of skills based on job ads
I have around 1,000 job ads in the filed of IT (in excel file). I want to find the skills which are mentioned in each of ads. and then find the similar jobs based on skills.
My method: I created 12 ...
- 23
1
vote
1
answer
258
views
Logistic Regression Cost Function Error
with regards to the Logistic Regression cost function of:
And hypothesis:
Is there a way to tell the +/- of the error for how "confident" the hypothesis is?
E.g. if the +/- of the error was 0.1, I ...
- 111
42
votes
10
answers
32k
views
Do data scientists use Excel?
I would consider myself a journeyman data scientist. Like most (I think), I made my first charts and did my first aggregations in high school and college, using Excel. As I went through college, ...
- 523
4
votes
3
answers
2k
views
Learning resources for data science to win political campaigns?
Does anyone know, where I can learn about applying data science to win a political campaign? I know the Obama campaign had 12 data scientists in 2008 and 165 data scientists in 2012. In 2012, they ran ...
8
votes
4
answers
284
views
What is the term for when a model acts on the thing being modeled and thus changes the concept?
I'm trying to see if there is a conventional term for this concept to help me in my literature research and writing. When a machine learning model causes an action to be taken in the real world that ...
- 83
1
vote
1
answer
96
views
how to modify sparse survey dataset with empty data points?
I am working on a data set where the categorical variables have lots of empty spaces (not "NA" but ""). For example, one variable has 14587 empty spaces out of 14644 observations. There are many such ...
- 1,067
0
votes
1
answer
254
views
Canonical VS Graph Isomorphism [closed]
I'm having a having a hard time understanding the difference between an isomorphism in graphs and canonical graphs. I have read through the Wikipedia articles, but it still isn't clicking.
Can ...
- 1,159
22
votes
3
answers
14k
views
How to generate synthetic dataset using machine learning model learnt with original dataset?
Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset ...
- 323
4
votes
2
answers
2k
views
Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib
I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose:
Storm alone and implement streaming naive bayes on my own in storm ...
- 141
3
votes
1
answer
3k
views
How to create and format an image dataset from scratch for machine learning?
I've only worked with ML with .csv formats. I've worked with image formats too but only premade imagesets (MNIST,etc). If I were to create an imageset from scratch, how are the class labels typically ...
1
vote
0
answers
204
views
Huge discrepancies in Logistic Regression and SVM using HOG features to identify an Object
I am doing some research on Logistic regression and SVM using different parameters using HOG features. I am facing a bit of problem while understanding each classifier with combination of different ...
- 111
2
votes
1
answer
266
views
Clustering not producing even clusters
I'm using k-means clustering to processes running on machines.
Dataset sample :
machine name, process
m1,java
m2,tomcat
m1,word
m3,excel
Build a matrix of ...
- 207
9
votes
1
answer
7k
views
How does SQL Server Analysis Services compare to R?
This may be too broad of a question with heavy opinions, but I really am finding it hard to seek information about running various algorithms using SQL Server Analysis Service Data Mining projects ...
- 213
0
votes
1
answer
1k
views
Which book is the best for introduction to analysis social network using python3 [closed]
I am a beginner studying social network analysis.
I installed python 3 just 2 weeks ago.
There are a lot of books for python and social network analysis.
I couldn't choose one of them.
I found one ...
- 3
0
votes
1
answer
518
views
Identity covariance matrix, decorrelated data?
Why would you want to decorrelated data?
As I am reading about PCA and whitening on image data for DNN, I wonder what is the purpose of achieving the identity covariance matrix in your data is?
Is ...
5
votes
3
answers
1k
views
Recommendation - item based vs user based [closed]
I have one clarification -
First the definitions-
User-based: Recommend items by finding similar users. This is often harder to scale because of the dynamic nature of users.
Item-based: Calculate ...
- 51
2
votes
2
answers
191
views
In a SVD with user/video bias, why is the UV contribute so small?
I'm testing a SVD-based collaborative filter on my data set, in which the label, $r_{ij}$, is a real value from 0 to 1.
Like the many papers suggested, to have a better performance, instead of using $...
- 147
1
vote
3
answers
827
views
Tools to preprocess a big data for dashboards?
I have a complex dataset with more than 16M rows coming from pharmaceutical industry. Regarding the data, it is saved in a sql server with multiple (more than 400) relational tables. Data got several ...
- 421
2
votes
0
answers
196
views
Which one is easier to debug between theano and caffe? [closed]
I am investigating whether to use theano or caffe for convnets. I would like to know which one provides a better debug environment. In caffe, it seems you don't write any code just a config in ...
- 121
2
votes
1
answer
430
views
Feature scaling
I am struggling with a conceptual problem related to feature scaling.
Let's assume I am building a classifier (e.g., a NN) and let's assume I rely on future scaling for the input features of my model....
- 21
3
votes
1
answer
3k
views
sk-learn - ValueError: array is too big.
I have a large dataset with characters and 90000 intances and I have the error ValueError: array is too big when I have the following code before the plot_kmeans_digits.py code:
data2=list(csv....
- 101
4
votes
2
answers
5k
views
R and Python, memory differences yielding performance differences
I think there are numerous posts regarding which one to use: R or Python. However, I'm curious about how their architecture differences yield differences in speed performance, not which one to use.
...
- 665
0
votes
1
answer
550
views
Best way to format data for supervised machine learning ranking predictions
I'm fairly new to machine learning, but I'm doing my best to learn as much as possible.
I am curious about how predicting athlete performance (runners in particular) in a race of a specific starting ...
- 103
0
votes
1
answer
49
views
What approaches are there to not display a search result that a user has no permission to see? [closed]
Question
What approaches are there to not display a search result that a user is not supposed to see?
Example
Suppose we have the following situation.
High-level view
In an enterprise, there are ...
user8743
46
votes
12
answers
54k
views
Data Science in C (or C++)
I'm an R language programmer. I'm also in the group of people who are considered Data Scientists but who come from academic disciplines other than CS.
This works ...
- 1,889
4
votes
1
answer
804
views
Terminology: SOMs, batch learning, online learning, and stochastic gradient descent
I'm not sure which word to use to differentiate a self-organizing map (SOM) training procedure in which updates for the entire data set are aggregated before they are applied to the network from a ...
- 141
60
votes
10
answers
108k
views
IDE alternatives for R programming (RStudio, IntelliJ IDEA, Eclipse, Visual Studio)
I use RStudio for R programming. I remember about solid IDE-s from other technology stacks, like Visual Studio or Eclipse.
I have two questions:
What other IDE-s than RStudio are used (please ...
- 5,424
1
vote
0
answers
168
views
procedure for gradient descent
I'm going to describe the procedure for gradient descent using the language of English:
1) For all of the files (e.g. documents) in our training corpus.
--> commence, now we are considering our ...
- 191
1
vote
1
answer
2k
views
Time series change rate calculations for displaying trend line chart
I'm struggling to find a solution to produce a line chart which shows a trend from a Time Series metric.
I try to make my line chart look similar to the following:
Basically, I want to make use of ...
- 113
1
vote
1
answer
729
views
Does pruning a decision tree always make it more general?
If I prune a decision tree, does that make the resulting decision tree always more general than the original decision tree?
Are there examples where this is not the case?
- 181
1
vote
0
answers
205
views
Better approach for handwriting recognition?
I am trying to write an ANN in python for handwriting recognition by mouse movements. ( like identify characters we draw in paint app n convert it to text)
The question might seem that I haven't ...
0
votes
2
answers
183
views
subsetting R data frame
I am subsetting some data frames and am sure there is a better way.
Essentially, I have two data frames. The first is the actual data. The second has some meta data and importantly a flag on ...
- 1
1
vote
1
answer
1k
views
Measuring similarity for sets with same cardinality
The Jaccard coefficient measures similarity between finite sample sets, and is defined as the size of the intersection divided by the size of the union of the sample sets.
I had 100 sets all of same ...
- 113
2
votes
0
answers
96
views
use classification to improve clustering
I am from information security field, and have some introductory level understanding of machine learning field. My problem is to identify behavioural patterns from network traffic. If I use supervised ...
- 23
3
votes
3
answers
17k
views
How to read file from user in Shiny and assign it to a variable in global.r?
I want to read a csv file as input from user in Shiny and assign it to a variable in global.r file.The code I have in ui.R is
...
0
votes
1
answer
2k
views
Implementation of Association Rules in Javascript
I have implemented an interactive visualization with d3.js, javascript to explore the frequency and various combinations of co-occurring item sets. I want to complement the interactive exploration ...
- 188
3
votes
1
answer
2k
views
Cost function for support vector regression
The optimisation problem for support vector regression is (see http://alex.smola.org/papers/2003/SmoSch03b.pdf):
minimise:
\begin{align*}
C\sum_{i=0}^{l}(\xi_{i} +\xi^{*}_{i})+ \frac{1}{2}\lVert w \...
- 31
0
votes
1
answer
112
views
Need help with python code as part of a data analysis project
I am new to python programming. As part of a data analysis project, I am trying to get a scatter plot of Salary vs Wins for each year of 4 consecutive years (so I am trying to get 4 different scatter ...
- 1,067
2
votes
2
answers
2k
views
Probability of similarity of two clusters
For example, there are 4 objects. Each object has some elements. There are 3 clusters. Each element of each object belongs to some cluster. Clustering was done before, and we know only how many ...
- 121
31
votes
3
answers
33k
views
General approach to extract key text from sentence (nlp)
Given a sentence like:
Complimentary gym access for two for the length of stay ($12 value per person per day)
What general approach can I take to identify the ...
- 421
0
votes
1
answer
3k
views
Forecasting sales and creating model [closed]
In a assignment we are given macro economic indicators like GDP, Consumer price index, Producer Price index and Industrial production index. Also we are given Crude oil, Sugar prices and FM-CG Sales. ...
- 1
1
vote
2
answers
754
views
How can the performance of a neural network vary considerably without changing any parameters?
I am training a neural network with 1 sigmoid hidden layer and a linear output layer. The network simply approximates a cosine function. The weights are initiliazed according to Nguyen-Widrow ...
- 123
1
vote
2
answers
1k
views
How do I get Twitter Dataset for Visualization [closed]
I am a newbie. I would like to do visualization on twitter data : top trends based on country (over map) and time variations (for each months in 1 year or for each year) . Can someone tell me where ...
- 111
2
votes
2
answers
239
views
How to make an effective sampling from a database of text documents?
Problem: I want to know methods to perform an effective sampling from a database.
The size of the database is about 250K text documents and in this case each text ...
- 21